Hand gesture synthesizer with MIDI export
Gesture Synth Weld is a free browser instrument: your webcam tracks your hands, your hands play chords,
and the performance comes back out as a standard .mid file you can open in any DAW.
No install, no sign-up, and no webcam required if you would rather use the keyboard.
MIDI export — the part no other gesture synth does
Every other browser gesture synth is a closed loop: you wave your hands, you hear a sound, and that
is where it ends. Here the note events are captured while you record, then written out as a
format-0 Standard MIDI File with real note-on and note-off pairs at 120 BPM.
- Hit ● Record and play — up to 30 seconds.
- When it stops, ⤓ Export MIDI appears next to the record button.
- Drop the
.mid into Ableton, Logic, FL Studio, Reaper, Bitwig or GarageBand.
- Re-voice it with whatever instrument you like — the gestures become a normal MIDI clip.
Because notes that carry across a chord change are held rather than retriggered, the exported clip
reads as musical phrasing instead of a wall of repeated one-frame notes.
Keyboard mode — no webcam needed
No camera, blocked permission, or just not in the mood to be on video? Keyboard mode plays the same
instrument with no camera prompt and no model download.
| Key | What it does |
|---|
| 1 – 7 | Scale degrees I – VII — hold to play, release to stop |
| [ / ] | Minor / major mode |
| 8 9 0 - | Triad · 1st inversion · 7th · 9th |
| Shift | Octave down |
| ↑ / ↓ | Volume sweep (hold) |
| ← / → | Filter sweep (hold) |
| Space | Stop all notes |
Gesture playbook
Left hand — harmony
| Fingers | Degree | Gesture | In A major |
|---|
| 1 | I | Any one finger | A |
| 2 | ii | Any two fingers | Bm |
| 3 | iii | Any three fingers | C♯m |
| 4 | IV | Any four fingers | D |
| 5 | V | All five fingers | E |
| VI | vi | Index + pinky only | F♯m |
| VII | vii° | Index + pinky + thumb | G♯dim |
| 0 | — | Fist | Mute |
Degrees follow real diatonic harmony rather than forcing every degree to a major triad — that is
why a I–vi–IV–V sweep already sounds like a song.
Right hand — expression
| Movement | Controls |
|---|
| Fingers (1–4) | Triad → 1st inversion → 7th → 9th |
| Hand height | Volume |
| Wrist tilt | Filter sweep — right brighter, left darker |
| Thumb extended | Drop one octave |
Nine scale modes — not just major and minor
Every other browser gesture synth ships a hardcoded table of major and minor triads, which
caps the instrument at two sounds. Here each mode is stored as its raw semitone set and
chords are generated by stacking thirds through that set, so the harmony is
derived rather than looked up — and the chord name is read back from the intervals that
actually came out.
| Mode | I – IV – V in C | Character |
|---|
| Major (Ionian) | C · F · G | Bright, resolved |
| Natural Minor | Cm · Fm · Gm | Dark, classic sad |
| Dorian | Cm · F · Gm | Minor with a major IV — funk, folk |
| Mixolydian | C · F · Gm | Major with a flat 7 — rock, blues-rock |
| Lydian | C · F♯° · G | Floating, film-score |
| Phrygian | Cm · Fm · G° | Spanish, metal |
| Harmonic Minor | Cm · Fm · G | Minor with a major V — the tense one |
| Major Pentatonic | C · G · A | Five notes, nothing can clash |
| Blues | Csus4 · G♭ · G | Six notes with the flat five |
Because the sevenths are generated too, a 7th chord comes out as
Cmaj7 in major and C7 in mixolydian without either being
written down anywhere.
Wrist tilt as a brightness lever
With Scale + wrist tilt enabled, tilting your left wrist drops to the
parallel darker mode over the same tonic — major↔minor,
lydian↔phrygian, mixolydian↔dorian, pentatonic↔blues. On a two-mode instrument that
control is a switch; across nine modes it becomes a performance gesture.
Three play modes
| Mode | Left hand | Right hand |
|---|
| Gesture | Scale degree I–VII, major/minor tilt | Chord richness, volume, filter, octave |
| Theremin | Height = volume | Height = pitch (C3–C6, logarithmic), tilt = filter |
| Piano | Fingers pick root / 3rd / 5th / octave / 9th | Triggers the note, height = volume |
Theremin pitch is mapped logarithmically, so the same hand movement gives the same musical
interval anywhere in the range instead of cramming the top two octaves into the last inch.
Virtual background — blur or replace the room
Most takes never get posted because of what is behind you. The Background
control blurs the room, drops in a flat studio ground or a glow, or swaps in an image of
your own — the same idea as Zoom or Google Meet, and it is baked straight into the
recording rather than added afterwards.
| Option | What you get |
|---|
| Blur / Blur + | The room stays, softened — two strengths |
| Studio | Flat ground in the current theme colour |
| Glow | Radial wash from the accent colour out to the ground |
| Image… | Any picture from your device |
Segmentation runs on-device and the model is 244 KB — and nothing is fetched until you
pick a background, so it costs a first-time visitor nothing. Hand tracking still reads the
untouched camera frame, because a fingertip clipped by the mask would be a wrong chord.
Off-the-shelf segmentation is built for video calls and loses a hand raised in the air,
which here is the entire subject. So the tracked hands are drawn back into the mask
directly — twenty-one landmarks per hand is better information than any general-purpose
segmenter has, and it means your hands stay sharp against the blur instead of dissolving
into it.
If you would rather not appear at all, Skeleton capture mode drops the
camera image entirely and records only the tracked hands.
Metronome & practice
40–240 BPM with tap tempo, four click sounds (click, wood, beep, hi-hat) and time signatures
from 3/4 to 7/8. The downbeat is accented so you can hear the bar line. It routes past the
instrument filter and reverb, so sweeping your wrist never smears the click — and it is
audible in recordings, which is useful for layering takes and a nuisance otherwise, so turn
it off before your final pass.
Why it sounds different
The synth engine is not a bank of bare oscillators. Each note is a pair of detuned oscillators with
its own ADSR envelope, feeding a shared resonant lowpass, a generated-impulse convolution reverb and
a limiter. Six presets — Warm Pad, Bright Synth, Retro Square, Ambient Pad, Harp Pluck and Electric Piano — change
the oscillator pair, envelope and reverb mix together rather than just swapping a waveform.
The arpeggiator sweeps chord tones as plucks at three speeds, and Auto Bass adds a sine root two
octaves down that tracks your left hand.
Troubleshooting
Camera permission was denied
- Chrome / Edge: click the 🔒 icon in the address bar → Camera → Allow → reload.
- Safari (Mac): Safari → Settings → Websites → Camera → Allow.
- Safari (iPhone/iPad): Settings → Privacy & Security → Camera → enable Safari, then reload.
- Firefox: click the permissions icon in the address bar → Camera → Allow.
Or skip it entirely — keyboard mode needs no camera.
No camera found
Close anything else using it (Zoom, Teams, FaceTime — a camera serves one app at a time), then reload.
Tracking feels laggy
Hand tracking runs on the GPU via WebGL. Close other GPU-heavy apps, and plug in a laptop — some browsers throttle GPU work on battery. Chrome and Edge give the best MediaPipe performance.
No sound
Browsers block audio until you interact with the page, so the synth only starts after you click the start button. Check the system output device too.
FAQ
Can I export what I play as MIDI?
Yes — record a take, then hit Export MIDI. It writes a format-0 .mid at 120 BPM with proper note-on/note-off pairs for every chord you held.
Can I use it without a webcam?
Yes. Keyboard mode plays the same instrument with keys 1–7 and needs no camera permission and no model download.
What is a gesture synth?
An instrument that turns webcam-tracked hand movement into notes — no MIDI controller, gloves or sensors involved.
Is my camera feed uploaded anywhere?
No. Tracking runs on-device through MediaPipe compiled to WebAssembly. Frames never leave your browser and recordings are written to your own device.
Can I blur or replace my background?
Yes — the Background control offers blur, a flat studio ground, a glow, or your own image, and it is baked into recordings. The 244 KB segmentation model only downloads once you pick one. See virtual background.
Does it work on a phone?
Yes, with single-hand mode enabled in Settings — a phone's front camera rarely frames two hands well.
Which browsers work?
Chrome 90+, Edge 90+, Firefox 90+ and Safari 15+. Chrome or Edge give the smoothest tracking.