Skip to content
Resume Contact

3D hand gesture recognition

A door lock you open with a sequence of hand gestures. It started as my high school senior capstone in 2022–2023, built by a team of four with a depth camera and TensorFlow. In October 2026 I rebuilt it on my own, with a Kinect depth camera, a small PyTorch model and a servo, and it runs on a Raspberry Pi, so far with simulated hands.

Code on GitHub All projects
  • Type High school capstone, rebuilt in 2026
  • When 2022–2023; rebuilt October 2026
  • Team Four in 2022–2023; the rebuild is mine
  • Sensor Xbox 360 Kinect (v1), depth only
  • Status Rebuild works end to end in simulation
  • Stack PyTorch · Kinect · Computer Vision · Python
A frame from the rebuild’s simulator: a synthetic person in false-colour depth, their right hand outlined making a peace sign, a side panel with the 64 by 64 crop the model sees, the status UNLOCKED, and a timeline marking point, open palm and peace as recognized

A frame from the rebuild’s simulator, not camera footage: the last gesture of the passcode, and the lock opening.

The high school version, 2022–2023

The idea was a passcode you make with your hands instead of typing on a keypad. Four of us built it in our senior year.

  1. A depth camera captures the hand in 3D, so each frame records how far away each part of the hand is, not just its outline.
  2. We used TensorFlow models, in Python, to recognize which gesture the hand is making.
  3. The system checks the sequence of gestures against the passcode and accepts or rejects it. The result drives a solenoid that acts as the lock on a real door, so the demo could actually lock and unlock the door.

The demo came close to fully working: it recognized gesture combinations and authenticated a sequence as a passcode. What held it back was depth perception. We wanted depth to be a native modality, something the model takes in directly as its own kind of input, and that is where we had trouble.

A drawing of how the 2022–2023 version worked: a depth camera captures the hand in 3D, TensorFlow names the gesture, a sequence check compares the gestures in order against the passcode, and a solenoid lock on a real door locks or unlocks. A note below marks depth perception as what held it back.

A drawing of how the 2022–2023 version worked, not a photo from the project.

October 2026

The rebuild

I came back to it on my own and built it again from scratch, this time with depth as the model’s only input, the thing the first version struggled with. Whether that holds up on real Kinect data is still untested.

  1. An Xbox 360 Kinect (v1) gives depth frames at 640 by 480, 30 times a second, in millimetres.
  2. Simple rules find the hand: the largest blob in a thin slab nearest the sensor, between half a metre and a metre and a half away, centred on the palm and cropped to a fixed real-world size, then scaled to a 64 by 64 patch of depth plus a mask.
  3. A small PyTorch CNN of about 1.2 million parameters names the gesture, with calibrated confidence, and a vote filter only counts a gesture once it wins 7 of the last 9 frames.
  4. A passcode of three gestures is checked only once all three are in, so no single step gives anything away. A fist locks it again, and it relocks by itself after 10 seconds.
  5. A servo turns the deadbolt, driven by an Arduino that keeps its own fail-safe relock timer, or straight from a Raspberry Pi. So far I have run a small servo from the Pi on a bench, not on a door.

On a Raspberry Pi 4, finding the hand and running the model through ONNX Runtime take 25.7 ms a frame, or 14.9 ms with a lighter model, against a 33 ms budget at 30 frames a second. There is a live view in the browser and a set of tests. I got the Kinect working on the Pi, which needed a one-line patch to libfreenect.

Demo

The rebuild in its simulator

These come from the simulator in the repository, which renders synthetic people and hands in place of the Kinect, so they are not camera footage. The left of each frame is the depth image, coloured from near to far; the panel on the right is what the model sees; the strip below shows what the model read against the truth and the lock’s state, and the timeline marks each gesture it counted.

The passcode, then a fist. A synthetic right hand points, opens its palm and makes a peace sign; each gesture is recognized and marked on the timeline, the status turns from LOCKED to UNLOCKED, and a fist locks it again. 14 seconds.
Three wrong codes, then the right one. The dots fill and reset after each wrong attempt, the lock itself never says which step was wrong, and it stays shut until the right sequence. Lockouts after failed attempts are turned off for the demo. 26 seconds.
Every gesture the model knows, made with a left hand about 1.1 m away: fist, open palm, peace, point, three and rock, plus “none” for anything else. 20 seconds.

Outcome

Where it is now

On 300 synthetic people the model had never seen, with different hand sizes, left and right hands and distances, it recognized 1,719 of 1,721 gestures, with 3 false inputs, and took about 300 ms on average from the start of a gesture to counting it. In a separate test of 100 people entering the passcode straight through, with no pause between gestures, all 100 unlocked it on the first try.

Those are simulated hands, and synthetic accuracy is not real-world accuracy. The model I have so far was trained on the simulator, so the next step is training and testing it on real hands recorded with the Kinect. The code is on GitHub.

Computer vision came back in my university capstone, on warning before violence starts in video.

View all 16 projects

Questions about this project?

I am happy to talk about how it works. nathanpaulgamache@gmail.com

Ask me about it