--- title: "Hand gesture door lock with a Kinect | Nathan Gamache" description: "A door lock you open with hand gestures: my high school capstone with a team of four, rebuilt on my own in 2026 with a Kinect, PyTorch and a servo." url: https://www.nathangamache.com/projects/hand-gesture/ --- # 3D hand gesture recognition A door lock you open with a sequence of hand gestures. It started as my high school senior capstone in 2022–2023, built by a team of four with a depth camera and TensorFlow. In October 2026 I rebuilt it on my own, with a Kinect depth camera, a small PyTorch model and a servo, and it runs on a Raspberry Pi, so far with simulated hands. [Code on GitHub](https://github.com/nathangamache/3d-hand-gesture) · [All projects](https://www.nathangamache.com/projects/) - Type: **High school capstone, rebuilt in 2026** - When: **2022–2023; rebuilt October 2026** - Team: **Four in 2022–2023; the rebuild is mine** - Sensor: **Xbox 360 Kinect (v1), depth only** - Status: **Rebuild works end to end in simulation** - Stack: **PyTorch · Kinect · Computer Vision · Python** ![A frame from the rebuild’s simulator: a synthetic person in false-colour depth, their right hand outlined making a peace sign, a side panel with the 64 by 64 crop the model sees, the status UNLOCKED, and a timeline marking point, open palm and peace as recognized](https://www.nathangamache.com/assets/img/hand-gesture/unlock-then-lock.webp) A frame from the rebuild’s simulator, not camera footage: the last gesture of the passcode, and the lock opening. ## The high school version, 2022–2023 The idea was a passcode you make with your hands instead of typing on a keypad. Four of us built it in our senior year. 1. A **depth camera** captures the hand in 3D, so each frame records how far away each part of the hand is, not just its outline. 2. We used **TensorFlow** models, in **Python**, to recognize which gesture the hand is making. 3. The system checks the **sequence of gestures** against the passcode and accepts or rejects it. The result drives a **solenoid** that acts as the lock on a real door, so the demo could actually lock and unlock the door. The demo came close to fully working: it recognized gesture combinations and authenticated a sequence as a passcode. What held it back was depth perception. We wanted depth to be a native modality, something the model takes in directly as its own kind of input, and that is where we had trouble. ![A drawing of how the 2022–2023 version worked: a depth camera captures the hand in 3D, TensorFlow names the gesture, a sequence check compares the gestures in order against the passcode, and a solenoid lock on a real door locks or unlocks. A note below marks depth perception as what held it back.](https://www.nathangamache.com/assets/img/p-hand-gesture.svg) A drawing of how the 2022–2023 version worked, not a photo from the project. October 2026 ## The rebuild I came back to it on my own and built it again from scratch, this time with depth as the model’s only input, the thing the first version struggled with. Whether that holds up on real Kinect data is still untested. 1. An **Xbox 360 Kinect (v1)** gives depth frames at 640 by 480, 30 times a second, in millimetres. 2. Simple rules find the hand: the largest blob in a thin slab nearest the sensor, between half a metre and a metre and a half away, centred on the palm and cropped to a fixed real-world size, then scaled to a **64 by 64** patch of depth plus a mask. 3. A small **PyTorch** CNN of about 1.2 million parameters names the gesture, with calibrated confidence, and a vote filter only counts a gesture once it wins 7 of the last 9 frames. 4. A **passcode** of three gestures is checked only once all three are in, so no single step gives anything away. A fist locks it again, and it relocks by itself after 10 seconds. 5. A **servo** turns the deadbolt, driven by an Arduino that keeps its own fail-safe relock timer, or straight from a Raspberry Pi. So far I have run a small servo from the Pi on a bench, not on a door. On a **Raspberry Pi 4**, finding the hand and running the model through ONNX Runtime take 25.7 ms a frame, or 14.9 ms with a lighter model, against a 33 ms budget at 30 frames a second. There is a live view in the browser and a set of tests. I got the Kinect working on the Pi, which needed a one-line patch to libfreenect. Demo ## The rebuild in its simulator These come from the simulator in the repository, which renders synthetic people and hands in place of the Kinect, so they are not camera footage. The left of each frame is the depth image, coloured from near to far; the panel on the right is what the model sees; the strip below shows what the model read against the truth and the lock’s state, and the timeline marks each gesture it counted. [Download the clip (MP4)](https://www.nathangamache.com/assets/video/hand-gesture/unlock-then-lock.mp4) The passcode, then a fist. A synthetic right hand points, opens its palm and makes a peace sign; each gesture is recognized and marked on the timeline, the status turns from LOCKED to UNLOCKED, and a fist locks it again. 14 seconds. [Download the clip (MP4)](https://www.nathangamache.com/assets/video/hand-gesture/wrong-codes-then-right-code.mp4) Three wrong codes, then the right one. The dots fill and reset after each wrong attempt, the lock itself never says which step was wrong, and it stays shut until the right sequence. Lockouts after failed attempts are turned off for the demo. 26 seconds. [Download the clip (MP4)](https://www.nathangamache.com/assets/video/hand-gesture/all-gestures.mp4) Every gesture the model knows, made with a left hand about 1.1 m away: fist, open palm, peace, point, three and rock, plus “none” for anything else. 20 seconds. Outcome ## Where it is now On 300 synthetic people the model had never seen, with different hand sizes, left and right hands and distances, it recognized 1,719 of 1,721 gestures, with 3 false inputs, and took about 300 ms on average from the start of a gesture to counting it. In a separate test of 100 people entering the passcode straight through, with no pause between gestures, all 100 unlocked it on the first try. Those are simulated hands, and synthetic accuracy is not real-world accuracy. The model I have so far was trained on the simulator, so the next step is training and testing it on real hands recorded with the Kinect. The code is [on GitHub](https://github.com/nathangamache/3d-hand-gesture). Computer vision came back in my university capstone, on warning before violence starts in video. ## Related projects ### [Streaming Violence Anticipation System](https://www.nathangamache.com/projects/capstone/) My university senior capstone: warning before violence starts in video, judged against a baseline that sees no video at all. Technologies: PyTorch, SigLIP 2, Computer Vision, Evaluation [View project: Streaming Violence Anticipation System](https://www.nathangamache.com/projects/capstone/) ### [Baylee Discord Bot](https://www.nathangamache.com/projects/baylee/) Another early project in making software seem to understand people: a Discord bot for a Minecraft server that matched keywords and pieced its replies together from templates. Technologies: Node.js, Express, MySQL, Discord API [View project: Baylee Discord Bot](https://www.nathangamache.com/projects/baylee/) [View all 16 projects](https://www.nathangamache.com/projects/) ## Questions about this project? I am happy to talk about how it works. [nathanpaulgamache@gmail.com](mailto:nathanpaulgamache@gmail.com) [Ask me about it](https://www.nathangamache.com/contact/) --- Nathan Gamache · [nathanpaulgamache@gmail.com](mailto:nathanpaulgamache@gmail.com) · [GitHub](https://github.com/nathangamache) · [LinkedIn](https://www.linkedin.com/in/nathan-p-gamache/) · [nathangamache.com](https://www.nathangamache.com/)