ML Hand Gesture

This is a browser-native platform for training custom hand gesture models in seconds and using them to control real-world hardware and interactive software. No cloud uploads, no latency, 100% privacy.

Video Walkthrough

---

I. Problem Statement

Most modern AI tools require sending data to a server for inference. For interactive experiences—like playing a virtual piano or controlling a robotic motor—even 200ms of latency feels "broken." Furthermore, privacy concerns often prevent users from sharing camera feeds with cloud providers.

ML Hand Gesture solves this by bringing the entire ML lifecycle—Training and Inference—directly into the browser.

---

II. Solution Overview

ML Hand Gesture provides a seamless "Zero-to-Hero" workflow for custom AI interaction:

1. Local Training (MLP): Users capture hand landmarks via webcam. A Neural Network (Multi-Layer Perceptron) is trained instantly in-browser using TensorFlow.js.
2. AI-on-AI Synthesis: Once trained, a cloud LLM (Gemini 2.0 Flash) analyzes the class names and predicts the "Gesture Recipe"—suggesting creative use cases and naming the interaction style.
3. Real-Time Control: Use your custom gestures to:
   • Play Music: Trigger a low-latency Web Audio piano.
   • Control Hardware: Drive LEGO Spike Prime motors via Web Bluetooth (BLE).
4. Community Sharing: Save models and gesture mappings to a serverless PostgreSQL cloud (Neon) to share with others.

---

III. AI Integration & Architecture

The "Edge-to-Cloud" Pipeline

Component	Technology	Role	Why?
Vision	MediaPipe	Hand Landmark Detection	Extract 21 3D points at 60fps.
Local Inference	TensorFlow.js	Custom MLP Classifier	Sub-10ms gesture recognition.
Logic/Auth	FastAPI	Python Backend	Secure session and model metadata.
Synthesis	Gemini 2.0	LLM Recipe Generation	Contextualize the "why" of the gesture.

IV. Design Decisions

• Edge vs. Cloud: I chose Client-Side MLP over Cloud CNNs. While CNNs are more robust, the landmark-based MLP is 100x faster for real-time interaction and requires zero server-side GPU costs.
• Stateless vs. Statefull: The backend is entirely Stateless (Docker on Cloud Run), allowing it to scale from zero to infinity without managing persistent VMs.

---

V. What did AI help you do faster, and where did it get in your way?

This project was built using a "Director-Level" development approach, orchestrating multiple specialized AI agents:

• GitHub Copilot (The Logic Pair): Handled the complex state machine for Web Bluetooth (BLE). BLE logic is notoriously difficult to debug; Copilot helped iterate through 7 versions of the hub-connection handshake in minutes.
• Google Jules (The Architect): Managed a massive multi-file refactor to migrate the backend to GCP Cloud Run and implemented the entire JWT/SendGrid password reset flow simultaneously across 5 files.
• Devin AI (The Documentarian): Acted as an autonomous SWE to generate the project's Internal Wiki (AGENTS.md) and verified the consistency of the folder structure.

Key Learning: Using agents shifted my role from "syntax-writer" to "system-reviewer." I focused on high-level constraints (like model serialization formats) while the agents handled the boilerplate implementation.

---

VI. Getting Started

1. Prerequisites

• Bun (Required for Frontend)
• Python 3.10+
• Google API Key (for AI Recipe generation)

2. Setup

# Clone the repo
git clone https://github.com/your-username/ml-hand-gesture.git
cd ml-hand-gesture

# Setup Backend
cd backend
pip install -r requirements.txt
# Add your GEMINI_API_KEY and DATABASE_URL to .env
cp backend/.env.example backend/.env

# Run backend
python main.py

# Setup Frontend (New Terminal)
cd ../frontend
bun install
bun run dev

---

VII. Demo

Explain how to use your application or run demos. Include screenshots, example commands, or links to videos if helpful.

VIII. Testing / Error Handling

• Hardware Robustness: The useSpikeDevice hook implements a robust "Heartbeat" and "Retry" logic to handle BLE disconnections without crashing the UI.
• Fail-Safe Inference: If the webcam fails or landmarks are lost, the prediction engine gracefully enters a "Wait" state rather than outputting garbage data.
• Input Validation: Pydantic models on the backend ensure that malformed model weights cannot be saved to the database.

---

IX. Future Improvements / Stretch Goals

• Multi-Hand Interaction: Extend the MLP to recognize two-handed gestures for more complex MIDI control.
• Gesture Recording: Record sequences of gestures to create "Action Macros."
• Unity/VST Integration: Export trained models as a bridge to professional music production software.

---

X. Link to website URL

https://ml-hand-gesture.netlify.app/

---

© All Rights Reserved Pascal Cyusa. 2026