2022 Computer Vision & Human-Robot Interaction

Neato Gesture Commands

An intuitive gesture-based control system for Neato robots, using MediaPipe for hand detection and machine learning for custom gesture recognition. This project enables natural human-robot interaction through hand movements, allowing users to control robot navigation and behaviors without physical interfaces.

View Code

Controls

Distinct gesture commands

Framework

ROS2

Robot Operating System integration

Recognition

Custom

Trainable gesture models

The Challenge

Creating a reliable and intuitive gesture-based control system presented several technical challenges:

Implementing robust hand detection that works reliably in varying conditions
Developing a system for training custom gesture recognition rather than relying on pre-trained models
Creating intuitive mappings between gestures and robot behaviors
Ensuring real-time processing with minimal latency for responsive control
Designing advanced features like path drawing that translate hand movements into robot trajectories
Integrating gesture recognition with ROS2 for reliable robot control

The Solution

We developed a comprehensive gesture recognition system using advanced computer vision techniques:

Implemented MediaPipe for accurate hand detection and tracking of key points (fingertips, knuckles)
Created a custom training system for recognizing user-defined gestures
Developed intuitive number-based gesture commands for basic robot control
Implemented specific behavior algorithms triggered by different gestures
Built a prototype path drawing feature that tracks finger movements to create custom robot paths

MediaPipe hand tracking visualization

Our approach to gesture control focused on user-friendliness and system extensibility:

Used MediaPipe's hand landmark detection to identify 21 key points on each hand
Implemented custom gesture training where users could create their own gesture dataset
Designed a speed control system that uses the distance between thumb and index finger as a variable control
Created an integrated system where the robot responds to gestures in real-time via ROS2 communication

Gesture Recognition System

Our implementation went through several distinct development phases:

Tool Selection: After evaluating various computer vision libraries, we chose MediaPipe for its robust hand detection capabilities. MediaPipe provided pre-built functionalities for identifying and tracking hands, including detailed finger position data that was essential for our gesture recognition system.

Custom Gesture Training: Rather than using pre-trained models limited to standard gestures, we implemented a system that allowed us to create and train recognition for custom gestures. This involved capturing images of our gestures in real-time and using them to build a custom dataset for machine learning-based recognition.

Behavior Integration: We mapped specific gestures to robot behaviors, initially using random gesture assignments before transitioning to a more intuitive number-based system for better usability.

Number-based gesture control system

Robot Control & Advanced Features

Our gesture control system evolved from basic commands to more sophisticated interactions:

Movement Control: We initially implemented basic movement controls using the robot's odometry to create consistent patterns. Rather than using time-based commands, we leveraged the robot's positioning system to achieve precise movements and turns for driving in various shapes.

Number Gesture System: We redesigned our gesture controls to use intuitive number-based commands, with each number triggering a specific robot behavior:

Number 1 (pointer finger): Forward movement
Number 2 (peace sign): Turn right
Number 3 (three fingers): Turn left
Number 4 (four fingers): Stop
Number 0 (fist): Continuous spinning
Open hand: Draw a square
Triangle gesture: Draw a triangle
Thumb and index finger pinch: Speed control (distance proportional to speed)

Path Drawing Prototype: We began work on an advanced feature that would allow users to "draw" paths in the air for the robot to follow. This involved tracking finger movement, recognizing the intended shape, and converting it to precise robot navigation commands.

Path drawing prototype visualization

Future Development

Path Drawing Enhancement

Waypoint-based drawing system to create more precise paths
Recognition of pauses in finger movement to mark corner points
Support for multiple path geometries (lines, zigzags, shapes)
Translation of detected paths into precise robot movement commands
Improved curve handling and path smoothing algorithms

System Improvements

Enhanced gesture recognition accuracy in varied lighting conditions
Implementation of more sophisticated gesture combinations
User-specific gesture calibration and preferences
Integration with other robot capabilities (mapping, object interaction)
Expansion of the gesture vocabulary for more complex commands

Results & Achievements

Successfully implemented a working gesture control system for the Neato robot
Achieved reliable recognition of multiple distinct hand gestures
Created intuitive mappings between gestures and robot behaviors
Developed a functional prototype for custom gesture training
Established a foundation for future development of path drawing functionality

Technical Learnings

Computer vision techniques for hand detection and tracking
Machine learning approaches for custom gesture recognition
ROS2 integration for translating gesture commands to robot actions
Real-time processing optimization for responsive control
Human-robot interaction design principles for intuitive interfaces

Next Project