Voice Activated Gesture Controlled Robot

Prof. Bhakti D. Kadam, Shreyas S. Tekawade, Ruta S. Deshpande, Onkar H. Joshi, Loukik J. Parab
Department of Electronics and Telecommunication Engg., Vishwakarma Institute of Technology Pune

Abstract—Even in today’s industrial processes, robots are receiver end which drives the motor to a particular direction.
controlled using the traditional method of control modules. This The robot moves forward, backward, right and left when we
paper presents design and implementation of a system which can tilt our palm to forward, backward, right and left respectively.
be controlled using natural human interactions – Hand gestures The robot stops when it is parallel to the ground.
and Speech. Thus, even non-expert programmers can easily
control a remote robotic system by using simple instructions. The
system is built around the feedback received from voice
commands of user and gesture recognition from data retrieved II. LITERATURE REVIEW
from a 3-axis accelerometer embedded on a handheld device.
[1] Gesture Based Robot Control V. S. Rao and C.
Keywords- accelerometer, hand gesture recognition, manual Mahanta. In this paper, they present a visual gesture
robot, microcontroller, voice recognition recognition system for controlling robots by using Fuzzy-
CMeans clustering algorithm. The proposed method is applied
I. INTRODUCTION for recognizing both static and dynamic hand gestures.
The idea of this project is to build a Hand gesture [2] A Hand-Gesture-Based Control Interface for a Car-
controlled Robot which will be activated by voice command. Robot Xing-Han Wu, Mu-Chun Su, and Pa-Chun Wang In
In many applications of controlling the robotic gadget it this paper, they introduce a hand-gesture-based control
becomes quite hard and complicated to control it with interface for navigating a car-robot. A 3-axis accelerometer is
different switches and controls. adopted to record a user’s hand trajectories. The trajectory
These days many a wireless robots are being developed data is transmitted wirelessly via an RF module to a computer.
and put to various applications and uses. In order to enhance [3] Research and Development of a Gesture-Controlled
the contribution of robots in our daily lives we need to find an Robot Manipulator System Karolis Root, Member, IEEE and
effective way of providing human feedback to robots. For this Renaldas Urniezius, Member, IEEE. In this paper they present
purpose, there have been certain developments in area of the results of research and development of a gesture-controlled
human-machine interaction. One common form of robot manipulator system. The accelerometer and gyroscope
communication is Gestures that are not only limited to face, inertia measurement unit helped estimate human arm’s joint
body and fingers but also hand gestures. In order to increase angles in stress test conditions.
the use of robot in places where conditions are not certain like
rescue operations, robots can be made to follow the [4] High-level programming and control for industrial
instructions of human operator and perform the task robotics: using a hand-held accelerometer-based input device
accordingly. for gesture and posture recognition Pedro Neto, J. Norberto
Pires, A. Paulo Moreira. This paper presents a robotic system
The main purpose of using gestures is that it provides a that allows users, especially non-expert programmers, to
more natural way of controlling and provides a rich and instruct and program a robot just showing it what it should do,
intuitive form of interaction with the robotic system and with a high-level of abstraction from the robot language.
Human hand gestures are natural and with the help of This is done using the two most natural human interfaces
wireless communication, it is easier to interact with the robot (gestures and speech), a force control system and several code
in a friendly way. The robot moves depending on the gesture generation techniques.
made by your hand and from a distance. The objective of this [5] Bluetooth communication controlled robot based on
paper is to build a wireless gesture control robot using Gesture recognition Rahul Kumar Singh, Archisman Sarkar,
Arduino, accelerometer, and Bluetooth communication. Debashish Chakravarty, Paritosh Goyal, Vaibhav Lodhi,
The Arduino Uno microcontroller reads the analog output Anurag Sharma. For this, an Android app of the algorithm has
values i.e., x-axis and y-axis values of the accelerometer and been developed which gives an interactive user interface
converts that analog value to respective digital value. The where the user will tilt the smartphone in the direction of his
digital values are processed by the Arduino Uno desired movement and the degree of tilt decides the speed of
microcontroller and according to the tilt of the accelerometer robot. The App uses smartphone’s inbuilt accelerometer,
sensor mounted on hand, it sends the commands to the processor and bluetooth as a processing unit and data
bluetooth module on the robot which is processed at the transmitter. By doing so, the user has the flexibility to control
the speed as well as direction of the robot.
 [6] Accelerometer-Based Hand Gesture Recognition
System for Interaction in Digital TV J. Ducloux, P. Colla, P.
Petrashin, W. Lancioni, L. Toledo. This paper presents the
design and implementation of a system of accelerometer-based
hand gesture recognition. This system will be embedded
within a modern remote control to improve human-machine
interaction in the context of digital TV of Argentina. As the
recognition of hand gestures is a pattern classification
problem, two techniques based on artificial neural networks
are explored: multilayer perceptron and support vector
machine.
[7] Accelerometer-Based Control of an Industrial Robotic
Arm Pedro Neto, J. Norberto Pires, Member, IEEE, and A.
Paulo Moreira, Member, IEEE. In this paper it is proposed an
accelerometer-based system to control an industrial robot
using two low-cost and small 3-axis wireless accelerometers.
These accelerometers are attached to the human arms,
capturing its behavior (gestures and postures). An Artificial
Neural Network (ANN) trained with a back-propagation
algorithm was used to recognize arm gestures and postures, Fig.1: Arduino Uno and Arduino Nano
which then will be used as input in the control of the robot.
[8] Android Based Wireless Gesture Controlled Robot
Altaf Ganihar, Shreyas Joshi, Rahul G, Rohini Hongal and
Uma Mudenagudi. In this paper they propose a framework for
the implementation of an android based wireless gesture
controlled robot. . In this context they propose a color
segmentation based gesture controlled robot along with an
ultrasonic sensor to monitor the 3D spatial coordinates in
region of operation. The gesture controlled robot is
implemented using Bluetooth communication protocol and
tested under various conditions of lighting.

III. METHODOLOGY

A. System Description
The system consists of 3 central components- the
Fig.2: Accelerometer Fig.3: Bluetooth module
transmitting device put on user’s hand, the application for
ADXL335 HC-05
voice control and the receiver section which is present on the
robot.
1. The handheld device 2. The Robot
The device which is on user’s hand consists of The robot consists of basic rover (chassis and wheels)
and electronics which includes an Arduino Uno for data
accelerometer ADXL335 for sensing movements,, an Arduino
processing, Bluetooth modules for reception, two 150 rpm
Nano for processing and an HC-05 bluetooth module for
transmitting input signal to the robot. A 5V regulated supply is motors, L298 motor driver. Power is supplied with a 12V
given to the Nano using a 9V battery and IC 7805. lead-acid battery.

3. Voice control application

The android application developed using MIT App
Inventor 2 takes voice input, recognizes and converts it into
text, and sends a corresponding character via Bluetooth to the
robot for activation
 .

Fig.5: Block Diagram of System

Fig.4.1: Voice recognition Application block diagram Part-1

IV. OBSERVATIONS AND RESULTS

Sr. X output Y output Z output Character sent

No.

1. 358 357 422 S(Stop)

2. 371 330 422 F(Forward)

3. 359 389 419 B(Backward)

4. 403 361 404 L(Left)

5. 320 359 417 R(Right)

Fig.4.2: Voice recognition Application block diagram

The analog outputs given in the range of 0-1023 were
Part-2
recorded and conditions were set based on the axis-wise
readings for character transmission. Testing showed a slight
B. Working of system
delay in transmission (upto 100ms) as the distance between
The robot locomotes based on the feedback received user and robot was increased.
from accelerometer, transmitted via Bluetooth. There are 3
bluetooth modules used - Master, Slave and Voice RX. Until
the Voice RX doesn’t receive a command to begin, Slave V. ADVANTAGES
doesn’t start taking feedback from Master. Also, Voice RX The system is wirelessly controlled, hence it can be
has higher precedence over Slave so that turning on/off can be used over a wide region. Since voice feedback is also
done as and when required. provided, potential situations can be avoided in which the
robot may go out of control. Other pre-programmed routines
The accelerometer can sense movements in 3 axial
can also be implemented via voice commands.
directions – X, Y, Z. Accordingly it gives analog output in the
range of 0-1023 which is read by the microcontroller ADC. By
VI. APPLICATIONS
setting limits on the output, the position of hand can be
checked and accordingly characters are transmitted by Master Due to the growing demand for natural Human Machine
module indicating the way in which robot should move. Interfaces and robot intuitive programming platforms,a robotic
system that allows users to control a robot using arm gestures
and postures was proposed. Such robots can be used in
military applications to operate robots and in medical
applications for the purpose of surgery. These robots can be
used in the construction field. These robots can also be used
in industries to control trolly and lift. Gestures can be used to
control interactions for entertainment purposes such as gaming
to make game player’s experience more interactive or Telecommunication Engg., VIT Pune for their support and
immersive. Gesture controlling is very helpful for guidance in the development of the project.
handicapped and physically disabled people to achieve certain
tasks, such as driving a vehicle.

VII. FUTURE SCOPE

References
Wireless modules consume very low power and is [1] Accelerometer-Based Control of an Industrial Robotic Arm Pedro
best suited for wireless, battery driven devices. Advanced Neto, J. Norberto Pires, Member, IEEE, and A. Paulo Moreira, Member,
robotic arms that are designed like the human hand itself can IEEE
easily controlled using hand gestures only. Proposed utility is
in fields of Construction, Hazardous waste disposal, Medical [2] High-level programming and control for industrial robotics: using a
science. Gesture controlling is very helpful for handicapped hand-held accelerometer-based input device for gesture and posture
recognition Pedro Neto, J. Norberto Pires, A. Paulo Moreira
and physically disabled people to achieve certain tasks, such
as driving a vehicle. Instead of Bluetooth module for [3] Android based wireless gesture controlled robot Altaf Ganihar, Shreyas
communication, WiFi or RF module can be used for shorter Joshi, G Rahul, Rohini Hongal and Uma Mudenagudi.
response time, high speed and increased range of
communication. Voice commands can be used for interfacing [4] Bluetooth communication controlled robot based on gesture recognition
the robot from anywhere. The mobile application can be added Rahul Kumar Singh,Archisman Sarkar, Debashish Chakravarty,
with more features for locomotion and control of robot. A Paritosh Goyal, Vaibhav Lodhi, Anurag Sharma
wireless camera can be placed on the robot so that the user
can get real time footage of the robot which could be [5] Gesture control robot using accelerometer Rashmi Vashisth, Akshit
displayed on the mobile application. Sharma, Shantanu Malhotra, Saurabh Deswal, Aman Budhraja

[6] Research and Development of a Gesture-Controlled Robot Manipulator

System Karolis Root, Member, IEEE and Renaldas Urniezius, Member,
Acknowledgment IEEE
This project was done as a Mini Project for Second
[7] Gesture Based Robot Control V. S. Rao and C. Mahanta
Year of Engg. The authors would like to thank Prof. Kadam
for her insight and expertise during the making of this project. [8] A Hand-Gesture-Based Control Interface for a Car-Robot Xing-Han
The authors also wish to thank Dept. of Electronics & Wu, Mu-Chun Su, and Pa-Chun Wang