Assess Motility in The Elderly With Wii
Assess Motility in The Elderly With Wii
Assess Motility in The Elderly With Wii
Elderly
Becca Groveman
Contents
1 Introduction 4
1.1 Problem and Importance . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Possible Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Background 7
2.1 Technology to Support Aging in Place . . . . . . . . . . . . . . . . . 7
2.2 3D Tracking, Learning, and Gesture Recognition . . . . . . . . . . . . 10
2.3 The Nintendo Wii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.1 Utilizing the Wii . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.2 Bluetooth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 My Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1
3.5.1 IR Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.5.2 Gesture Data Analysis . . . . . . . . . . . . . . . . . . . . . . 29
3.5.3 Data Visualization . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
A Sample Scripts 42
A.1 Single Camera Calibration Script . . . . . . . . . . . . . . . . . . . . 42
A.2 Triangulation Script for Data Visualization . . . . . . . . . . . . . . . 44
2
List of Figures
3
Chapter 1
Introduction
The first generation of the “baby boomer” cohort will turn 65 in 2011 [10]. By 2050,
there will be three times as many elders over the age of 85 as there are today [30].
As a result, in-home elderly care is becoming an important issue. Currently, much of
the technology available to assist the elderly takes the form of technology that can
be worn on the body, such as necklaces or bracelets with emergency buttons or GPS
to alert caregivers of a problem. However, technology in this form is obtrusive, in
the sense that the elder is constantly aware of its presence, and that the elder must
remember to wear it in order for it to be effective.
An alternative to this technology is in the socially assistive robotics currently
being developed. Socially assistive robotics is a phrase referring to technological sys-
tems that offer assistance in the form of social, rather than physical interaction [30].
While this technology sometimes includes the presence of an embodied robot [22] [23],
more often it takes the form of a distributed sensor system.
A distributed sensor system is a system that utilizes sensor technology such as
video capture, fall detection, and other sensors such as bed or stove sensors. These
sensors are installed in a given living space, providing a level of care to the person
living in the space, while still allowing the user to feel independent.
4
There are many high-end systems that assess motility. Motility analysis, or anal-
ysis of how well a person is able to move, is an important topic in eldercare. Motility
analysis gives an assessment of a person’s physical well-being; if a person is doing
well, a physical therapist or doctor is able to say with certainty that the person is
physically able to live alone, which helps elders to stay independent for longer. How-
ever, such systems are expensive and require specialized equipment. My project asks
the question of whether it is possible to utilize an inexpensive and unobtrusive sys-
tem to capture a person’s movements sufficiently to assess motility. To that end, my
project explores the viability of using the Nintendo Wii as an inexpensive alternative
to the high-end systems available.
The Nintendo Wiimote is a sophisticated piece of equipment that is inexpensive
(at $40) compared to other sensor systems available. It has an infrared camera and
three accelerometers, allowing it to detect acceleration along the X, Y, and Z axes.
In addition to being an inexpensive alternative to high-end systems, the Wii can
also be used as an unobtrusive tool to monitor the elderly. The Wii was designed to
appeal to a broad audience [32], and elders are among its users [33].
Since many elders are playing the Wii, and the Wiimotes used can generate such
advanced and useful data, utilizing the Wii to monitor motility in the elderly becomes
an option worth exploring.
It is important to note that while one Wiimote is inexpensive, my system uses
three Wiimotes, a Wii, and a computer, for a minimum cost of $700, more if using
a more expensive computer. Nevertheless, $700 is considerably less than the cost
of the larger research systems, especially because this setup can be used at home
or at a senior center, and allow the user to play the Wii. Other systems require an
elderly person to go somewhere to perform activities for the sole purpose of assessing
physical well-being.
5
gather data with sufficient accuracy to be useful.
If successful, this project could be a welcome addition to the more advanced
systems currently in use. Elders could play the Wii while the system collects data,
and physical therapists or doctors could examine the data visually.
6
Chapter 2
Background
My project is based around two seemingly different areas: the Nintendo Wii and
eldercare. Technology to assist the elderly often takes the form of embodied robotics
systems, utilization of video data, and distributed sensor systems. Some of this
technology is discussed in Section 2.1. Machine learning and gesture recognition
are discussed in Section 2.2. The Nintendo Wii is discussed in Section 2.3, and my
project is laid out in Section 2.4.
7
believed that the physical embodiment of a robot makes a key difference in the
system’s effectiveness [22]. In this section, I describe a few of the distributed sensor
systems that have been designed to help elders to age in place.
The ASSIST framework [10] is a collaboration between the University of Mas-
sachusetts, Amherst, and Smith College. It seeks to bring together computer scien-
tists, social scientists and gerontologists to examine the way in which technology is
adopted by the elderly. It also looks at the impact of technology on the well being
of the elderly and on health care delivery.
Elderly focus group participants were enthused about the usage of video tech-
nology, which has prompted the ASSIST group to develop the uBot, a mobile robot
able to follow the user around. The uBot [6] has a touch screen that allows it to
interact with the user and act as the embodiment of the distributed system. It is
able to communicate through gesture, motion, and verbal commands, and can act
as a substitute for health care and service providers who cannot be present [10]. An
image of the uBot can be seen in Figure 2.1.
8
In addition to the uBot, the ASSIST system is built on another key technology:
the distributed sensor array. Activity modeling occurs through the use of this array.
Activities are defined as observational temporal patterns consisting of auditory, vi-
sual, and haptic feedback that can be extracted from the environment and analyzed.
In the ASSIST framework, Hidden Markov Models (HMMs) are used in order to
determine the underlying process or pattern that results in these patterns of activ-
ity. (Hidden Markov models are discussed in more detail in Section 2.2.) Through
HMMs, it can be determined whether an activity sequence is a fall or not, based on
the likelihood of a given sequence.
Further, through the help of a remote doctor, the uBot can determine whether or
not a fall is stroke-related. If a distributed sensor network with embodied component
is present, the system would be able to rule out false alarms and perform remote
diagnoses and triage [10].
The idea of the distributed sensor network is further discussed by Williams, Xie,
Ou, et al [31]. Contrary to many aging-in-place aids currently in use, which involve
devices that are worn or attached to objects, this group relies heavily on the analysis
of video data. They have developed a fall detector and an object finder using a central
processing node and multiple camera nodes in a given living space. A visualization of
this setup can be seen in Figure 2.2. In addition, in most major living spaces the user
will have a display, speakers, and an input device, which will allow communication
between the user and the central processing node. If a fall is detected, the central
processing node initiates the alert procedure.
Video data is also used in a collaborative project between the Sinclair School of
Nursing and the Department of Electrical and Computer Engineering at the Uni-
versity of Missouri [7]. They designed a system to increase effectiveness of exercise
routines for elders. The system uses video data and tracks the contours of human
bodies and extracts the silhouettes of the subjects, as seen in Figure 2.3. The study
compared posture and smoothness of gait in the two case study participants.
Distributed sensor systems for the elderly are also employed at TigerPlace, a
retirement community in Missouri designed to help the elderly to age in place [27].
They use a sensor network with an event-driven architecture, and consists of bed,
motion and stove sensors with a video sensor system. The framework is used to
detect falls, restless sleep, change in gait or activity levels, and in users’ routine. The
system includes a reasoning component involved with pattern recognition to allow it
to detect these changes.
9
Figure 2.2: Analysis of video data for fall detection and object finding [31]
10
Figure 2.3: Use of video data to track body contours and exercise effectiveness [7]
of dynamic Bayesian networks [12] in which some parameters are known and some
are hidden. The goal is to deduce the hidden parameters, given the parameters that
are observable [26].
A Bayesian network is a graphical representation of the probabilistic relationships
in a set of variables. Bayesian networks are significant in that they help one to
learn about causal relationships, and allow for the examination of incomplete data
sets [17]. Hidden Markov Models are particularly useful in the field of speech [26]
and gesture [20] [28] recognition.
Bennewitz, Burgard, and Thrun [8] utilize Hidden Markov Models and an
expectation-maximization (EM) algorithm to enable a mobile robot to follow people
and avoid obstacles. This task is accomplished through a machine learning (ML)
algorithm that learns the typical trajectories and patterns that comprise human mo-
tion. This ML might allow a mobile robot to make decisions and detours that would
minimize the chance of a collision.
Machine learning and pattern recognition are also used in the design of the Inde-
pendent Lifestyle Assistant (I.L.S.A.) developed at Honeywell Laboratories. Through
11
machine learning techniques, I.L.S.A. will be able to accurately model human be-
havior, which will allow it to recognize human actions and respond accordingly [13].
An application of Hidden Markov Models for gesture recognition can be seen in
work by Lee and Xu [20]. Lee and Xu have designed a system in which a user signs
letters of the American Sign Language alphabet while wearing a ‘cyberglove.’ The
cyberglove allows the computer to observe the gestures, which can be interpreted
and classified interactively as specific letters through the use of HMMs.
To use the Wii, the user places the sensor bar centered on top of the television
screen. The sensor bar consists of two groups of infrared LEDs. The infrared camera
at the head of the Wiimote observes these LEDs, allowing the user to point at the
12
TV screen with the Wiimote. When the user moves the Wiimote, infrared data is
sent to the Wii via Bluetooth, and the cursor on the screen moves as well.
In addition to the infrared camera, the Wiimote is also equipped with three
accelerometers, one for each dimension. The accelerometers allow the Wiimote to
detect acceleration along the X, Y, and Z axes, and to detect degree of pitch, roll, and
yaw, as seen in Figure 2.5. During gameplay, the accelerometer data streams back
to the Wii, and the Wii uses gesture recognition to assess motions. For example, it
might compare the accelerometer data from a user’s bowling motion with its internal
parameters for bowling to determine how many pins the user has knocked down.
Wiigee
Schlomer, Poppinga, et al [28] have designed a system called Wiigee designed to
harness and analyze the accelerometer data generated by the Wiimote. Unlike the
gesture recognition performed by Lee and Xu [20], the gesture recognition performed
in Wiigee is batch rather than interactive, and utilizes the technology of the Wiimote
rather than a separately designed piece of technology (such as the Cyberglove used
by Lee and Xu). The methodology in the two projects is similar (both use Hidden
13
Markov Models to analyze their data), but because the Wiigee project is open source
and utilizes the inexpensive Wiimote, it can be viewed as a very approachable and
extendable system. The Wiigee project recognizes gestures as follows: first the user
enters a training stage, in which a gesture is repeated some number of times (at
least eight repetitions are necessary to allow sufficient accuracy). After the user has
trained as many gestures as is desired, the user can enter the recognition stage, in
which the system attempts to pair whatever gesture is performed with a gesture
already in the system.
2.3.2 Bluetooth
Bluetooth is a short-range wireless communication method, used as a way for devices
to communicate when they are in proximity of each other [18]. Device discovery is
a phrase referring to the process in which Bluetooth enabled devices find each other
and then connect to one another.
Bluetooth is significant because it utilizes standard hardware, and makes short-
range wireless connection possible. The Nintendo Wii and Wiimotes are Bluetooth
14
enabled, and the Wii uses Bluetooth to connect to Wiimotes in use. Since Bluetooth
is ubiquitous, it can be used to allow other programs to capture data from the
Wiimotes. While the nature of Bluetooth makes it impossible to connect a Wiimote
to both the Nintendo Wii and to the computer, there has been work in the area
of Bluetooth signal sniffing [25], which allows a computer to monitor the Bluetooth
signal being sent from the Wiimote to the Wii.
2.4 My Project
My project utilizes the Wiigee software [28] for gesture recognition using a Wiimote.
For stereo triangulation, I have reimplemented the algorithms used by Hay, Newman,
and Harle [16]. In addition, I have written several programs to interact with the
Wiimotes, synchronize the data streams, and visualize the collected data. In addition
to being a Wiimote project, my project provides one possibility for caring for the
growing number of elders in society. Though much has been done [10], [31], [27]
in this area, I hope that by using the Wii my project will lead to an inexpensive
alternative or addition to these projects.
15
Chapter 3
The basic setup of the project can be seen in Figure 3.1. As can be seen from the
figure, the elder holds one Wiimote. This Wiimote is called the ‘playing Wiimote’
throughout this and subsequent chapters. In addition to the playing Wiimote, the
project uses two additional Wiimotes as cameras. These Wiimotes, which watch
infrared points on the elderly user’s body, are referred to as ‘camera Wiimotes’ or
‘watching Wiimotes’ interchangeably. User interaction, including the use of the sys-
tem by technicians, elderly users, and caregivers, is discussed in Section 3.1. The user
interface, the program written to connect to the watching Wiimotes and stream data
from them, is discussed in Section 3.2. Camera calibration is covered in Section 3.3.
The process of gathering data is discussed in Section 3.4, and data analysis is covered
in Section 3.5. Lastly, the actual experimental setup is discussed in Section 3.6.
16
Figure 3.1: Elder interaction with Wiimotes for Project
3.1.1 Technicians
In order to gather meaningful data, a technician must first calibrate the two watching
Wiimotes, visualized by the first state in the state diagram in Figure 3.2. The data
gathered in this step allows Matlab’s Camera Calibration Toolbox to calculate the
locations of the camera Wiimotes, in 3D space and relative to one another. Once
this information has been calculated, the system can triangulate the 3D locations of
points on the elder’s body.
A state diagram of the technician’s role can be seen in Figure 3.3. The watching
Wiimotes are mounted securely on a flat surface, with overlapping viewpoints for
stereo vision analysis [16]. The technician uses the Wiimotes’ Bluetooth capabilities
to connect the two watching Wiimotes wirelessly to the computer. This process is
accomplished using a program I wrote to interface with the Wiimotes, collect data,
17
Figure 3.2: State diagram representing user interaction
and record it to file. The technician then performs the calibration step. After the
calibration data has been gathered, the technician utilizes my program to convert
the raw infrared data into data that can be processed by the Matlab Camera Cal-
ibration Toolbox, and uses Matlab to calibrate each camera Wiimote individually.
The technician then calibrates the stereo system. After the stereo system has been
calibrated, the system is ready to use.
Provided that the camera Wiimotes remain stationary, this initial setup phase
need only be performed once.
18
A state diagram of the elderly user’s role can be seen in Figure 3.4. The elderly
user first trains the system with the gestures that he or she will be using. For
example, if the elder will be playing Wii Bowling, he or she will train the system
using a bowling motion. The elder then plays the Wii as normal, as seen in Figure 3.1.
During gameplay, the two camera Wiimotes and the playing Wiimote stream data
back to the computer, where it is saved to file. After gameplay is complete, the
elderly user can remove the reflective tape, and can disconnect the Wiimotes from
the computer.
3.1.3 Caregivers
The caregiver’s role in this process is to examine the data that was saved to the files,
so it is not necessary for the Wiimotes to be connected for this step. Using programs
I wrote, the caregiver can visually examine the gesture data, the infrared data, or
the infrared and gesture data together. This last program correlates infrared and
gesture data, allowing the caregiver to examine segments of IR data as they relate
to specific gestures or motions. The points that the caregiver monitors depends on
where the reflective tape is placed. For example, the caregiver could examine the
movement of the elderly user’s shoulders and hips during a specific bowling motion.
The placement of reflective tape on the shoulders and hips was suggested by Cynthia
Jacelon [19].
19
of synchronizing all Wiimotes such that the data streams will be accurate to one an-
other.
20
Figure 3.5: Screenshot of Wiimote Interface
Section 3.3.
Otherwise, the technician connects the third Wiimote to the Wiigee interface
(discussed in general in Section 2.3 and more specifically in Section 3.4.2) and helps
the elder to affix the reflective tape to points on his or her body, and the elderly user
21
plays the Wii. This step is described in Section 3.4.
After the elder has played the Wii or the calibration square step (Section 3.3.1) is
complete, the technician or the elder should press the ‘reset synchronization’ button.
This button stops data from being collected.
Within the context of this program, the elder or technician is able to resynchronize
the Wiimotes any number of times to allow different data to be collected. This feature
is especially useful if more than one elder will be playing the Wii.
When no more elders wish to play the Wii and data collection is finished, the
technician can disconnect the Wiimotes and exit the program.
This process of gathering infrared data can be seen in Figure 3.6.
22
The single camera calibration is described in Section 3.3.2, and the stereo calibration
is described in Section 3.3.3.
Figure 3.7: Screenshot of Calibration Square (from “Hacking Wiis for 3D tracking”
YouTube video [14])
The technician should continue to gather data in this way for several minutes,
making sure that all four points on the calibration square have been seen. After the
several minutes have passed, he or she should reset the synchronization, to stop the
data collection, and should then perform the single camera calibration.
23
3.3.2 Single Camera Calibration
The technician first opens my program to convert raw infrared data into calibration
scripts that can be understood and utilized by the Camera Calibration Toolbox for
Matlab [9]. The raw IR data files were generated by my program, as discussed in
Section 3.2.2. A sample of the data from one of these files can be seen in Figure 3.8.
The first number represents the time in milliseconds since synchronization. Within
the brackets, the first number identifies one of the four IR LEDs on the calibration
square. The second two are the X and Y pixel locations of the point seen, and
the final number represents the strength of the signal received. This signal strength
ranges from 0 (not very strong) to 5 (very strong). Note that in order to successfully
calibrate the cameras, the data in the calibration script must represent all four
infrared sources, so some of the timestamps in Figure 3.8 would not be usable.
The process of removing inaccurate or unusable data is discussed in greater detail in
Section 3.5.1.
7325: [0 255 414 1]
7325: [1 373 408 1]
7325: [2 258 533 1]
7325: [3 377 526 1]
7385: [0 256 411 0]
7411: [0 254 410 2]
7411: [1 374 408 1]
7412: [2 377 521 1]
7412: [3 260 531 2]
7420: [0 254 410 2]
7420: [1 374 407 1]
7421: [2 377 522 1]
7421: [3 260 531 2]
7421: [0 254 409 2]
7421: [1 374 408 1]
7422: [2 377 523 1]
7422: [3 259 531 2]
24
Figure 3.9: Diagram of camera calibration for one Wiimote
Once the calibration script has been generated, it is possible to run the script
through Matlab by simply typing the script name at the Matlab prompt.
The script contains the “real” and “observed” locations of the points. The system
expects a square shape, and the “real” length of each side is assumed to be 1.
Since the shape is a square, no matter the skew of the points seen, Matlab is able
to calibrate each camera individually with considerable precision, with no physical
measurement necessary. A sample of the IR data in the calibration script can be
seen below:
Q_1 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_1 = [ 351 333 213 227 ; 465 354 365 483 ];
The first set of information for each number, using the uppercase Qs, represents
the “real” locations of the points, and that this set of information never changes.
The second set of information, using the lowercase qs, are the observed x and y pixel
locations of the IR LEDs seen.
After a camera Wiimote has been calibrated individually, Matlab generates a
file containing all the intrinsic and extrinsic parameters of that camera, called (by
default) Calib_Results.mat. The intrinsic parameters are the parameters internal
to the camera, such as lens distortions, and the extrinsic parameters are the rotations
and translations of the calibration square, as seen by the camera.
Before proceeding to the stereo calibration step, the technician should rename the
Calib_Results.mat files to Calib_Results_left.mat for the left camera Wiimote,
and Calib_Results_right.mat for the right camera Wiimote.
25
Calibration Toolbox [9], in Matlab. It asks for the left and right calibration files,
which are by default Calib_Results_left.mat and Calib_Results_right.mat.
After it has those, it performs the stereo calibration, and generates a file called
Calib_Results_stereo.mat. This file contains more precise intrinsic and extrin-
sic parameters for each camera, and also contains all the information necessary to
perform stereo triangulation. The process of generating this file is visualized in
Figure 3.10.
26
camera can only see in 2 dimensions. Since both camera Wiimotes are looking at
the same place, they are able to see the same points. The calibration square step
(Section 3.3.1) generates all the data necessary to calibrate the cameras individually
and together, and once that is accomplished, the physical locations of the cameras
in space relative to each other are known, and so the Camera Calibration Toolbox
can be used to calculate the actual 3D locations of all the points seen.
Hay, Newman, and Harle [16] use the CWiid library [29] and the Camera Cali-
bration Toolbox for Matlab [9] with two Wiimotes with overlapping fields of view to
determine 3D locations of visible points in real time.
CWiid is a Linux-based library. Since my project is built on Mac OS X, and the
Apple Bluetooth implementation and the Linux Bluetooth implementation are very
different, I was unable to use it. Instead, I designed my own framework to allow for
similar stereo triangulation. Their system, written in C, calls the Camera Calibra-
tion Toolbox from within their program, using the Matlab C compiler. Their project
was interactive: they performed stereo triangulation on each new set of points seen,
as soon as they had been seen. To contrast, my project follows a batch implementa-
tion. After data collection is complete, my system takes the infrared data from each
camera Wiimote, and creates a new IR file for each camera. Each of these new files
contains only the points seen by both cameras. Using these two files, it is possible
to triangulate the 3D locations of the points seen using the stereo_triangulation
function included in the Camera Calibration Toolbox.
27
Figure 3.11: Diagram of Wiigee and gesture recognition
data can be broken down into smaller chunks according to gesture. By also grouping
the infrared data accordingly, the system can identify points of interest to make
visualization clearer and more useful. Knowing what gesture was being performed
allows for a context for the locations of the infrared points. Without the context
that gestures provide, it would be much more difficult to determine causation or
correlation of specific movements, making meaningful motility analysis unrealistic.
I had hoped to use the Wiigee library exclusively for my project, but it became
apparent that such a thing was not feasible. I found I was unable to use the Wiigee
library with more than one Wiimote, and that I was unable to generate infrared
data from the connected Wiimote consistently. Nevertheless, Wiigee still proved
extremely useful to me for its ability to train and recognize gestures.
In using Wiigee I found it necessary to modify and extend it to achieve cer-
tain additional functionality. Since I use a separate program to handle the infrared
data from the camera Wiimotes, a method of synchronizing the two data streams
generated by the separate programs was necessary.
In addition to adding this synchronziation functionality, I also modified Wiigee
such that it would write a gesture’s name to the text output file when a gesture
is recognized. This addition makes later identification of periods of interest more
straightforward.
28
of examining only the gesture data is described in Section 3.5.2, and the process of
visualizing both sets of data together is given in Section 3.5.3.
Each file does not represent every timestamp seen, but rather contains only those
timestamps representing all 4 points, seen by both cameras at the same time. If only
one camera Wiimote saw a given IR light source, or the IR points were not seen
at the same time, or not all 4 points were represented, the data is useless, and the
program discards it. Stereo triangulation is only possible if both camera Wiimotes
have seen the same IR point, at the same time. In this way, it can be guaranteed
that, although each camera saw the point at a different 2D location, the location in
3D space is actually the same.
Once the matrices have been created, my program generates a script containing
all the information necessary for the stereo_triangulation call. This script can
be seen in Appendix A.2. The caregiver can then run that script through Matlab,
which will calculate the 3D locations of the points given and write that information
to file. The script then graphs that information in 3 dimensions for the caregiver.
29
The gesture visualization program can be seen in Figure 3.12. The user selects
a file containing acceleration and gesture information as generated by Wiigee, using
the Choose New File button. The program then parses the data file and separates
it out into gestures, which the user can then select using the drop down menu at
right. Once the user has selected a gesture (in the case of the figure, the gesture
chosen is “bowl 1”), the acceleration data appears in the text box at the lower right.
The user can then press the Graph Data button, which graphs the acceleration data
over time in the x, y, and z directions. The user is also able to clear the graph, and
select other gestures within this file, or other files entirely.
A sample of the accelerometer output can be seen below:
0.08,0.9583333333333334,1.0
0.04,1.0,0.9166666666666666
0.16,1.1666666666666667,0.6666666666666666
0.28,1.125,0.7083333333333334
0.44,1.1666666666666667,0.9166666666666666
0.6,1.0833333333333333,1.25
In Figure 3.12, the gesture graphed is a bowling motion. A sample of this ac-
celeration data generated during this bowling motion can be seen in the text box
in the lower right corner. This data takes the form x, y, z. The graph shows the
acceleration data in all three axes during the bowling motion, and that the motion
had a very pronounced peak, the point at which the user was swinging the Wiimote
with the most speed.
30
Figure 3.12: Gesture visualization program
A second view of the IR LED square pattern can be seen in 3.14. Within Matlab,
a physical therapist, doctor, or anyone examining the data can move the view around
to see the data points from all angles.
31
Figure 3.13: Head-on view of IR LED square pattern during single bowling motion
32
Figure 3.14: 3D view of IR LED square pattern during single bowling motion
33
3.6 Experimental Setup
Bluetooth is designed for point-to-point communication, rather than broadcast com-
munication, so it is not possible to connect a given Wiimote to both the Nintendo
Wii and to the computer. This setback makes it impossible to gather data from an
elderly user who might simply be playing the Wii. While this setback makes the
program less unobtrusive for elders, the computational problem remains the same.
Using our setup, the elderly user would not play the Wii, but would perform similar
motions with a Wiimote connected to the computer. A project at Rice University,
having encountered the same problem, has devised the following workaround: they
have attached two Wiimotes to each other. One of the Wiimotes interacts with the
computer, while the other interacts with the Wii [24].
34
Chapter 4
The project did not progress to the point of collecting data while a user bowled,
because this project concentrated on building the fundamental infrastructure for the
motility analysis system. Now that this infrastructure is complete, capturing such
data is possible. However, as a result, the evaluation focused on the success and
accuracy of smaller parts of the system, rather than on the usability of data collected
as determined by a physical therapist. This system evaluation, and a discussion of
meaningful ways to present the collected data, are discussed in Section 4.1. Usability
issues from the elder’s perspective are discussed in Section 4.2. Finally, the fit of the
system for elder motility analysis is discussed in Section 4.3.
35
solution means successfully pairing events from the camera Wiimotes with one an-
other, making calibration of the stereo system much more successful. This solution
implies that the project will be able to make considerable advances in the future.
36
4.3 Fit of the Wii
Originally the Wii seemed to be a natural fit for elder motility analysis. Elders were
playing the Wii [33] and, in other arenas, Wiimotes were being used in unconven-
tional ways [21]. It seemed that elder motility analysis using the Wii was a clear
progression of these two separate occurrences. However, due to many stumbling
blocks encountered over the duration of this project, the suitability of the the Wii
and Wiimotes for motility analysis is debatable.
One key limitation is that, due to the nature of Bluetooth, the user cannot connect
a Wiimote to both the computer system and the Wii console simultaneously. This
problem eliminates the notion of an “unobtrusive” system. If the project were to
be expanded, a workaround such as the one utilized by those at Rice University [24]
could be used. Though such a workaround was outside of the scope of my project,
some possibilities are discussed in Section 5.2.
Finally, while Wiigee was suitable for gesture recognition, sometimes after train-
ing it with gestures, it was unable to recognize them, either saying that a gesture
that should be familiar was “unknown” or that a motion performed was a separate
gesture entirely (for example, “recognizing” a bowling motion as a circle). When
trained with only one gesture, Wiigee showed an 85% success rate, over 20 ges-
ture recognition attempts. When trained with 2 or more gestures, that success rate
dropped to 61.9%, over 42 attempts.
Using the Wii provides a unique way to perform motility analysis. Additionally,
the system I designed remains far less expensive than the research systems available.
Now that the infrastructure of this system is in place, further experimentation will
determine the fit of the Wii for motility analysis.
37
Chapter 5
5.1 Conclusions
I had hoped that my project would culminate in a collection of software that could
be used by physical therapists and doctors to analyze the motility of elder Wii users.
However, the scope of the project was simply too large for that of a one-year project.
As a result, while much was accomplished, there is still considerable work to be done
before this project is usable as initially imagined.
The system I created is not as unobtrusive as originally hoped, and the problems
encountered make it difficult to determine its ability to assess motility. However,
the problems encountered are not insurmountable, and with additional work, this
system could sufficiently capture a person’s movements for motility analysis.
The project was successful in that I was able to connect all three Wiimotes to
the computer, and to write programs that could interact with these Wiimotes in
real time. In addition, I was able to synchronize the streams of data generated by
each program. I was then able to use additional programs, both that I wrote and
otherwise, to visualize the data graphically.
One success, achieved at the very close of my project, is in that of increasing the
38
success and accuracy of camera calibration, as discussed in 4.1. This was a stumbling
block I struggled with for a large portion of the project, and now that it has been
overcome, the project is that much closer to completion.
The final stumbling block is to find some way to observe IR points on the user’s
body, since the use reflective tape was largely unsuccessful. This problem is an area
that requires immediate work for the system to be considered successful or complete.
Additional work would allow the system to be more effective, and more likely to be
used and useful by elders and physical therapists. This additional work is discussed
in Section 5.2.
39
cams could be adjusted so they only ‘see’ infrared, just as the Wiimotes do. The
viability of using low-cost webcams instead of Wiimotes as cameras in this project
is an option worth exploring.
The Wiigee project’s gesture recognition library proved extremely useful to me
throughout the course of my project, and my project would not have been possible
without it. However, Wiigee does not support the saving of gestures between gesture
recognition sessions. To extend Wiigee and add this functionality would make a huge
impact on my project. If users did not have to retrain the system at each new session,
the project would become more streamlined and intuitive. Further, it would pave
the way for additional changes, all of which would serve to improve the usefulness
and relevance of the project.
After recognizing a gesture, Wiigee adds that instance of the gesture to its library,
to more finely tune each gesture. This feature is very useful when it works, because
the system is being constantly refined. However, if Wiigee incorrectly recognizes a
gesture, it still adds that motion to the system. This issue means that with only
one incorrect recognition, the system becomes inaccurate. One way to combat this
problem would be to extend Wiigee to allow the user to accept or reject recognition of
gestures. With this added functionality, Wiigee’s success rate would improve, which
would improve the success of the project.
In addition to adding the saving and loading gestures functionality to Wiigee, the
final stumbling block to transform my system into one which is truly unobtrusive
would be in devising a way to connect the playing Wiimote to both the computer sys-
tem and the Wii. Due to the nature of Bluetooth, this goal as it stands is unrealistic.
However, there are two possible solutions.
One solution is to build a device that is equipped with Bluetooth and has three
accelerometers, as a Wiimote has. This device could be attached to the playing
Wiimote, and it could stream data back to the computer while the Wiimote talks to
the Wii. Adding this functionality would make the system unobtrusive, because the
system would become a background feature while the user plays the Wii.
A second possibility is to utilize Bluetooth signal sniffing [25], which would allow
the computer to passively ‘eavesdrop’ on the Bluetooth signal being sent from the
Wiimote to the Wii.
Once a workaround has been developed to allow the playing Wiimote to connect
to both the computer and the Wii, the system I designed can be expanded in several
ways. The system could support multiple users, who could each train the system
with their gestures once. Each time the user wishes to play the Wii, he or she could
select his or her name from a menu, and select the desired Wii game. Depending
on the game selected, the system would load the required gestures and key button
40
presses, so the user would not need to do any additional work. Finally, once gameplay
is complete, a physical therapist or doctor could use the system to analyze the user’s
motility, examining both individual sessions and over time.
In order for the system to be the most useful for doctors and physical therapists,
it would be important to work with them to understand what types of data visu-
alizations would be the most useful, and how these visualizations could be created
using the data gathered by this system.
After these changes have been made, it would be necessary to test the new and
updated system on elders. One could gather data over time and, with the aid of
physical therapists and doctors, analyze the data.
41
Appendix A
Sample Scripts
clear;
nx = 1024;
42
ny = 768;
Np = 4;
est_alpha = 0;
Q_1 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_1 = [ 351 333 213 227 ; 465 354 365 483 ];
Q_2 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_2 = [ 351 331 210 229 ; 464 349 365 483 ];
Q_3 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_3 = [ 351 331 210 229 ; 463 348 365 483 ];
Q_4 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_4 = [ 351 328 209 231 ; 459 347 366 483 ];
Q_5 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_5 = [ 353 327 207 235 ; 456 343 368 484 ];
Q_6 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_6 = [ 354 325 206 236 ; 455 342 369 484 ];
Q_7 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_7 = [ 357 323 206 239 ; 449 340 370 484 ];
Q_8 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_8 = [ 359 323 206 242 ; 448 340 370 484 ];
Q_9 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_9 = [ 359 323 206 243 ; 448 339 369 483 ];
Q_10 = [ 0 1 1 0 ; 0 0 1 1 ; 0 0 0 0 ];
q_10 = [ 359 327 209 244 ; 449 339 368 482 ];
n_ima = 10
% Set the toolbox not to prompt the user (choose default values)
dont_ask = 1;
43
% Saves the results into a file called Calib_Results.mat:
saving_calib;
xL =
623 622 742 743
503 385 387 504
XL =
-1.2141593e+00 -1.2205303e+00 3.4750352e-01 3.5666067e-01
1.4573386e+00 -1.8561229e-01 -1.5284220e-01 1.4290919e+00
2.4846113e+01 2.4959722e+01 2.3935121e+01 2.4982121e+01
xL = load(‘/Users/becca/Documents/research/thesis/bowlL_0.m’)
44
xR = load(‘/Users/becca/Documents/research/thesis/bowlR_0.m’)
load(‘Calib_Results_stereo.mat’)
45
Bibliography
[6] Robot developed by computer scientists to assist with elder care. http://www.
umass.edu/loop/print.php?articleID=74082, April 2008.
46
[11] R. V. Emmerik. Motility analysis at the kinesiology lab. Personal communica-
tion, August 2009.
[13] V. Guralnik and K. Z. Haigh. Learning models of human behaviour with se-
quential patterns. American Association for Artificial Intelligence Workshop
”Automation as Caregiver”, 2002.
[16] S. Hay, J. Newman, and R. Harle. Optical tracking using commodity hardware.
In Proceedings of the 7th IEEE and ACM International Symposium on Mixed
and Augmented Reality, 2008.
[20] C. Lee and Y. Xu. Online, interactive learning of gestures for human/robot
interfaces. Technical report, The Robotics Institute, Carnegie Mellon University.
[22] M. J. Mataric. The role of embodiment in assistive interactive robotics for the
elderly. In American Association for Artificial Intelligence Fall Symposium on
”Caring Machines: AI for the Elderly”, Arlington, VA, November 2005.
47
[24] M. O‘Malley. Wii bit of fun at Rice University has serious intent. Personal
communication, January 2009.
[33] M. Yam. Wii invades retirement home: Nintendo console has officially gone
geriatric. Daily Tech online magazine: http://www.dailytech.com/article.
aspx?newsid=6191, February 2007.
48