Novel Human Computer Interaction Principles For Cardiac Feedback Using Google Glass and Android Wear
Novel Human Computer Interaction Principles For Cardiac Feedback Using Google Glass and Android Wear
Novel Human Computer Interaction Principles For Cardiac Feedback Using Google Glass and Android Wear
Robert Richer1 , Tim Maiwald2 , Cristian Pasluosta1 , Bernhard Hensel2 and Bjoern M. Eskofier1
1
Digital Sports Group, Pattern Recognition Lab, Department of Computer Science
2
Max-Schaldach-endowed Professorship for Biomedical Engineering
Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany
Email: robert.richer@fau.de
Abstract—This work presents a system for unobtrusive allow an ubiquitous and pervasive monitoring of vital signs
cardiac feedback in daily life. It addresses the whole pipeline in the patients’ daily environment without any restrictions in
from data acquisition over data processing to data visualization activity or modification in behavior, during physical activity
including wearable integration. ECG signals are recorded
with a novel ECG sensor supporting Bluetooth Low Energy, or at rest.
which is able to transmit raw ECG data as well as estimated Several research groups presented mobile applications for
heart rate. ECG signals are processed in real-time on a ECG recording. Oresko et al. [3], Yen et al. [4], and Makki
mobile device to automatically classify the user’s heart beats. et al. [5] proposed ECG analysis for detecting cardiac abnor-
A novel application for Android-based mobile devices was malities, whereas Valchinov et al. [6] and Richer et al. [7]
developed for data visualization. It offers several modes for
cardiac feedback, from measuring the current heart rate to used ECG analysis for sports and fitness assessment. Gradl
continuously monitoring the user’s heart status. It also allows et al. [8] presented an application for Android-based mobile
to store acquired data in an internal database as well as devices with an algorithm for real-time ECG monitoring and
in the Google Fit platform. Further, the application provides automated arrhythmia detection. These approaches mainly
extensions for wearables like Google Glass and smartwatches focused on the algorithm development, and did not provide
running on Android Wear. Hardware performance evaluation
was performed by comparing the course of heart rate between a solution for daily life conditions. There is definitely a
the novel ECG sensor and a commercial ECG sensor. The mean need for developing a user-friendly application that does not
absolute error between the two sensors was 4.83 bpm with a distract the user during its everyday activities. Nowadays, it
standard deviation of 4.46 bpm, and a Pearson correlation of is possible to display information about the user’s vital signs
0.922. A qualitative evaluation was performed for the Android as well as interacting between wearable and mobile phone
application with special emphasis on the daily usability and the
wearable integration. When the Google Glass was integrated, at the same time.
the subjects rated the application as 2.8/5 (0 = Bad, 5 = The purpose of this paper is to develop a solution that
Excellent), whereas when the application was integrated with incorporates novel human computer interaction principles
a smartwatch the rating increased to 4.2/5. with cardiac feedback that is usable in daily life situations.
Keywords-Body Sensor Networks; Electrocardiography; This work presents a novel low-power data acquisition
Wearable Computing; Android Application; Human Computer ECG hardware using Bluetooth Low Energy. The main
Interaction contribution however is the implementation of an application
called DailyHeart for Android-based mobile devices with
I. I NTRODUCTION special focus on human computer daily life usability and
Changes in the demographic structure are characterized wearable integration.
by lower reproduction rates, higher life expectancy, and a
II. M ETHODS
decrease in mortality [1], leading to an increase of chronic
diseases. Therefore, a need to continuously monitor the indi- A. Data Acquisition
vidual’s cardiac functions throughout the day for preventing The sensor module presented here measures the user’s
fatal disorders becomes increasingly important. ECG, which is sampled with a 12-bit resolution using the
Because the influence of the environment on the measure- integrated ADC of a TI MSP430-FR5969 microcontroller
ment of patients’ physiology (commonly manifested as the (Texas Instruments, Dallas, TX, USA), whose clock fre-
“white coat syndrome” [2]) is not negligible, mobile devices quency was dropped to 1 MHz due to energy saving aspects.
like smartphones or tables are an ideal platform, given their The sensor system was configured to measure the user’s
integration into daily life. ECG, but it can moreover be configured for measuring
Wearable healthcare technology is a promising way to physiological signals like EMG or respiratory rate [9].
improving the quality of life for chronic disease patients and ECG data was transmitted to the mobile device by a
elderly people as well as healthy individuals. These solutions TI CC2541 system-on-chip (SoC), which is an integrated
QRS Detection The raw ECG signal was processed with
a pipeline of digital filters proposed by Pan & Tompkins
RA [13]. It consists of a cascaded Bandpass filter, a five-point
-
DRL derivative filter, a squaring operation, and a moving window
•
integrator. QRS complexes were then isolated from the
Lead II algorithm output.
Template Formation In order to allow an automatic
classification of heart beats, a feature computation using two
LL QRS complex templates which were automatically found in
+
the ECG signal and adapted over time was implemented [8].
Feature Extraction The features for the heart beat classi-
fication were extracted from 400 ms windows centered on
Figure 1. Lead positioning of the BLE-ECG Stamp. Schematic drawing every isolated R peak [8]: The difference in absolute area
of the sensor placement and the lead positioning; RA: Right Arm, LL: Left
Leg, DRL: Driven Right Leg
and the maximal Pearson correlation between the templates
and the current beat as well as the the width of the QRS
complex, and the R-R interval between the last two QRS
solution for Bluetooth Low Energy (also referred as BLE, complexes.
or Bluetooth LE) applications, combining a high power Beat Classification The isolated heart beats were classi-
Bluetooth antenna with an improved 8051 microcontroller. fied according to the decision tree proposed by Gradl et al.
The Bluetooth Low Energy Application of the sensor [8], distinguishing between abnormalities in waveform and
system named BLE-ECG Stamp provides two different Blue- abnormalities in rhythm/pace.
tooth Services for transferring data to a mobile device: an
EcgService and a HeartrateService. The EcgService streams C. Data Visualization
the sampled ECG signal directly. In order to reduce the The data acquired by the BLE-ECG Stamp was transmit-
power consumption of the CC2541 SoC, the raw values are ted to the proposed application for Android-based mobile
wrapped into packages with a size of 10 samples before devices called DailyHeart. Android (Google Inc., Mountain
transmission. By performing a heart rate calculation on the View, CA, USA) was used because of its open source char-
sensor side and only transmitting the current heart rate acteristics, the portability of code since the programming
for the HeartrateService, the data rate (and thus the power language is Java, and because it is the most widely spread
consumption) is significantly reduced. The HeartrateService mobile operating system up today [14] with a market share
implements the Heart Rate Profile (HRP) as proposed by of 83.1 %. Furthermore, the BLE APIs of Android provide
the Bluetooth SIG, and hence provides compatibility for support for Bluetooth LE since software version 4.3 (API
other applications supporting this Profile [10]. Here, the
EcgService was used since the entire ECG signal was needed Lowpass
Highpass
for subsequent heart beat classification. Band
R-Peak Detector
The sensor recorded a 1-channel-ECG at a sampling Differentiation
Squaring
frequency of 256 Hz with 3 electrodes that were placed ac- Wnd-Int [150 ms]
Yes Peak
cording to Lead II of Einthoven’s triangle [11]. The negative Int
placed on the left costal arch (Figure 1). The third electrode Thr R
was placed on the right breast and used for the driven right
leg circuit, which is often added to physiological signal
amplifiers in order to eliminate electromagnetic interference
[12]. Template
plate
pla te 1
Template 2 Waveform Window
B. Data Processing
MaxCorr ArDiff R-R QRS
QRSwidth
The algorithm used for data processing was proposed by
Gradl et al. [8]. It is able to perform a real-time detection of Thresholds
QRS complexes in an ECG signal, followed by an automated
Abnormal Classes Normal Classes
classification of normal and abnormal heart beats. The
pipeline is visualized in Figure 2 and consists of four stages: Figure 2. Algorithm pipeline. Overview of the pipeline that is used for
QRS Detection, Template Formation, Feature Extraction, QRS detection and heart beat classification (modified with permission from
and Beat Classification. [8])
Figure 3. User interface of DailyHeart. Different screens of the application. From left to right: Main menu for selecting the recording mode; Activity
to measure the user’s current heart rate; Activity consisting of two tabs to monitor the user’s ECG.
level 18). The DailyHeart application was implemented ECG. In addition, a DailySimulate mode was implemented
using the Android SDK 5.0 (API level 21), which was for re-playing pre-recorded data. All activities plot the live
released with Android 5.0 Lollipop in November of 2014. ECG signal and display the current heart rate to the user. The
It introduced a completely new design language called DailyMonitor mode provides also the average heart rate, the
material design that is characterized by Google as bold, course of heart rate since the beginning of recording, as well
colorful, and responsive, providing a new and intuitive user as further features obtained from the ECG processing. At the
experience [15]. end of the measurement, a summary of the acquired data is
shown as well as buttons to restart the measurement or save
The application consists of several components: a BLE the results to the internal database and/or Google Fit. The
Service for data delivery, a User Interface for informing history screen accesses the database and lists all entries.
the user and receiving input, a Data Storage unit for saving
the recording results, and a Wearable Extension for Google Data Storage The Android operating system is shipped
Glass and Android Wear-based smartwatches. with a SQLite database that allows to store results into the
database for retrieving it again at a later time. The Dai-
BLE Service This component is implemented as an An- lyHeart application uses this feature for local data storage
droid background service. It provides ECG data for the and also provides support for Google Fit, a health-tracking
processing unit and delivers the results to the user interface. platform where users can access their fitness and health data
The data can either be obtained via live mode or simulation acquired and uploaded by different devices and applications.
mode. In live mode, the service establishes a connection
to the BLE-ECG Stamp and receives the acquired data Wearable Extension Android Wear is a modified version
via Bluetooth LE, whereas in simulation mode, real-time of the Android operating system for smartwatches and
sampling is simulated by reading data from the external other wearables by integrating smartphone features with a
storage. It is possible to simulate pre-recorded data from watch form factor. Information on smartwatches running
DailyHeart, or data from the MIT-BIH Arrhythmia databases on Android Wear can be displayed either by extending
[16]. Handlers are used to pass data to the service’s signal regular Android system notifications for wearables, that are
processing unit and the user interface. By using a service, then synchronized between smartphone and wearable, or
data delivery and processing runs independently from any by creating custom applications for Android Wear that can
foreground process, which allows the user to switch between communicate with the smartphone application. The extended
applications without terminating the connection. notifications allow the user to fully control the application
after it has been launched. They furthermore continuously
User Interface The application was designed following update the screen with new information. In this work, a
the material design guidelines in order to provide a clear, custom application for smartwatches was implemented in
intuitive and appealing user interface for cardiac feedback. order to enable voice capabilities. This allows the user to
It features modes that were designed for different daily life start the application by saying the phrase “Okay Google,
situations (Figure 3 shows screen examples): a CurrentHR start DailyHeart!”, or to directly measure the current heart
mode to perform a snapshot heart rate measurement, a rate by saying “Okay Google, what’s my heart rate?” or
AnalyzeECG mode to analyze the ECG over a period of time, “Okay Google, what’s my bpm?”.
and a DailyMonitor mode to continuously monitor the user’s
Table I
Google Glass is a wearable device with an optical head- R ESULTS OF THE S ENSOR E VALUATION . MAE = M EAN A BSOLUTE
mounted display (OHMD), a camera, and several sensors E RROR , SD = S TANDARD D EVIATION , PC = P EARSON C ORRELATION
like IMU, ambient light sensor, proximity sensor, and a GPS
MAE SD PC
receiver [17]. It also includes chips for Wi-Fi and Bluetooth (bpm) (bpm)
communication. Google Glass can be controlled by voice Subject 1 2.43 2.91 0.957
commands or by swipe-and-click gestures performed on a Subject 2 6.54 6.58 0.920
touchpad located at the right of the frame. By using the same Subject 3 3.85 3.45 0.912
Subject 4 5.64 4.29 0.913
extended notifications as for smartwatches, the DailyHeart Subject 5 5.68 5.05 0.906
application also provides support for Google Glass. The Mean 4.83 4.46 0.922
OHMD offers cardiac feedback by overlaying information
directly into the user’s field of view. 50
40
Average Rating
4
The application layout of DailyHeart was reduced to only
3
the essentials in order to provide clarity and to prevent
2 unnecessary loss of time by getting used to the handling,
1 which was the main purpose of introducing the material
0
design. Future work could use more detailed queries that
are compiled against the SQLite database implemented for
e
ea res
an in g
tra ion
sa ity
y
lit
p e an c
ce
es
l U il
in
io
bi
- H dl
t
u
al ab
c
an
r
at
Ap e a r
dl
ct
tu
er Us
r
ea Ha
ea Fe
ar
ct
p
is
e a Ap
F
-
Ov all
sc
W s-
r - er
Di
o n las
s
W s-
-
r
a
ea Ov
on lass
r-
-
l
G
o n la s
G
W
ar
could be combined with additional data from GPS and
ea n
W -
n
e
G
G
o
on ss
on
W
ily ar t
ea o n
ily a r t
ea n
r t la
rt
rt
Da He
ea
ily a r t
ily a r t
rt
H on
rt
H
H
il y
ily
e
Da He
H
ily r t
Da
Da
H
H
ly
ea
ily
Da He
Da
Da
Da
ily
[3] J. J. Oresko, Z. Jin, J. Cheng, S. Huang, Y. Sun, H. Duschl, [16] G. B. Moody and R. G. Mark, “The impact of the MIT-BIH
and A. C. Cheng, “A wearable smartphone-based platform Arrhythmia database,” Engineering in Medicine and Biology
for real-time cardiovascular disease detection via electrocar- Magazine, IEEE, vol. 20, no. 3, pp. 45–50, 2001.
diogram processing,” Information Technology in Biomedicine,
IEEE Transactions on, vol. 14, no. 3, pp. 734–740, 2010. [17] S. Torberg, “Google Glass teardown,” http://www.catwig.
com/google-glass-teardown/, 2013, accessed 2015-02-23.
[4] T.-H. Yen, C.-Y. Chang, and S.-N. Yu, “A portable real-
time ECG recognition system based on smartphone,” in [18] S. Thomas, J. Reading, and R. J. Shephard, “Revision of the
Engineering in Medicine and Biology Society (EMBC), 2013 Physical Activity Readiness Questionnaire (PAR-Q). Cana-
35th Annual International Conference of the IEEE. IEEE, dian journal of sport sciences, 1992.
2013, pp. 7262–7265.
[19] J. M. Bland and D. Altman, “Statistical methods for assessing
[5] M. M. Makki, G. A. Saade, A. G. Altouma, S. Al-Terkawi, agreement between two methods of clinical measurement,”
A. Baobeid, and R. Tafreshi, “Acquiring and analyzing The lancet, vol. 327, no. 8476, pp. 307–310, 1986.
electrocardiograms via smartphone to detect cardiovascular
abnormalities,” in Biomedical and Health Informatics (BHI), [20] J. I. Smith, “Google’s eye doctor admits Glass
2014 IEEE-EMBS International Conference on. IEEE, 2014, can cause pain,” http://observer.com/2014/05/
pp. 277–280. googles-eye-doctor-admits-glass-can-cause-pain/, 2014,
accessed 2015-02-02.
[6] E. Valchinov, A. Antoniou, K. Rotas, and N. Pallikarakis,
“Wearable ECG system for health and sports monitoring,” [21] G. Marshall, “Google Glass: Say goodbye to your
in Wireless Mobile Communication and Healthcare (Mobi- privacy,” http://www.techradar.com/news/mobile-computing/
health), 2014 EAI 4th International Conference on. IEEE, google-glass-say-goodbye-to-your-privacy-1134796, 2013,
2014, pp. 63–66. accessed 2015-02-02.