Breakout: Design and Evaluation of a Serious Game for Health Employing Intel RealSense

Clinical evaluations in recent years depict a major potential of the employment of motion tracking technologies coupled with other media in healthcare application. Benefits include the enhancement to clinical scores of partly disabled patients, a broader community integration associated with a more positive attitude and more independence in scheduling the therapy are registered as benefits [1]. Among the means of implementation, serious games for health attracts the most attention. Owed to its game characteristics uniform to commercial games, it allows for more immersion into the game and a secondary focus on entertainment, thus securing motivation for long term engagement and more success if employed as means of an accompanying rehabilitation method [2]. Most recent approaches to serious games in rehabilitation are based on Microsoft Kinect, a depth-camera featuring motion-tracking, which is chosen for its convenience in handling, affordability and reasonable pricing. Fewer clinical studies employ other sensors or customized motion tracking systems [3].

The objective of this research is to depict critical interaction design aspects encountered in employing the motion capture system Intel RealSense, to design a serious game prototype for upper body rehabilitation with focus on game design. The fundamentals are based on commercial game design intertwined with serious games characteristics, enabling a more immersive gameplay than comparable serious games for health developed with Microsoft Kinect. The differences in design requirements compared to Microsoft Kinect originate in the implemented 3D virtual world and close-range interaction with the motion capture system.

The following Sect. 2 will point out distinctions between Intel RealSense and Microsoft Kinect motion capture system and refer to related research in the thematic field of serious games for upper body rehabilitation. Subsequently, in Sect. 3 the concept for the serious game is introduced: gameplay settings and interaction design including control and feedback are addressed. The successive Sect. 4 presents the evaluation method carried out in this study. The second last Sect. 5 aggregates the evaluation findings and discusses issues encountered in the game design. Concluding with Sect. 6, a short summary about the research is given and prospective future work is proposed.

In the first Sect. 2.1, attributes and application area of motion capture system Microsoft Kinect V2 and front-facing Intel RealSense cameras are introduced. Sequentially, in Sect. 2.2 the concept of serious games is addressed and exemplary implementation in form of serious games for health is presented.

2.1 Motion Capture System

The employed depth-camera with specifications like Microsoft Kinect is optimized for close range interaction and allows for more accurate tracking of motoric movements in direct comparison to Microsoft Kinect. Table 1 depicts core attributes of each system.

Intel RealSense is available either bundled in a developer kit or can be purchased as an integrated unit substituting the ubiquitous camera unit in notebooks and tablets. The precise camera specifications depend on whether it is employed as a front- or rear-facing camera, albeit here only the former ones are reviewed. This condition reflects the advantage of Intel RealSense’s small size and weight, allowing for ex works integration and eliminates the necessity of adapters as seen with Microsoft’s Kinect V2 [4]. The most recent (front-facing) version is the SR300, succeeding the F200 version employed in this research and offers novel features to the system such as a new tracking mode labeled “Cursor Mode” for accurate point tracking, person tracking, increased range and tracking speed [5].

Microsoft Kinect deploys a Time-of-Flight sensor to measure the distance between an object and the sensor, while Intel RealSense generates the depth map via triangulation of a projected infrared grid. Another distinct difference is found in the field of application, also reflected in the respective working distances. Microsoft Kinect is used for far-range applications and allows full body tracking of up to six persons simultaneously with 25 skeleton joints. Further on, simple gesture detection as thumbs up, closed and open hand can also be recognized. Intel RealSense however, is focused on near-field application and enables subtler identification of features: hand gesture recognition with single joints and face tracking with up to 78 landmarks allow to detect precise motor movements. This is reflected in the accompanying Intel SKD featuring a multitude of predefined gestures like two fingers pinch, full pinch and victory sign [4, 5].

2.2 Serious Game Design and Implementation

Serious games follow the idea of guiding the user, also known as the player, with inherent game mechanics to a predetermined objective distinct from entertainment purposes such as transfer of knowledge or skills. The gamification nature hereby supports motivation [2].

The penultimate goal of the implemented serious game is the improvement of mobility for upper body, both in the sense of rehabilitation as well as exercising. Therefore, a principal aspect in game design is an easy and intuitive interaction which is adoptable to an individual’s needs and scalable to the progress over the course of playing the game. This also poses high requirements to the hardware which is preferably low-cost, customizable and easy to install and use. Next generation depth-cameras, initially seeing a widely-spread use in gaming consoles like Xbox 360, offer great opportunities and synergy in the context of serious games for health: as a means of easily accessible motion capture technology, it facilitates recording training sessions and analysis of data [3].

This idea has been picked up in research and many approaches to combine Microsoft Kinect and serious games for health can be enumerated. Thematic fields range from rehabilitation and restoration of mobility after contracting diseases to the enhancement of process monitoring for distinct exercises and improving fitness by exercising.

Around 2012, low-cost serious game frameworks for rehabilitation covering essential aspects such as high configurability to adapt to patients’ requirements and a dynamic adaption of game difficulty in relation to their progress have surfaced [7]. Exemplary implementations of serious games for upper body rehabilitation using low-cost motion capture systems such as Intel RealSense or Microsoft Kinect are presented below. Most approaches develop the serious game using basic motion sequences as starting points and can be categorized as matching games in a 2D virtual environment.

In “Post office trouble” the player grabs packages with a grasping gesture and matches them to boxes with predetermined topics. Game framework parameters such package size and distance to boxes are adoptable to players’ motion capabilities. An initial small study with 8 healthy people aged between 52 and 79 indicates good engagement, while usability is restricted due to unnatural movements. This is owned to setup restrictions with Intel RealSense, such as the requirement for the palm to face forwards [8]. A motion-wise similar game is a jigsaw puzzle, implemented with Microsoft Kinect: matching color blocks must be identified and moved to their corresponding position in the jigsaw puzzle, whereas the reach distance to grab the puzzle pieces is adjustable. The player’s movement during the game can be recorded using inbuilt software functions, but requires further development for analysis. The game has been tested under supervision of a physiotherapist with a patient suffering from post-stroke impaired hand movement. Results point towards some acclimatization time but underline the general playability, although a capability of balancing one’s extremities has been identified as a requirement to operate the game [9]. In “PhysioMate”, the players are first taught motion sequences called routines in a preliminary imitation game, which awards points for mimicking predetermined motions devised by a physiotherapist. These movements are then performed in the principal game: a matching game, where appearing objects are to be moved to the designate waste container. The required motion sequence reflects the pre-taught routines. Data regarding game progress is recorded and can be accessed by the physiotherapist [10].

An approach to more immersive gameplay with Microsoft Kinect has been presented with the serious game “The Sorcerer’s Apprentice” in a 3D game environment. The game is targeted at patients suffering from one-sided Shoulder Impingement Syndrome and intends to improve overall mobility through exercises. Specific gestures, performed with impaired arm, trigger actions in the game required to advance in a level while the healthy arm assumes all other motions. The implementation of meaningful play through a story differs from predominant implementations of serious games for health and enables more immersive gameplay, while further evaluation towards usability and engagement are needed [11].

In view of general functionality and capability of motion capture systems as a means of objectively assessing aspects of movement, a comparative analysis from 2016 of published papers reviewing Microsoft Kinect points towards a sufficient precision, especially for gross spatial movement [12]. The data basis is published papers, which employ additional motion capture devices or sensors to validate the data collected with Microsoft Kinect. With a view to stroke rehabilitation applications, a maximum average normalized root mean squared error of 1.74 cm in comparison to the research-grade motion capture system OptiTrack has been measured, verifying the camera’s potential as low-cost progress tracking system [13]. Since camera specifications for Microsoft Kinect and Intel RealSense are largely comparable, the findings are to some extend valid for the latter motion capture system as well.

Research groups also attempt to increase the accuracy by altering existing algorithms for image analysis: for instance, the accuracy of Kinect skeletal joint coordinates has been improved by implementing constraints on recognized body segment lengths and orientation in the existing code, thus reducing body segment variance by up to 72% [14].

To garner knowledge on critical interaction design issues for near-field applications using motion capture technology in healthcare sector and to highlight potential assets and drawbacks, a serious game for rehabilitation using Intel RealSense F200 is developed and presented in this paper. The game design is inspired by commercial game design while integrating core aspects of serious game design [3, 15,16,17]: intuitive user-centered game design, extensive configurability and control feedback, an adequate challenge, and balancing a meaningful play while keeping the objective of health improvement. Based on these aspects, the game concept will be briefly introduced focusing on core elements and is followed by an evaluation performed in the context of user testing.

The game was developed in Unity 5, a cross-platform engine with support for Intel SDK, and a developer version of Intel RealSense F200. The assets applied in game design, namely models, animations, music and fonts, are taken from openly accessible sources in the Unity asset store, i.e. Unity essential packages, and from Intel SDK with the intention of speeding up prototyping phase.

The prototype “Breakout” follows the archetype of a survival game, in which players need to protect a given objective from enemy attacks and hereby accumulate points for their achievements tracked in a high score. The game design is based on commercial game structure adapted to serious games for health, especially with regard to meaningful play through an immersive story and environment. In the following sub sections, the game concept as well as multiple game characteristics are explained in detail.

3.1 Gameplay Design

The gameplay is the compound framework of rules for setting the game’s environment. It describes the player’s range of actions and covers essentials such as a concept scheme, in which the previously described story line is fitted. In the following Fig. 1, a sketch of the game concept can be found.

Concept scheme for game play

Full size image

“A” represents the game area and “C” the fixed position of an object, the player must guard. Over the course of time, enemies, represented by figures “1” and “2”, randomly spawn on the map and move per their programmed AI to destination “C” while the player must prevent their advance. This is achieved by controlling an ingame hand object “H”: by touching enemies, damage is inflicted to them and they vanish when a damage threshold is exceeded. Disabling enemies generates a score equal to their difficulty setting, whereas distinct enemy types are distinguishable through their appearance (overview in Fig. 2).

Concept for ingame objects including enemies

Full size image

The current iteration of the game is situated in a non-time-limited game mode. A game session is terminated, when the player cannot prevent the enemies from approaching the objective, thus deducting life points from the player’s health bar until its reduction to zero. The present accumulated score is then saved to a high score chart and is visible in the main game menu.

The rough game interface is represented in Fig. 3. In the lower right corner, the health bar for the object to be protected is displayed. When damage is sustained, the bar is reduced and the screen briefly flashes to support visual feedback. The current game score is tracked in the upper middle part of the screen. The remaining two UI elements support the control via visual feedback: the upper left symbol represents a hint when the motion capture system cannot detect the player’s hands during a game session and the lower right rectangular box transmits the player’s live stream as recorded by the camera.

Concept for game interface

Full size image

3.2 Game Control Design

The game can be controlled contact-free without the necessity of further peripheral devices like a mouse and keyboard to facilitate intuitive and immersive control. The ingame camera view is set as perspective due to the size of the terrain in the virtual environment to better convey the depth aspect of the map. It is in a fixed position and located in third person aerial view to allow a better overview over the entire field of action.

Within a game session, the player takes control of hand shaped objects. Their behavior precisely reflects the player’s own hands in the 3D environment in real-time and are operated contact-free as seen in Fig. 4: the ingame hand object imitates the player’s full-pinch gesture. It is possible to use either one or both hands to play the game, while this decision should be reflected in the game settings to adapt the game difficulty. Touching enemies can be done in any manner, while specific gestures trigger predetermined actions. For instance, the game menu is invoked by signaling a thumbs-up gesture or alternatively by pressing the Esc-button on the keyboard.

Gameplay demonstration

Full size image

The menu on the other hand can be operated with either mouse, keyboard or handsfree, although it is recommended to adopt game settings manually due to the multilayered menu structure. To choose a menu option, a separately superimposed cursor is hovered above the menu item. After a short time-interval, the item is selected. An example is demonstrated in Fig. 5 where the ingame menu is invoked. Control is possible with either the left or right hand, or in case of two-handed usage with the hand closer to the camera.

Invoked game menu during a game session

Full size image

3.3 Feedback Design

The game’s natural control by using hands in a 3D game environment requires supplementary feedback to both convey the depth aspect of the map as well as give insight to operational range limitations.

Visual control feedback is conveyed via a live camera feed of the depth map in the lower right corner of the screen (see Fig. 5). The stream is intentionally chosen to represent a mirror-inverted view, since this is the most natural way of self-monitoring. Further on, visual hints are superimposed in the upper left corner to enhance the feedback in certain events, for instance if no hand is detected by the camera during a game session (see Fig. 3). To support the game flow while preventing excessive abuse of this function, the game speed is slightly slowed down if this event is flagged.

In case of enemy contact to the object to be protected, the screen briefly flashes red to indicate loss of health points. On the other hand, a short, subtle audio clip is replayed every time the player touches an enemy to support the perception of enemy contact.

3.4 Design of Challenge

The game’s challenge is situated in personal reflexes, agility and reaction time required to ward off enemies as well as implicit challenges as the selection of which enemies to engage in the right order. This is owed to distinct enemy types identified via their appearance: one type of enemy may be slower and more resilient while others are quick but easy to disable. The types of enemy spawn as well as their spawning location are randomized. Predetermined, feasible locations differ in distance from the center of the map to introduce a factor of randomness in gameplay.

As already touched upon in Sect. 3.1, currently only one game mode with three difficulty settings (“easy”, “normal” and “hard”) is implemented, affecting count of enemy spawn in a set time frame and both their damage threshold and the damage they can deal. During game progression, the difficulty is dynamically scaled dependent on three different factors: elapsed time, current player health and score. As a thumb of rule the more time elapses, the more frequent enemy spawn instantiation is observed until the point is reached, where the player is overwhelmed.

3.5 Configurability of Gameplay

In terms of configurability there are three further settings apart from difficulty to customize the gameplay per the player’s requirements: game speed, hand focus and object scaling. They are locally saved to the hard drive and the game boots with the last settings by default.

Game speed setting affects the overall speed of the game and is not to be confused with difficulty setting. While an increase in game difficulty equals to a higher count of enemy instantiations at the same time, a high game speed also accelerates their movement speed and attack rate. Therefore, it is possible to have a high number of slow moving enemies on the map. The threshold for speed setting is currently set to 70%, 100% and 130% (each in accordance with “slow”, “normal” and “fast”).

The Hand focus setting allows to choose with which hand the game is played. The options “left hand”, “right hand” and “both hands” affect the principal spawn location of enemies during a game session. Generally spoken, enemy spawning is possible from predetermined locations distributed over the map with variable distances to the center of the map. For instance, “left hand” option enables a dominant instantiation of enemies on the left side of the map.

Object scaling changes the size of spawned enemies: the bigger an object is, the easier it is to spot and to touch since the collision box scales with object size. It is possible to scale objects in 50% intervals from 100% up to 200% size. Special monster spawns are excluded from this option, otherwise their size will be too large. The underlying reason is found in the implemented game physics: if enemies become too large, they will be stuck in between narrow building complexes and impede game experience.

A user testing is conducted after developing a functional prototype version of the game to gather user feedback on game design, means of control and usability in context of improving upper body movements.

4.1 Test User Profile

For the preliminary test, a total of 10 users, respectively 5 German and 5 Chinese students, are asked to participate and compensated for their expenses. The students’ background is in engineering with different specializations. Students with a HCI background can give pointers towards improving user-centered design, especially with a view to game control and control feedback. Further benefit is drawn from two students’ experience in designing serious games for health with Microsoft Kinect, and additional two students’ expertise in depth cameras. One student is in possession of a trainer license and several years of training experience, and can give first insight to the game’s potential in upper body exercising and rehabilitation. The remaining students without prior knowledge in game design or technology, assist in providing feedback from a layman’s point of view.

4.2 Evaluation Procedure

The testing procedure is equal for all test users independent of potential prior knowledge and is estimated to take about 45 min per person. After a brief introduction to the Intel RealSense camera and the game framework, the participants are distributed a leaflet covering all orally relayed information. The participants are free to test the prototype game in whatever manner they see fit for a duration of about 15 min and are free to inquire simple questions. Meanwhile, observations regarding gameplay and associated challenges are noted by an observer.

A subsequent survey and brief interview investigate vital criteria as entertainment factor, gameplay, degree of challenge, perspective of long-term motivation, user-centered design and feedback on the new non-physical user-computer-interaction via the motion capture system. The survey results are depicted in Fig. 6.

Survey results, depicted with standard deviation (Chinese: grey, German: black)

Full size image

Evaluation findings from different sources – observation, survey and interview – are pooled together and address different facets of the game.

5.1 General Feedback

Many participants enjoyed the game and exceeded the given time frame for testing. Generally, the prototype game is better received by Chinese in view of entertainment, long-term motivation and possible frequency of playing the game in comparison to the German students.

Both groups rate the ease of play in medium difficulty, nevertheless a prior introduction to the game and control is recommended due to its unfamiliarity. The game concept is simple and the manner of control is perceived as intuitive but sometimes cumbersome. In some instances, ingame control did not properly follow the input action or hands were not correctly recognized, which is touched upon in the next sections. The gameplay experience however is discerned to be challenging.

5.2 Operational Range Perception

A major issue is observed with many subjects: the continuous tracking did not work flawlessly and the virtual hand object kept disappearing after initial fail of tracking. It is evident in case the participant’s hands are outside the camera’s operational boundary, which admittedly is difficult to assess as a layman. The range is exemplary illustrated in Fig. 7, resembling a distorted cuboid volume when considering minimum working distance of the motion capture system.

Potential improvement to operational range perception via haptic feedback

Full size image

Still, the absence was sometimes observed even though the hand was within operational range. This was especially true for female participants with smaller hands wearing bracelets, wrist watches or rings, seemingly interfering with the camera’s tracking capability. The depth map, the fundamental basis to all employed algorithms, is based on the analysis of the distortions when projecting an infrared light grid on an object’s surface. Most likely, an issue arises in conjunction with polished surfaces, affecting the projected grid and thereby the calculation of the depth map.

As a means of improvement to facilitate the perception of boundaries, it is suggested to implement smart devices indicating the operational limit via vibrations as seen in Fig. 7. Thus, it is possible to mimic tactile feedback as is given with legacy peripheral devices like a keyboard. Another proposal addresses a perception enhancement of the operational range with a tutorial: the user traces a predefined path along the borders of operational reach to raise the sensitivity for both the visual boundary of control inside the game and the real operational boundary set by the hardware.

5.3 Spatial Mapping

Issues regarding spatial perception based on a mismatch of 3D input for control and the game display with a conventional (2D) screen are monitored. In the perspective view, moving one’s hand forward and upwards seems to imply a similar movement (see Fig. 8 for ingame hand object coordinate system), although the change in object size and the relative position to the environment indicate a disparity in illustration. The hand object does not directly interact with the environment and can slip through all environmental objects: it can both sink through the ground if the hand is moved to low or be partly obstructed by buildings.

Issue in spatial mapping when moving the hands

Full size image

It is observed, that participants sometimes show confusion about the inability to touch the enemies, even though in fact they were hovering the hand above the enemies instead of moving forward. Initially this phenomenon is perceived as a design error, but surprisingly the participants are eager to figure out the mechanism behind the disparity of perception and regard it as an additional challenge. After a period of acclimatization though, the participants experienced a more satisfying gameplay and displayed a steep learning curve in performance.

5.4 Difficulty Design and Configurability

There are more distinct requirements to gameplay design, especially in view of interaction and control for a serious game in upper body rehabilitation in e.g. post-stroke treatment in comparison to healthcare fields as exergaming. In this case, the principal issue is the limitation on limb control due to part palsy as an after-effect of stroke, whereas individual patients display different grades of severity. Therefore, special accommodation to their needs can be achieved by modifying game mechanics as described in Sect. 3.5: independent from the chosen difficulty level which regulates the enemy spawn and the dynamical scaling over time, the game speed can be increased or decreased to cope with personal handicaps. Further, changes regarding enemy object size scaling and the focus which hand is mainly used for control can be set. This overall process should be supervised by healthcare experts. Physical therapists can estimate the load on the patients and recommend movement patterns to support with the rehabilitation prior to conducting clinical studies. This approach can be adopted to fine-tune game mechanics as well. For instance, monster may require additional conditions like both a special gesture and a direct contact to be disabled.

5.5 Support of Natural Interaction

The interaction between player and enemies gives insight to the favored means of interaction. At first, participants attempt to push the enemies away with a palm strike and some instinctively tried picking them up with a pinch-like movement, but failed in this approach due to the game mechanics not supporting the underlying physics. Therefore, the request to a more realistic physics engine was voiced to go along with the natural control which is otherwise well perceived.

This paper describes the experiences in designing and evaluating a prototype for a serious game in upper body rehabilitation and exercising employing Intel RealSense and concentrates on critical interaction design issues. The game itself is employed with special consideration to meaningful gameplay in conjunction with a natural user interaction and high adaptability to the player’s physical capabilities. The findings garnered from early trial runs with healthy people support an additional value as a serious game for health, especially in the aspects of immersive gameplay and natural interaction. Further on, the evaluation process highlights critical game design elements affecting the usability, for instance the requirement of extensive user feedback to support spatial perception and estimation of operational range.

The present game design allows a natural control by tracking the player’s hands and projecting them to the 3D virtual game environment, while gameplay does not require specific upper extremity movements. The player is free to interact with ingame elements in a feasible and desired manner. Still, to check the suitability of the game for patients with impaired upper body movement and to improve the fine-tuning of game configurability, a cooperation with a physiotherapist is desired. Apart from implemented customization options such as game speed and ingame object size, more game elements can be modified. For instance, the game environment and camera settings can be changed to exclusively coerce sideways movement of upper extremity, if that is the recommended motion sequence for the patient without changing the fundamental game structure. In a subsequent step, clinical studies under the supervision of physiotherapists with patients suffering from impaired upper body movement, can be conducted to gather meaningful data about the impact of a serious games to support in upper body rehabilitation.