Keywords

1 Introduction

Providing gaming experience adaptive to the player’s real-time mental states is important to game designing [2]. For example, games can be adapted to match players’ emotions in gaming process [6]. Recent study shows that emotion output can be used to give user more immersive experience and more satisfaction in video gaming [9]. However, the relationship between emotion changes and game events remains unclear. In this paper, we hypothesize that emotion changes are consistently triggered by game events. Our goal is to answer how are emotion changes and game events are correlated which can inspire game designers using detectable human emotions to provide better gaming experience.

A practical way to quantify human emotion in gaming system is to use electroencephalogram (EEG), which has been tested as part of many gaming systems as a user input or a supplement to traditional inputs [4, 6, 10]. Existing literature has proven the feasibility to measure “emotions” from EEG [8]. Therefore, in this study, we carried out an experiment of 20 participants with moderate gaming experience and analyzed their emotion changes along with events in the game. Three representative games were used to provide good coverage of this study.

Data analysis focusing on two typical emotions in gaming, namely excitement and frustration, a highly linear correlation (\(R^2=0.97\) for both emotions) was found between event occurrences and emotion changes in video gaming. On average, a game event is followed by an emotion peak detected in EEG after 35.58 s for excitement, and 24.77 s for frustration.

This research is the first on this topic to the best of the authors knowledge. It proves the strong correlation between emotion changes and triggering gaming events. We hope the conclusion drawn from this study can help game designers altering game characteristics to improve user immersion and satisfaction.

2 Methodology

2.1 Data Acquisition

Emotiv EPOC+ EEG headset was used to record human EEG and to extract the emotions using Emotiv’s APIs [9]. Many research studies in the past few years have supported the reliability and accuracy of this head set [1, 5] and its capability to detect emotions [3, 7]. The sampling rate of 14-channel EEG is 128 Hz. Five emotions can be extracted by Emotiv’s APIs: 1. meditation, 2. engagement/boredom, 3. long-term excitement, 4. short-term excitement, and 5. frustration. Emotiv’s API measures emotion using emotion intensity, a normalized value between 0 and 1.

Table 1. Game experience survey from 20 subjects
Full size table

Twenty participants were recruited for the experiment, consisting of 18 males and 2 female between the ages of 18 and 45, with an average age of 25.5 years. Participants were randomly assigned to one of the 3 games.

Before the test, each participant completed a game experience survey. The summary of their game experience is shown in Table 1.

The participants’ average gaming experience was moderately good. Their facial expression and game screen are also recorded for event-emotion correlation analysis.

Our experiment environment was set up to simulate a natural gaming environment like a living room. The gaming device was an XBox One connected to a 32-inch TV. One camera recorded participant’s facial expression while the TV screen was also recorded for event annotation afterwards.

2.2 Experimental Design

For each participant, the experiment consists of four states (Fig. 1), beginning with picking one of the 3 games: Battlefield 4 (shooting), Forza 5 (racing), and Pool Game (table). Participants rest for 7 min with eyes closed after every 10 min game playing period.

Fig. 1.
figure 1

Experiment flowchart

Full size image

We also record participants’ facial expression and game screen for event-emotion correlation analysis. The simultaneously captured video of participant’s facial expression and the game screen are shown in Fig. 2.

2.3 Events & Emotions

Since events and emotion intensity are represented differently–the former is a collection of discrete time points while the latter is a signal of constant sampling rate, some conversion is needed in order to study the correlation between them. Our approach is converting emotion intensity into a time series first.

Event occurrences are annotated on recorded video, e.g., hitting a target in shooting game (Battlefield 4) and relative time points of occurrences are extracted using usability software Morae. We define several events which may cause high emotion intensity and extract the events’ time from the recorded game. We used three different games in this work, which means the events vary from game to game. In the 5 emotions mentioned above, we choose excitement and frustration in event-emotion correlation analysis. We choose these two emotions because the events they correspond to are easy to be annotated in recorded video. Events associated with short-term excitement and frustration for 3 games are annotated based on video recording of the experiment trials as shown in Table 2.

Table 2. Events used in annotation
Full size table
Fig. 2.
figure 2

Game screen (top left), player recording (bottom left) and emotion intensity (right)

Full size image

2.4 Data Analysis

If game events can regularly stimulate detectable emotion changes, game designers can use emotions as the objective feedback of players to improve game performance. An ideal condition is a single game event can trigger a dependent emotion peak with high likelihood which means the relationship between the time series of event occurrences and emotion peaks should be linear. To test this assumption, linear regression is used for event-emotion correlation analysis.

Events associated with short-term excitement and frustration for 3 games are annotated based on video recording of the experiment trials as shown in Fig. 2. Events vary from game to game. Events associated with short-term excitement and frustration for 3 games are annotated based on video recording of the experiment trials as shown in Table 2. We choose these two emotions since they are easy to be captured and annotated in video recording. Video totally in 6 h 40 mins are annotated manually by two researchers. An event is considered “true event” only if it is marked by both researchers. We use the time points of annotated events to establish a square-wave which is the Y vector of the same length as vector X. Then, we calculate the correlation coefficient between X and Y.

If we denote the event time points as a vector \(\mathbf {Y}=[y_1, \dots , y_N]\) for one type of event in one trial, the emotion intensity for one type of emotion in the same trial is a function I from time points \(\{t_1, \dots , t_M\}\) to intensity values \(\{a_1, \dots , a_M\}\), such that \(a_i = I(t_i), \forall i\in [1..M]\).

Then we convert the emotion intensity into a vector of time series \(\mathbf {X} =[x_1, \dots , x_N]\). \(\forall y_i \in \mathbf {Y}\), \(x_i = \arg \max ({a_p, \dots , a_q}) \) such that \( y_i \le t_i\le y_{i+1}, \forall i \in [p..q]\). In other words, for any event time points \(y_i\), \(x_i\) is the time point of the maximum emotion intensity (called emotion peak) between an event point \(y_i\) and the next event point \(y_i+1\).

With N-point vectors \(\mathbf {X}\) representing the emotion peaks and the event vector \(\mathbf {Y}\), we can compute the cross-correlation coefficient between event and emotion by

$$\begin{aligned} \small r_{xy}=\frac{\sum _{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum _{i=1}^{n}(x_{i}-\bar{x})^{2}(y_{i}-\bar{y})^{2}}} \end{aligned}$$
(1)

where \(\bar{x}\) and \(\bar{y}\) are the means of \(\mathbf {X}\) and \(\mathbf {Y}\).

3 Results and Discussion

As introduced in Subsect. 2.4, we use time points of annotated events to establish a square-wave which is the Y vector (blue dots in Fig. 3) of the same length as emotion peaks vector X (red dots in Fig. 3). Then, we calculate the correlation coefficient between X and Y. The time points of event occurrences are treated as the independent variable, while the time points of emotion peaks are treated as the dependent variable.

Fig. 3.
figure 3

Emotion intensity of one subject with events and emotion peaks marked. (color figure online)

Full size image

We present results from a typical subject in Fig. 3. Data from other subjects shows similar results. The results reveal a high correlation between events and emotion peaks.

Table 3. Result of regression analysis (20 subjects)
Full size table

The linear regression between the two vectors reveals a high linear relationship between events and emotion peaks, with an average goodness-of-fit \(R^2=0.97\) among all subjects for both excitement and frustration (Column \(R^2\) in Table 3). Some subjects do not produce enough emotion data points of excitement or frustration for linear regression (fewer than 5 data points), results of these subjects are denoted as NaN in Table 3.

A consistent slope of approximately 1 (Column Slope in Table 3) of linear regression results show the time-invariant delay between emotion peaks and events. The delay can be represented by the intercept of linear regression (Column Intercept in Table 3). The average delays across all subjects for excitement and frustration are \(-35.58\) and \(-24.77\) s, respectively, meaning an average delay of less than half minute. The proximity between a blue circle and its following red circle in Fig. 3 illustrates the duration of the delay.

The consistency of delay can be measured by the standard deviation (Column STD in Table 3) of prediction error of the linear regression model on all samples. For most subjects, the standard deviation is under 2 min, meaning the maximum 2 min error between true emotion peak and predicted emotion peak. The maximum standard deviation is 132.35 s for Subject 14 on excitement.

Although the regression result is promising, we need to eliminate the possibility that the high correlation is due to how we constructed the emotion peak vectors (e.g., local maximums between two consecutive events). Hence we calculate the ratio between game event related emotion peaks and all emotion peaks (Column Ratio in Table 3). The average ratio is 68.43 % for excitement and 71.64 % for frustration, indicating that emotion peaks are very likely to be yielded by game events.

4 Conclusion

In this work, we investigated how human emotion respond to game events during video gaming. Two important conclusions drawn from the emotion-event correlation analysis are: first, the human brain is very sensitive to events in video gaming as demonstrated by emotion peaks appearing around half a minute after onset of event; second, the one-to-one correspondence between game events and emotion peaks can be quantitatively and reliably established. The strong correlation between game events and human emotion shows promising evidence that game designers could use event-triggered emotion to design adaptive immersive game for better user experience.