Keywords

1 Introduction

Control rooms can be found in fields where safety is highly critical, such as aviation, traffic, health care, or power plant control. In these environments, they act as centers in which the control operations are being coordinated and constantly monitored in order to guarantee the stability and safety of the systems [1]. Due to an increase in automation, monitoring activities in the control room have gained importance. Monitoring requires the operator to visually scan the system status for any abnormal system behavior and be ready to act if a failure occurs [2]. In the control room, monitoring is often based on prior knowledge and follows certain strategies, rather than simply waiting for a failure to occur [3]. Because of the complexity of system control rooms, teams of operators monitor a system together. Collaborative monitoring requires team members not only to visually scan for failures, but to interact with each other and exchange relevant information at all times in order to keep track of the system’s current status [4]. Staying highly aware under such circumstances is difficult. Therefore, in order to prevent the occurrence of human error in control room teams, ways to maintain and train situation awareness in control room teams are needed [5].

The psychological concept of situation awareness is widely known and commonly used in human factors research [6, 7]. The term was first used in the context of aircraft piloting and has been extensively investigated in the field of aviation ever since [e.g. 8,9,10]. In recent years, it has been transferred to other domains such as driving [11], medicine [12, 13], nuclear power plant control [14, 15], and cyber security [16]. In this research, the main finding is that high situation awareness is generally linked to better performance. For example, Bell [8] found that fighter pilots with a higher situation awareness make fewer decision errors in a combat scenario than pilots with lower situation awareness. Conversely, low situation awareness is still found to be the major cause of human error. According to Jones and Endsley [17], over 70% of pilot errors that are related to situation awareness can be traced back to level one of the three levels of situation awareness, which involves the initial perception of information. At level two, the information that has been perceived is then understood and processed on a deeper level. At least 20% of pilot errors can be attributed to difficulties with level two situation awareness. With level three situation awareness, future events can be anticipated [18]. According to Endsleys’ model [6, 7] it takes all three levels of information processing for an individual to make a decision in the end. In order to gain insight into an individual’s level of situation awareness, appropriate measures are needed.

Measures of situation awareness mainly include task performance measures, memory probes, and subjective as well as objective rating techniques [19,20,21]. However, because situation awareness is a more fluid concept that is achieved over time, more process-oriented measures are needed. In this context, eye tracking has been found to be beneficial as an online indicator of situation awareness. In aviation, it has become a frequently used method [22, 23]. Furthermore, in a wide range of domains, such as problem solving and reasoning [24, 25], driving [26, 27], medicine [28], education [29], music [30] and chess [31], various studies have integrated eye tracking in order to understand and compare visual attention processing [32, 33]. Little research has been done on this topic in the context of control rooms. For example, Sharma and colleagues [5] used eye tracking in a chemical plant environment and were able to find differences among operators concerning their fixation patterns.

Another process-oriented measure of situation awareness is communication [34, 35]. Authors have stressed the importance of effective teamwork and communication at work [36]. Specifically in the control room, communication is crucial for teams to be successful [4]. Here, it is not only critical that operators become aware of their own situation, but the situation of their team members as well. By exchanging relevant information, they can ultimately achieve higher situation awareness [37]. In other words, communication not only allows one’s own situation awareness to be modified and enhanced, but it also promotes sharing it, ultimately affecting the situation awareness of the whole team [38]. Therefore, measuring the level and quality of communication between team members can help assess the level of achieved SA [39] and contribute as an additional measure for evaluating situation awareness in control room teams.

Taking previous research into account, studies have mostly focused on not one, but multiple measures for investigating situation awareness, thus demonstrating the necessity of using multiple measures when investigating teams, specifically in a complex environment such as the control room. As this study concentrates on examining teams in a simulated control center task, it therefore uses a variety of measures of situation awareness. We assumed that the measures of performance, gaze, and communication behavior would be related, and that they could be used to help gain insight into the situation awareness of the operators:

Hypothesis I:

There is a relationship between performance, gaze, and communication behavior.

By examining the link between different situation awareness measures, we further assumed that operators who are better at detecting failures while monitoring should differ from low performers with regards to their situation awareness:

Hypothesis II:

High performers differ from low performers with regard to their gaze and communication behavior.

Continuous measures such as eye tracking and verbal communication are beneficial when it comes to investigating situation awareness over time. They allow tracking the state of situation awareness at all times. Taking multiple measures into account should further help investigate if and how situation awareness was achieved:

Hypothesis III:

The increase of situation awareness over time can be observed in different measures.

2 Method

The reported study was part of the DLR project COCO (Collaborative Operations in Control Rooms). The overall aim of COCO was to evaluate different psychological and physiological factors that influence control room teams. The study was conducted using ConCenT (generic Control Center Task Environment), a simulation in which control center activity was simulated [40].

2.1 Simulation

In ConCenT, teams of three have to supervise several distributed production facilities and monitor the environment to determine whether or not a failure has occurred (monitoring task). If there is a failure, they have to determine the cause of the failure (diagnosis task) and find a solution to the problem (remedy task). In order to complete all three tasks, operators are required to collaborate and exchange information at all times. Because only the monitoring task was taken into consideration for the analysis, the diagnosis and remedy tasks will not be described here further.

In order to understand how the collaborative monitoring task was conducted, the structure of ConCenT is illustrated in Fig. 1. The control center that needs to be supervised consists of nine production plants which are distributed over three different sites: Alpha, Bravo, and Charlie. Each plant comprises three overlapping assembly lines, adding up to a total of 27 lines. Joint power stations supply their corresponding sites with energy. Operators A, B, and C are each in charge of monitoring the status of nine of the 27 assembly lines. Their status is indicated by individual gauges on their associated control panel (Fig. 2). Failures are indicated by a gauge’s actual value falling below or exceeding the limits of its associated tolerance range, indicating a failure in the associated plant. A failure was the consequence of a critical situation that the team members could only anticipate by exchanging information about the system status. Rules on how to anticipate a critical situation were given to the team members beforehand. This enabled team members to gather expectations about critical situations and possible failures within the system. Whenever a failure occurred, all operators had to report it within a time interval of four seconds.

Fig. 1.
figure 1

Structure of ConCenT. Three sites with three plants . Each plant comprises three overlapping assembly lines . Every site is supplied with energy by one power station .

Full size image
Fig. 2.
figure 2

Control Panel. Each operator must monitor 12 gauges. The top three gauges represent the energy levels of the joint power stations. The nine gauges below indicate the status of the assigned assembly lines. Next to each gauge, there is a button for reporting failures in case the associated black arrow exceeds or falls below the tolerance area.

Full size image

2.2 Eye Tracker

Eye movements were recorded with the Eye Follower System manufactured by LC Technologies, Inc. Management of raw data was conducted using NYAN software. Subjects were seated in front of a 24-inch LCD computer display at a distance of approximately 60 cm. The system operated at 120 Hz and was combined with the simulation tool ConCenT to ensure that both systems used the same timestamp. The fixation-detection algorithm was set to require a minimum of six gazes on a particular point on the screen (within a deviation threshold of 25 pixels). All successive fixations falling on an AOI (area of interest) were categorized as gaze duration.

2.3 Participants

Twenty-one teams (total N = 63 participants), were tested. Participant age ranged between 18 and 34 years (M = 21.57, SD = 3.39). 47.6% of the participants were female and 52.4% male. 41 individuals were applicants for air traffic control training at DFS (German Air Navigation Service Provider), 22 individuals were students and graduates from various universities. Participants were compensated €25 for participating in the 2.5-h test.

2.4 Procedure

Three participants were seated next to each other. Room dividers prevented direct eye contact between the participants. Each operator position included a separate computer and eye-tracking system. After reading comprehensive instructions and completing a ten-minute practice scenario, teams performed a 72-min test scenario. Over the course of the scenario, a failure occurred at six different times during the monitoring task. Each failure had to be reported by every single team member within four seconds. If it was not reported by all members within the four-second interval, the system would automatically switch into the diagnosis task and it would individually be counted as a miss. Rules on system behavior and on how to detect a failure were given to the participants beforehand. This allowed team members to exchange relevant information and enabled them to anticipate failures in time.

2.5 Variables

Performance, gaze, and communication data were collected. For the present study, only data from the monitoring task was analyzed. Performance measures included number of failures detected and response time in seconds. The analysis of gaze was based on four distinct monitoring phases, which were predefined as taking place before and during the occurrence of a failure. These phases were derived based on the normative model on how to identify operators monitoring appropriately that was used in previous research on monitoring tasks [22, 41]. During the first two phases, operators had to identify the relevant information [identification phase] and verify this information within the team [verification phase]. During the third phase, a failure could be anticipated by the team [anticipation phase]. During the fourth and final phase, the failure was visible [detection phase]. For each phase, the information that was needed to be perceived and shared was marked as relevant. This information also represented the relevant areas of interests (AOIs) for eye movement analysis. In order to conduct the gaze analysis, fixation-based eye-movement parameters were used. These included relative fixation count (ratio between number of fixations on relevant AOIs and all fixations within a given time span) and relative gaze duration (ratio between gaze duration on relevant AOIs and total gaze duration within a given time span). The time to first fixation was used only during the anticipation and detection phases. Verbal communication was analyzed with respect to the accuracy of information that was shared by each team member in order to anticipate and detect each of the six failures. In order to define an individual’s quality of communication, participants could score on a scale from 0 (no or false communication of relevant information) to 1 (correct communication of relevant information) within each of the four monitoring phases.

3 Results

The data gathered from 52 participants were used for analysis. Eleven data sets were excluded due to missing data or failed manipulation checks. Analysis was conducted on the individual level and focused on the time before and during the occurrence of each of the six failures (monitoring phases). The six failures were referred to as items. On a scale from 0 to 6, an average of 4.33 (SD = 1.37) failures were detected. The average response time was 2.17 s (SD = 0.56). On a scale from 0 (no gaze on relevant information) to 1 (all fixations on relevant information) relative fixation count and relative gaze duration varied between M = .41 and M = .50 (SD = 0.09-0.19) within each of the four monitoring phases. On a scale from 0 to 6, the average communication quality was 3.30 (SD = 0.89). The level of communication varied between the four monitoring phases (see Fig. 3).

Fig. 3.
figure 3

Mean communication quality from 0 to 1 for each monitoring phase over all six failures.

Full size image

3.1 Linking Performance, Gaze and Communication

Multiple correlations between detection performance, gaze data, and communication quality were calculated. Figure 4 shows the overall results in a simplified manner. A statistically significant relationship between all eye movement parameters (relative gaze duration, relative fixation count, and time to first fixation) and the number of detected failures was revealed, F(10,41) = 3.34, p < .01, R = .67. In particular, eye movement parameters in the anticipation and detection phases were correlated to detection performance (p < .01). In order to investigate the relationship between detection performance (in terms of numbers of failures detected) and communication quality, a non-parametric correlation of Kendall’s tau-b was run. A positive relationship was identified, rƮ(51) = .26, p < .05. For response time, this relationship turned out to be negative, rƮ(48) = -.35, p = .001. Multiple correlations between gaze data and communication quality revealed no statistically significant relationship F(10,38) = 1.78, p > .05, R = .57. However, non-parametric rank correlations of Kendall’s tau-b showed a significant relationship between eye movement parameters and communication quality during the anticipation phase (p < .05).

Fig. 4.
figure 4

Simplified illustration of the link between the three measures of situation awareness. * p < .05, **  p < .01.

Full size image

3.2 High Vs. Low Performers

Participants were split post-hoc into two groups, high and low performers, based on their total number of detected failures using a median split (Mdn = 4.4). Two groups were created, one with 26 high performers and the other with 26 low performers. High performers detected a significantly higher number of failures than low performers (U = 0, z = −6.40, p = < .001) and their mean response time was significantly lower than that of low performers, t(49) = 2.39, p < .05 (Fig. 5). In order to test whether high and low performers differed with regard to their eye movement data, a one-way multivariate analysis of variance (MANOVA) was conducted. A statistically significant MANOVA effect was obtained, Wilks-λ = .63, F(10,41) = 2.44, p < .05, partial η2 = .37. High performers had a higher relative fixation count and higher relative gaze duration than low performers in the last three monitoring phases [verification, anticipation, detection] and a lower time to first fixation in the last phase [detection]. In terms of their overall communication quality, high performers (Mdn = 3.25, n = 24) did not significantly differ from low performers (Mdn = 2.75, n = 25), U = 207, z = −1.88, p > .05.

Fig. 5.
figure 5

Descriptive performance results of high and low performers.

Full size image

3.3 Performance and Communication Over Time

In addition to linking the different measures and comparing them for the two groups of high and low performers, consideration was also given to the development of performance measures and communication quality over time. Figures 6 show the descriptive results. A Cochran’s Q test determined a statistically significant difference over time in the proportion of participants who detected a deviation successfully, p < .001. This applied both to high and low performers, p < .001. Pairwise comparisons using continuity-corrected McNemar’s tests with Bonferroni correction revealed that significantly more high performers detected failures two to six than failure one, p < .01. Correspondingly, it was shown that low performers detected failures three to six significantly more often than failures one and two (p < .001). High and low performers differed significantly in their mean response times as determined by independent-samples t-tests when detecting failure three, t(42) = 3.28, p < .01, and five, t(40) = 2.55, p < .05. In terms of the communication quality of individual participants, statistically significant deviations were displayed between the six items (Friedman test: χ2(5) = 15.15, p = .01, n = 47). This applied to low performers, χ2(5) = 20.48, p = .001, n = 23, but not to high performers, χ2(5) = 0.10, p > .05, n = 24. Post-hoc analysis of low performers revealed a statistically significant difference between items two and five, z = −3.63, p < .01, as well as two and six, z = −3.00, p < .05, as item two had the lowest communication quality, whereas five and six had the highest. Mann-Whitney U tests revealed that high and low performers differed significantly with regard to item two, U = 187, z = −2.24, p < .05 and four, U = 198, z = 2.28, p < .05.

Fig. 6.
figure 6

Performance and communication quality over the course of the six failures.

Full size image

4 Discussion

The present study addressed the benefit of using multiple measures of situation awareness in control room teams. In a simulated control center task, teams of three were asked to monitor different production facilities and detect failures. Information was distributed among team members in a way that made communication necessary within the team. Performance was measured by counting the number of failures detected and collecting the associated response times. Eye tracking was integrated to examine where individuals directed their gaze before and during the occurrence of each of the six failures. Additionally, communication was recorded for the same time intervals and analyzed based on the previously defined information that needed to be shared with the team members in order to anticipate and detect failures.

4.1 Linking Multiple Situation Awareness Measures

The results of this study show that both gaze and communication behavior is linked to monitoring performance. It was found that when the gaze was directed towards the relevant information during each monitoring phase more, failures were detected and reported faster and more consistently. This entailed longer and more frequent fixations on relevant information during time intervals before and during the occurrence of a failure. Therefore, relevant information was processed more intensively and more often than irrelevant information [42, 43]. Further, participants that fixated a failure earlier were able to report it more accurately and much faster. This shows that by following certain strategies, and ultimately being more aware of the situation, operators are able to anticipate and detect failures in time during control room monitoring [3, 5, 41].

Besides investigating situation awareness by means of performance and gaze measures, the quality of information exchange within the team was analyzed as well. When more relevant information was communicated accurately and at the appropriate time, system failures were detected earlier and more consistently. Therefore, this work supports previous results that showed communication is strongly linked to performance [44, 45]. It indicates that the more aware someone is, the more accurate his or her communication will be, which ultimately affects the final outcome.

Lastly, it was assumed that there would be a link between gaze and communication behavior. No overall relationship was found except for a link between the two measures for the distinct monitoring phase of anticipation, supporting the proposed importance of anticipatory processes in complex fields where safety is critical [46]. Right before the occurrence of a failure, the words being spoken by the participants concerned where they were looking, emphasizing the need to share task-relevant information immediately before a system failure occurred. If no communication had occurred at this point, it would have made it impossible for team members ultimately to detect a failure. The fact that no relationship was found for the other three distinct monitoring phases might be explained by consistently high communication quality among participants during the first phase. Moreover, the detection phase was only a few seconds long, so it was nearly impossible for all team members to share and confirm the detection of a failure. Furthermore, communication analysis was based on assumptions on what kind of information had to be shared in order to anticipate and detect a failure, but not on what kind of information needed to be received. Taking this into account when analyzing communication on the team level might result in finding a relationship between gaze and communication behavior.

4.2 High and Low Situation Awareness

Post-hoc analysis showed significant differences between high and low performers in terms of their eye movements. Overall, high performers were better at directing their gaze toward relevant information at the right time [cf. 32]. In three out of four monitoring phases, they focused longer and more frequently on relevant information. Also, they directed their gaze at the failures earlier. Having more fixations on task-relevant areas and shorter times to first fixate relevant information is consistent with previous findings on expertise in eye tracking research [22, 33, 47]. Results showed that high performers were more efficient and knew how to anticipate and detect a failure better, indicating they had higher situation awareness than low performers. However, with regard to their communication quality, high and low performers did not differ significantly. Though, for two failures communication was found to differ between high and low performers, indicating that there must have been some difference between the two groups at some point. For future research it would be of interest to consider analyzing not only the quality but also the quantity of communication, allowing a more objective approach to analyzing communication in control room teams.

4.3 The Development of Situation Awareness

Results show an increase in performance and communication quality over time. Teams were able to improve when and what kind of information was exchanged while their performance improved as well. This is in line with Cooke [48], who found that command-and-control teams improve their interactions over time, accompanied by improvements in team performance. With regards to this study, the increase in communication quality can be attributed to the group of low performers. High performers started off the simulation with a better quality in their communication and therefore started off the scenario with higher situation awareness. Both groups improved their performance, which is likely to happen with participants who did not have any previous experience with the simulation. In this context, the first ten minutes of the task were mostly needed to adapt to the task. This is consistent with previous research showing that performance increases early in the team development stages [48]. However, it must be said that high performers adapted to the situation more quickly than low performers. They started off the simulation with a higher performance and were better at detecting failures throughout the scenario. They performed better and communicated more effectively in the early stages of the scenario than low performers. This indicates that they had higher situation awareness right from the beginning and were able to maintain it over time.

4.4 Conclusion and Outlook

Our findings provide important implications for the integration of different measures of situation awareness in control room teams and the development of situation awareness in complex teamwork scenarios. Performance, gaze, and communication behavior were found to be useful when investigating situation awareness in team-based monitoring. Efficient gaze behavior and the quality of communication are closely linked to the amount of system failures detected and the speed of detection. High situation awareness is demonstrated by directing one’s gaze toward relevant information faster and more frequently, and by sharing information accordingly when having to anticipate a failure. While an increase in situation awareness could be seen in overall performance and communication behavior, high performers started off with higher situation awareness than low performers. The noticeable increase of situation awareness at the beginning of the task suggests the need for knowledge enhancement and training for control room teams right before and at the start of their cooperation. Since the analysis was conducted on an individual level, analyses must also be carried out on a team level by using advanced methods in order to investigate team cognition patterns and draw conclusions about team situation awareness [39]. Current and future research focuses on applying reported single team outcomes to multi-team situations, in which individuals of not one but several organizations work together in a collaborative manner.