Human-Centered Design of Flight Mode Annunciation for Instantaneous Mode Awareness

The continuous reoccurring of accidents and incidents involving insufficient situational awareness and mode confusion underlines the need in the aerospace industry to develop a more simplistic and easy to interpret way of flight mode annunciation. A well-known example of such an event is the accident involving a Boeing 777 aircraft at SFO airport. During a visual approach in clear weather conditions, flight crew actions led to several mode changes of the automation, that were not perceived and interpreted correctly by the flight crew, ultimately resulting in the aircraft hitting the sea wall short of the runway [1]. The investigation clearly showed, that the flight crew did not have sufficient awareness of the current status of the automation [1]. The proposed concept was implemented on a Boeing 777 PFD, in order to investigate how it would affect the effort of interpretation in similar situations.

The key idea behind the modified display style is to merge the FMA with raw flight data on the PFD and thus to embed it in the natural scanning pattern of a pilot. Additionally, the cognitive work of interpreting the FMA and correlate it with the raw data shall be significantly reduced by simply showing with a “green border”, whether or not a certain parameter is controlled by the autopilot flight director system (AFDS). Björklund et al. [2] found that pilots are paying very little attention to the FMA when not expecting automation mode changes. The accident in SFO confirmed, that a flight crew can easily miss such an unexpected change [1]. A conventional PFD (B777) as shown in Fig. 1 displays raw flight data, such as airspeed (left), attitude (center), altitude (right) and navigation information (i.e. ILS deviation scales). The FMA is located on top of the display and shows “active” modes in green, as well as “armed” modes in white. The FMA is divided into three columns for auto throttle, roll-mode and pitch-mode and one AFDS status field [1]. The red boxes in Fig. 1 list all the possible modes for each column and the AFDS status field respectively. The high cognitive effort required for interpretation can be demonstrated as follows: Assuming the flight crew desires to find out, if the airspeed is automatically controlled, they need to do the following:

Complex nature of flight mode annunciation in current display style (Color figure online)

Full size image

• Read the FMA auto throttle column text

• Interpret the text, as there are modes that cause the auto throttle to be “engaged”, but not controlling the airspeed (e.g. “HOLD” or “THR”)

• If the text is anything else than “SPD”, the scanning continues to the pitch-mode, as this channel can also be used to control the airspeed [1]

• Interpret the pitch-mode (check if it is FLCH SPD or VNAV SPD)

• Check the AFDS status indication, to ensure that the autopilot is engaged in case of airspeed being controlled via the pitch-mode.

The example of Fig. 1 demonstrates that the complex nature of control-mode interdependence of current PFD creates a situation that cannot be readily conveyed to pilots using text only. One of the major problems with the current flight mode annunciation philosophy is that the basic layout did not change for decades, while the capabilities of the automation systems evolved dramatically [3]. For a basic function, like “altitude hold” or “heading hold” a very basic annunciation concept using text labels was capable of providing adequate information. With the advent of more sophisticated automation concepts and the introduction of the auto throttle, the situation became much more complex. A basic flight parameter such as indicated airspeed can now be controlled by multiple systems (i.e. autopilot pitch mode or auto throttle) [3]. The location of the FMA on top of the PFD is in peripheral vision most of the time. The current FMA however, is designed for foveal vision in terms of the saliency it provides [4]. Furthermore, the current philosophy does not allow the flight crew to readily identify if a certain parameter is controlled by the automation [3], as the cognitive task of relating the flight mode annunciation text to the physical behaviour of the aircraft is left to the flight crew. This has led to several occurrences of so-called “controlled flight into stall” (CFIS) [3]. Sherry and Feary [5] found that another cause for inaccurate understanding of automation behaviour lies in the differences between flight crew training documents and manufacturers technical specifications. These differences are then often rectified by gaining practical experience during the line-training [6].

The proposed modified design shown as Fig. 2 aims to remedy this by relating the flight mode annunciation to the physical flight parameters. This new PFD layout consists of only small changes to the general appearance in order to keep transition training for pilots to a minimum. At the same time there is a distinct change in philosophy: Instead of just displaying the flight mode as in the legacy design, the new design also highlights the parameter that is controlled. In simple terms, this could be described as showing what is controlled, rather than just how it is controlled. The basic FMA box is retained, while green borders are added for the parameters that are actively controlled by the automation. In Fig. 2 the active flight modes “SPD”, “LOC” and “G/S” are augmented with the respective parameters being highlighted. It is therefore instantly clear, whether or not a certain parameter is controlled by the automation, simply by looking for the presence or absence of green borders.

Proposed modified flight mode annunciator by green color border (Color figure online)

Full size image

2.1 Participants

The experiment involved 20 participants, aged between 22 and 47 years (M = 27.7, SD = 7). Thereof 12 were qualified pilots and 8 were aerospace engineering professionals. Flight experience reached from 0 to 11000 h (M = 946, SD = 2567). All participants were provided with a consent form before the experiment. Furthermore, the experiment process was reviewed and approved by the Cranfield University Research Ethics System (CURES), reference number 2475.

2.2 Apparatus

Flight Simulator: A virtual replica of the B777 instrument panel was used to create the basic scenarios. The Precision Manuals Development Group (PMDG) “B777 expansion pack” allowed authentic recreation of the B777 PFD and ND. The experiment was conducted in a segregated room that was quiet and free from optical disturbances. Windows were blanked off and the illumination level kept constant.

Eye Tracker “Pupil” is a wearable, light-weight eye-tracking device that can be used for automated eye-movement analysis in everyday life [7]. It consists of a headset including cameras and software packages for capture and analysis. The headset is connected to any convenient computing device (e.g. laptop) using an USB connection. The Laser-sintered headset hosts two cameras, one facing the right eye of the participant (eye-camera), the other capturing the field of vision (scene-camera) [7]. The eye-camera has a resolution of 800 × 600 pixel and a frame rate of 60 Hz. The detection of the pupil is based on the “dark-pupil” concept, using the infrared spectrum. The scene-camera captures the user’s field-of-view at a high-resolution (1920 × 1080 pixel) with a frame rate of 60 Hz.

2.3 Scenario

The research involved developing a new display concept for flight mode annunciation and verifying it using an eye-tracking experiment and a questionnaire. Five typical scenarios were developed, using both the current and modified display style for each of them. Table 1 depicts the essential description of each simulated stage of flight.

2.4 Hypothesis

Of particular interest is the flight crew workload during the task and the ease of interpretation of the FMA. The combination of objective (eye-tracking) and subjective data (NASA-TLX) serves as a basis for this assessment. These two designs (current and modified) are compared using the following set of experimental null hypotheses:

1. 1.

   There is no significant difference in fixation duration

2. 2.

   There is no significant difference in pupil size

3. 3.

   There is no significant difference in perceived workload.

2.5 Research Design

Participants were split into qualified and unqualified groups in order to allow a mixed-design analysis to be carried out. The qualified group included all participants with any kind of flight experience, reaching from some flight training on single engine piston aircraft up to senior captain of multi-engine jet aircraft. The unqualified group consisted of all other participants, mainly aerospace engineering professionals. In order to counterbalance the “learning effect” each participant may experience during the experiment and also to remedy any bias based on the sequence of display styles, the scenarios and display styles were randomly assigned to participants. Following the guidance provided in [8], a “Williams design” was developed for the five scenarios and two display styles. Two dedicated pilot tasks were created to generate a realistic workload. The key aspect here is that the participants were told to be the “pilot monitoring”, simulating a multi-crew environment and requiring them to check the progress of the flight with respect to given constraints. This represents a typical real-world situation were the aircraft is controlled by the “pilot flying” or the autopilot and the “pilot monitoring” has to verify the adherence to published procedures. Additionally this was deemed to create an acceptable amount of cognitive workload apart from monitoring the FMA, in order to avoid prolonged fixation on the FMA.

The first task involved monitoring of airspeed. Subjects were asked to callout every 10 kts of change in airspeed. As a second task, any altitude change of 100 ft had to be called out. These two tasks existed in addition to monitoring of the FMA. Any change on the FMA had to be called out and was recorded by the operator. As not all of the participants were familiar with the intricacies of the B777 flight modes, the emphasis was laid on the notification of the flight mode text change, rather than the understanding of the physical meaning of the respective flight mode. All participants evaluated their perceived workload using NASA-TLX after each scenario.

A mixed-design repeated measures two-way ANOVA was carried out for the eye-tracking parameters listed in this section. The within-subject factors were display style of FMA (current and modified) and scenario (1–5), while flight experience (experienced vs non-experienced) was used as a between-subject factor.

3.1 Fixation Duration

There is a significant difference on fixation duration between conventional FMA and modified FMA, F (1, 16) = 6.81, p < .01, partial η² = 0.299. Evidence of a significantly lower fixation duration was found for the modified display style. The greater amount of saliency provided by the modified display style seems to facilitate the process of comprehending a new automation state, possibly due to the related change in the participant’s saliency map [9]. The lower fixation duration also serves as an objective confirmation of the faster processing time and lower subjective workload ratings mentioned by many participants. It was observed that the qualified group had a smaller variation in fixation duration than the unqualified group. One of the key elements in pilot training is to establish a useful scanning pattern and avoiding to fixate too long on only one specific area of the PFD [10]. It is very likely that this training bias manifests itself in the smaller and more consistent fixation duration. Differences in fixation duration between experienced and novice pilots were also observed by Yu et al. [11]. The null hypothesis for fixation duration can therefore be rejected.

3.2 Pupil Size

There is no significant difference on pupil dilation between conventional FMA and modified FMA, F (1, 16) = 2.67, p = 0.125, partial η² = 0.141. Although a smaller pupil size was measured in modified PFD, this effect was not significant. Ahlstrom and Friedman-Berg [12] found that pupil-size does change with workload, however the change in pupil-size for the medium-workload region was minimal. It was therefore not possible to underline the lower subjective workload with this measurement. The unqualified group had a much smaller pupil size. It should be noted that the parameter of interest is the change in pupil size rather than the absolute value of the pupil size [13]. Yu et al. [11] found significant differences in pupil size of pilots depending on the flight scenario and task. The observed effect shows that it is possible to “merge” the flight mode annunciation with the raw flight data and thus co-locate important information. This facilitates the scanning pattern and improves the perception of information in accordance with the proximity compatibility principle [14, 15].

3.3 Subjective Workload

There is a significant difference on subjective workload between conventional FMA and modified FMA, F (1, 16) = 11.67, p < .01, partial η² = 0.422. Both pilot groups showed a lower subjective workload when using the modified display style. The merging of raw data with flight mode annunciation data seems to reduce the perceived effort in interpreting the displayed information. It should be mentioned, that the unqualified group adapted much better to the modified display style, than the qualified group. This is not surprising, bearing in mind that the qualified pilots have undergone significant hours of training using the legacy display philosophy. The simple design of the modified display style allowed for a rapid transition and quick adaption even for the experienced pilots. A very significant difference in subjective workload allows the rejection of the corresponding null hypothesis.

3.4 Scenario and Style Results Summary

The overall effects of the modified display style caused a decrease in all three cases, although statistical significance was only reached for Fixation duration and subjective workload, as described in the previous sections. The use of objective (eye-tracking) and subjective criteria (NASA-TLX) combined proofed to be a useful hybrid solution, aiming to eliminate any bias in either of the parameters. The experiment also showed that the introduction of a modified display style does not show equal benefits for each one a given set of different flight situations. In particular, it could be shown that the current design works quite satisfactorily when the workload is low (scenario 5). However, if the workload gets high, the increased scanning demand causes a deterioration of mode awareness. This is precisely the situation where the modified display style is fundamentally different and advantageous, as it incorporates the automation state in the natural scanning path and thus reduces the crew effort to establish a good mode awareness.

3.5 Attention Distribution

Based on the eye-tracking data, so-called “heat-maps” were calculated, depicting the gaze distribution (Red reflects large amount of time, relative to the total scenario duration). Figure 3 represents a classic example of a professional pilot’s behaviour. Attention is given to the raw data fields primarily, and the flight modes were only checked occasionally. It is worth noting that the particular subject was able to maintain a good situational awareness throughout all scenarios. Most of the saccades performed towards the FMA seem to be based on expectancy of a mode change. This shows the vulnerability to miss “uncommanded” or automation-induced mode changes. The modified FMA is much more likely to get the attention of the pilot, as the green borders appear/disappear directly around the raw data fields shown as Fig. 3. It cannot be over-emphasized that scanning the raw data is important and crucial for the safety of flight. It really is the limited scanning of the FMA box that should raise concerns about the conventional design.

Heatmap of attention distribution. (Color figure online)

Full size image

Figure 4 depicts a typical sequence of fixations during the experiment. After checking the AFDS status field (1) the subject scans the altitude (2) and the airspeed (3), before reaching the LNAV deviation scale (4). This example also underlines the need for the automation mode annunciation to be incorporated in the raw data field, as it is obvious from the picture, that scanning raw data necessitates a diversion from the FMA in the current design philosophy. In fact it can be seen, that by fixation (3) and (4), the pilot automatically gets the current automation state with the new design using the green borders.

Scanning path for 3 s (Color figure online)

Full size image

The experiment showed that the modified display style causes a significant reduction in workload, by applying the proximity compatibility principle [14]. The flight crew’s ability to rapidly gain situational awareness is vital for flight safety. The modified display style reduced the cognitive and temporal workload when interpreting the automation status on the PFD. Additionally, the modified display style is much more likely to catch the flight crew’s attention when an “unexpected” mode change occurs, as the salient stimulus is applied directly in the raw data fields, which are frequently looked at by the flight crew. The proposed method was tested for the “autopilot engaged” situations only during this experiment, however an expansion to the “flight director only” regimes is also possible. It should also be noted that the use of green borders is by no means a very sophisticated way of incorporating the automation state in the raw data fields. The use of color-coded or solid and hollow deviation bars and indications would be a next design step. This was also underlined by subjective qualitative feedback obtained from participants. Professional pilots highly appreciated the idea of the concept, but provided critical comments on the implementation. An improved experiment design could incorporate a “trial” phase first, to eliminate most of the implementation problems before the actual experiment.

An interesting discussion developed when participants were asked about an aural flight mode annunciation: The professional pilots were strongly opposing this idea. Nevertheless research in this area should continue, as the introduction of data link communication for air traffic control [16] will only further overload the optical channel and reduce the use of aural information exchange between the pilots and air traffic control (ATC). Furthermore, Zhang et al. [17] demonstrated that a timely aural warning can significantly improve a human’s ability to quickly fixate on a critical object. A good example of combining aural advisories with visual perception in a cockpit has been tested by Purcell and Andre [18]. They were able to reduce head-down-time during taxi operations by using an aural cue when to look at the moving map. A similar concept could also work for the FMA. Caution should be exercised when using similar aural alerts for multiple different events. Kearney et al. [19] found that in the case of air traffic controllers, the acoustic alert led to a slower reaction than a semantic alert, as the same aural sound was used in multiple instances.