Keywords

1 Introduction

1.1 Introduction to Research Problem

Crew information needs (and the requirement for workload and decision support) vary according to the crew composition, the specific experiences of the two crew members (i.e. familiarity with type, familiarity with route and time elapsed since last flown that route), and the specific flight circumstances on the day (traffic and weather).

Given automation advances over the last decade, Pilots share responsibility for different flight tasks with cockpit systems. Adaptable systems are systems which require human delegation of task and ‘function authority’ to automation during real-time operational performance (i.e. the task distribution is controlled by the user) [1]. Adaptive automation (AA) is defined as a ‘form of automation that allows for dynamic changes in control function allocations between a machine and human operator based on states of the collective human–machine system’ [2, 3]. As such, task distribution changes can be controlled autonomously by the system.

The air accident and flight safety literature reports on the many still-open issues in relation to automation design. For example: Flight Air France 447 (2009) [4], Flight Spanair 5022 (2008) [5], Flight Helios Airways HCY 522 (2005) [6], Flight China Airlines 140 (1994) [7], and Flight Air Inter 148 (1992) [8]. Critically, several human factors problems have been documented. This includes: automation surprises, degraded situation awareness, unintentional blindness, workload concerns and issues pertaining to over-reliance on automation.

With increasing flight hours, fatigue and increased traffic growth, all crews can benefit from an “experience aid”. Ideally, the user and the “experience aid” (or assistance system) constitute a cooperative system - they share tasks and perform them as a team.

1.2 Introduction to A-PiMod Project

The Applying Pilots’ Model for Safer Aircraft (A-PiMod) project aims to address problems relating to crew/automation teamwork and workload management. The high level goal of the project is to design a new adaptive automation concept based on a hybrid of three elements – (1) Multi-Modal Pilot Interaction, (2) Operator Modelling, and (3) Real-Time Risk Assessment. This research is funded by the European Commission and has been undertaken since September 2013.

2 Research Design and Status

The high level Human Machine Interaction (HMI) design/evaluation methodology combines formal HMI design/evaluation activities (i.e. interviews and simulator evaluation), informal HMI design/evaluation approaches (i.e. participatory design activities), along with an integrated stakeholder approach to evaluation [9]. The concept of a Community of Practice (COP) proposed by Wenger underpins the stakeholder evaluation approach [10]. The current COP panel comprises fifteen participants (see Fig. 1 below). Stakeholder participation involves consultative interaction along with engagement in technical research tasks [11]. Overall, nineteen COP sessions and two phases of simulator evaluation have been undertaken. The first phase of simulator evaluation involved eight participants, while the second phase involved twelve participants.

Fig. 1.
figure 1

Current state of stakeholder competency knowledge in A-PiMod

Full size image

The Radar Diagram below (see Fig. 1) shows the two overlaying levels of expertise both from the internal and external stakeholders. The composition of the internal stakeholders is represented in blue, while the composition of the external stakeholder is represented in amaranth. The red dotted line corresponds to the 2-level expertise.

3 Key Results and Emerging Concept

3.1 Human Factors Problems and Proposed Approach

Field research with the pilots resulted in the identification of several categories of human factors problems. This includes (1) human factors specific to automation, (2) more general HF/operational problems - that might be addressed by improved automation design and (3) existing technology/information gaps in terms of cockpit design – that might be addressed by an improved automation design. For more information, please see Appendix 1.

The theoretical starting point for addressing human factors issues with automation is the assessment of crew/automation/aircraft state in relation to the achievement of the mission level goal (i.e. mission level risk assessment), and the identification of a suitable task distribution at cockpit/agent level, to achieve this (see Appendix 2). Automation has a role in relation to (1) real-time risk assessment, (2) the identification of a course of action, (3) the selection/implementation of a course of action (i.e. the delegation of work to automation is part of the course of action selection), and (4) the identification of a suitable task distribution based on crew state .

3.2 Automation Concept

The emerging concept can be conceptualised on several levels - (1) a task/experience aid, (2) a proactive risk/safety tool, (3) a crew state monitor and (4) a decision support system. The goal is to support crew in situations when they may need help irrespective of experience, and/or in situations when the crew has less experience, and/or in situations where the crew is experiencing high workload, under pressure and potentially fatigued. The team comprises the pilot flying (PF), the pilot monitoring (PM) and automation. Automation is a virtual team-member. The team co-operates in relation to mission level decisions. The system continuously monitors the operational situation and the allied crew/automation/aircraft state, to determine the tasks the team has to perform together, and how to best distribute them between the crew and automation. A-PiMod flags potential risks - providing operational guidance in relation to managing those risks. Pilots have final control (i.e. make the final decisions), but are responsible and accountable for their decisions/actions. The crew forms their own judgement/ideas as to risk status of situation and the appropriate course of action. The crew are not mandated to follow the decision support provided by A-PiMod (this is an aid, not a requirement). Figure 2 below depicts the overall A-PiMod architecture – including the mission, cockpit and agent levels.

Fig. 2.
figure 2

Architecture Concept and Technology Components/Modules

Full size image

3.3 Pilot Interaction in the Cockpit

Pilot interaction in the cockpit can be characterized in relation to the following points:

  • User friendly and flexible information/decision support

  • The crew interact using voice/touch and traditional controls

  • This interaction is tracked by the system (i.e. what tasks performing, level of fatigue, involvement in activity): this is referred to as ‘crew state monitoring’

  • The crew obtain feedback via a new cockpit user interface (Mission and Cockpit Level Management Display - MCMD) as to:

    • The risk status of the operational situation (this includes an assessment of the status of joint crew/automation system)

    • What to do – including the provision of best options/alternatives based on different ‘technical’ contributing factors (i.e. fuel remaining, status of alternates etc.)

  • The proposed MCMD features two related sub-displays – the mission and cockpit level displays

  • The crew can over-ride system proposals/decisions – except in certain critical situations (i.e. incapacitation)

4 Discussion

4.1 Cockpit Centred .V. Task Centred Approach

A-PiMod adopts a team centred approach as opposed to a crew centred approach. We are focussing on the outcome; considering what is best for the safe and efficient completion of the mission/flight, and not particularly trying to adapt to human needs. As indicated in the architecture concept [12], if the pilot flying/pilot monitoring is overloaded and this threatens the completion of the mission, the task distribution is adapted at the agent level.

4.2 Crew State Monitoring

The real gain in A-PiMod relates to crew state monitoring – that is focussing the pilot’s attention on their state (i.e. crew state) along with that of their crew member - and on the current and future state of the aircraft. If overloaded and/or under pressure, pilots may forget or not consider all the safe options. However the 3rd crew member (automation) will not, so a quick check will refresh the possible options, to allow a safe decision to be made. In this context, a key challenge is how to get the two human crew members to share their ‘current state’ with the 3rd crew member such that it is meaningful, informative but not self-incriminating in any post hoc analysis. Normal human interactions can easily accommodate this in simple pre-flight social interactions. Formalising it such that the 3rd crew member can make useful sense of it may be more problematic.

The assessment of crew state is not just about workload, it’s about the crew experience, flight hours, familiarity with route, when last flown there, training background etc. If the Pilots are not familiar with the route, then the crew state might be assessed as less optimal. From a Pilots perspective, the starting point for crew state monitoring is the crew briefing/flight planning. This might occur a week before the flight. Or at least, at the time of the pre-flight, flight planning and briefing task. For crew state monitoring to work, we need to establish a picture/sense of the crew state from the very beginning of the flight. The A-PiMod system needs to know what the join crew status is and any threats associated with this. Potentially, we will need the crew to provide feedback about their state in advance of the flight. Further, it takes into account real-time crew behaviour. This involves monitoring the crew state via the assessment of (1) crew activity (gesture), and (2) crew interaction with cockpit systems including new multi-modal input (i.e. touch, voice and gesture) and traditional controls.

4.3 Interpreting Crew State Information

Careful attention needs to be given to the means/basis by which crew state information (i.e. eye tracking data, gesture data, voice, and touch data) is used to form an assessment of crew state. The evaluation of this feedback depends on what we know about the crew. For example, if crew are not looking at the right area of the screen and/or blinking a lot, it may not be a problem. In this case, the crew might be very familiar with the route and also, on the first day of their roster. However the system might interpret this behaviour differently if the crew are unfamiliar with the route, and on the last day of their roster (i.e. if expect more or less fatigue).

4.4 Teamwork Concept and Nature of Automation

The question of how the system interprets quietness in the cockpit is controversial. One of the crew members might be taking a controlled rest (i.e. flight over the Atlantic). In this case, one would not expect any briefing/verbalizations between the crew members. Maybe A-PiMod has a role at this time, to ensure the other crew member (PF) is kept in a safe state. Accordingly, A-PiMod might issue soft alerts so that they remain active and engaged. If A-PiMod is to become a ‘trusted 3rd crew member’, it needs to behave as one does in ‘normal’ situations. Potentially, it needs to engage in some form of ‘social’ interaction as much as technical interactions in order to be fully integrated in the ‘team’. This latter issue has not been explored in A-PiMod and warrants more attention.

How assertive a team member is automation? When and how can it voice its concerns – and potentially, over-ride? In most (but not all situations), the cockpit crew have the final authority and can veto automation. As such, this reflects an ‘adaptive systems’ logic. However, A-PiMod’s crew state monitoring technology will detect certain situations, when it is necessary from a safety perspective for automation to ‘take charge’ (for example, situations of crew incapacity). Specifically, there may be different levels of crew state monitoring. For example, (1) passive support, (2) active support and (3) intervention/over-ride. In this sense, A-PiMod represents an adaptive automation approach.

4.5 Adaptive Automation and Roadmap to Single Crew Operations

The proposed third Pilot/adaptive automation concept has the potential to support single crew operations and remote crew operations (i.e. provision of ground support). In principle, the implementation of single crew operations necessitates a fully adaptive automation approach. As indicated in this research, a fully adaptive automation approach requires detailed and reliable/robust modelling and assessment of both crew and aircraft states. The former (i.e. crew states) links to the new cockpit display (MCMD) and specifically the integration between crew modelling and crew monitoring (and specifically real time monitoring of crew use of the proposed HMMI - touch, voice, gesture and eye-tracking). The latter (i.e. aircraft state) may require complete integration between A-PiMod and other aircraft systems, and potentially wider ATM and ground systems (i.e. ATM system picture/multiple missions). Not all of this may be fully achievable in terms of what is demonstrated at the end of the A-PiMod project, and may require additional research/development.

5 Conclusions

Overall, the stakeholder evaluation/validation approach adopted has facilitated the preliminary specification and evaluation of a new adaptive automation concept and associated technology requirements. Specifically, the integration of a range of formal and informal HMI methods, with a stakeholder evaluation approach has proved effective in terms of enabling both operational and safety validation. Critically, the emerging adaptive automation concept is predicated on feedback in relation to flight crew experience with automation (and associated problems). Automation is conceptualized as a third crew member – providing support to crew in both high and low workload situations – to optimise flight safety and ensure the mission level goal is achieved. A-PiMod cannot supplant experience. However, it is ready to provide extra information in relation to risks/hazards and potential courses of action – if required by crew. In this way, as noted by a COP pilot, the proposed A-PiMod system features different “levels” of response, similar to the way a Captain would have with different co-pilots of varying experience.