Representations and operations: parts of the problem and the solution

As the discussion led by Dekker and Woods (2002), Dekker and Hollnagel (2004), Parasuraman et al. (2008), Dekker et al. (2010) shows, active discussion concerning the scientific status of human factors concepts is currently being pursued. First off, the fact that this type of discussion has even been initiated is a quite important step for our discipline; however, some assertions from the involved parties should be cautious. Dekker and Hollnagel (2004) show strong reluctance to investigate cognitive mechanisms and to “look for measurements of hypothetical internal states [instead of] measurements of the performance that admittedly is determined by the internal states” (p. 83). They promote an approach that focuses on the functioning and the observable performance of the joint man–machine system. The main constituents thereof are the orderliness of performance which depends on the degree of control. This enrichment of perspectives—cognition in the head and cognition in the world—is certainly prosperous. Care should be taken to not disregard one perspective from the outset, as some remarks of Dekker and Hollnagel (2004) suggest. We consider this discussion as positive and stimulating for the discipline of HFE as a whole: new concepts are brought into consideration; established ones are refined; theoretical dependencies are (re-)established. Controversies of this kind are the fuel that drives research. We will resume this issue to suggest an integrating account later on. Although certainly welcome as contribution to the discussion, there is no immediate need to resolve this controversy but even to continue and intensify it.

This thesis might be the quintessence of de Winter’s explanations, which deserves and requires some consideration. There are different categorizations of types of measurement models apart from the representational-pragmatic (used synonymously to operational) as Hand (2004) readily concedes. One could, for example, differentiate between direct and indirect measurement. Hand chooses representational and pragmatic as two extremes of a continuum for reasons of convenience. He uses this one-dimensional format to locate very different measurement models and practices of very different disciplines—natural as well as social sciences—to make them comparable. The pragmatic aspects of measurement bridge “the gap between the model and the reality” (p. 15; Hand 2004). As might be noted, Hand adopts the delineation of representational and operational (or pragmatic) measurement from Michell (1986; for a broad discussion see Michell 1999), who introduces classical measurement as a third category. Michell strictly separates representational and operational measurements by taking a more rigid notion of measurement itself. De Winter seems to adhere to Michell’s argumentation by suggesting the same separation. However, unlike Michell (1999) who advocates a quantitative representational approach, de Winter suggests to use the degrees of freedom that this definition offers and declares HFE concepts as pragmatic. So, the above mentioned controversies on HFE’s scientific status could be resolved “and any discussion about empirical reality (…) is irrelevant” (p. 4). His argumentation culminates in declaring “that human factors constructs are not part of an empirical reality” (p. 5); an assessment that he (in our opinion, mistakenly) proposes to share with Parasuraman et al. (2008).

De Winter adopts Hand’s representational-pragmatic continuum but misinterprets to our opinion its descriptive nature. Hand (2004) states: “In this book we have deliberately set out to construct a model of measurement practice, not of what measurement (…) should be. That is, we seek to be descriptive about measurement practice, rather than proscriptive” (p. 12; emphasis in original). Quite to the contrary of de Winter’s suggestion, Hand describes any measurement activity as exhibiting representational and pragmatic aspects. By prescribing a purely pragmatic account of measurement, de Winter, deliberately or not, leaves the descriptive approach of Hand (2004).

De Winter describes a kind of operationalism as initially conceived of by Bridgman (1927): “the proper definition of a concept is not in terms of its properties but in terms of actual operations” (p. 6). According to an unequivocal assessment by Michell (1999), this kind of “operationalism is false (…) Bridgman confused the knowing of something, with the thing known” (p. 170). Michell further evaluates Stevens’ approach similarly: “Operationism commits an elementary confusion: it confuses ‘the act or process of measuring with the object of the act, namely the quantity in question’ (Byerly 1974, p. 376). Once this confusion is exposed, Stevens’ definition of measurement is revealed for the charade it is” (p. 177). The distinction between theoretical concepts and their effects is so either ignored or, worse, a priori not existent.

De Winter recommends on taking special care of “the construction of the measuring instrument, and the precision of the definition” (p. 4). However, if the measuring procedure defines the concept, a precise definition is not needed. Although this issue might seem to be marginal, it illustrates a fundamental shortcoming of operationalism when recommended as a solution for the discussion of the role of human factors concepts. Defining concepts exclusively by procedures might bereave them of their theoretical relationships to other concepts and dramatically reduce their meaningfulness. According to Balzer (1997), the apparent simplicity of operationalism actually leads to an inflation of theoretical measures, reduces them to feasible activities, and increases interpretative complexity.

According to Saint-Mont (2011, 2012), scientific progress manifests itself in the form of the development of attributes that are scale independent from those that are scale dependent. The diversity of operational definitions is thus reduced by generalizations, which are increasingly independent of special procedures. Specifically, an initially pragmatic-operational situation would develop into increasingly general theories which could be interpreted realistically.

This development is depicted in an iterative interpretation of a research circle (based on Saint-Mont 2011). The process depicted in the research cycle starts from empirical experiences in the form of data. Bottom-up generalizations lead to hypotheses and then interrelated theories or even axioms. These hypotheses, theories, axioms direct top-down acquisition and evaluation of data. This scheme illustrates the interplay of deduction and induction, necessary for scientific progression.

This structure is already quite well-known and in its most basic form, was already described by the ancient Greek philosopher, Aristotle. It is necessary and might be helpful to refresh this idea because representational measurement promised a shortcut for behavioural and social sciences like HFE. Using well-established mathematical tools and frameworks, which proved valuable in other disciplines like the natural sciences, offered a chance to jump directly to the deductive side of this cycle. For various reasons, however, representational measurement theory could not live up to its promises in the social sciences. Cliff (1992) concluded: “(…) regrettably, the promise of the axiomatic measurement movement has not been fulfilled” (p. 187). The title of his article evaluates measurement theory as a “revolution that never happened.” (Cliff 1992; cited in Saint-Mont 2011, p. 61).

With the help of the research circle, critical remarks about measurement can be located. Saint-Mont (2011; p. 61) discusses the reason the representational measurement theory fails and blames it on an over emphasis of the deductive side of measurement. He suggests a more inductive-oriented approach to measurement. Jaynes (2003) criticises the missing relationship between mathematical measurement theory and reality: “nothing could be more pathetically mistaken than the prefatory claim (…) that mathematical rigor ‘guarantees the correctness of the results’. On the contrary, much experience teaches us that the more one concentrates on the appearance of mathematical rigor, the less attention one pays to the validity of the premises in the real world, and the more likely one is to reach final conclusions that are absurdly wrong in the real world.” (Jaynes 2003; p. 674; cited in Saint-Mont 2011, p. 46). Tukey emphasises the necessity of both aspects of scientific endeavour—with the slightest bit of polemics against deduction: “We know a lot about the phenomena involved in the use of our techniques. Some of what we know has been learned deductively, using assumptions and mathematics. We do learn from practice, as well as from deduction and from experimental sampling. We can practice a science. We need not hide behind a mysterious shield of false-to-fact deduction!” (Tukey 1982; cited in Saint-Mont 2011, p. 351).

The inflationary use of an operational perspective that we can observe in social sciences and that de Winter suggests as rationale for HFE, might thus be interpreted as the pendulum that swings back: Disillusioned from the “revolution that never happened”, maybe alienated by the rigid mathematical nature of measurement theory, maybe attracted by the freedom that it offers, researchers seem to stick to operationalism. In terms of the research cycle discussed above, de Winter’s suggestion is not already implemented. Researchers do in fact use different procedures, but they also usually try to define the concepts they intend to measure by referring to existing theories. They do not define their concepts in terms of their measurement procedures. In the case of cognitive workload, for example, most researchers who use theoretical frameworks do derive these measures which they want to be integrated into methods for a network of concepts. So pupillometric approaches to the measurement of cognitive workload are explicitly based on the knowledge of physiological processes, theories of attention and mental effort of Kahneman (1973). In general, one has to consider that in the area of human factors and ergonomics one important expertise is to analyse existing human–machine systems and working situations; but another prominent task is to specify future ones. In particular, the specification and synthesis of systems benefits from the usage of theoretical frameworks instead of pure operationalism.

Based on these considerations, we want to conclude with some recommendations for the researcher at the scientific front line:

• Keep in mind that research is a process: operationalism and representationalism have the strong potential to slow down or stop a priori the cycle from necessary iterations. These iterations are fuelled by discussions like the one we saw between the two camps “cognition in the head” versus “cognition in the world”.

• Strengthen the inductive side of the research process 1: mathematical rigor in derivations can be comforting. Inductive approaches necessarily have an aura of uncertainty, and any form of inductive insight requires a leap of faith to bridge the inductive gap. Successful predictions of the same behaviour using different procedures might lead to the connection of seemingly different concepts in one theoretical framework. As a by-product, some form of methodological invariance might have been established.

• Strengthen the inductive side of the research process 2: transform the discussion of invariance from the daunting mathematical interpretation of legal transformation to a discussion of practical invariances between different procedures. Bring it from the mathematical-deductive side to the exploratory-inductive side of the process.

• Strengthen the inductive side of the research process 3: use exploratory data analytic approaches in the spirit of Tukey (1977); look for patterns everywhere; calculate “impermissible” statistics; be bold when on the exploratory-inductive side of research.

• Keep in mind that the goal of that iterative process of scientific research should be to increase the structure of the discipline concerned. The natural sciences did not start with axioms; they were the result of various interrelated inductive–deductive research activities, which succeeded in developing high-level structures beginning with observations. Increasing knowledge about the underlying structures enabled scientists to then formulate their expertise by formulating axiomatic systems. As the development of conjoint measurement (Luce and Tukey 1964) impressively demonstrates, original new lines of thought can lead to a similarly solid structure even in the social sciences.

• Look beyond the discipline’s borders. For example, in the case of cognitive workload, there seems to be some remarkable resemblance to cognitive load theory (Plass et al. 2010) and to the discussion of a limited resource versus attentional process in ego depletion process of self-control (Inzlicht and Schmeichel 2012).

One final recommendation as a direct reply to de Winter’s restriction of HFE concepts to predictive purposes: Build models to categorise, describe, explain, and predict. Refine concepts into models according to the iterations described above. Prediction is a valuable goal and it might be reached without understanding mechanisms; however, it is reached easier with.