When good observers go bad: Change blindness, inattentional blindness, and visual experience

PSYCHE an interdisciplinary journal of research on consciousness When Good Observers Go Bad: Change Blindness, Inattentional Blindness, and Visual Experience Ronald A. Rensink Cambridge Basic Research Nissan Research & Development, Inc. 4 Cambridge Center Cambridge MA 02141-1494 U.S.A. rensink@pathfinder.cbr.com Copyright (c) Ronald A. Rensink 2000 PSYCHE, 6(09), August 2000 http://psyche.cs.monash.edu.au/v6/psyche-6-09-rensink.html KEYWORDS: visual attention; inattentional blindness; visual memory; perceptual experience; neural correlate of consciousness (NCC); implicit perception. COMMENTARY ON: A. Mack and I. Rock. (1998) Inattentional Blindness. MIT Press. 288 pp. ISBN: 0262632039. Price: $US 25 pbk. 1. Introduction Several studies (e.g., Becklen & Cervone, 1983; Mack & Rock, 1998; Neisser & Becklen, 1975) have found that observers attending to a particular object or event often fail to report the presence of unexpected items. This has been interpreted as inattentional blindness (IB), a failure to see unattended items (Mack & Rock, 1998). Meanwhile, other studies (e.g., Pashler, 1988; Phillips, 1974; Rensink, O'Regan, & Clark, 1997; Simons, 1996) have found that observers often fail to report the presence of large changes in a display when these changes occur simultaneously with a transient such as an eye movement or flash of the display. This has been interpreted as change blindness (CB), a failure to see unattended changes (Rensink et al., 1997). In both cases there is a striking failure to report an object or event that would be quite visible under other circumstances. And in both cases there is a widespread (although not universal) belief that the underlying cause has to do with the absence of attention. The question then arises as to how these effects might be related. Is CB the same thing as IB? If not, what is the relation between them? And given that these phenomena deal with failures of subjective perception, what can they teach us about the nature of our visual experience? In particular, what can they teach us about the role played by visual attention? 2. Change Blindness and Inattentional Blindness When considering how CB and IB might relate to each other, it is useful to begin by reviewing exactly what these terms denote. Strictly speaking, CB simply denotes the failure to report the presence of significant changes in the visual input under particular experimental conditions. This failure is not caused by the physical aspects of the display (such as size or contrast), since -- by definition -- significant changes are obvious once noticed. Instead, CB must originate in the way we construct our representation of the world. The particular implications of CB for the nature of our representations is the subject of much current debate (see e.g., Rensink, 2000a, 2000b; Simons, 2000a). But no matter what implications may be drawn about our perception of various aspects of the visual input, the phenomenon itself pertains only to change. More precisely, CB concerns itself with second-order information (i.e., transitions between quantities) -- it does not concern itself with the simple presence of quantities. Furthermore, it pertains only to visual experience (i.e., subjective impressions, or qualia), and not to other aspects of seeing, such as visuomotor response. (See Rensink (2000b) for a more detailed discussion of these matters.) Meanwhile, IB denotes the failure to report the presence of significant items in the visual input under particular experimental conditions. As in the case of CB, this failure is not caused by physical factors, since the stimuli can be readily seen if the observer is given appropriate instructions. But in contrast to CB, IB is concerned with seeing the presence of a stimulus at any moment in time; it says little about the perception of change per se. Note that IB may be -- and often is -- found with a changing input. But a changing input need not be experienced as changing: it could, for example, be experienced as a succession of images, with no history to connect them. In other words, IB is largely concerned with first-order information (i.e., the presence of quantities in the input) rather than second-order information (i.e., the transitions themselves). In practice, this distinction may sometimes seem to blur: for example, would a response to an unexpected word be due to its presence or its appearance (i.e., the fact that it suddenly appeared from nowhere)? But there are -- at least in principle -- ways to distinguish between the two. For example, it is possible to maintain the amount of presence (i.e., the fraction of time the stimulus is displayed) while altering the rate of transition (e.g., have it appear/disappear more frequently). Conversely, it is possible to maintain the rate of transition while altering the amount of presence (e.g., giving the stimulus a greater duration each time it appears). The distinction between firstand second-order information would then translate into the issue of which quantity (presence or transition rate) affects performance. Note that this distinction is more than purely descriptive of the task: it also has consequences for the mechanisms responsible. The perception of a change requires a sequence of operations: load information into visual short term memory (VSTM), hold it across the blank interval, compare the stored to the visible information in the new display, and -- if search needs to be continued -unload VSTM and shift attention to a new location. CB could arise from the failure of any of these operations. In contrast, IB pertains to the perception of the here-and-now first-order aspects of the input. Such quantities do not require VSTM or comparison operations for their determination. Consequently, different mechanisms may be responsible for the two effects. This possibility is supported by empirical findings showing that the two phenomena have very different sensitivities to manipulations of expectation. Mack and Rock (1998) showed that IB vanishes once observers suspect they will be tested on the item that is introduced; this suggests that divided visual attention is necessary (or at least sufficient) to see a stimulus. Although CB is stronger when the change is unexpected (e.g., Levin & Simons, 1997), a lack of expectation is not needed to induce it. Several CB studies have given observers explicit instructions to look for the change and have even given them practice in doing so. But in spite of these manipulations, a high degree of CB remains (e.g., Rensink et al., 1997). Divided attention is not a problem here; evidently, the visual experience of change requires focused visual attention. Thus, CB and IB differ in several ways: (i) at the level of the task (first- vs. second-order information), (ii) at the level of the mechanisms involved (VSTM, comparison operations), and (iii) their dependence on the expectations of the observer (and thus on whether attention is focused or divided). It therefore appears that these are somewhat different phenomena. The critical word here is 'somewhat'. Although it is important to establish that CB and IB differ, it is also important to keep in mind that they are similar in several ways. To begin with, there is an interesting degree of parallelism in their natures: one involves the failure to report secondorder quantities when focused attention is absent, the other involves the failure to report firstorder quantities when divided attention is absent. There is also considerable agreement among the mechanisms both posit for processing that occurs in the absence of attention. For example, IB experiments indicate that unattended items can provide a context capable of influencing the perception of attended structures (Moore & Egeth, 1997). This suggests that representations of considerable detail and sophistication can be created at low levels even though they are not reported. Similarly, accounts of CB often assume that volatile low-level representations with some degree of sophistication are generated in the absence of focused attention (e.g., Rensink, 2000a). In both cases, then, explanations rely on an assumption that unattended stimuli can give rise to sophisticated (but volatile) representations at early levels. As such, CB and IB might best be regarded as two different aspects of an induced blindness created by the diversion of attentional resources. What distinguishes them is the type of attention involved, and the particular operations (e.g., comparison) associated with those resources. Note that such a division may correspond to the two modes proposed for visual experience: an object mode associated with focused attention, and a background mode operating as default (Braun & Sagi, 1990; Iwasaki, 1993). 3. Blindness vs. Amnesia An important issue raised by Wolfe (1999) concerns the perceptual status of the unattended items in an induced-blindness experiment: does the failure to report them really correspond to blindness (i.e., a failure to perceive the unattended items) or to amnesia (i.e., a failure to remember them). In one sense, this question was answered by Moore and Egeth (1997), who showed that unattended items are indeed perceived, at least as far as having an effect on reported items. But what about blindness in terms of visual qualia: do we still have a fleeting -- but nevertheless conscious -- visual experience of unreported items and events? The answer to this requires careful consideration of the way that the report is made. A useful distinction in this regard is that between on-line and off-line reports (see e.g., Moore & Egeth, 1997). An on-line report is made at the instant of the event; assuming that the observer has adequate visuo-motor coordination, a failure of an on-line report necessarily indicates a failure to respond to the event. In contrast, an off-line report is made some time after the event. Given that it relies on memory, the failure of an off-line report can have two possible causes: (i) a genuine failure to respond to the event, or (ii) a failure to remember it. Disentangling the two is a difficult problem, one that hobbles many studies of conscious experience (see e.g., Dennett, 1991). Another distinction -- one often used in discussion of implicit perception -- is that between direct and indirect reports (see e.g., Holender, 1986). A direct report is one triggered by the conscious visual experience of the observer; it can be a verbal report, or an action that is at least potentially verbalizable, such as a volitional press of a button. In contrast, an indirect report is one where the observer has no conscious visual experience, so that the response must be made via mechanisms not triggered by conscious command. (Here, the entity "reporting" is not the conscious mind of the observer, but rather, some other system.) The two main types of indirect report are behavioral (e.g., priming effects of stimuli not consciously seen by the observer) and physiological (e.g., differential activity of various brain regions). A key issue here is whether the conscious experience of an event can cause a direct report to be made reliably. If a direct report is made after the occurrence of the event, there is no question of conscious experience. But what is critical concerns the converse situation: if a direct report is not made, can it be reliably inferred that the observer did not have a conscious experience of it? An observer experiencing an event may be unwilling or unable to respond to it by overt behavior or by entry into a durable memory of some sort. Unwillingness may result from the belief of the observer that a response is inappropriate or irrelevant; inability may result from the observer giving priority to some other task, and so being unable to make an appropriate response before the stimulus was forgotten. Concerns such as these must be addressed when establishing reliability. Consider now the case of CB. Change detection is often measured by asking the observer to respond to the change as soon as possible -- i.e., a direct on-line report. In this situation, the observer is set to respond as soon as a change is experienced. Since all that is needed to trigger the response is a minimal conscious experience, an inability to report the change must indicate an inability to consciously experience it. As such, change blindness is not really "change amnesia" (i.e., a failure to remember a perceived change), but is a true blindness -- a true failure to have a conscious visual experience of the change. The case of IB is more problematic. IB vanishes if the observer expects the target item. Thus, giving the observer instructions to respond to the stimulus (as can be done for CB) is not possible. The problem then is to arrange things so that the presence of the stimulus will reliably cause a direct response from the observer even though they do not expect it. Attempts have been made along these lines. For example, Simons and Chabris (1999) investigated whether an observer would spontaneously report (or at least remember) the appearance of an unexpected stimulus when it was highly unusual -- for example, a gorilla. Interestingly, observers often failed to report such a stimulus. A somewhat similar study was carried out by Haines and colleagues (Haines, 1991), who examined how experienced pilots used a head-up display on an aircraft simulator. Just before the (simulated) landing, a large airplane was placed onto the runway at the point of touchdown. But the pilots often failed to detect this airplane, even though it was highly relevant and should have triggered an immediate avoidance response. Although these effects are impressive, they are not sufficient to establish that the observers had no visual experience of the unexpected stimuli. For example, even though the observers in the Simons and Chabris study likely did not see the unexpected object as a gorilla, they still could have experienced the stimulus itself -- as an array of colors and lines, for example. More generally, the observers may have failed to assign the proper category to the input, and so found nothing unusual about the stimulus. (If so, this might be termed inattentional agnosia -- see Simons, 2000b.) Another possibility is that they may have perceived the stimulus correctly but were somehow unable to make the appropriate response. Until these concerns are adequately addressed, it is not possible to determine whether the failure to report unexpected items in IB experiments is due to blindness or amnesia. 4. Visual Attention and Visual Experience Given the above, what can be said about the relation of visual attention to visual experience? Work on CB has led to the assertion that focused attention is necessary to see -- i.e. visually experience -- change (Rensink et al., 1997). Although it may appear that defining attention in terms of the ability to see change is tautological, such is not the case -- attentional effects can be tested in a number of independent ways. If attention is needed to see change, two (logically equivalent) consequences follow: (i) if a change is experienced, the changing stimulus is attended, and (ii) if the stimulus is not attended, the change is not experienced. Evidence for the first of these is the finding that attentional priming always occurs at the location of an item seen to be changing (Fernandez-Duque & Thornton, 2000). Evidence for the second is obtained from studies that control the length of time the changing stimulus remains unattended. For example, CB is reduced in items described as interesting (Rensink et al., 1997), an effect consistent with the high-level drawing of attention. CB is also reduced by exogenous cues at the location of the change (Scholl, 2000), consistent with the low-level drawing of attention. Finally, the characteristics of the change detection process itself (such as its speed, capacity, and selectivity) are consistent with what is known of focused attention (Rensink, 2000c). Taken together, then, all experimental results to date support the assertion that focused attention is needed to visually experience change. Work has also begun on the way that attention relates to nonconscious (or implicit) perception of change (e.g., Fernandez-Duque & Thornton, 2000). Here, the observer is shown a pair of stimuli (one presented after the other), and asked to report if there was a change between the two. Interestingly, even when observers had no visual experience of change, indirect reports (forcedchoice guessing) showed above-chance accuracy in determining which of two test items was the one that changed. Furthermore, in such cases attentional effects did not show up. Although absence of evidence is not evidence of absence, this latter result points toward an interesting hypothesis: conscious visual experience of change is mediated by focused attention, whereas nonconscious perception of change is not. Turning to the case of IB, things are less certain. The assertion here is that attention is needed to visually experience a stimulus. As in the case of CB, two consequences follow: (i) if a stimulus is experienced, it is attended, and (ii) if the stimulus is not attended, it is not experienced. The first of these has yet to be tested. In principle, this could be done by a straightforward determination of whether attentional effects always exist whenever a target is visually experienced. But note that such tests involve divided -- and not focused -- attention. Looking for attentional effects of this type may be relatively difficult. Testing the second consequence requires establishing that the stimulus is not experienced whenever attention is absent. This was done to some degree by Mack and Rock (1998), who showed that some exogenous and endogenous cues can reduce the degree of IB. However, other tests are still needed to provide a more thorough verification of this point. Most et al. (in press) explored the issue of attentional capture by having observers keep track of a set of black items moving among (irrelevant) white items, and then introducing an unexpected item having a unique shape and a variable luminance. It was found that IB still occurred, with the strongest effects when the unexpected item had a luminance similar to that of the distractors. This indicated the existence of active suppression: rather than selecting for the relevant features of the monitored stimuli, observers inhibited the features of the other stimuli. However, the conditions of this experiment were modeled after those of Neisser and Becklen (1975), and not Mack and Rock (1998). There is a critical difference between these two kinds of studies. Studies of the Neisser and Becklen type are selective: the observer needs to select from a number of simultaneously presented stimuli. In contrast, studies of the Mack and Rock type are non-selective: the only stimuli present (until the appearance of the unexpected item) are those being attended. Although selective experiments argue for the suppression of unexpected stimuli, it is not clear that these conclusions apply to non-selective ones. In other words, the attentional suppression occurring in selective experiments may not have the same consequences for visual experience as the absence of attention that presumably occurs in non-selective ones. As such, it remains unknown whether divided visual attention is needed to visually experience an item. 5. Prospects It has been argued above that CB experiments show that focused attention is needed for the visual experience of change. It has also been argued that IB experiments have been unable to determine if divided attention is needed to visually experience the presence of the items. The issue considered now is the reason for this latter limitation. Is it something inevitable, perhaps a consequence of some fundamental constraint against knowing the nature of our own experience? Or have we just not yet designed the right kinds of experiments? As argued above, the bottleneck originates in the way that visual experience is investigated: it is difficult to ensure that conscious experience of an event reliably causes a direct report. Ideally, the observer would reliably generate a response whenever the stimulus is consciously experienced, but do so without expectations that allocate attention to the target item. Two options are possible here: (i) no expectations at all, and (ii) incorrect expectations as to what the target will be. The first of these -- direct on-line report without prior expectation -- might be obtained by triggering a response at the moment the unexpected stimulus is encountered. Simons and Chabris (1999) used this approach, but as argued above, their results may only show the existence of inattentional agnosia, and not blindness. A better approach is that of Most et al. (in press), who looked at IB as a function of a simple stimulus parameter (i.e., luminance). Here, the fact that a particular choice of stimulus parameters always generates a response shows that the failure to respond to an item is not due to a failure to attach meaning to it, or to respond to it, but instead is due to a failure to visually experience it. The major limitation of this approach is that it is selective, involving the active suppression of irrelevant items. However, it might be possible to modify it into a non-selective task like that of Mack and Rock (1998). If target detection were found to vary with similarity to the items in the display, this would constitute convincing evidence that (divided) attention is needed for the visual experience of an item. The second option is to prepare the observer to make a response to a particular stimulus, but then use a trigger different from the one expected. This has the advantage that the observer is set to make an appropriate response, thereby eliminating the possibility that the response would not be made because of its low priority. But the drawback is that the (incorrect) expectation about the appearance of the target may cause suppressive effects that would not otherwise exist. However, it might be possible to test for this by examining how detection of the unexpected items improves as a function of its similarity to both the expected trigger item and the other items in the display. A rather different strategy might be to avoid direct on-line reports altogether, and use indirect reports. For example, conscious visual experience may correlate with a particular pattern of visuo-motor behavior, or galvanic skin response, or pupillary reflex. Another possibility would be some form of neural activity. The critical step would be to establish that the response is correlated in a one-to-one manner with the visual experience itself, and not with anything else (e.g., a particular stimulus parameter). Note that doing so for the case of neural activity is essentially the problem of determining the neural correlate of consciousness (NCC), i.e., determining which neural states correspond to which states of conscious perceptual awareness (Crick & Koch, 1998). Finally, there may also be a way around the problem of off-line reports viz., the inability to determine if a failure to report indicates a failure of memory or a failure of perception. In the case of direct report, memory failure presumably occurs because the attention needed to record the event is directed elsewhere. However, visual experience may have residual effects on other systems, making it possible to measure such effects by indirect off-line reports. Thus, for example, a brief visual experience may cause (or alter) some form of priming for subsequent events over the next several seconds. Or perhaps some trace of neural activity may exist for a while after the experience, and could serve as a reliable marker. As in the case of on-line reports, it is important to establish that the creation of such residual effects correlates with visual experience alone. But if such a correlation exists, there would appear to be a good chance of demonstrating this experimentally. 6. Summing Up It has been argued here that IB and CB are related but nevertheless distinct phenomena. Although both involve an inability to report visual stimuli that are obvious once attended, IB pertains primarily to first-order aspects of visual input (i.e., presence), while CB pertains entirely to second-order aspects (i.e., transitions). This difference leads to the potential involvement of different mechanisms -- in particular, CB may result from a failure of visual short-term memory or comparison processes that are simply irrelevant to IB. Finally, it was argued that the two phenomena differ in their sensitivity to expectation effects, and that this implies the involvement of different kinds of attention: IB requires the absence of divided attention, whereas CB requires the absence of focused attention. These differences have important ramifications for what can be concluded about attention and visual experience. Because CB is not greatly affected by expectation, direct on-line reports can reliably indicate when the observer does or does not have a visual experience of change. Such reports show that CB is not due to forgetting, but to a true failure to visually experience the change. And given that focused attention is what allows CB to be overcome, it follows that focused attention is necessary to visually experience change. The situation is more problematic for IB. Because IB disappears when the target is expected, it is difficult to determine whether the absence of a direct on-line report indicates the absence of a visual experience. This in turn makes it difficult to determine if attention is necessary to visually experience the stimulus. And to compound matters, it has proven difficult to devise experimental measures that could decisively settle this issue. One possibility is that the solution to this problem is inherently beyond our reach. But it has been argued here that this view is needlessly pessimistic, and that even at our current state of knowledge there exist several experimental approaches that appear viable, and should be explored. We have already made considerable progress on the general problem of conscious experience, in that we now understand the role of attention in the visual experience of change. With more work -- and a bit of luck -- we may be able to understand other aspects of our visual experience as well. Notes <1> I would like to thank Dan Simons, Carol Yin, and two anonymous reviewers for their comments on an earlier draft of this article. Correspondence may be addressed to R.A. Rensink, Cambridge Basic Research, Nissan Technical Center North America, Inc., 4 Cambridge Center, Cambridge, MA 02142-1494 USA. Email: rensink@cbr.com. References Becklen, R., & Cervone, D. (1983). Selective looking and the noticing of unexpected events. Memory & Cognition, 11, 601-608. Braun, J., and Sagi, D. (1990). Vision outside the focus of attention, Perception & Psychophysics, 48, 45-58. Crick, F. & Koch, C. (1998). Consciousness and neuroscience. Cerebral Cortex, 8, 97-107. Dennett, D.C. (1991). Consciousness Explained. Boston : Little, Brown and Co. Fernandez-Duque, D., & Thornton, I.M. (2000). Change detection without awareness: Do explicit reports underestimate the representation of change in the visual system. Visual Cognition, 7, 324-344. Haines, R.F. (1991). A breakdown in simultaneous information processing. In G. Obrecht and L. Stark (eds.), Presbyopia Research. (pp. 171-175). New York : Plenum. Holender, D. (1986). Semantic activation without conscious identification in dichotic listening, parafoveal vision, and visual masking: A survey and appraisal. Behavioral and Brain Sciences, 9, 1-23. Iwasaki, S. (1993). Spatial attention and two modes of visual consciousness. Cognition, 49, 211233. Levin, D.T., and Simons, D.J. (1997). Failure to detect changes to attended objects in motion pictures. Psychonomic Bulletin & Review, 4, 501-506. Mack, A., & Rock, I. (1998). Inattentional Blindness. Cambridge, MA: MIT Press. Moore, C.M., & Egeth, H. (1997). Perception without attention: Evidence of grouping under conditions of inattention. Journal of Experimental Psychology: Human Perception and Performance, 23, 339-352. Most, S.B., Simons, D.J., Scholl, B.J., Jimenez, R., Clifford, E., & Chabris, C.F. (in press). How not to be seen: The contribution of similarity and selective ignoring to sustained inattentional blindness. Psychological Science. Neisser, U., & Becklen, R. (1975). Selective looking: Attending to visually significant events. Cognitive Psychology, 7, 480-494. Pashler, H.E. (1988). Familiarity and visual change detection. Perception & Psychophysics, 44, 369-378. Phillips, W.A. (1974). On the distinction between sensory storage and short-term visual memory. Perception & Psychophysics, 16, 283-290. Rensink, R.A. (2000a). The dynamic representation of scenes. Visual Cognition, 7, 17-42. Rensink, R.A. (2000b). Seeing, sensing, and scrutinizing. Vision Research, 40, 1469-1487. Rensink, R.A. (2000c). Visual search for change: A probe into the nature of attentional processing. Visual Cognition, 7, 345-376. Rensink, R.A., O'Regan, J.K., and Clark, J.J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8, 368-373. Scholl, B.J. (2000). Attenuated change blindness for exogenously attended items in a flicker paradigm. Visual Cognition, 7, 377-396. Simons, D.J. (1996). In sight, out of mind: When object representations fail. Psychological Science, 7, 301-305. Simons, D.J. (2000a). Current approaches to change blindness. Visual Cognition, 7, 1-15. Simons, D.J. (2000b). Attentional capture and inattentional blindness. Current Trends in Cognitive Sciences, 4, 147-155. Simons, D.J., & Chabris, C.F. (1999). Gorillas in our midst: Sustained inattentional blindness for dynamic events. Perception, 28, 1059-1074. Wolfe, J.M. (1999). Inattentional amnesia. In V. Coltheart (Ed.), Fleeting Memories. (pp. 71-94). Cambridge, MA: MIT Press.