Trends in

Cognitive Sciences
Opinion

Perception in real-time: predicting the present,

reconstructing the past
1,2, ,@
Hinze Hogendoorn *

We feel that we perceive events in the environment as they unfold in real-time. Highlights
However, this intuitive view of perception is impossible to implement in the We feel that we perceive our environ-
nervous system due to biological constraints such as neural transmission delays. ment in real-time, despite the constraints
imposed by neural transmission delays.
I propose a new way of thinking about real-time perception: at any given
moment, instead of representing a single timepoint, perceptual mechanisms Due to these constraints, the intuitive
represent an entire timeline. On this timeline, predictive mechanisms predict view of perception in real-time is impossi-
ahead to compensate for delays in incoming sensory input, and reconstruction ble to implement.
mechanisms retroactively revise perception when those predictions do not
I propose a new way of thinking about
come true. This proposal integrates and extends previous work to address a real-time perception, in which perceptual
crucial gap in our understanding of a fundamental aspect of our everyday life: mechanisms represent a timeline, rather
the experience of perceiving the present. than a single timepoint.

In this proposal, predictive mechanisms

predict ahead to compensate for neural
Understanding perception in real-time delays, and work in tandem with
Perception is among the most fundamental functions of the nervous system. The function postdictive mechanisms that revise the
of perception is to organise, identify, and interpret sensory information from the sense timeline as additional sensory information
organs, in order to represent and understand the environment [1]. For humans, like becomes available.

many other animals, this sensory environment can be highly dynamic, changing rapidly Building on recent theoretical, computa-
over time. Successfully interacting with objects and agents in the environment therefore tional, psychophysical, and functional
requires the perceptual system to accurately represent the sensory environment as it unfolds in neuroimaging evidence, this conceptual-
real-time. isation of real-time perception for the first
time provides an integrated explanation
for how we can experience the present.
Understanding how real-time perception is implemented in the human nervous system has
turned out to be a surprisingly thorny problem. Our introspective experience appears
straightforward: at any given instant, our perception of the environment seems to simply mirror
the state of the external environment at that instant (Figure 1). This intuition is so pervasive that
the majority of experimental research on perception has largely ignored the fact that both the
sensory environment, as well as the neural mechanisms that internally represent that environment,
are continually evolving. In a typical laboratory experiment, we might present participants with brief,
isolated stimuli, and then record neural responses to each stimulus, or wait until the participant can
report some aspect of that stimulus. Such experimental designs are effective because even very
brief stimuli (as short as a few milliseconds) can evoke cortical responses that last for a second
or more [2], and presenting individual stimuli spaced well apart in this way prevents the responses
1
to different stimuli from running together. However, sensory input outside the laboratory is not so Melbourne School of Psychological
Sciences, The University of Melbourne,
neatly spaced apart. Instead, multiple streams of information from different modalities relentlessly Parkville, VIC 3010, Australia
2
pour into the brain to inform it about the continually evolving sensory environment. Studying http://timinglab.org
perception using individual stimuli therefore both highlights and neatly avoids a fundamental gap
in our understanding of the mechanisms underlying perception: how do perceptual mechanisms
in the brain cope with the continuous stream of sensory input [3] to generate perception in *Correspondence:
hhogendoorn@unimelb.edu.au
real-time? Here, I propose a new way to think about how neural mechanisms might enable (H. Hogendoorn).
real-time perception. @
Twitter: @hinzehogendoorn

128 Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 https://doi.org/10.1016/j.tics.2021.11.003
© 2021 Elsevier Ltd. All rights reserved.
Trends in Cognitive Sciences

Three challenges to the intuitive view of real-time perception Glossary

Neural delays Duration channels: a symbolic
There are three key areas where the intuitive view that our perception mirrors the outside representation of time in which specific
world at any given instant falls short. The first is that the transmission and processing of neurons respond to stimuli with specific
durations.
information in the nervous system takes time. During this time, events in the environment Physical time: time as measured on an
continue to unfold, such that sensory information becomes outdated while in transit. In the objective time-scale, which can be used
case of visual motion, for example, a moving object continues moving while sensory informa- to indicate when physical events
tion about its position flows through the nervous system. These delays are substantial: it (including neural processes) occur. This
is distinct from represented time.
takes several dozen milliseconds for information from the retina to reach visual cortex Postdiction: a perceptual mechanism
[4,5], and at least ~120 ms before it is possible to use visual information to initiate voluntary in which the perception of a given event
actions [6,7]. In ball sports such as tennis, cricket, or baseball, such delays would is modified by sensory information which
is presented after that event. Postdiction
correspond to mislocalising the ball several meters behind its true position (Figure 1B).
appears to violate causality, but does not
Even a relatively slowly moving object, such as a passing cyclist, would be perceived up because it affects the perceptual repre-
to half a meter behind its true position. Humans are nevertheless able to play ball sports sentation of the past, rather than the
and navigate through traffic, and laboratory experiments confirm that humans are remarkably past itself.
Prediction: a perceptual mechanism
accurate at interacting with dynamic environments, achieving approximately zero lag for
by which perceptual information is
even fast-moving objects [8]. So how do perceptual mechanisms compensate for their preactivated even though no sensory
own delays? information is yet available, using previ-
ous sensory information and prior
knowledge.
Desynchronisation
Represented time: time in our
The second key challenge to understanding real-time perception is that sensory information subjective experience, corresponding to
becomes desynchronised at it is processed. This is because the time required to process sensory the time at which we perceive an event
input differs across sensory modalities (e.g., audition precedes vision [9]) and even across to happen, or how long we perceive an
event to take. Time in our experience is
features within a modality (e.g., within vision, colour precedes motion [10]). At the neural level, distinct from the physical timing of the
there is enormous variability in the response latency of neurons even within the same visual neural mechanisms that create that
area [11]. As a result, information from a single sensory event becomes available to perception subjective experience.
not at any single moment, but over a range of moments. For the same reason, the most recent Sensory horizon: the point on a
perceptual timeline beyond which no
information available to perception at any given moment will have originated from different sensory information is available, due to
moments in the environment for different features, and therefore do not belong together to the neural transmission and processing
same percept (Figure 1C). This causes a temporal binding problem: as the brain processes the delays.
Short-term synaptic plasticity (STP):
continuous stream of desynchronised sensory input, how does it infer what happened when?
the mechanism by which the relative
strength of a neural connection briefly
Causality changes as a result of recent activity of
The final challenge to understanding how perception operates in real-time is the existence of a that connection.
Symbolic representation: encoding a
number of robust perceptual phenomena, in which the second of two sequentially presented
sensory feature (such as time) as a
stimuli affects the perception of the first. This is problematic because it appears to violate the different dimension of neural activity
fundamental rule of physics that causes must precede their consequences; an event cannot (for example, as a spatial pattern). The
cause something to retroactively happen ‘back in time’. Nevertheless, there are several known alternative is that the timing of a sensory
stimulus is encoded only as the timing of
examples where this does appear to happen. In vision, backward masking is one such example,
the neural activity signalling that stimulus.
where presenting a second stimulus shortly after an earlier stimulus can render the first stimulus
invisible. Conversely, cuing attention to the location of a subthreshold stimulus even after its
disappearance can cause the stimulus to be perceived [12]. In the Colour Phi phenomenon
(Figure 1D) [13], two differently coloured discs are presented in different positions shortly after
one another. This sequence is perceived as a single disc moving smoothly from the first to the
second position, changing colour midway. The paradox is that until the second disc is detected,
perceptual mechanisms do not have access to its new colour or position, and so should not be
able to create a percept of motion and changing colour during the gap. Logically, this would only
be possible after the presentation of the second disc. Given that we require the brain to obey
the physical law of causality, how do real-time perceptual mechanisms appear to allow sensory
information to ‘go back in time’ to affect the perception of earlier events?

Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 129

 Trends in Cognitive Sciences

(A) Intuive model of (B) (C) (D) Apparent violaons

Neural delays Desynchronisaon
real-me percepon of causality
Present Present Present Present
Environment

Environment

Environment

Environment
Past Future Past Future Past Future Past
Brain

Brain

Brain

Brain
Physical me Physical me Physical me Physical me

(E) (F)

presented
Physically
Physical me

Perceived
Snapshot of external world Snapshot of represented sensory
(visual features are aligned in me) informaon (asynchronous
Perceived me
processing causes misalignment)
Trends in Cognitive Sciences

Figure 1. The intuitive model of real-time perception, and three challenges that it fails to address. (A) In the intuitive view of perception, we perceive events in the
environment as they unfold in the present: the contents of perception are aligned in time with the external world. (B) Due to transmission delays, sensory information about
the present moment (black) will only be available in cortex at some point in the future. As a result, perceptual representations that are currently available (red) contain
information that is slightly outdated. If this were not somehow compensated, perception would lag behind the present, causing moving objects to be mislocalised
behind their true position. (C) Because neural delays are variable across features and modalities, information about the present becomes available over a range of
future timepoints (black). As a result, the most recent sensory information available at any given instant (red) does not originate from the same timepoint. (D) Various
stimuli can retroactively change the perception of preceding events; a phenomenon called postdiction. In the intuitive model, this means sensory information needs to
travel back in time to influence representations that existed earlier, violating the physical law of causality. (E) Desynchronisation of sensory features should cause the
perception of different features (e.g., colour, motion, form, etc.) to become misaligned in time, in contrast to our everyday experience. (F) In Colour Phi, the presentation
of a red disc after an earlier green disc retroactively creates a percept of motion during the gap, as well as a colour change midway along the trajectory. The intuitive
model of real-time perception cannot explain how the disc appears to change colour before the second disc has been presented.

A new conceptualisation of real-time perception

The intuitive model of real-time perception fails to account for important biological constraints and
key experimental observations. I propose a new way of thinking about perception in real-time,
that addresses the three challenges outlined above.

The key innovation of this proposal is that rather than representing a single timepoint, perceptual
mechanisms represent a timeline covering an extent of time. This timeline contains an ongoing
best estimate of past, present, and future (Figure 2). It is not a temporal buffer, in which informa-
tion accumulates before individual timepoints are sequentially perceived; rather, it is an editable
sensory narrative, in which sensory interpretations become available to perception as soon as
they are detected. Moreover, information on this timeline never becomes definitive, thereby
allowing future interpretations to overwrite earlier ones as if those never happened.

This conceptualisation solves a key problem that any account of temporal perception must solve.
On the one hand, sensory information rapidly becomes available to guide behaviour (e.g., <150 ms

130 Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2

Trends in Cognitive Sciences
Memory

Sensory horizon present

Re
pr
es
e nte
dp
ast

Represented
future

Trends in Cognitive Sciences

Figure 2. A new conceptualisation of real-time perception. The key feature of this proposal is that perceptual
mechanisms instantaneously represent an entire timeline of events, rather than a single timepoint. Different sensory
features (e.g., texture, colour, motion, luminance, etc.) are illustrated here as separate rows on a metaphorical film reel.
Sensory input to this timeline is limited by neural delays, imposing a sensory horizon on incoming input that can vary
across different features (broken lines). Nevertheless, predictive mechanisms allow sensory information to be represented
on this timeline for timepoints beyond this horizon, for which no input is yet available.

[6,7] for saccades). On the other hand, some percepts (such as visual motion) require integration
over time, or can be altered by later events (such as Colour Phi). This integration window can
extend several hundred ms (e.g., [14,15]). Many other features are perceived only as part of their
immediate temporal context (e.g., musical notes in a melody, phonemes in speech, and objects
in apparent motion). In an attempt to address this, a recent model of postdictive perception pro-
posed a two-stage model with an unconscious integration window of ~300 ms [15], but this
requires that conscious perception lags the external environment by an equivalent duration.
Does perception really wait that long before committing to a percept?

In the current proposal, perception does not wait at all. Instead, sensory information becomes
available immediately, whilst remaining editable. Although counterintuitive, this provides the
best of both worlds: rapid perception is possible when input is available, whilst sensory features
that require temporal integration or lead to reinterpretation of earlier input can overwrite the initial
percept when they become available. Because there is no separate memory of the initial percept,
the timeline always reflects the most coherent and up-to-date interpretation of the occurrence of
events.

Representing time
If perception represents a timeline, how does it represent time? In considering how this might be
achieved in real-time, it is important to note that the timing of neural activity representing a sensory
event is conceptually distinct from the timing of that event, and that both are distinct from the
perceived timing of those events. In other words, when an event occurs in the environment, when
a sensory representation of that event is activated in the brain, and when the event is perceived to
happen are three different questions. Furthermore, whereas the first two can be established on
the same objective timeline (physical time; see Glossary), the third can only be answered on a
subjective timeline (the person’s subjective experience or represented time).

Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 131

 Trends in Cognitive Sciences

It is often implicitly assumed that the perceived timing of events is encoded simply as the timing of
the neural activity itself, such that when an event is perceived to happen is determined by when
sensory information about that event is processed. However, this encoding strategy of using
time to represent time [16–18] cannot explain experimental dissociations of these two timelines
[19–21], or how desynchronised sensory information nevertheless leads to an integrated,
coherent percept (Figure 1C) [16,22,23].

In the current proposal, the timing of sensory events is instead represented symbolically, as the
activity of specialised neural systems that encode the relative time of external events just like other
features of the event. In this way, when an event is perceived to happen (represented time), is
encoded explicitly as a perceptual feature, and is not deduced from (and therefore principally
independent of) the time at which sensory information about that event becomes available
(physical time). This works in a similar way to a date-stamp on a letter. Due to unknown variations
in postage times, the arrival time of a letter is not necessarily an accurate reflection of when it
was sent. However, even if two letters are received around the same time, their date-stamps
make it possible to ascertain which was sent first. In the same way, even though different sensory
features of the same event may be processed at different latencies, and therefore become
available to be incorporated into the perceptual timeline at a different physical time, they can
still be represented as belonging to the same moment on the internal timeline [24].

A sensory horizon
Neural delays impose an important constraint on the sensory information that is available to be
represented at any given moment. Sensory information about a new event is not available to
be incorporated into the timeline until a given amount of time has passed, creating a sensory
horizon. For this reason, the most recent sensory evidence available to perception always lags
slightly behind the present.

I propose that perception compensates for the limitations imposed by the sensory horizon by
allowing perceptual representations to be activated by other neural processes, including local
computations and feedback from other areas. In particular, prediction mechanisms can use
available information about past timepoints to estimate the contents of timepoints, for which
limited or no sensory information is yet available (Figure 3A and Box 1). In this way, prediction
allows a perceptual representation of a particular feature to be formed at a shorter latency than
would normally be possible given neural transmission delays.

Furthermore, perceptual representations never become final or definitive. Instead, the timeline is
an ongoing best estimate that can change as new evidence becomes available or new interpre-
tations emerge. This means that representations of any timepoint (future, past, or present) can
always be updated or revised. Where prediction mechanisms might activate the representation
of a future event, postdiction processes [14,25] can reconstruct the perceptual past, for
example to correct predictions that did not eventuate (Figure 3B and Box 2).

A final implication of the current proposal is that there is no hard natural boundary between
perception and memory. Rather, there is a continuum between the two: as perceptual repre-
sentations become older, they become degraded, compressed, or summarised, gradually
becoming experiences of a past event in a way that is typically called episodic memory. This
continuum between perception and memory is consistent with previous discussions of
consciousness more broadly [26] and postdiction specifically [14], where retroactive revisions
of past events are known to take place on timescales ranging from subsecond [14,27] to
months or years [28].

132 Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2

Trends in Cognitive Sciences

(A) (1) Smulus detecon Physical me (2) Trajectory predicon Physical me

Available sensory input Objecve present Available sensory input

Sensory
horizon

Sensory delay Predicted

moon trajectory

(3) Predicon of future input Physical me (4) Trajectory percepon Physical me

Available sensory input Predicted future input Available sensory input Predicted future input

Perceived
moon trajectory

Physical me
(B)
Objecve
New Predicon present

Overwrien Predicon
Previous
predicted
! trajectory
Predicon error
New predicted
trajectory

Trends in Cognitive Sciences

Figure 3. Complementary roles for prediction and postdiction in updating the perceptual timeline. (A) Due to neural delays, (1) available sensory input about the
position of a moving object lags behind the object’s physical position (black), but can be used to extrapolate the object’s expected trajectory. (2) This allows future sensory
input to be predicted (green open circles), such that perceptual representations of timepoints beyond the sensory horizon can be activated. (3) In this way, perception
represents a timeline (4), encompassing the ball’s entire trajectory, (i.e., past, present, and future). (B) When predictions do not match incoming sensory input, for
example because backspin on a tennis ball causes it to deviate from its anticipated trajectory, prediction error results. In this situation, postdiction mechanisms
overwrite representations of past or previously predicted events (red), and new predictions are formed (green). The trajectory that was initially perceived is overwritten,
and only the new trajectory is perceived and remembered. Importantly, these postdiction mechanisms satisfy causality because they do not affect past
representations, but rather overwrite current representations of past events.

Relationship to previous work

Concurrent representation of multiple timepoints
This proposal both integrates and extends previous discussions of perception over time. Firstly,
the idea that the nervous system might represent an entire timeline at any given instant builds on a
previous proposal that the brain might construct time through trajectory rather than state estima-
tion [17]. If each of the many ways in which the sensory environment can vary constitutes a
dimension in a hugely multidimensional space, then the state of the environment at any given
instant can be represented by a single point in that state space. As our sensory environment

Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 133

 Trends in Cognitive Sciences

Box 1. Prediction compensates for neural delays

In the proposed conceptualisation of real-time perception, although sensory information about the environment is not available
for timepoints beyond the sensory horizon, these timepoints can nevertheless be activated by prediction. In the case of visual
motion, for example, it is possible to extrapolate the previous trajectory of a moving object to predict its real-time position in the
present (or even the future), despite neural delays [45,47,88]. Naturally, prediction is only effective at compensating delays when
sequences are predictable. If sequences are unpredictable or predictions are violated, then lag is inevitable.

Figure IA shows an example in which an observer views a bar that rotates smoothly over time. The bar unexpectedly
reverses direction between timepoints t0 and t1. Rows in Figure IB indicate the contents of the perceptual timeline at the
four (physical) timepoints t0, t1, t2, and t3. Broken squares indicate timepoints for which no sensory input is available at that
moment. Asterisks mark represented timepoints that correspond to physical time (i.e., the present).

At time t0, predictive mechanisms allow the position of the moving bar in the present to be predicted despite neural delays.
However, when the stimulus reverses direction, predictions at time t1 initially overshoot because the reversal is not yet
detected. When incoming sensory information at time t2 contradicts the predicted representation, the old prediction is
discarded (red broken lines), and lag results because no new prediction is yet available. Finally, by time t3 additional sen-
sory input allows a new prediction to be formed, and the representation ‘catches up’ again.

Figure I. Prediction and

(A)
prediction error.

t0 Unexpected t1 t2 t3
reversal
Physical time

(B)

t0 Delay

Prediction

t1 Delay

Prediction
Delay
t2
? ? ? ?

Prediction
error
t3 Delay

New prediction
Represented time
Trends in Cognitive Sciences

evolves, subsequent moments can be represented as additional points in that same state space.
The evolution of our sensory environment over time can then be described by connecting these
points into a trajectory: a curve in the system’s state space. I propose that the brain continuously
represents and updates that curve, such that the entire perceptual timeline is represented at any
given instant.

At the neural level, the implementation of such a process requires that sensory information from
different timepoints can be simultaneously represented in a way that still allows them to be

134 Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2

Trends in Cognitive Sciences

separated. How can the perceptual system represent sensory information from multiple
timepoints without having that information blur together? Taking vision as an example modality,
one solution to this problem lies in the hierarchical nature of the visual system. Because the trans-
mission of information between layers takes time, at any given instant higher layers represent infor-
mation from older timepoints than lower layers [29]. Importantly, sensory information is also
transformed as it flows through the hierarchy, such that the same information is organised differ-
ently at each step along the way [30]. Across its different levels, the hierarchy as a whole therefore
automatically represents sensory information from multiple distinct timepoints simultaneously
(Figure 4A). A series of recent time-resolved electroencephalogram (EEG) experiments showed
this very elegantly by demonstrating that when multiple stimuli are presented in quick succession,
several stimuli can be separately decoded from the pattern of brain activity at a single instant
[3,31,32].

Symbolic representation of time

This proposal requires that perceptual mechanisms represent time symbolically, as a neural
‘time-stamp’. There are at least two ways in which time might be encoded for this purpose.
One line of evidence indicates that time perception is encoded by dedicated time-sensitive
neurons, such as populations of neurons tuned to specific durations (i.e., duration channels),
just as there are channels tuned to features such as orientation, velocity, and spatial frequency
[33–35]. In this interpretation, the timing of a sensory event is a feature much like its other features,
that directly encodes the appropriate position of the represented event on the perceptual timeline.
In support of this notion, a number of recent studies have revealed patterns of adaptation to
durations that closely mirror the effects of adapting to other features [33–36].

A complementary neural mechanism by which time could be represented is through short-term

synaptic plasticity (STP). STP causes recently-active synapses in a neural circuit to subtly
change their connectivity. As a result, the exact response of a given neuron to any particular stim-
ulus will depend on its recent history (Figure 4B). At the level of neural circuits, this means that the
population response to a stimulus intrinsically encodes the events that preceded the stimulus
[37–40]. Simulations in artificial neural circuits show that in this way, the response pattern evoked
by a given stimulus in even a small neural population can encode both the identity of previous
stimuli as well as how long ago those stimuli were presented [41]. In this way, time (and recent

Box 2. Postdiction reconstructs the perceptual past

A key feature of this proposal is that the perceptual timeline can be updated, revised, reinterpreted, and overwritten as new
information (sensory or otherwise) becomes available. This means that the subjective experience of past events can be
affected by later events. Importantly, in this account, these postdictive mechanisms do not violate the law of causality
because it is the represented past, not the physical past, that is revised.

Figure I illustrates how this allows the presentation of a second disc to affect the perception of events leading up that event
in the Colour Phi effect [13]. In this phenomenon, observers view two differently coloured discs presented in different
positions in quick succession (Figure IA). This creates the percept of a single disc jumping from one position to the other,
changing colour midway. As in Box 1, rows in Figure IB indicate the contents of the perceptual timeline at the three
(physical) timepoints t0, t1, and t2. Broken squares indicate timepoints for which sensory input is not yet available, and
asterisks mark the represented present.

At t0, the first available sensory evidence indicates that a disc has been detected. This is represented at the appropriate
moment. Future representations may also be activated, depending on prior expectations of the disc’s duration. At t1,
subsequent sensory evidence suggests the disc was an isolated flash. Any earlier prediction is discarded and empty space
is represented for the moments following the flash. When the second disc is detected at t2, the timeline as a whole is
postdictively reinterpreted as a moving disc. The timeline is revised, such that the disc is represented in intervening
locations at intermediate moments.

Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 135

 Trends in Cognitive Sciences

Figure I. Postdiction.
(A)

t0 t1 t2

Physical time

(B)

t0 Delay

t1 Delay

Postdiction
t2 Delay

Represented time
Trends in Cognitive Sciences

sensory history) are inherently encoded in subtle variations in the spatial pattern of neural
responses to the most recent stimulus, without dedicated timing neurons.

Either of these mechanisms would allow perception to encode time-stamps alongside or as part
of incoming sensory information. How then might these time-stamps be used to realign sensory
input in (represented) time? For one, perception is likely to detect and adapt to temporal offsets in
correlated inputs across features, as the sensorimotor system does for voluntary action [42].
Feedback and recurrent connections across and within hierarchical layers allow information
from different timepoints to nevertheless be represented concurrently. Indeed, an intriguing
EEG study showed that visual luminance sequences continue to replay, or ‘echo’, in occipital
cortex for up to ~1 s [43], providing ample opportunity for coincidence detection and alignment

136 Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2

Trends in Cognitive Sciences

(A) (C) Continuous Reversal

Next Next
750
Training time (ms)

500

250

Detection task (fast)

Interference
0 1000 2000 along original
Testing time (ms) trajectory
(B)

Discrimination task (slow)

Interference
along revised
trajectory

0 100 200 300 0 100 200 300

Time (ms) Time (ms)

Trends in Cognitive Sciences

Figure 4. Convergent evidence for the current proposal. (A) A temporal generalisation matrix of EEG data [30] reveals
neural representations activated when observers viewed a rapid sequence of oriented visual images. Because individual
images evoke waves of neural activity that flow through the visual hierarchy for up to 750 ms, multiple stimuli can be
decoded at any given testing time. Reproduced with permission from [3]. (B) Short-term plasticity (STP) causes subtle
changes in neural connectivity, such that the response of a neural population to a given stimulus depends on its recent
history. Left and right panels show the multidimensional network response to one or two identical brief input pulses,
respectively (reduced to three principal component dimensions for illustration). In the right panel, when two identical input
pulses are presented, the second pulse (broken arrow) evokes a subtly different neural response to the first pulse (solid
arrow), because the network has not yet returned to baseline. In this way, STP causes neural responses to implicitly
encode recent sensory history. Reproduced with permission from [40]. (C) Apparent motion reveals both prediction and
postdiction. The represented position of an object in apparent motion during the gap between presentations can be
inferred from interference on perceptual tasks in intervening positions. When two black discs continue on their clockwise
trajectory (left panels), maximal interference is observed ahead of the most recent position, irrespective whether
interference is probed with a fast detection task (top) or a slow discrimination task (bottom). When the apparent motion
sequence reverses (right panels), at short latencies the object is still represented along its original trajectory (top), but at
longer latencies, its represented position is revised such that it now causes interference along the new trajectory (bottom).
Reproduced with permission from [77]. Abbreviations: EEG, electroencephalogram; PC, principal component.

with other features. A more recent high-speed fMRI study further revealed that prediction alone is
sufficient to preplay sensory sequences in primary visual cortex [44]. Together with mechanisms
such as STP, this allows information from different timepoints to be represented concurrently
without losing track of where it belongs on the perceptual timeline.

Predicting the present

The proposition that the perceptual system can mitigate the consequences of neural transmis-
sion delays through predictive mechanisms has a long history [45–47]. Prediction is essential to
accurately interact with a dynamic environment despite these delays, but in principle those delays
could be compensated in the motor system [46,48–50], rather than in perception. Nevertheless,
for the case of visual motion, convergent evidence from visual illusions (reviewed in [51]), compu-
tational modelling [29,52–55], and animal neurophysiology [56,57] suggests that the perceptual
system does use prediction to approximate the present [58].

Importantly, recent human neuroimaging work using different sensory and imaging modalities
directly supports the notion that predictive mechanisms preactivate neural representations of
expected stimuli. For example, the orientation of anticipated gratings can be decoded from the

Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 137

 Trends in Cognitive Sciences

activation of early visual cortex using both fMRI [59] and magnetoencephalography (MEG) [60].
Likewise, the future position of an apparent motion stimulus can be decoded using both fMRI
[44] and EEG [32]. Similar effects have been observed in the auditory system, with MEG revealing
preactivation of tonotopic neural representations corresponding to expected but absent tones
[61]. Ultra-high field laminar fMRI indicates that such predictions may be driven by feedback to
deep cortical layers [62,63]. Importantly, when combined with subsequent afferent sensory
information, predictive preactivation both accelerates [64,65] and facilitates [66,67] perception,
consistent with theoretical accounts of how predictions [55,68] and memory more generally
[69], might integrate with sensory input.

Reconstructing the past

The proposal that the perceptual representation of an event can be modified by the subsequent
occurrence of a later event also builds on previous theoretical and experimental work. There are
numerous experimental examples of this postdiction phenomenon, including the Flash-Lag
[25,70] and Flash-Grab effects [71,72] as well as backward masking and apparent motion [14].
Retro-perception [12] is a further example of later events changing the conscious experience of
earlier events [73,74]. Although at first sight paradoxical, postdiction does not entail information
going back in time in a way that would violate causality; it merely involves updating a representa-
tion of a previous event [14,18,46,75]. Conceptually, prediction and postdiction go hand-in-hand,
for the simple reason that predictions do not always eventuate, and failed predictions therefore
need to be corrected if the brain is to represent an accurate chronology of events [46].

Importantly, inherent in this reconstruction role of postdiction is that although the original repre-
sentation is ultimately overwritten, before that happens there is a moment at which the original
(predicted) representation is active, however briefly. Although the original representation does
not survive to be consciously reported, if it would be possible to probe the perceptual system
at precisely the right moment, it should be possible to demonstrate the brief preactivation of
that representation. A recent EEG study showed that when an object in apparent motion unex-
pectedly reverses, this is precisely what is observed [32,76]. When the object changes direction,
information about the trajectory change arrives too late to prevent predictive mechanisms from
activating perceptual representations along the original trajectory. When sensory information
about the new trajectory eventually arrives, the failed prediction is then postdictively overwritten,
undoing the preactivation along the original trajectory.

This pattern of neural evidence closely parallels a previous behavioural study [77] that probed the
represented position of objects along an apparent motion path using an interference paradigm
[78]. Observers carried out a perceptual task on the positions between the possible positions
of two discs in apparent motion (Figure 3C). When the discs moved along their original trajectory,
maximal interference was observed just ahead of the most recent apparent motion station, as
expected. Crucially, when the discs unexpectedly reversed direction, the pattern of interference
depended on the perceptual task. With a fast detection task (~ 350 ms), interference was still
observed along the original trajectory. However, with a slower discrimination task (~ 620 ms),
interference ahead of the last position was lifted, and interference was now observed along the
new trajectory (Figure 4C). Together, this pattern of results is consistent with preactivated
representations being overwritten by postdictive reconstruction mechanisms when predictions
do not eventuate.

Existing models of temporal perception

There is an extensive literature in both philosophy and cognitive neuroscience debating the tem-
poral structure of (conscious) perception [79]. In the first place, there exists a distinction between

138 Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2

Trends in Cognitive Sciences

three classes of models of the perceived present [80]. In cinematic models, perception takes Outstanding questions
place at sequential instants, and each percept contains information about a single instant. In Is there a neural sense in which
retentional models, perception also takes place at sequential instants, but each instant includes sensory features processed in distinct
brain areas ‘all come together’, or is
information from an extent of time (thereby allowing integration over time). Finally, in extensional
conscious perception nothing more
models, both perception and its contents are extended in time. My conceptualisation of real- than the output of each of the
time perception as representing a curve (rather than a point) in a perceptual state space aligns individual areas? Sensory processing
with the class of extensional models [81,82]. is known to be distributed across
numerous distinct brain areas, whereas
our perceptual experience is generally
An important concept in this context is the specious present: the time period within which per- integrated and coherent. The current
cepts are experienced as being ‘in the present’ [83]. The present proposal extends this concept proposal presents a mechanism by
by challenging the notion that perception is made up of sequential experiences of the present, which different features can be
temporally aligned, but how (and if)
with previous experiences relegated to memory. Instead, I propose that perception is continuous
they combine to produce conscious
with memory, such that the (recent) past is, and the predicted future are, experienced alongside the awareness remains an open question.
present. This does not cause a ‘too-many-percepts’ problem [15], because past, present, and
future do not overlap as ‘now’, rather, we perceive a point (or small range, see Outstanding What are the precise quantitative
computations underlying prediction
questions) on this timeline as the present, and also experience the recent past and near future and postdiction mechanisms, and
as having just happened and being about to happen, respectively. This also solves the paradox their integration with sensory input? It
of how perceptual features that are defined over time, such as visual motion or auditory melodies, seems plausible that some kind of
generalised Kalman filter would be
can be perceived at a single instant [84]. With perception conceived as representing a timeline
involved in generating and updating
rather than a single timepoint, it is the object’s trajectory in state space that is perceived, rather predictions, as has been successfully
than a sequence of instantaneous positions [17], thereby solving the paradox. modelled for visual motion. However,
whether comparable computations
operate on other features and
Another influential debate has centred on the question of whether perception is discrete or
modalities remains to be investigated.
continuous [85]. The notion that our apparently continuous stream of perception is made up of
a sequence of discrete perceptual frames has been related to neural oscillations [86], but percep- How is ‘now’, the experienced present
tual moments might equally occur ‘on demand’ [15,87]. Importantly, the current proposal does moment, marked and updated on
the perceptual timeline? One point
not require discretization or multistage processing [15,84] to solve the apparent contradiction (or small range) is presumably
between the speed of perception and the long integration window of phenomena such as experienced as ‘now’, but is the
postdiction. This is because under this proposal, postdiction does not impose an upper bound position of that point constant relative
on the latency of conscious perception. to the flow of physical time? If the
position of this marked present on the
timeline is somehow variable, this
A final question is whether overwritten sensory representations are (briefly) consciously perceived might lead to distortions of perceived
before they are overwritten. When observers are unable to report the original (unrevised) percept time, such as those that occur around
in a postdiction paradigm, it is a mistake to conclude that the original input was never perceived. the time of saccades and blinks. It will
be interesting to see whether such
That conclusion assumes that if observers would perceive the original input, they would also have phenomena can be understood in this
access to a memory trace of that percept by the time they need to report it. However, in most framework.
postdiction paradigms, overt responses to a stimulus are executed (at least) several hundreds
How does the experience of past
of milliseconds after the initial event. This means observers’ reports of the event are based on a
events change as time passes? How
recent representation of that event. This representation is susceptible to postdictive revision different features of an experience
[27], and no separate memory trace exists to indicate whether an earlier (now overwritten) gradually evolve from perceiving now
representation of that same event was ever perceived. As noted above, the brief existence of to perceiving the past, to what feels
like remembering the past, is an open
the original sensory representation can be demonstrated both behaviourally [77] and neurally
question. One might wonder if such
[32], but whether it can be said to be briefly consciously perceived (and thus subsequently (presumably gradual) transitions map
forgotten) remains an interesting open question (see Outstanding questions). onto different types of memory, such
as iconic, working, and long-term
memory, which are also associated
Concluding remarks with different latencies and capacities.
Despite our everyday introspective experience, the intuitive notion that real-time perception
simply reflects the external environment from moment to moment is inconsistent with biological
constraints and experimental observations. Here, I address the key challenges caused by neural
delays, temporal desynchronisation, and retroactive effects on perception by proposing a new

Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 139

 Trends in Cognitive Sciences

way to conceptualise real-time perception. The key innovation is that at any given moment,
perceptual mechanisms represent an entire timeline, rather than a single timepoint. The proposal
is consistent with known properties of the nervous system: neural delays impose a sensory
horizon on incoming sensory input to this timeline, but perceptual representations beyond this
horizon can nevertheless be preactivated by predictive mechanisms. Perceptual representations
of past events can be postdictively updated and revised as new information warrants, without
violating principles of causality. Together, predictive and postdictive mechanisms generate a
conscious perceptual experience of the present, despite the intrinsic delays in the nervous
system.

Acknowledgments
I gratefully acknowledge support from the Australian Research Council (DP180102268 and FT200100246). I am also grateful
to Dr Marjolein Kammers and three anonymous reviewers for invaluable comments on earlier versions of this manuscript.

Declaration of interests
No interests are declared.

References
1. Schacter, D.L. et al., eds (2010) Psychology, 2nd edn Worth 19. Johnston, A. (2010) Modulation of time perception by visual adap-
Publishers tation. In Attention and Time, pp. 187–200, Oxford University Press
2. Carlson, T. et al. (2013) Representational dynamics of object vi- 20. Johnston, A. et al. (2006) Spatially localized distortions of event
sion: the first 1000 ms. J. Vis. 13, 1 time. Curr. Biol. 16, 472–479
3. King, J.-R. and Wyart, V. (2021) The human brain encodes a 21. Hogendoorn, H. et al. (2010) Spatially localized time shifts of the
chronicle of visual events at each instant of time thanks to perceptual stream. Front. Psychol. 1 181–8
the multiplexing of traveling waves. J. Neurosci. 41, 22. Moutoussis, K. and Zeki, S. (1997) A direct demonstration of
7224–7233 perceptual asynchrony in vision. Proc. Biol. Sci. 264, 393–399
4. Maunsell, J.H. and Gibson, J.R. (1992) Visual response latencies 23. Zeki, S. and Bartels, A. (1999) Toward a theory of visual con-
in striate cortex of the macaque monkey. J. Neurophysiol. 68, sciousness. Conscious. Cogn. 8, 225–259
1332–1344 24. White, P.A. (2020) The perceived present: what is it, and what is
5. Maunsell, J.H. et al. (1999) Visual response latencies of it there for? Psychon. Bull. Rev. 27, 583–601
magnocellular and parvocellular LGN neurons in macaque 25. Eagleman, D.M. and Sejnowski, T.J. (2000) Motion integration
monkeys. Vis. Neurosci. 16, 1–14 and postdiction in visual awareness. Science 287, 2036–2038
6. Thorpe, S. et al. (1996) Speed of processing in the human visual 26. Edelman, G. (2006) Consciousness: the remembered present.
system. Nature 381, 520–522 Ann. N. Y. Acad. Sci. 929, 111–122
7. Kirchner, H. and Thorpe, S.J. (2006) Ultra-rapid object detection 27. Intraub, H. and Dickinson, C.A. (2008) False memory 1/20th of a
with saccadic eye movements: visual processing speed second later. Psychol. Sci. 19, 1007–1014
revisited. Vis. Res. 46, 1762–1776 28. Loftus, E.F. and Palmer, J.C. (1974) Reconstruction of auto-
8. Smeets, J.B.J. et al. (1998) Visuomotor delays when hitting run- mobile destruction: an example of the interaction between
ning spiders. In EWEP 5 - Advances in Perception-Action Cou- language and memory. J. Verbal Learn. Verbal Behav. 13,
pling (Bril, B. et al., eds), pp. 36–40, Editions EDK 585–589
9. Alais, D. and Burr, D. (2003) The “flash-lag” effect occurs in au- 29. Hogendoorn, H. and Burkitt, A.N. (2019) Predictive coding with
dition and cross-modally. Curr. Biol. 13, 59–63 neural transmission delays: a real-time temporal alignment hy-
10. Arnold, D.H. et al. (2001) Asynchronous processing in vision: pothesis. eNeuro 6 ENEURO.0412-0418.2019
color leads motion. Curr. Biol. 11, 596–600 30. King, J.-R. and Dehaene, S. (2014) Characterizing the dynamics
11. Schmolesky, M.T. et al. (1998) Signal timing across the macaque of mental representations: the temporal generalization method.
visual system. J. Neurophysiol. 79, 3272–3278 Trends Cogn. Sci. 18, 203–210
12. Sergent, C. et al. (2013) Cueing attention after the stimulus is 31. Grootswagers, T. et al. (2019) The representational dynamics
gone can retrospectively trigger conscious perception. Curr. of visual objects in rapid serial visual processing streams.
Biol. 23, 150–155 Neuroimage 188, 668–679
13. Kolers, P.A. and von Grünau, M. (1976) Shape and color in 32. Blom, T. et al. (2020) Predictions drive neural representations of
apparent motion. Vis. Res. 16, 329–335 visual events ahead of incoming sensory information. Proc. Natl.
14. Shimojo, S. (2014) Postdiction: its implications on visual Acad. Sci. U. S. A. 117, 7510–7515
awareness, hindsight, and sense of agency. Front. Psychol. 5, 33. Heron, J. et al. (2012) Duration channels mediate human time
196 perception. Proc. R. Soc. B Biol. Sci. 279, 690–698
15. Herzog, M.H. et al. (2020) All in good time: long-lasting 34. Maarseveen, J. et al. (2019) The duration aftereffect does not
postdictive effects reveal discrete perception. Trends Cogn. reflect adaptation to perceived duration. PLoS One 14,
Sci. 24, 826–837 e0213163
16. Johnston, A. and Nishida, S. (2001) Time perception: brain time 35. Maarseveen, J. et al. (2017) An investigation of the spatial selec-
or event time? Curr. Biol. 11, 427–430 tivity of the duration after-effect. Vis. Res. 130, 67–75
17. Grush, R. (2005) Internal models and the construction of time: 36. Maarseveen, J. et al. (2018) Attention gates the selective
generalizing from state estimation to trajectory estimation to ad- encoding of duration. Sci. Rep. 8, 2522
dress temporal features of perception, including temporal illu- 37. Buonomano, D. and Merzenich, M. (1995) Temporal information
sions. J. Neural Eng. 2, S209–S218 transformed into a spatial code by a neural network with realistic
18. Dennett, D.C. and Kinsbourne, M. (1992) Time and the observer: properties. Science 267, 1028–1030
the where and when of consciousness in the brain. Behav. Brain 38. Motanis, H. et al. (2018) Short-term synaptic plasticity as a
Sci. 15, 183–201 mechanism for sensory timing. Trends Neurosci. 41, 701–711

140 Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2

Trends in Cognitive Sciences

39. Buonomano, D.V. and Maass, W. (2009) State-dependent com- 63. Aitken, F. et al. (2020) Prior expectations evoke stimulus-specific
putations: spatiotemporal processing in cortical networks. Nat. activity in the deep layers of the primary visual cortex. PLoS Biol.
Rev. Neurosci. 10, 113–125 18, e3001023
40. Karmarkar, U.R. and Buonomano, D.V. (2007) Timing in the ab- 64. Hogendoorn, H. and Burkitt, A.N. (2018) Predictive coding of
sence of clocks: encoding time in neural network states. Neuron visual object position ahead of moving objects revealed by
53, 427–438 time-resolved EEG decoding. Neuroimage 171, 55–61
41. Paton, J.J. and Buonomano, D.V. (2018) The neural basis of 65. Pinto, Y. et al. (2015) Expectations accelerate entry of visual
timing: distributed mechanisms for diverse functions. Neuron stimuli into awareness. J. Vis. 15, 13–15
98, 687–705 66. Aitken, F. et al. (2020) Prior expectations of motion direction
42. Haggard, P. et al. (2002) Voluntary action and conscious awareness. modulate early sensory processing. J. Neurosci. 40, 6389–6397
Nat. Neurosci. 5, 382–385 67. Roach, N.W. et al. (2011) Visual motion induces a forward pre-
43. VanRullen, R. and Macdonald, J.S.P. (2012) Perceptual echoes diction of spatial pattern. Curr. Biol. 21, 740–745
at 10 Hz in the human brain. Curr. Biol. 22, 995–999 68. de Lange, F.P. et al. (2018) How do expectations shape perception?
44. Ekman, M. et al. (2017) Time-compressed preplay of anticipated Trends Cogn. Sci. 22, 764–779
events in human primary visual cortex. Nat. Commun. 8, 15276 69. Nobre, A.C. and Stokes, M.G. (2019) Premembering experience: a
45. Nijhawan, R. (2008) Visual prediction: psychophysics and neuro- hierarchy of time-scales for proactive attention. Neuron 104,
physiology of compensation for time delays. Behav. Brain Sci. 132–146
31, 179–239 70. Rao, R.P.N. et al. (2001) Optimal smoothing in visual motion per-
46. Eagleman, D.M. (2008) Prediction and postdiction: two frame- ception. Neural Comput. 13, 1243–1253
works with the goal of delay compensation. Behav. Brain Sci. 71. Blom, T. et al. (2019) When predictions fail: correction for extrap-
31, 205–206 olation in the flash-grab effect. J. Vis. 19, 3–11
47. Nijhawan, R. (1994) Motion extrapolation in catching. Nature 72. Cavanagh, P. and Anstis, S.M. (2013) The flash grab effect. Vis.
370, 256–257 Res. 91, 8–20
48. Nijhawan, R. and Wu, S. (2009) Compensating time delays with 73. Rimsky-Robert, D. et al. (2019) Retrospective auditory cues can im-
neural predictions: are predictions sensory or motor? Philos. prove detection of near-threshold visual targets. Sci. Rep. 9 18966–
Trans. R. Soc. A Math. Phys. Eng. Sci. 367, 1063–1078 11
49. Nijhawan, R. and Kirschfeld, K. (2003) Analogous mechanisms 74. Thibault, L. et al. (2016) Retrospective attention gates discrete con-
compensate for neural delays in the sensory and the motor path- scious access to past sensory stimuli. PLoS One 11, e0148504
ways. Curr. Biol. 13, 749–753 75. Sergent, C. (2018) The offline stream of conscious representations.
50. Kerzel, D. and Gegenfurtner, K.R. (2003) Neuronal processing Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci. 373, 20170349
delays are compensated in the sensorimotor branch of the visual 76. Blom, T. et al. (2021) The time-course of prediction formation
system. Curr. Biol. 13, 1975–1978 and revision in human visual motion processing. Cortex 138,
51. Hogendoorn, H. (2020) Motion extrapolation in visual processing: 191–202
lessons from 25 years of flash-lag Debate. J. Neurosci. 40, 77. Hogendoorn, H. et al. (2008) Interpolation and extrapolation on
5698–5705 the path of apparent motion. Vis. Res. 48, 872–881
52. Kwon, O.-S. et al. (2015) Unifying account of visual motion and posi- 78. Yantis, S. and Nakama, T. (1998) Visual interactions in the path
tion perception. Proc. Natl. Acad. Sci. U. S. A. 112, 8142–8147 of apparent motion. Nat. Neurosci. 1, 508–512
53. Burkitt, A.N. and Hogendoorn, H. (2021) Predictive visual motion 79. Kent, L. and Wittmann, M. (2021) Time consciousness: the
extrapolation emerges spontaneously and without supervision at missing link in theories of consciousness. Neurosci. Conscious.
each layer of a hierarchical neural network with spike-timing-de- 2021, niab011
pendent plasticity. J. Neurosci. 41, 4428–4438 80. Dorato, M. and Wittmann, M. (2020) The phenomenology and
54. Khoei, M.A. et al. (2017) The flash-lag effect as a motion-based cognitive neuroscience of experienced temporality. Phenomenol.
predictive shift. PLoS Comput. Biol. 13, e1005068 Cogn. Sci. 19, 747–771
55. Feuerriegel, D. et al. (2021) Predictive activation of sensory 81. Hoerl, C. (2014) Time and the domain of consciousness. Ann. N.
representations as a source of evidence in perceptual decision- Y. Acad. Sci. 1326, 90–96
making. Cortex 136, 140–146 82. Dainton, B. (2011) Time, passage, and immediate experience.
56. Schwartz, G. et al. (2007) Synchronized firing among retinal gan- In Oxford Handbook of Philosophy of Time (Callendar, C., ed.),
glion cells signals motion reversal. Neuron 55, 958–969 pp. 382, Oxford University Press
57. Berry, M.J. et al. (1999) Anticipation of moving stimuli by the 83. Andersen, Holly K. and Grush, Rick (2009) A brief history of time-
retina. Nature 398, 334–338 consciousness: historical precursors to James and Husserl.
58. White, P.A. (2018) Is the perceived present a predictive model of J. Hist. Philos. 47, 277–307
the objective present? Vis. Cogn. 26, 624–654 84. Herzog, M.H. et al. (2016) Time slices: what is the duration of a
59. Kok, P. et al. (2014) Prior expectations evoke stimulus templates percept? PLoS Biol. 14, e1002433
in the primary visual cortex. J. Cogn. Neurosci. 26, 1546–1554 85. VanRullen, R. and Koch, C. (2003) Is perception discrete or
60. Kok, P. et al. (2017) Prior expectations induce prestimulus sensory continuous? Trends Cogn. Sci. 7, 207–213
templates. Proc. Natl. Acad. Sci. U. S. A. 114, 10473–10478 86. VanRullen, R. (2016) Perceptual cycles. Trends Cogn. Sci. 20,
61. Demarchi, G. et al. (2019) Automatic and feature-specific 723–735
prediction-related neural activity in the human auditory system. 87. Brenner, E. and Smeets, J.B. (2000) Motion extrapolation is
Nat. Commun. 10, 3440 not responsible for the flash-lag effect. Vis. Res. 40,
62. Kok, P. et al. (2016) Selective activation of the deep layers of the 1645–1648
human primary visual cortex by top-down feedback. Curr. Biol. 88. Nijhawan, R. (2002) Neural delays, visual motion, and the flash-lag
26, 371–376 effect. Trends Cogn. Sci. 6, 387

Trends in Cognitive Sciences, February 2022, Vol. 26, No. 2 141