The structure of the prediction problem depends on the structure of the world around us. In a world of completely random stimuli, for example, prediction is impossible. Consider a simple visual world such that, in the small patch of space represented by the neurons from which we record, there is just one object (a dark horizontal bar against a light background) moving along a trajectory $x_t$. We want to construct trajectories that are predictable, but not completely; the moving object has some inertia, so that the velocities ${\mathit{\upsilon }}_t$ are correlated across time, but is also “kicked” by unseen random forces. A mathematically tractable example (Eqs. 4 and 5 in Materials and Methods) is shown in Fig. 1A, along with the responses recorded from a population of ganglion cells in the salamander retina.

Open in a separate window

Fig. 1.

Information about position of a moving bar. (A) Trajectory $x_t$ and spiking responses recorded from several cells simultaneously (36); responses in a single small window of time $\mathrm{\Delta }\tau$ can be expressed as binary words $w_t$. (B) Information that N-cell words provide about bar position (Eq. 1), as a function of the delay $\mathrm{\Delta }t$, averaged over many N-cell groups. Estimation errors and SEMs over groups both are negligible (~0.01 bits/spike); we stop at $N=7$ to avoid undersampling. (C) Normalized autocorrelation of the trajectory, $C_{xx}$, vs. time delay, $t-t^{\prime }$. The peak has been shifted to align with the peak information in B.

If we look at neural responses in small windows of time, e.g., $\mathrm{\Delta }\tau =1/60\hspace{0.17em}\text{s}$, almost all ganglion cells generate either zero or one action potential. Thus, the activity of a single neuron, labeled $\text{i}$, can be represented by a binary variable ${\sigma }_{\text{i}}\left(t\right)=1$ when the cell spikes at time t and ${\sigma }_{\text{i}}\left(t\right)=0$ when it is silent. The activity of N neurons then becomes a binary “word” $w_t\equiv \left\{{\sigma }_1\left(t\right),\hspace{0.17em}{\sigma }_2\left(t\right),\hspace{0.17em}\cdots ,\hspace{0.17em}{\sigma }_N\left(t\right)\right\}$. If we (or the brain) observe the pattern of activity $w_t$ at time t, how much do we know about the position of the moving object? Neurons are responding to the presence of the object, and to its motion, but there is some latency in this response, so that $w_t$ will be maximally informative about the position of the object at some time in the past, $x_{t^{\prime }<t}$. On the other hand, we know that the brain is capable of predicting the future position of moving objects and that these ganglion cells provide all of the visual data on which such predictions are based, so it must be true that $w_t$ also provides some information about $x_{t^{\prime }>t}$.

We can make these ideas precise by estimating, in bits, the information that the words $w_t$ provide about the position of the object at time $t^{\prime }$ (5–8):

where $P_W\left(w\right)$ describes the overall distribution of words generated by the neural population, $P_X\left(x\right)$ describes the distribution of positions of the object across the entire experiment, and $P\left(x_{t^{\prime }}|w_t\right)$ is the probability of finding the object at position x at time $t^{\prime }$ given that we have observed the response $w_t$ at time t. Results are shown in Fig. 1B, where we put the information carried by different numbers of neurons on the same scale by normalizing to information per spike.

As expected, the retina is most informative about the position of the object $t-t^{\prime }=t_{\text{lat}}\sim 80\hspace{0.17em}\text{ms}$ in the past. At this point, the information carried by multiple retinal ganglion cells is, on average, redundant, so that the information per spike declines as we examine the responses of larger groups of neurons. Although the details of the experiments are different, the observation of coding redundancy at $t_{\text{lat}}$ is consistent with many previous results (9–17). However, the information that neural responses carry about position extends far into the past, $t^{\prime }\ll t-t_{\text{lat}}$, and more importantly this information extends into the future, so that the neural response at time t predicts the position of the object at times $t^{\prime }>t$. This broad window over which we can make predictions and retrodictions is consistent with the persistence of correlations in the stimulus, as it must be (Fig. 1C). As we extrapolate back in time, or make predictions, the redundancy of the responses decreases, and there are hints of a crossover to synergistic coding of predictions far in the future, to which we return below.

Even if we keep a perfect record of everything we have experienced until the present moment, we cannot make perfect predictions: all of the things we observe are influenced by causal factors that we cannot observe, and from our point of view the time evolution of our sensory experience thus has some irreducible level of stochasticity. Formally, we imagine that we are sitting at time $t_{\text{now}}$ and have been observing the world, so far, for a period of duration T. If we refer to all our sensory stimuli as $s\left(t\right)$, then what we have access to is the past $X_{\text{past}}\equiv s\left(t_{\text{now}}-T<t\le t_{\text{now}}\right)$. What we would like to know is the future, $X_{\text{future}}\equiv s\left(t>t_{\text{now}}\right)$. The statement that predictive power is limited is, quantitatively, the statement that the predictive information, $I_{\text{pred}}\left(T\right)\equiv I\left(X_{\text{past}};X_{\text{future}}\right)$, is finite (4). This is the number of bits that the past provides about the future, and it depends not on what our brain computes but on the structure of the world.

Not all aspects of our past experience are useful in making predictions. Suppose that we build a compressed representation Z of our past experience, keeping some features and throwing away others. We can ask how much predictive information is captured by these features, $I_{\text{future}}\equiv I\left(Z;X_{\text{future}}\right)$. Notice that, in building the representation Z, we start with our observations on the past, and so there is some mapping $X_{\text{past}}\to Z$; this feature extraction captures a certain amount of information about the past, $I_{\text{past}}\equiv I\left(Z;X_{\text{past}}\right)$. The crucial point is that, given the statistical structure of our sensory world, $I_{\text{future}}$ and $I_{\text{past}}$ are related to one another. Specifically, if we want to have a certain amount of predictive power $I_{\text{future}}$, we need to capture a minimum number of bits $\left(I*\right)$ about the past, $I_{\text{past}}\ge I_{\text{past}}*\left(I_{\text{future}}\right)$. Conversely, if we capture a limited number of bits about the past, there is a maximum amount of predictive power that we can achieve, $I_{\text{future}}\le I_{\text{future}}*\left(I_{\text{past}}\right)$, and we can saturate this bound only if we extract the most predictive features. Thus, we can plot information about the future vs. information about the past, and in any particular sensory environment this plane is divided into accessible and impossible regions; this is an example of the information bottleneck problem (18, 19). In Fig. 2A, we construct this bound for the simple sensory world of a single moving object used in our experiments (see Materials and Methods for details). To be optimally efficient at extracting information is to build a representation of the sensory world that is close to the bound that separates the allowed from the forbidden.

Open in a separate window

Fig. 2.

Bounds on predictive information. (A) Any prediction strategy defines a point in the plane I_future vs. I_past. This plane is separated into allowed and forbidden regions by a bound, $I_{\text{future}}*\left(I_{\text{past}}\right)$, shown for the sensory world of a moving bar following the stochastic trajectories of Fig. 1. (B) We can capture the same information about the past in different ways, illustrated by the black “error ellipses” in the position/velocity plane. If we know that the trajectory is inside one of these ellipses, we have captured I_past = 0.42 bits. However, points inside these ellipses propagate forward along different trajectories, and after $\mathrm{\Delta }\tau =1/60\hspace{0.17em}\text{s}$, these trajectories arrive at the points shown in purple and green. Using the same number of bits to make more accurate statements about position leads to more predictive information (purple; $0.40\hspace{0.17em}\text{bits}$) than if we use these bits to make more accurate statements about velocity (green; $0.18\hspace{0.17em}\text{bits}$).

Building the maximally efficient predictor is nontrivial, even in seemingly simple cases. For an object with trajectories as in Fig. 1, knowledge of the object’s position and velocity at time t provides all of the information possible about the future trajectory. However, knowing position and velocity exactly requires an infinite amount of information. If, instead, we know the position and velocity only with some errors, we can draw an error ellipse in the position–velocity plane, as shown in Fig. 2B, and the area of this ellipse is related to the information that we have captured about the past. Points inside the error ellipse extrapolate forward to a cloud of possible futures. The key point is that error ellipses with the same area but different shapes or orientations—using, for example, the limited number of available bits to provide information about position vs. velocity—extrapolate forward to clouds of different sizes. Thus, to make the best predictions, we have to be sure that our budget of bits of about the past is used most effectively, and this is true even when prediction is “just” extrapolation.

The statement that the neural response w provides information about a feature f in the stimulus means that there is a reproducible relationship between these two variables (5–8). To probe this reproducibility, we must present the same features many times, and sample the distribution of responses $P\left(w|f\right)$. The information that w provides about f is as follows:

where the features are drawn from the distribution $P\left(f\right)$, and the overall distribution of responses is given by the following:

In the case of interest here, the feature f is the future of the stimulus. To measure the information that neural responses carry about the future, we thus need to repeat the future. More precisely, we need to generate stimulus trajectories that are different but converge onto the same future. Given that we can write the distribution of trajectories $P\left[x\left(t\right)\right]$, we can draw multiple independent trajectories that have a “common future,” as shown schematically in Fig. 3A (20) (Materials and Methods).

Open in a separate window

Fig. 3.

Direct measures of the predictive information in neural responses. (A) Many independent samples of the trajectory $x_t$ converge onto one of several common futures, two of which are shown here (red and blue). The time of convergence is indicated by the vertical dashed line. (B) Mean spike rates of a single neuron in response to the stimuli in A. Shaded regions are ±1 SEM. (C) Information about the common future for one group of five cells, as a function of the time, $\mathrm{\Delta }t$, until convergence. Solid line shows the bound on $I_{\text{future}}\left(\mathrm{\Delta }t\right)$ for this group’s $I_{\text{past}}$. (D) Information about the future vs. information about the past, for many groups of different size, N with Δt = 1/60 s; group A as in C. Error bars include contributions from the variance across groups and the SD of the individual information estimates. Solid line is the bound from Fig. 2A.

If trajectories converge onto a common future at time $t=0$, then for $t\ll 0$ the neural responses will be independent of the future, and we can see this in single cells as a probability of spiking that is independent of time or of the identity of the future (Fig. 3B). As we approach $t=0$, the neurons respond to aspects of the stimulus that are themselves predictive of the common future stimulus, and hence the probability of spiking becomes modulated. Quantitatively, we can use Eq. 2 to estimate the information carried by responses from $N=\mathrm{1,2},\cdots ,7$ neurons about the future, as shown in Fig. 3C for a particular five-cell group. This group of cells captures $0.78\hspace{0.17em}\text{bits}/\text{spike}$ of information about the past of the sensory stimulus, or $I_{\text{past}}=0.11\hspace{0.17em}\text{bits}$, computed by taking the stimulus feature, f, to be the past. Fig. 2A tells us that this amount of information about the past can lead to a maximum of $I_{\text{future}}^{*}\left(I_{\text{past}}\right)=0.097\hspace{0.17em}\text{bits}$ about the future. We can compute the predictive information in this group of cells via Eq. 2 and compare it to this bound. In fact, this group of cells achieves $I_{\text{future}}/I_{\text{future}}^{*}=0.98\pm 0.39$, so that it is within error bars of being optimal. We can also generalize the bound in Fig. 2, to ask what happens if we make predictions not of the entire future, but only starting $\mathrm{\Delta }t$ ahead of the current time; we see that the way in which predictive power decays as we extrapolate further into the future follows the theoretical limit set by the structure of the sensory inputs (Fig. 3C).

The results for the five-cell group in Fig. 3C, which has a modest amount of information about the future, are not unusual. For each of the 53 neurons in the population that we monitor, we can find the group of cells, including this neuron, that has the most future information. These groups also operate close to the bound in the $\left(I_{\text{past}},I_{\text{future}}\right)$ plane, as shown in Fig. 3D. Not all groups that contain this neuron sit near the bound, but we do not expect a random sampling of cells to have this property. For example, two cells might sample different parts of visual space that are not connected via a predictable stimulus trajectory. The fact that every cell in this recording participated in some group that sits near the bound is intriguing. This continues to be true as we look at larger and larger groups of cells, until our finite dataset no longer allows effective sampling of the relevant distributions. At least under these stimulus conditions, populations of neurons in the retina thus provide near-optimal representations of predictive information, extracting from the visual input precisely those bits that allow maximal predictive power.

Could near-optimal prediction result from known receptive field properties of these cells? To test this, we have made conventional linear/nonlinear (LN) models of the individual neurons in our dataset (21, 22): image sequences are projected linearly onto a template (spatiotemporal receptive field), and the probability of spiking is a nonlinear function of this projection. We fit these models to the responses of each neuron to a long movie with same statistics as in Fig. 3, and we adjust the nonlinearity to match the mean spike probability and the information captured about the past by single cells (details in Linear–Nonlinear Model). We then analyzed the performance of the model populations in exactly the same way that we analyzed the real populations. Populations of LN neurons fall far below the bound on predictive information, and this gap grows with the number of neurons (Fig. S1), in marked contrast to the real data (Fig. 3D). Interestingly, the models are not so far from the performance of an optimal system that has access only to data from ~100 ms in the past, comparable to the delay one might guess from Fig. 1B. Rather than being a consequence of the receptive fields for individual neurons, the near-optimal performance that we see in Fig. 3D thus is evidence that conventional models are missing the ability of the retina to overcome apparent delays in the encoding of dynamic inputs.

It seems natural to phrase the problem of prediction in relation to the visual stimulus, as in Fig. 3, but the brain has no direct access to visual stimuli except that provided by the retina. Could the brain learn, in an unsupervised fashion, to predict the future of retinal outputs? More precisely, if we observe that a population of retinal ganglion cells generates the word $w_t$ at time t, what can we say about the word that will be generated at time $t+\mathrm{\Delta }t$ in the future? The answer to this question is contained in the conditional distribution of one word on the other, $P\left(w_{t+\mathrm{\Delta }t}|w_t\right)$.

In Fig. 4A, we show an example of $P\left(w_{t+\mathrm{\Delta }t}|w_t\right)$ for $N=4$ cells, as the retina responds to naturalistic movies of underwater scenes (see Materials and Methods for details). This conditional distribution is very different from the prior distribution of words (shown to the right), which means that there is significant mutual information between $w_t$ and $w_{t+\mathrm{\Delta }t}$. In Fig. 4B, we show the distribution of this predictive information between words, for groups of $N=2$, $N=4$, and $N=9$ cells. We have normalized the information in each group by the mean number of spikes, and we see that the typical bits per spike is growing as we look at larger groups of cells. Thus, the total predictive information in the patterns of activity generated by N cells grows much more rapidly than linear in N: predictive information is encoded synergistically.

Open in a separate window

Fig. 4.

Mutual information between past and future neural responses. (A) Conditional distribution $P\left(w_{t+\mathrm{\Delta }t}|w_t\right)$, at time $\mathrm{\Delta }t=1/60\hspace{0.17em}\text{s}$, for the group of four cells with the maximum information (1.1 bits/spike), in response to a natural movie. The prior distribution of words, $P\left(w\right)$, is shown adjacent to the conditional. Probabilities are plotted on a log scale; blank bins indicate zero samples. (B) Distributions of $I\left(W_t;W_{t+\mathrm{\Delta }t}\right)$ for $N=2$, $N=4$, and $N=9$ cells, with $\mathrm{\Delta }t=1/60\hspace{0.17em}\text{s}$. (C) Information between words as a function of $\mathrm{\Delta }t$. Inset shown information vs. N at $\mathrm{\Delta }t$ marked by arrows. (D) Information between words for groups of $N=9$, as a function $\mathrm{\Delta }t$ for different classes of stimuli: a natural movie, the moving bar from Fig. 1, and a random flickering checkerboard refreshed at 30 fps. Shaded regions indicate $\pm 1$ SD across different groups of cells.

With these naturalistic stimuli, larger groups of cells carry predictive information for hundreds of milliseconds, as shown in Fig. 4C, and the maximum predictive information is above 1 bit/spike on average across the thousands of groups that we sampled. Smaller groups of cells do not carry long-term predictive power, and for short-term predictions they carry roughly one-half the information per spike that we see in larger groups.

The large amounts of predictive information that we see in neural responses are tied to the structure of the sensory inputs (Fig. 4D). Naturalistic movies generate the most powerful, and most long-ranged, predictable events; the responses to random checkerboard movies lose predictability within a few frames; and motion of a single object (as in Fig. 1) gives intermediate results. The internal dynamics of the retina could generate predictable patterns of activity even in the absence of predictable structure in the visual world, but this does not seem to happen. This raises the possibility that trying to predict the future state of the retina from its current state can lead us (or the brain) to focus on patterns of activity that are especially informative about the visual world.

The predictive information carried by N neurons is more than N times the information carried by single neurons, but even at $N=9$ it is less than 1 bit in total. Can a neuron receiving many such ganglion cell inputs compress the description of the state of the retina at time t, while preserving the information that this state carries about what will happen at time $t+\mathrm{\Delta }t$ in the future? That is, can we do for the retinal output what the retina itself does for the visual input? In particular, if we can write down all of the predictive information in one bit, then we can imagine that there is a neuron inside the brain that takes the N cells as inputs, and then a spike or silence at the output of this “predictor neuron” $\left({\sigma }^{\text{out}}\right)$ captures the available predictive information.

Compressing our description of input words down to 1 bit means sorting the words $w_t$ into two groups $w_t\to {\sigma }^{\text{out}}$. If this grouping is deterministic, then with $N=4$ neurons there are $\mathrm{65,536}$ possible groupings, and so we can test all of the possibilities (Stimulus Information in σ^out for One Group and Fig. S2.). It indeed is possible to represent almost all of the predictive information from four neurons in the spiking or silence of a single neuron, and doing this does not require the predictor neuron to generate spikes at anomalously high rates; this result generalizes across many groups of cells (Fig. 5A). We also find that the optimal rules can be well approximated by the predictor neuron thresholding an instantaneous weighted sum of its inputs—a perceptron (Fig. 5B)—suggesting that such predictor neurons are not only possible in principle, but biologically realizable.

Open in a separate window

Fig. 5.

Predictor neurons. (A) Maximum efficiency $I\left({\sigma }_t^{\text{out}};W_{t+\mathrm{\Delta }t}\right)/I\left(W_t;W_{t+\mathrm{\Delta }t}\right)$ as a function of the output firing rate, for 150 four-cell groups. Average over all groups is indicated by the dashed line; solid black line indicates perfect capture of all of the predictive information. (B) Efficiency of a perceptron rule relative to the best possible rule, for the same groups as in A. (C) The information that ${\sigma }^{\text{out}}$ provides about the visual stimulus grows with the predictive information that it captures. Results shown are the means over all possible output rules, for 150 four-cell input groups; error bars indicate SDs across the groups. (D) Average velocity triggered on a spike of the predictor neuron for one four-cell group; light gray lines show the triggered averages for the input spikes; the predictor neuron selects for a long epoch of constant velocity. In A–C, Δt = 1/60 s; in D, Δt = 1/30 s.

Predictor neurons are constructed without reference to the stimulus—just as the brain would have to do—but by repeating the same naturalistic movie many times, we can measure the information that the spiking of a predictor neuron carries about the visual input, using standard methods (15, 23). As we see in Fig. 5C, model neurons that extract more predictive information also provide more information about the visual inputs. There is some saturation in this relationship, perhaps because the most effective predictor neurons are more efficient in selecting the relevant bits of the past. Nonetheless, it is clear that, by solving the prediction problem, the brain can “calibrate” the combinations of spiking and silence in the ganglion cell population, grouping them in ways that capture more information about the visual stimulus.

If we return to the simple world of a single bar moving on the screen, as above, then we can see that the spikes in predictor neurons are associated with interesting patterns of motion. One example is in Fig. 5D, where we see that a spike corresponds to an exceptionally long period of nearly constant velocity motion, followed by a reversal. Other examples include periods of high speed, independent of direction, or moments where the bar is located at a particular position with very high precision (see Feature Selectivity in Predictor Neurons and Fig. S3 for details). These results, which need to be explored more fully, support the intuition that the visual system computes motion not for its own sake, but because, in a world with inertia, motion estimation provides an efficient way of representing the future state of the world.

Information theory defines the capacity of a signal to carry information (the entropy), but information itself is always information about something; successful applications of information theoretic ideas to biological systems are cases where it is clear which information is relevant. However, how can we use information theory to think about neural coding and computation more generally? It is difficult to guess how organisms will value information about particular features of the world, but value can be attached only to bits that have the power to predict the organism’s future experience. Estimating how much information neural responses carry about the future of sensory stimuli, even in a simple world, we have found evidence that the retina provides an efficient, and perhaps nearly optimal, representation of predictive information (Fig. 3).

Efficient representation of predictive information is a principle that can be applied at every layer of neural processing. As an illustration, we consider the problem of a single neuron that tries to predict the future of its inputs from other neurons, and encodes its prediction in a single output bit—spiking or silence. This provides a way of analyzing the responses from a population of neurons that makes no reference to anything but the responses themselves, and in this sense provides a model for the kinds of computations that the brain can do. Predictive information in the patterns of activity is coded synergistically (Fig. 4), maximally efficient representations of this information involve spiking at reasonable rates, without any further constraints, and the optimal predictor neurons are efficient transmitters of information about the sensory input, even though the rules for optimal prediction are found without looking at the stimulus (Fig. 5). Thus, solving the prediction problem would allow the brain to identify features of the retina’s combinatorial code that are especially informative about the visual world, without any external calibration.

The idea that neural coding of sensory information might be efficient, or even optimal, in some information theoretic sense, is not new. Individual neurons have a capacity to convey information that depends on the time resolution with which spikes are observed, and one idea is that this capacity should be used efficiently (24, 25), in part by adapting coding strategies to the distribution of the inputs (26–28). Another idea is that the neighboring cells in the retina should not waste their capacity by transmitting redundant signals, and minimizing this redundancy may drive the emergence of spatially differentiating receptive fields (29, 30). Similarly, temporal filtering may serve to minimize redundancy in time (31), and this is sometimes called “predictive coding” (32). Reducing redundancy requires removing any predictable components of the input, keeping only the deviations from expectation. In contrast, immediate access to predictive information requires an encoding of those features of the past that provide the basis for optimal prediction. The retina actively responds to predictable features of the visual stimulus (33) and, in the case of smooth motion, can anticipate an object’s location in a manner that corrects for its own processing delay (34, 35). Our current results suggest that, even for irregular motion, the retina can efficiently extract the features of the stimulus that allow it to encode all available predictive information. Efficient coding of predictive information is therefore a very different principle from most of those articulated previously, and one that illustrates the surprising computational powers of local neural circuits, like the retina.

Although there has been much interest in the brain’s ability to predict particular things, our approach emphasizes that prediction is a general problem, which can be stated in a unified mathematical structure across many contexts, from the extrapolation of trajectories to the learning of rules (20). Our results on the efficient representation of predictive information in the retina thus may hint at a much more general principle.