Subjects. Fifteen healthy right-handed participants (21-28 years, seven female and eight male) took part in the experiment after providing a written statement of informed consent.

Stimuli. The stimuli used in the pattern learning paradigm consisted of 50-ms chords presented continuously during the sparse sampling fMRI acquisition in an eyes-closed condition via headphones (≈80 dB) with an onset asynchrony of 2 s. Infrequent “targets” (500 Hz, 25% probability) were interspersed with frequent “standards” (250 Hz, 75% probability). For a total of 216 targets, alternating sequences of six consecutive targets were presented either with a random target-to-target interval (TTI) ranging from 4 to 22 s or with a regular 8-s TTI (Fig. 5). Each of these 12-target cycles lasted on the average 96 s. When detecting a target, participants were instructed to press a response button in the middle of the interval between the target and the next standard stimulus. The delayed-response mode was chosen to focus on stimulus-related perceptual and cognitive effects associated with predictability. The instruction hampers the expected behavioral effect, i.e., response-time speeding, and thus minimizes the confounding effect of motor-related potentials on the auditory evoked potential. Participants received a training session with random targets and were not informed about the presence of regularity beforehand.

fMRI Data Acquisition and Preprocessing. Imaging was performed on a 1.5-T Siemens (Erlangen, Germany) scanner. Scanning of anatomy was done with a T1-weighted MPRAGE sequence. Thereafter, 300 BOLD-sensitive echo planar images (EPI) were collected in two sessions, with a 10- to 15-min break in between. EPI volumes were anterior-posterior comissure line aligned and consisted of 18 axial slices with 5.5-mm thickness including a 0.5-mm interslice gap [flip angle 90°; echo time 60 ms; field of view 220 × 220 mm; matrix 64 × 64 voxel). We used a sparse-sampling acquisition design (30) with 8 s repetition time (TR) and 2 s acquisition time, leaving a 6 s silent gap. This allowed EEG recording without scanner noise and gradient artifacts. Baseline data were collected at the beginning and end of each session, nullevents were defined as EPI volumes with only standard stimuli during the TR. Preprocessing and statistical analyses were carried out by using spm2 (Wellcome Department of Imaging Neuroscience, University College London, London) running in matlab (Mathworks, Natick, MA). All images were realigned to the first image in the time series to correct for head movement and normalized to the Montreal Neurological Institute reference space. Normalized data were resliced to a voxel size of 3 mm³, smoothed with an 8-mm full-width half-maximum Gaussian kernel, and high-pass-filtered (256 s).

EEG Data Acquisition and Preprocessing. EEGs were recorded at 5-kHz sampling frequency with a MR-compatible amplifier (Brain Products, Munich) placed inside the MR scanner. Subjects were fitted with an elastic cap (BraincapMR, FMS, Falk Minow Services, Herrsching, Germany) containing 28 Ag/AgCl electrodes (FP1, FP2, F7, F3, Fz, F4, F8, T7, C3, Cz, C4, T8, P7, P3, Pz, P4, P8, O1, OZ, O2, FC5, FC1, FC2, FC6, CP5, CP1, CP2, and CP6). Vertical eye movement was acquired from below the right eye; the electrocardiogram was recorded from the subject's back. Channels were referenced to FCz, with a forehead ground and impedances kept below 5 kΩ. EEGs were downsampled offline to 500 Hz and filtered from 1 to 45 Hz. Target epochs from -312 to 712 ms around stimulus onsets were subjected to independent components analysis (ICA), implemented in eeglab (31) (Institute for Neural Computation, University of California, San Diego) running in matlab. Components related to pulse and eye-movement artifacts were removed from the data. After recalculation to average reference, single trials were wavelet-denoised (26). Coefficients were selected on the basis of ICA-corrected ERPs and were the same for all participants and electrodes. For targets (9%) that were presented within the echo planar image volume acquisition, the ERP was estimated as the mean of two surrounding targets. The data were then downsampled to 125 Hz, smoothed to account for intra- and intersubject latency variability, and high-pass-filtered across trials (216 s). For all these 8-ms frames from -100 to 600 ms (n = 88) around stimulus onset, separate single-trial amplitude vectors were extracted and entered into the joint ERP-fMRI analysis.

Joint ERP-fMRI Analysis. The fMRI time series were modeled with a design that was deployed sequentially for all frames of the ERP time series and replicated for four frontal-central electrodes (Fz, FC1, FC2, and Cz), i.e., those electrode sites where paradigm-induced amplitude modulations were maximally expressed. For each of these designs, two regressors were formed by convolving stimulus functions with a canonical hemodynamic response function. The first stimulus function encoded a generic obligatory response to target stimuli of constant amplitude, applicable to regional fMRI responses associated with “exogenous” features of the auditory evoked response and the motor task. The second stimulus function encoded the amplitude of the single-trial ERPs measured at each frame to find brain regions whose responses could be predicted by paradigm-induced amplitude modulations at that time frame/electrode, thus sensitive to predictability/pattern learning. This function was decorrelated (Schmidt-Gram orthogonalization) from the first, ensuring that activation related to the second function was specific to the electrophysiological measure and not to some general feature in the evoked response to targets. The regressors were entered into single-subject fixed-effects regression analyses; on the group level, random effects analyses were performed by entering the contrast images of each subject into one-sample t tests. fMRI activation to targets (first regressor) is significant at P < 0.05, family-wise error corrected, extent 10 voxel. AM-related activations (second regressor) are significant at P < 0.001 on the voxel level, cluster extent threshold P < 0.01, unless otherwise stated uncorrected for multiple comparisons. This threshold appears adequate in this experiment, because we were interested in the profile of responses and their colocalization with the auditory evoked responses per se, with maximal sensitivity. To minimize the risk of reporting Type I false-positive activation, we applied a descriptive criterion: results were considered reliable and reported only when same/similar activation patterns are replicated in adjacent time points and in two or more of the electrodes. A schematic of the analysis procedure is given in Fig. 2.

Open in a separate window

Fig. 2.

Flowchart of the single-trial ERP and fMRI analysis. Data are decomposed with independent components analysis (a), and artifact topographies (cardioballistic, eye movement) are removed. Effects of component removal on the ERPs are shown in a representative subject (Upper Center). Subsequently, wavelet denoising (b) is applied to the single trials. AM vectors are derived separately for each time point and electrode. To ensure specificity, shared variance between target presentation and AM is removed by orthogonalization. The regressors are convolved with canonical hemodynamic response functions (HRF) to account for the neurovascular coupling before voxelwise correlations with the fMRI signal (c).

Curve Fitting. To illustrate the principal ERP amplitude modulations, hypotheses were tested by nonlinear regression analysis of N1, P2, N2, and P3 amplitudes in the frontocentral region of interest (i.e., the average of Fz, FC1, FC2, and Cz) and the response times. H0 assumed that the measure is insensitive to patterning, represented by a straight horizontal line; H1 assumed that the measure is sensitive to patterning and decreasing or increasing its amplitude, best described as a sigmoid function.

Upon debriefing, all participants noted that targets had occasionally appeared rhythmically, indicating that they explicitly apprehended regular target sequences in the experiment. However, none of them was able to recollect whether these regular patterns were of constant length, or whether regular patterns alternated with random target sequences in succession, suggesting that the overall order of the experiment remained either unrecognized or was implicitly acquired.

Average ERPs. The sequence of cerebral processes leading to discrimination of a target stimulus in an active oddball condition may be indexed by a number of generic ERP components: N1, P2, mismatch negativity (MMN), N2b, P3a, and P3b (32). The extent to which components are detectable in the waveforms depends on experimental parameters. N1 and P2 typically tend to be enhanced under “attend” compared with “ignore” conditions (33). MMN, being an automatic response to changes in auditory stimulation, may be difficult to estimate because of overlapping components like the N2b, which is elicited by infrequent events in attended input, or when the difference between standard and deviant stimuli is relatively large, as in a standard oddball paradigm (32). N2b is usually followed by P3a, indicating a passive shift of attention, and the P3b (also labeled P3 or P300), which is particularly sensitive to task relevance, target probability, sequence, and TTI (23, 28, 29, 34-36). Fig. 3 displays the grand-average ERPs to standards, regular and random TTI target categories, along with results from a pointwise t statistic. After a sequence of midlatency responses and P1 (70-80 ms), a broad centrally distributed N1 (100-120 ms) emerges, followed by a more central-parietal P2 (160-180 ms). The dominant feature in the ERPs to both target categories in comparison with the standard is a frontal-central N2 (200-220 ms), followed by a double-peaked P3 (270-360 ms). The earlier peak at 270 ms is more prominent frontocentrally, the later peak is prominent at parietal sites. TTI regularity in the averaged waveforms most strongly affects N2 and P3 amplitudes but is also seen as a P2 decrement and reduced N1 enhancement.

Open in a separate window

Fig. 3.

Grand-average ERP at frontocentral sites. Waveforms are shown from -100 to 600 ms around stimulus onset for all targets at the third to sixth position of all random TTI cycles (blue), all targets at the third to sixth position of all regular TTI cycles (orange), and all standards not immediately before or after a target (gray dotted). Effects of target predictability appear most prominently as amplitude reductions of N2 and P3 and, to a lesser degree, P1, N1, and P2 are also affected. Above the waveform, significant differences (P < 0.05) from a pointwise t statistic are plotted as blue rectangles for random target vs. standard comparison and in orange for the regular target vs. standard comparison. Black rectangles below the waveform indicate significant differences between the random and regular target categories.

Curve Fitting. Response times were on average 905 ms (SD 200) and remained unaffected by the presence of patterns (F = 0.22, not significant), indicating that participants followed the delayed-response instruction.

N1 amplitudes were also insensitive to regularity (F = 0.31, not significant). All three subsequent components were found to be sensitive to patterns, showing amplitude effects that were best fitted with sigmoid curves: P2 (F = 5.60, P < 0.005), N2 (F = 15.64, P < 0.0001), and P3 (F = 29.89, P < 0.0001). The estimated turning points of these functions were all between the second and third target presentation in regular sequences. The effect strength gradually increased across components, indicative of either higher intra- and intersubject consistency at later timepoints or more stable single-trial estimates. Although the global effect could be well approximated with a sigmoid learning curve, the raw data expressed unique shape variations (Fig. 4 Left).

Open in a separate window

Fig. 4.

AM-correlated fMRI results. Render views and maximum-intensity projections of the general target related activation and positive (red) and negative (blue) correlations with the respective AM. Each correlation map shows for each voxel the maximum t value from the four electrodes (FZ, FC1, FC2, and Cz). To the left of each rendering of the AM-correlated fMRI, the average AM (empty circles ± SEM) and the fitted sigmoid curves are shown. Top row, target-related activation, P < 0.05 (FWE), cluster size ≥10; second row, P2 (170 ms); third row, N2 (200 ms); and fourth row, P3 (320 ms). All AM-related activations were thresholded at P < 0.001 (uncorrected), cluster extent threshold P < 0.01.

fMRI-Target Processing. Areas constantly contributing to target processing in a uniform fashion were found in the superior temporal gyri of both hemispheres extending into the insula and hippocampal formation, the inferior parietal lobe, anterior cingulate gyrus, supplementary motor area, pre- and postcentral gyri (left>right), cuneus, and the middle and superior frontal gyri (right>left), (Fig. 4 and Table 1, which is published as supporting information on the PNAS web site).

AM-Correlated fMRI. The entire spatiotemporal activation results for all four electrodes and timepoints, along with plots of the average scalp topography, waveform, and AM, were compiled into Movies 1 and 2, which are published as supporting information on the PNAS web site.

Here, we focus on the maxima of the three most consistent AM-correlated fMRI activation patterns: P2 (≈170 ms), N2 (≈200 ms), and P3 (≈320 ms).

Inverse relations between BOLD and AM on P2 were seen in posterior cingulate, precuneus, supramarginal gyri, left parietal, and frontal areas (Fig. 4; see also Table 2, which is published as supporting information on the PNAS web site). Inverse relations were also seen for N2, with the most consistent region across electrodes being located in the right medial frontal gyrus. Additional clusters were in the right and left superior frontal gyri, left subcallosal gyrus, left hippocampus, and right amygdala (Fig. 4; see also Table 3, which is published as supporting information on the PNAS web site). Note that these latter results stem from the Cz electrode, where the clusters pass FDR correction, but are also seen at FC1 and FC2 at a lower cluster extent threshold. We observed positive linear relations between BOLD responses and the P3 AM (≈320 ms), mainly in the middle and inferior frontal gyri, inferior parietal lobule, and middle temporal gyri in the right hemisphere. Smaller additional activations were also observed in the right insula, right postcentral gyrus, left supramarginal, and middle frontal gyri (Fig. 4; see also Table 4, which is published as supporting information on the PNAS web site). There was no consistent amplitude modulation of N1 (100 ms), such that it did not differ from the stimulus function and thus did not support a significant regression.

Except for a close spatial relationship between P2- and P3-related regions in the left supramarginal gyrus (mm distance X 0-3, y, 9; and z, 8), there was no considerable overlap among the AM activations. There was also no direct match between any of the AM- and target/response-related local maxima. Note that the delayed-response instruction used in this experiment effectively pruned the salient speeding of response times induced by target predictability. Consequently, we did not observe ERP-related fMRI activation in areas in the motor system(s) that were found sensitive to sequence learning elsewhere (37, 38).

When studying the neuronal substrates of cognitive processes, the researcher typically considers both their spatial and temporal properties. There is, however, a disparity between the major methods in human cognitive neuroimaging, focusing either on the “where” (e.g., fMRI) or “when” (e.g., ERPs), thus providing only a limited window into the neuronal correlates. We propose here that a key to merging both methods is to exploit the functional resolution, that is, how signatures of an experimental manipulation are correlated. The crucial aspect of this approach for spatiotemporal integration is to make effective use of singletrial variability in the entire ERP time series to predict regional fMRI activations, i.e., using time-variant effects induced by a manipulation as a vehicle to achieve a temporal expansion of the fMRI. The prospect of this conjunction is that it allows application of an electrophysiologically derived temporal order to fMRI activation that aids in determining the hierarchy and “serial” functional connectivity of brain regions associated with a process, in this case recognition of temporal patterns in the auditory environment.

Later components in the ERPs are often attributed to “endogenous” or “top-down” processing (32). Recent models of brain function in the context of perceptual inference and learning focus on the hierarchical nature of cortical systems and suggest that these components derive from high levels of processing (ref. 39; for an overview, see ref. 40). We therefore expected that the regionally specific correlates of target predictability would most likely be located outside the sensory regions, in multimodal higher-order cortical areas. Conversely, the fMRI correlates of earlier exogenous components (32), insensitive to the manipulation, would be localized in the vicinity of sensory regions. For this reason, we did not constrain our search for AM-related effects to the main effects of auditory stimulation but used a whole-brain search for the latency-specific correlates. The regional deployment of our activations conformed roughly to our general prediction that later components were coherent with metabolic or synaptic activity in higher cortical areas.

Activation associated with auditory target processing, insensitive to predictability, was seen maximally expressed in the superior temporal gyri and in further areas associated with auditory and visual target detection (12, 41, 42).

Three independent stages separated from peak to peak by 30 ms (P2-N2), 120 ms (N2-P3), and 150 ms (P2-P3) were additionally identified, where the amplitude modulations of single-trial ERP sequences selectively predicted fMRI activation.

The first stage reached maximum intensity during P2 (≈170 ms) after target onset. It is worth noting that the regions mediating the P2 effect overlap with those being associated with “default mode” brain activation (18, 43). P2 hosts processing negativities that indicate matching processes between the sensory input and a neuronal representation of stimuli selected for further processing and as such are markers of sensory memory and selective attention (32). The main sources of these components reside bilaterally in the temporal and frontal lobes (32). It is, however, conceivable that activated brain regions have a modulating effect on these components, allowing for optimization of resource allocation when target occurrence is predictable. This interpretation would also be consistent with the role appointed to the “default mode” (18, 43). In addition, fMRI/positron-emission tomography results of spatial and temporal attention, and sequencing are overlapping with the sites seen here (37, 38). The fMRI activation in the supramarginal and posterior cingulate gyri ≈170 ms matches with the onset latency of a widespread waveform that has been reported from intracranial recordings (5).

The second spatiotemporal stage during the N2 (≈200 ms) was located in the anterior frontomedian cortex and parahippocampal regions. Portions of the N2 reflect the attentive detection of a mismatch between stimulus features and an actively generated memory template. fMRI correlates of this memory process are observed in the same brain regions as activated in the present study (38, 42). Moreover, intracranial recordings in the vicinity of these regions have documented depth N2s in the same peak latency range (6, 7). Further, the scalp N2 to auditory targets is strongly reduced in patients with bilateral hippocampal damage (44), and the sigmoid AM is consistent with recent fMRI findings showing rapid prefrontal and hippocampal habituation to novel events (45).

The overall strongest and most extensive spatiotemporal stage was related to the P3 (≈320 ms) and yielded activations in frontal, temporal, and parietal regions most prominent in the right hemisphere. For all these regions, intracranial recordings have evidenced depth P3s with about the same peak latency (5-7). P3 has been suggested to index a mechanism that is elicited when a memory representation of the recent stimulus context is updated upon detection of deviance from it (36, 46, 47). The effects of a variety of manipulations (e.g., task relevance, information content, probability, and sequence) have been delineated in support of this view (36, 46, 47). fMRI activation in the P3-related regions is seen in a variety of related cognitive operations, including target processing (12, 23, 41, 42, 48), attention, working memory (38, 49) and sequencing (37). Although the rightward lateralization is not strictly predicted from scalp (46, 47) and intracranial measurements (4-7), hemodynamic activity to auditory target/novel stimuli has been shown to be greater in frontal, temporal, and parietal regions of the right hemisphere (42, 50). Also, our data co-localize with fMRI studies reporting right-lateralized attentional mechanisms that would host much of the functionality that is probed by target detection in general (49, 51-53) and, specifically, by manipulating target predictability (54, 55). One should note, however, that a portion of the lateralization might also be attributable to the left-lateralized activation around the central sulcus induced by the motor task, which could have minimized the relative contribution of the predictability effect in adjacent areas to the total variance of the fMRI signal.

The common feature in all three sequential spatiotemporal stages was the sigmoid-shaped response amplitude modulation coherently expressed in the ERP and fMRI, because the target-to-target interval was repeated and became predictable. One classic psychophysiological example for such behavior is the orienting response/reflex, which displays rapid habituation to regularly presented stimuli and dishabituation to deviants from a pattern of preceding stimuli (56, 57). Similarly, this principal mode of responding is overlapping with that of the mismatch negativity (32) and P3 components (36, 42, 46, 58). At the core, all these neuronal processes encompass detection of a salient change in the environment, comparison against a stored representation, and the elicitation of an adequate response. Models accounting for these effects, however, to an extent are conceptually incomplete in the sense that they focus more on why and how the brain responds to unexpected events than on how the representation, i.e., a prediction, is established in the first place. This aspect, however, can be accounted for by linking these orienting response/reflex-type responses with a Bayesian scheme that defines neuronal systems as reciprocally connected hierarchical generative models that construct context-dependent expectancies (39). The amplitude behavior of ERP components (and, correspondingly, the fMRI signal) would here represent the state of prediction error in the model, indicating to which degree “surprise” about the sensory input is suppressed (39): to detect the presence (or absence) of patterns in the environment means to extract contingency rules with highly salient predictive value for the anticipation of future events (39, 55, 59). In this context, our data capture the spatiotemporal dynamics associated with such perceptual inference and learning.

Author contributions: T.E., K.S., M.M., M.L.A.J., R.Q.Q., H.N., and K.H. designed research; T.E., K.S., and M.M. performed research; T.E., K.S., M.M., and M.L.A.J. analyzed data; R.Q.Q. contributed new reagents/analytic tools; and T.E., H.N., and K.H. wrote the paper.

This work was presented in part in poster form at the Helsinki School in Cognitive Neuroscience, March 2-11, 2005, Lammi, Finland, and at the Annual Meeting of the Organization for Human Brain Mapping, June 12-16, 2005, Toronto, ON, Canada.

Conflict of interest statement: No conflicts declared.

This paper was submitted directly (Track II) to the PNAS office.

Data deposition: The neuroimaging data have been deposited with the fMRI Data Center, www.fmridc.org (accession no. 2-2005-120AE).

Abbreviations: ERP, event-related potential; fMRI, functional MRI; BOLD, blood oxygenation level-dependent; EEG, electroencephalogram; AM, amplitude modulation; TTI, target-to-target interval.