Syllables in Sync Form a Link: Neural Phase-Locking Reflects Word Knowledge During Language Learning

Participants

A total of 21 participants (10 female) contributed data to this study. Participants were recruited at Northwestern University and were paid $10/h. They were all fluent English speakers, between 19 to 23 years old (mean = 20.5 y) and had no history of neurological problems. The study was undertaken with the understanding and written consent of each participant.

Data from other tasks completed by this sample of participants have been reported in a previous publication (implicit training group; Batterink, Reber and Paller, 2015). Briefly, the goal of this previous study was to test whether the ability to predict incoming stimuli, a key function of statistical learning, can be enhanced through explicit training. This previous study did not analyze EEG data recorded during the statistical learning exposure period. This previous study also included an additional group of participants assigned to an explicit training condition, whose data are not included here.

Stimuli

Stimuli in the exposure phase were modelled after previous auditory statistical learning studies (e.g., Saffran et al., 1996, 1997). This language consists of 11 syllables combined to create six trisyllabic nonsense words (babupu, bupada, dutaba, patubi, pidabu, tutibu). Some members of the syllable inventory occur in more words than others, which produces varying transitional probabilities between the syllables within the words, as in natural language. Each nonsense word was repeated 300 times in pseudorandom order, with the restriction that the same word never occurred consecutively. Because the speech stream contained no pauses or other acoustic indications of word onsets, the only cues to word boundaries were transitional probabilities, which were higher within words than across word boundaries (cf. Saffran et al., 1996, 1997).

A speech synthesizer (Mac text-to-speech application, female voice “Victoria”) was used to generate a continuous speech stream composed of the six trisyllabic nonsense words. To achieve more natural sounding speech, speech synthesis technology makes use of automated techniques to produce acoustic variations in the speech output. Thus, as in speech produced by a human talker, individual tokens of a given word type in the speech stream were acoustically variable (e.g., each instance of “babupu” was not uttered in an identical manner). Descriptive statistics summarizing token durations for each of the six words are shown in Table 1. As can be seen from the table, there was considerable acoustic variability across the tokens within each word type. Across the synthesized stream, the average syllable to syllable latency was 235 ms (std = 39.8 ms).

The speech stream was edited to include a total of 31 pitch changes. Each pitch change represented either a 20-Hz increase or decrease from the baseline frequency (~160 Hz). Pitch changes occurred randomly, rather than systematically on certain syllables, and thus could not provide a cue for segmentation. Syllables that spanned pitch changes were excluded from EEG analysis. The detection of infrequent pitch changes was used as a cover task during the learning period, in order to ensure adequate attention to the auditory stimuli.

EEG event codes were sent at the onset of each syllable. The timing of syllable onsets in the continuous speech stream was determined by three trained raters using both auditory information and visual inspection of sound spectrographs, with the mean rating used. Any discrepancy >20 ms among one or more raters was resolved by a fourth independent rater. The first 30 syllables of the stream were not coded and thus not included in the analysis to avoid auditory onset effects.

For the recognition test phase, six nonword foils were created (batabu, bipabu, butipa, dupitu, pubada, tubuda). The nonwords consisted of syllables from the language’s syllable inventory that never directly followed each other in the speech stream, even across word boundaries. The frequency of individual syllables across words and nonword foils was matched.

Procedure

At the beginning of the experiment, participants were fitted with an elastic EEG cap embedded with electrodes. After EEG setup, participants were informed that they would listen to a speech stream of nonsense words. They were instructed that the speech stream contained occasional pitch changes and that they should detect these pitch changes using the keypad, with one button indicating a low pitch change and another indicating a high pitch change. To increase interest in the task, participants earned a small amount of additional money (12 cents) for each successfully detected pitch change. Due to technical issues, behavioural data for the pitch- detection task from two participants could not be analyzed. Overall, the remaining participants performed well on the pitch-detection task, detecting 94.6% (SD = 4.2%) of the 31 pitch changes. The total exposure stream of approximately 21 min was divided into three equal blocks and participants were given a brief break between each block.

After finishing the listening phase of the experiment, participants were informed that the nonsense language that they had just listened to was composed of individual words. They were then given a free recall task in which they were asked to recall the six words by writing them down on a piece of paper. Overall performance was very poor on this task (mode of 0 words correct across participants) and was not analyzed further.

Participants then completed a forced-choice recognition judgment task. Each trial included a word and a nonword foil. Participants gave two responses for each trial, (1) indicating which of the two sound strings sounded more like a word from the language, and (2) reporting on their awareness of memory retrieval, with “remember” indicating confidence based on retrieving specific information from the learning episode, “familiar” indicating a vague feeling of familiarity with no specific retrieval, and “guess” indicating no confidence in the selection. Each of the six words and six nonword foils were paired exhaustively for a total of 36 trials. In half of the trials the word was presented first, while in the other half the nonword foil was played first; presentation order for each individual trial (whether presented first/second) was counterbalanced across participants. Participants then completed a final speeded target detection task, designed to assess implicit memory of the syllable patterns. Behavioural and EEG data for the remember/familiar/guess component of the recognition test and for the target detection task have been analysed previously (Batterink et al., 2015) and are not included in the current paper.

Behavioural Data Analysis (2AFC Recognition Task).

For each participant (1–21) and word type (1–6), I computed a “recognition score,” which represents the total number of correct trials out of six for each word. These 126 values were then analyzed as the dependent variable in the main linear mixed effects model (described below), in order to examine the relationship between recognition and neural phase-locking at the item level.

In addition, a one-sample t-test was used to test whether recognition performance was above chance, with 50% correct representing chance-level performance. Finally, a repeated-measures ANOVA with word type (1–6) as a within-subjects factor was used to test whether recognition performance differed across the different component words of the language.

EEG Recording and Analysis

Behavioural Results (2AFC Recognition Test)

Recognition performance was significantly above chance across participants (mean = 59.3%, 11.7%, t(20) = 3.62, p = 0.002), providing evidence of successful word segmentation due to statistical learning. Recognition performance varied significantly across the six component words of the language words (word type: F(5,100) = 2.76, p = 0.049; Figure 2B), indicating that some words were learned better and recognized at higher levels than others.

EEG Results (Exposure Period)

Neural Entrainment and Word Knowledge May Interact Bidirectionally

The current results are correlational in nature and cannot directly disentangle the causality between neural phase-locking and word learning. However, based on previous findings, I propose that there may be bidirectional interactions between phase-locking and linguistic knowledge: (1) neural phase-locking to underlying patterns may influence the formation of high-level linguistic representations; and (2) word representations may exert a top-down influence on phase-locking of ongoing oscillations. The first idea—that modulation of neural phase may influence word learning—is supported by several recent transcranial alternating current stimulation studies (Riecke et al., 2018; Wilsch et al., 2018; Zoefel et al., 2018). By directly manipulating neural oscillations, these studies demonstrated that the phase lag between brain and speech rhythms influenced the neural responses to intelligible speech in superior temporal gyrus (Zoefel et al., 2018) as well as speech comprehension (Riecke et al., 2018; Wilsch et al., 2018). These results indicate that the phase alignment between neural oscillations and an ongoing speech signal plays a causal role in high-level speech processing, and by extension could also (in principle) influence speech segmentation and statistical word learning.

A major mechanism underlying neural phase-locking to speech is phase-resetting of low frequency oscillations in the auditory cortex to “acoustic landmarks” in the speech envelope, such as speech onsets or sharp acoustic transients (Doelling et al., 2014; Gross et al., 2013). Certain words may thus be more learnable because they contain acoustic features that elicit stronger or more consistent phase-resetting across word presentations. Consistent with this idea, in the present study I found that some words elicited higher ITPC than others, and that word-level differences in ITPC accounted for word-level differences in recognition (Figure 2B). Further, these word-level ITPC differences emerged very early on and were relatively stable across exposure; words that showed high ITPC values during early learning continued to elicit relatively higher ITPC values throughout the first block of exposure (Figure 4). These findings suggest that “baseline” acoustic features influence phase-locking, and that degree of phase-locking predicts whether a given word is more learnable. Over multiple word exposures, phase-locked oscillations at the word frequency could mediate the binding of syllables into larger temporal chunks, thereby supporting word learning. This idea is consistent with the proposal that neural entrainment functions to align phases of neural excitability to repeated temporal patterns, providing a mechanism for identifying specific patterns in upcoming sensory input (Schroeder and Lakatos, 2009).

A second possibility, which is not mutually exclusive, is that high-level word knowledge has a top-down influence on neural phase-locking. This idea is supported by the present finding that ITPC significantly increased over exposure, reflecting the gradual acquisition of word knowledge that in turn may facilitate predictive processing. This significant increase in phase-locking over time cannot be accounted for by bottom-up factors alone, given that the stimulus stream did not differ systematically over exposure, and replicates previous findings showing that neural phase-locking to words (or phrases) in an artificial language increases gradually over the course of exposure (Getz et al., 2018; Batterink & Paller, 2017, 2019; Choi, Batterink et al., 2020). Together, these results converge with mounting evidence that neural phase-locking is critically modulated by top-down processes such as selective attention and expectations (e.g., Ding and Simon, 2012; Horton et al., 2013; Lakatos et al., 2008; Rimmele et al., 2015; Zion Golumbic et al., 2013). Mechanistically, a recent MEG study demonstrated that top-down signals from frontal brain areas causally influences the phase of speech-coupled oscillations in auditory cortex, enhancing speech-brain coupling (Park et al., 2015). The idea that neural phase-locking is influenced by top-down processing is also compatible with theoretical proposals that neural phase-resetting provides an instrument for sensory selection by enabling phases of higher neural excitability to align with important stimulus events (Schroeder and Lakatos, 2009; Thut et al., 2012). In the context of speech segmentation, high-level word knowledge enables predictions to be made about upcoming syllables. In turn, these top-down predictions may function to optimally align ongoing neural oscillations with the most important or informative moments of the speech signal, acting to increase sensitivity to relevant acoustic cues and thereby facilitating speech processing (Peelle and Davis, 2012).

The current results also suggest that bottom-up acoustic factors may interact with statistical learning and top-down knowledge, with words that are the most initially “trackable” also benefitting the most from exposure and showing continual gains in learning (Figure 5). The time course analysis demonstrated that words with the highest ITPC estimates at the beginning of the exposure period (i.e., bupada and dutaba) showed significant increases in ITPC over exposure. In contrast, words with low initial-phase locking (i.e., babupu and tutibu) did not show increases in neural phase-locking over this period, suggesting that words that are not initially trackable (as measured by baseline ITPC estimates) do not benefit from exposure. In sum, both bottom-up and top-down factors appear to contribute to the observed relationship between item-level ITPC and subsequent word learning, and further, these different mechanisms are likely to interact with one another.

Word Variability Across Utterances Influences Both ITPC and Word Learning

At the behavioural level, I found a significant impact of word type on recognition performance, indicating that some words were more easily learned than others. This finding aligns well with previous findings that language-specific knowledge influences linguistic statistical learning, with words that more closely follow the phonotactic regularities of a participant’s native language being learned better (Finn & Hudson Kam, 2015; Siegelman et al, 2018). A more novel, unexpected finding was that words with more variable durations in the stream were learned better compared to words that had less variability. Some caution is warranted in interpreting this result, given that there were only six word in the language and that a full exploration of acoustic differences between word is beyond the scope of this paper. Nonetheless, this finding is consistent with prior evidence showing that variability facilitates speech learning and generalization to novel instances (e.g., Bradlow & Bent 2008; Clopper & Pisoni, 2004; Singh, 2008; Greenspan, Nusbaum, & Pisoni, 1988). For example, the perception of new sentences produced with synthetic speech improves when participants are exposed to a larger set of training stimuli compared to a restricted set (Greenspan, Nusbaum, & Pisoni, 1988). Within the context of statistical learning, Gomez (2002) demonstrated that infants’ and adults’ learning of nonadjacent dependencies (e.g., pel-X-jic) depends on sufficient variability, occurring only when the middle, nonpredictive element of the dependency (i.e., X) is drawn from a sufficiently large pool. Taken together, these results indicate that exposure to a greater variety of exemplars allows learners to better ignore irrelevant features and identify the most predictable, informative or invariant structures in a stimulus stream. In the current study, words with greater variability across utterances may promote the acquisition of more abstract word representations, as opposed to more specific, stimulus-based, acoustic representations (Vouloumanos et al., 2012). This in turn may facilitate generalization and better recognition performance when the same word presented in a new context (i.e. in isolation during the 2AFC recognition task, rather than embedded in a continuous speech stream as during the exposure phase).

Word variability in duration across utterances also influenced ITPC, with greater word variability predicting higher ITPC values. This finding does not follow from a straightforward bottom-up mechanistic account of neural phase-locking, which would predict that ITPC should be higher to items that have a more stable (i.e. less variable) duration across presentations. Rather, this finding suggests that greater word variability facilitates word learning, which in turn leads to stronger phase-locking to the embedded words, providing additional support for top-down influences on neural phase-locking.

Neural Mechanisms Underlying Statistical Learning

On a more specific note, the current findings also provide new insights into the neural mechanisms that underlie statistical learning in the context of word segmentation, extending previous work in this area. As described in the Introduction, prior studies have shown that neural tracking of repeating nonsense words predicts statistical learning performance on subsequent behavioural tests at the individual level; participants who show stronger neural entrainment responses to the underlying linguistic structures perform better on subsequent learning tests (Batterink and Paller, 2017, 2019; Buiatti et al., 2009). The current results conceptually replicate these results, demonstrating that average ITPC during learning predicts subsequent overall recognition performance across individuals. At the same time, the current findings go beyond a demonstration of interindividual correlations, showing that item-level neural entrainment predicted item-level recognition more strongly than individual-level average entrainment. Further, the effect of individual participant did not significantly account for variability in item-level word recognition when item-level ITPC was accounted for.

Taken together, these results indicate that neural phase-locking in the context of language learning primarily reflects the discovery and perception of individual items in the language inventory, rather than indexing more general interindividual differences that would operate similarly across all items, such as the tendency to “spontaneously synchronize” one’s behaviour to external stimuli (Assaneo et al., 2019). In other words, it appears that the previously documented relationship between neural entrainment and statistical learning performance (Batterink and Paller, 2017, 2019; Buiatti et al., 2009) primarily reflects the specific content of linguistic knowledge, and can be accounted for by better learners’ higher rates of word learning.

ITPC results also hint that learners may have engaged in a suboptimal parsing strategy for words that were not successfully learned (i.e., “lowest recognition items” with a score of 50% accuracy or below on the 2AFC test a recognition score). As shown in Figure 3B, low recognition items show a peak at approximately 2.1 Hz, which corresponds to the average bigram rate in the speech stream. This finding suggests that poorly learned items may be parsed as bigrams on some proportion of trials. For example, for a triplet such as “babupu,” participants may segment the bigram “babu” on some occurrences, “bupu” on other occurrences, and neither possible bigram on still other occurrences. Across all trials, this would produce a weak signature of bigram tracking. Because overall ITPC values corresponding to the bigram presentation rate are similar across highest recognition and lowest recognition items (Figure 3B), it appears that some (relatively weak) degree of erroneous bigram parsing also occurs for better learned words. This finding highlights that ITPC at the word rate specifically is a signature of statistical word learning, and that phase-locking at other low frequencies more generally (< 10 Hz) does not distinguish between better learned and poorly learned items.

Temporal Trajectory of Item-Level Neural Entrainment

The temporal dynamics of ITPC provide additional insights into the neural mechanisms that support statistical learning of novel words. As a given word unfolds, ITPC showed a steep increase beginning immediately after word onset (see Figure 5). The rapid nature of this effect converges with previous demonstrations that word onsets modulate ongoing neural responses very quickly. For example, a recent MEG study modelled neural responses to continuous narrative speech, and found a highly significant effect of word onsets with a peak latency of 103 ms (Brodbeck et al., 2018). This finding was interpreted as evidence that word boundaries are detected essentially as they occur, rather than after incorporating cues occurring subsequent to word onset. Similarly, an ERP study found an early sensory-related N100 effect to onsets of nonsense words in continuous speech (Sanders et al., 2002). This effect was observed only in learners who showed the strongest behavioural evidence of word knowledge, suggesting that high-level linguistic knowledge is a prerequisite for this early response. In the context of neural entrainment frameworks (e.g., Schroeder and Lakatos, 2009), word onsets may represent privileged sensory events, as syllables occurring at the beginning of a word are relatively information-rich and highly predictive of subsequent syllables. Successful word learning may therefore be accompanied by the rapid alignment of neural oscillations to these informative word onsets.

Although showing a rapid increase soon after word onset, the neural entrainment response did not peak until ~420 ms, which coincides to roughly 200 ms after the onset of the second syllable. This peak was followed by a decrease in entrainment, which statistically reached baseline levels by 701 ms, very close to the mean word duration of 704 ms (see Table 1). A similar neural tracking trajectory was reported by Ding and colleagues (2016; see their Figure 4), who found that neural activity reached its peak during the second word of artificial grammar phrases, and then progressively decreased with each additional word in the phrase. Taken together, these findings indicate that ITPC tracks the entire time course of a higher-level unit, rather than being a transient response occurring only at unit boundaries (cf. Ding et al., 2016). It is also interesting to note that the observed peak in ITPC is similar to the typical latency of the N400 effect (Kutas and Federmeier, 2011). This suggests that neural tracking of a given word may decline once the word is processed to the point of recognition.

In contrast to ITPC at the word presentation rate, ITPC at the syllable rate peaked very soon after word onset (~110 ms; Figure 5B). Overall, the trajectory of neural entrainment at the syllable rate resembles a symmetrical, steep parabolic curve centered just after word onset, consistent with a sensory-evoked response that is not strongly modulated by high-level knowledge.

Conclusions

In summary, the main finding of the study is that neural phase-locking accompanies an individual’s subjective perception of an individual word in continuous speech, as acquired in real time during statistical learning. These results indicate that the association between neural phase-locking and statistical learning is not limited to perfectly isochronous syllable sequences (e.g., Batterink and Paller, 2017, 2019; Buiatti et al., 2009; Ding et al., 2016; Getz et al., 2018), but is generalizable to continuous speech containing nonidentical word tokens. The demonstration that neural phase-locking is sensitive to recognition strength of individual words opens up the possibility of tracking the contents of learning in real time. For example, by monitoring the EEG of language learners exposed to a continuous stream of foreign language input, it may be possible to predict which words have been successfully learned and which words require additional training. This neural phase-locking approach may also be applied to investigate other aspects of language that involve the concatenation of smaller linguistic elements into larger units, such as the learning and processing of grammatical rules, as well as perceptual aspects of language acquisition, such as phonetic category learning. Thus, new applications of this approach may significantly advance our understanding of other neural mechanisms underlying language acquisition and processing.