Evidence in dual pathway coding of speech with prediction errors

1004 

Abstract Submission 

Baihan Lyu^1,2, Xiuyi Wang², Yi Du^1,2,3

¹Department of Psychology, University of Chinese Academy of Sciences, Beijing, China, ²CAS Key Laboratory of Behavioral Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, China, ³Chinese Institute for Brain Research,, Beijing, China

Baihan Lyu  
Department of Psychology, University of Chinese Academy of Sciences|CAS Key Laboratory of Behavioral Science, Institute of Psychology, Chinese Academy of Sciences
Beijing, China|Beijing, China

Xiuyi Wang  
CAS Key Laboratory of Behavioral Science, Institute of Psychology, Chinese Academy of Sciences
Beijing, China

Yi Du  
Department of Psychology, University of Chinese Academy of Sciences|CAS Key Laboratory of Behavioral Science, Institute of Psychology, Chinese Academy of Sciences|Chinese Institute for Brain Research,
Beijing, China|Beijing, China|Beijing, China

The human brain integrates external information with predictions from long-term memory to facilitate language comprehension. Research has elucidated phonological and semantic representations in natural language processing contexts. However, the manner in which these representations are affected when predictions are violated remains elusive. This study delves into the neural representations of phonological and semantic information in two scenarios: when information is missing but can be predicted based on context and long-term memory, and when incongruent information is presented that contradicts both context and long-term memory. We aim to determine whether the same mechanisms or distinct pathways are involved in representing these divergent forms of information.

In an fMRI experiment, 29 native Chinese speakers (19 females; mean age = 21 years) were presented with Chinese idiomatic phrases, each comprising 4 characters (e.g., "羊入虎口" - "sheep enters tiger's mouth"). We designed 3 conditions by manipulating the final character (target) of each phrase. In the 'Expected' condition (EP), participants heard the correct phrase. In the 'Missing' condition (M), the target was omitted. In the 'Unexpected' condition (UP), an incorrect target was provided (see Fig. 1A for an example). Catch trials were included to maintain participant engagement, where participants indicated whether the previous target matched the current character on the screen (see Fig. 1B). We used a set of 18 phrases, with 3 phrases sharing the same target, resulting in 54 experimental trials and 12 catch trials per run. All items were presented once per run, with a total of 8 runs. We extracted the whole-brain activation beta map for each phrase using a general linear model. We conducted univariate analysis to compare the activation maps across conditions. To reveal regions representing phonological and semantic information in each condition, we conducted the representation similarity analysis (RSA) by fitting linguistic (phonological and semantic) representation distance matrices against the neural RDM through multiple regression. In the UP condition, we built two RDMs: one for the predicted but unpresented correct word, and the other for the perceived error word (see Fig. 1C)

Greater activation was found in inferior frontal gyrus (IFG) pars opercularis and superior temporal gyrus (STG) in M than in EP, while auditory cortex and semantic memory regions (e.g., IFG pars triangularis, angular gyrus, middle temporal gyrus) showed the reverse pattern (Fig. 2A). Left frontal-temporal semantic regions exhibited greater semantic representations in M than in EP; conversely, the opposite pattern was observed in precentral gyrus (Fig.2B). Thus, understanding correct phrases relies on the long-term semantic memory system. However, when information is absent, the ventral temporal semantic pathway becomes crucial for information prediction and recovery. In UP, auditory, attention and general control systems showed greater activation than in EP, but no region showed the reverse pattern (Fig. 2A). The semantic representation of perceived words was observed in ventral pathway areas, including anterior STG and IFG. Conversely, the semantic representation of predicted words was found in precentral, inferior parietal area and posterior cingulate cortex (Fig. 2B). These suggest bottom-up semantic processing for actual words and top-down memory retrieval for predicted words when predictions are violated.

Our preliminary results identified distinct pathways for understanding normal phrases and predicting information from long-term memory to recover missing words. We also revealed separate pathways for representing semantic information related to perceived error words and predicted words. While we mainly observed distinct sensitivity to semantic representations, we will further explore in relation to phonological representations.

Language Comprehension and Semantics ¹

Speech Perception ²

Activation (eg. BOLD task-fMRI)

Multivariate Approaches

Cognition

FUNCTIONAL MRI

Language

Multivariate