The Representational Dynamics of Speech Prosody: An MEG Study
Poster No:
1038
Submission Type:
Abstract Submission
Authors:
SEUNG-CHEOL BAEK1, Seung-Goo Kim1, Burkhard Maess2, Maren Grigutsch2, Daniela Sammler1
Institutions:
1Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Hesse, 2Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Saxony
First Author:
Co-Author(s):
Maren Grigutsch, Dipl.-Phys.
Max Planck Institute for Human Cognitive and Brain Sciences
Leipzig, Saxony
Max Planck Institute for Human Cognitive and Brain Sciences
Leipzig, Saxony
Introduction:
Speech prosody is crucial for communication as it conveys meaning, such as intentions, beyond the propositions. Prosody is processed in a bilateral frontotemporal network (Belyk & Brown, 2014), and is thought to involve multiple steps along the ventral auditory stream (Schirmer & Kotz, 2006). These steps are likely to include the sensory analysis in bilateral posterior temporal areas, the identification and evaluation of prosodic forms in right anterior temporal and frontal areas. Additionally, right premotor cortex (PMC) as part of the dorsal auditory-motor stream was found to play a causal role in prosody categorization (Sammler et al., 2015). However, it remains to be shown where and when speech prosody is actually represented in its acoustic and/or abstract categorical form, and if prosodic representations are dominant in the right hemisphere.
Methods:
We collected magnetoencephalogram (MEG) data from 29 native Germans while they listened to single words that gradually varied in prosody and word-initial phoneme along orthogonal continua generated by audio morphing (Fig. 1A). Participants categorized these words into either prosody (statement vs. question) or phoneme (/b/ vs. /p/) in alternating blocks (Fig. 1B).
The MEG data were analyzed in source space (eLORETA) by time-resolved representational similarity analysis to explore the dynamics of prosodic representations (Kriegeskorte et al., 2008). We first defined ten regions of interest (ROI) based on a multimodal cortical parcellation (Glasser et al., 2016; Fig. 2A), including the bilateral primary auditory (PAC), anterior and posterior superior temporal (a/pSTC) cortices, PMC and inferior frontal gyri (IFG). Per ROI, time-varying neural dissimilarity patterns (window size/step = 24/4ms, Crossnobis distance) were computed over different morphing levels of prosody and compared with dissimilarity models of their acoustic forms and perceived categories (behavior; Fig. 1C). Then, we estimated the variance of neural dissimilarity patterns explained by a non-negative linear combination of acoustic and categorical models, and the unique variance explained by each model. Fisher-Z-transformed variance explained was inferred against baseline (-0.2-0s) by cluster permutation tests. The same analysis was carried out for phoneme representations.
In case representations were significant in only one of the homologous ROIs, the difference between these ROIs was tested to examine hemispheric lateralization.
The MEG data were analyzed in source space (eLORETA) by time-resolved representational similarity analysis to explore the dynamics of prosodic representations (Kriegeskorte et al., 2008). We first defined ten regions of interest (ROI) based on a multimodal cortical parcellation (Glasser et al., 2016; Fig. 2A), including the bilateral primary auditory (PAC), anterior and posterior superior temporal (a/pSTC) cortices, PMC and inferior frontal gyri (IFG). Per ROI, time-varying neural dissimilarity patterns (window size/step = 24/4ms, Crossnobis distance) were computed over different morphing levels of prosody and compared with dissimilarity models of their acoustic forms and perceived categories (behavior; Fig. 1C). Then, we estimated the variance of neural dissimilarity patterns explained by a non-negative linear combination of acoustic and categorical models, and the unique variance explained by each model. Fisher-Z-transformed variance explained was inferred against baseline (-0.2-0s) by cluster permutation tests. The same analysis was carried out for phoneme representations.
In case representations were significant in only one of the homologous ROIs, the difference between these ROIs was tested to examine hemispheric lateralization.
Results:
Combined acoustic and categorical representations of prosody emerged first in bilateral PAC and a/pSTC around 300ms after word onset, followed by left IFG and right PMC. Phoneme representations emerged earlier (from 70ms) and only in the bilateral temporal areas.
The unique variance analysis showed that acoustic representations of prosody emerged earlier (340ms; Fig. 2B) than categorical representations (480ms) in bilateral PAC and pSTC (Fig. 2C), followed by the right aSTC (420ms) and left IFG (464ms), both significantly lateralized (Fig. 2B). Notably, PMC exhibited additional categorical representations which were right-lateralized during word presentation (384ms) and bilateral after word offset (532ms; Fig. 2C).
These temporal and interhemispheric dynamics substantiate the representational abstraction of prosodic pitch contours along the ventral auditory stream (Schirmer & Kotz, 2006), and a two-staged role of PMC in the perceptual categorization of prosody and motor-based decision-making (Liebenthal & Möttönen, 2018). While early auditory and late decision-making processes were bilateral, intermediate perceptual processes in aSTC and PMC showed right-hemispheric predominance, in line with cue-dependent models of prosody perception (van Lancker & Sidtis, 1992).
The unique variance analysis showed that acoustic representations of prosody emerged earlier (340ms; Fig. 2B) than categorical representations (480ms) in bilateral PAC and pSTC (Fig. 2C), followed by the right aSTC (420ms) and left IFG (464ms), both significantly lateralized (Fig. 2B). Notably, PMC exhibited additional categorical representations which were right-lateralized during word presentation (384ms) and bilateral after word offset (532ms; Fig. 2C).
These temporal and interhemispheric dynamics substantiate the representational abstraction of prosodic pitch contours along the ventral auditory stream (Schirmer & Kotz, 2006), and a two-staged role of PMC in the perceptual categorization of prosody and motor-based decision-making (Liebenthal & Möttönen, 2018). While early auditory and late decision-making processes were bilateral, intermediate perceptual processes in aSTC and PMC showed right-hemispheric predominance, in line with cue-dependent models of prosody perception (van Lancker & Sidtis, 1992).
Conclusions:
This study draws a comprehensive picture of the dynamic frontotemporal representations of speech prosody. At the network level, the representational transfer will be further investigated by a directed information transfer analysis.
Language:
Speech Perception 1
Modeling and Analysis Methods:
EEG/MEG Modeling and Analysis
Multivariate Approaches
Novel Imaging Acquisition Methods:
MEG 2
Perception, Attention and Motor Behavior:
Perception: Auditory/ Vestibular
Keywords:
ADULTS
Cortex
Hearing
Hemispheric Specialization
Language
MEG
Modeling
Multivariate
Perception
Source Localization
1|2Indicates the priority used for review
Provide references using author date format
Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., ... & Van Essen, D. C. (2016). A multi-modal parcellation of human cerebral cortex. Nature, 536(7615), 171-178.
Liebenthal, E., & Möttönen, R. (2018). An interactive model of auditory-motor speech perception. Brain and Language, 187, 33-40.
Kriegeskorte, N., Mur, M., & Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 4.
Schirmer, A., & Kotz, S. A. (2006). Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends in cognitive sciences, 10(1), 24-30.
Sammler, D., Grosbras, M. H., Anwander, A., Bestelmeyer, P. E., & Belin, P. (2015). Dorsal and ventral pathways for prosody. Current Biology, 25(23), 3079-3085.
Van Lancker, D., & Sidtis, J. J. (1992). The identification of affective-prosodic stimuli by left- and right-hemisphere-damaged subjects: All errors are not created equal. Journal of Speech and Hearing Research, 35, 963-970.