Harry Potter meets Markov: Detecting event boundaries in narrative reading by a data-driven model
Poster No:
1037
Submission Type:
Abstract Submission
Authors:
Huidong Xue1, Filipp Dokienko2, Giovanni Menon1, Francesco Gentile1, Bernadette Jansma1
Institutions:
1Maastricht University, Maastricht, Limburg, 2Imperial College London, London, Greater London
First Author:
Co-Author(s):
Introduction:
In narrative reading, we integrate a series of words into meaningful mental representations, so-called events. Recent neuroimaging studies applying data-driven analysis have shown a temporal cortical hierarchy of event structure during movie watching and story listening [1, 2]. How event segmentation takes place within the neural reading network remains elusive. A Hidden Markov model-based event segmentation technique allows inferring unobserved temporal event structure from neuroimaging data without annotations[1]. We applied this data-driven model to fMRI data acquired during reading of a Harry Potter story to detect the event boundaries within the neural reading network, and compared the obtained event structure with the human annotation.
Methods:
We used public-release fMRI data of eight volunteers reading chapter nine of the book "Harry Potter and the Sorcerer's Stone"[3] (Figure 1). Around 5200 words were presented in four runs by rapid serial visual presentation with 0.5s presentation duration per word. Imaging data was acquired using a T2*-weighted echo planar imaging pulse sequence with repetition time (TR) of 2 seconds. Wehbe et al. preprocessed the data in SPM8 with slice-time and motion correction, 3 × 3 × 3mm isotropic spherical Gaussian kernel smoothing, and detrending[3].
We extracted fMRI data from six ROIs, namely the Angular Gyrus (AG), Posterior temporal lobe (PTL), Anterior temporal lobe (ATL), Middle frontal gyrus (MFG), Inferior frontal gyrus (IFG), Inferior frontal gyrus orbital (IFGorb) according to the reading network mask[4] in subject space. To reduce data dimensions and align the fMRI data across subjects, we used the shared response model to project all subject's space data into shared 90-dimensional space for each ROI[5].
We obtained the human annotations by asking five independent volunteers to read the chapter and to mark boundaries of scenes with shifts in topic, location, time, or other crucial elements.
In order to detect the event boundaries for each ROI, we applied a HMM-based event segmentation model in BrainIAK software[6]. We evaluated the model using the leave-one-subject-out method. The optimal number of events for each region was determined by computing the maximum t-distance (t-value), defined as difference in distributions between within-event and across-event correlations[7]. We investigated the temporal alignment of the event boundaries across regions using optimal number of events. The alignment of HMM boundaries and human annotations were tested using the human annotations as upper limit in the model. Significance of alignment was computed by means of permutation testing.
We extracted fMRI data from six ROIs, namely the Angular Gyrus (AG), Posterior temporal lobe (PTL), Anterior temporal lobe (ATL), Middle frontal gyrus (MFG), Inferior frontal gyrus (IFG), Inferior frontal gyrus orbital (IFGorb) according to the reading network mask[4] in subject space. To reduce data dimensions and align the fMRI data across subjects, we used the shared response model to project all subject's space data into shared 90-dimensional space for each ROI[5].
We obtained the human annotations by asking five independent volunteers to read the chapter and to mark boundaries of scenes with shifts in topic, location, time, or other crucial elements.
In order to detect the event boundaries for each ROI, we applied a HMM-based event segmentation model in BrainIAK software[6]. We evaluated the model using the leave-one-subject-out method. The optimal number of events for each region was determined by computing the maximum t-distance (t-value), defined as difference in distributions between within-event and across-event correlations[7]. We investigated the temporal alignment of the event boundaries across regions using optimal number of events. The alignment of HMM boundaries and human annotations were tested using the human annotations as upper limit in the model. Significance of alignment was computed by means of permutation testing.
Results:
Human annotation unveiled 58 events on average, with 31 events being common across individuals. The HMM detected 37 (MFG) to 67 (AG) events across ROIs (Figure 2A). Their boundaries were significantly aligned with eachother (Figure 2B).
ATL (p=0.013) and AG (p=0.042) boundaries were aligned with human annotation (Figure 2C&D). Interestingly, in the first run, all regions showed a higher alignment with human annotation (p value for each: AG=0.009, PTL=0.087, ATL=0.016, MFG=0.022, IFG=0.015, IFGorb=0.044), and only the AG (p=0.015) showed significantly aligned in the fourth run (Figure 2D).
ATL (p=0.013) and AG (p=0.042) boundaries were aligned with human annotation (Figure 2C&D). Interestingly, in the first run, all regions showed a higher alignment with human annotation (p value for each: AG=0.009, PTL=0.087, ATL=0.016, MFG=0.022, IFG=0.015, IFGorb=0.044), and only the AG (p=0.015) showed significantly aligned in the fourth run (Figure 2D).
Conclusions:
The present results show a nested event structure in the narrative reading network with shorter events in AG and longer events in frontal and temporal areas, and high-frequent boundaries partially nested in low-frequency boundaries. Results of ATL and AG support their role in conceptual representation[8, 9], Their event boundaries were aligned with human annotation, suggesting a link between conceptual relevance and event structure. The AG shows good tracking with human annotation along the entire reading time, suggesting an involvement in context integration over time[10]. These findings provide neuroscientific insight into event segmentation in narrative reading.
Language:
Reading and Writing 1
Modeling and Analysis Methods:
Multivariate Approaches 2
Keywords:
Data analysis
Language
Modeling
MRI
Other - Event segmentation
1|2Indicates the priority used for review
Provide references using author date format
[2] Soares, A. D. (2023). "Top-down attention shifts behavioral and neural event boundaries in narratives with overlapping event scripts." bioRxiv: 2023.08.08.552465.
[3] Wehbe, L. (2014). "Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses." PLoS One 9(11)
[4] Fedorenko, E. (2010). "New method for fMRI investigations of language: defining ROIs functionally in individual subjects." J Neurophysiol 104(2): 1177-1194.
[5] Chen, P. H. (2015). "A Reduced-Dimension fMRI Shared Response Model." Advances in Neural Information Processing Systems 28 (Nips 2015) 28.
[6] Kumar, M. (2021). "BrainIAK: The Brain Imaging Analysis Kit." Apert Neuro 1(4).
[7] Geerligs, L. (2021). "Detecting neural state transitions underlying event segmentation." Neuroimage 236: 118085.
[8] Correia, J. (2014). "Brain-based translation: fMRI decoding of spoken words in bilinguals reveals language-independent semantic representations in anterior temporal lobe." Journal of Neuroscience 34(1): 332-338.
[9] Acunzo, D. J. (2022). "Deep neural networks reveal topic-level representations of sentences in medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus." Neuroimage 251: 119005.
[10] Branzi, F. M. (2021). "The left angular gyrus is causally involved in context-dependent integration and associative encoding during narrative reading." Journal of cognitive neuroscience 33(6): 1082-1095.