Transient cortical activity predicts spontaneous attention shifts
Poster No:
2455
Submission Type:
Abstract Submission
Authors:
Masakazu Inoue1, Yasuo Kabe1, Shuntaro Sasai1
Institutions:
1ARAYA Inc., Chiyoda, Tokyo
First Author:
Co-Author(s):
Introduction:
We encounter a massive influx of environmental inputs in our daily lives, yet our attention is spontaneously allocated to only a subset thereof. With controlled experimental settings, parietal cortex was found to play an important role in attentional shift [1]. However, the neural mechanisms underlying spatial attention remains unclear in natural and voluntary settings. Using fMRI data obtained during movie-watching, we examine whether fMRI signals predict spontaneous changes of fixated locations. We localize brain regions contributing to the prediction and characterize the spatiotemporal neural dynamics underlying the spontaneous attentional allocation in natural settings.
Methods:
dataset
We used a publicly available, StudyForrest dataset [4] consisting of simultaneous fMRI (TR=2) and eye tracking (temporal resolution=1000 Hz) data during movie watching (Forrest Gump, 2h-long). We used data from twelve participants successfully completed their experiments. Twenty two ROIs were extracted according to Glasser et al. [5].
fMRI-based prediction of fixated locations
We newly developed fMRI-based prediction of fixated locations as follows (Fig 1).
Visual features of entire image and fixated location. We resampled the movie clips at a rate of 0.5Hz and generated masks blacking out non-fixated locations. We used five hidden layers' outputs of image encoder (ViT-g/14 from EVA-CLIP [6]) to construct visual features from entire images and masked images, respectively. For the masked image features, average pooling across spatial dimension was applied. After this procedure, we obtained patched features of entire image whose size was 5x16x16x1408 and features whose size was 5x1x1x1408 at each time point.
Visual feature decoding from fMRI signal. We constructed linear models to predict the feature of fixated locations from BOLD signals of each ROI after voxel selection [7].
Prediction of fixation and evaluation. Pearson correlation between predicted feature vectors and those of each image patch was calculated. Patches with the top-k correlation were selected as predicted fixated locations. We used Intersection over Union (IoU) between these predicted patches and the true fixated patches as metrics for evaluation. The chance level was obtained by the same procedure using temporally shuffled fMRI signals.
We used a publicly available, StudyForrest dataset [4] consisting of simultaneous fMRI (TR=2) and eye tracking (temporal resolution=1000 Hz) data during movie watching (Forrest Gump, 2h-long). We used data from twelve participants successfully completed their experiments. Twenty two ROIs were extracted according to Glasser et al. [5].
fMRI-based prediction of fixated locations
We newly developed fMRI-based prediction of fixated locations as follows (Fig 1).
Visual features of entire image and fixated location. We resampled the movie clips at a rate of 0.5Hz and generated masks blacking out non-fixated locations. We used five hidden layers' outputs of image encoder (ViT-g/14 from EVA-CLIP [6]) to construct visual features from entire images and masked images, respectively. For the masked image features, average pooling across spatial dimension was applied. After this procedure, we obtained patched features of entire image whose size was 5x16x16x1408 and features whose size was 5x1x1x1408 at each time point.
Visual feature decoding from fMRI signal. We constructed linear models to predict the feature of fixated locations from BOLD signals of each ROI after voxel selection [7].
Prediction of fixation and evaluation. Pearson correlation between predicted feature vectors and those of each image patch was calculated. Patches with the top-k correlation were selected as predicted fixated locations. We used Intersection over Union (IoU) between these predicted patches and the true fixated patches as metrics for evaluation. The chance level was obtained by the same procedure using temporally shuffled fMRI signals.
Results:
We examined whether fMRI signals in each ROI can predict changes in fixated locations through spontaneous attentional shifts. To do so, we focused time points where large eye movements occurred.
We found that posterior, parietal ROIs can significantly predict the fixated locations before the occurrence of the eye movements (Fig. 2). In addition to these ROIs, activities in paracentral, middle cingulate, posterior cingulate, and dorsolateral prefrontal ROIs can successfully predict the fixated locations after the eye movements. Interestingly, paracentral and middle cingulate ROIs successfully decoded the relocated areas earlier than other cortices.
We found that posterior, parietal ROIs can significantly predict the fixated locations before the occurrence of the eye movements (Fig. 2). In addition to these ROIs, activities in paracentral, middle cingulate, posterior cingulate, and dorsolateral prefrontal ROIs can successfully predict the fixated locations after the eye movements. Interestingly, paracentral and middle cingulate ROIs successfully decoded the relocated areas earlier than other cortices.
Conclusions:
We confirmed that the fixation with voluntary shift of attention could be significantly decoded from the BOLD signals of multiple time points. We also found that the neural activity in Paracentral lobular and mid cingulate can decode earlier than visual cortex, suggesting the ROI may be involved with generation or processing of voluntary attention.
Modeling and Analysis Methods:
Methods Development 2
Motor Behavior:
Brain Machine Interface
Perception, Attention and Motor Behavior:
Attention: Visual 1
Keywords:
Data analysis
FUNCTIONAL MRI
Machine Learning
Vision
Other - Spatial Attention
1|2Indicates the priority used for review
Provide references using author date format
Jake Son (2020), ‘Evaluating fMRI-Based Estimation of Eye Gaze During Naturalistic Viewing’, Cerebral Cortex, Volume 30, Issue 3, Pages 1171–1184.
O’Connell (2018), 'Predicting eye movement patterns from fMRI responses to natural scenes', Nat Commun 9, 5159
Hanke, M. (2016), 'A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation' Sci Data 3, 160092
Glasser, M. (2016), 'A multi-modal parcellation of human cerebral cortex' Nature 536, 171–178
Fang, Y. (2022), 'Eva: Exploring the limits of masked visual representation learning at scale' arXiv preprint arXiv:2211.07636
Horikawa, T. (2017), 'Generic decoding of seen and imagined objects using hierarchical visual features' Nat Commun 8, 15037