Spatiotemporal dynamics of object-based attention: an MEG-fMRI-DNN fusion study

2454 

Abstract Submission 

Chun-Hui Li¹, Bo-Cheng Kuo¹

¹National Taiwan University, Taipei, Taiwan

Chun-Hui Li  
National Taiwan University
Taipei, Taiwan

Bo-Cheng Kuo  
National Taiwan University
Taipei, Taiwan

Individuals typically receive continuous and unlimited sensory inputs from the external environment. The ability to selectively choose the most relevant information is contingent upon current behavioral goals. Recent advancements in multivariate methods have significantly contributed to unveiling the informational contents of object-based attention. Employing these sophisticated methods allows the recording of spatial and temporal information in neural responses related to object-based attention through various neuroimaging modalities. Despite this progress, it remains uncertain whether and how information can be integrated across different imaging modalities and analysis methods. In this study, we adopted representational similarity analysis to investigate the spatiotemporal dynamics of neural representations that underlie object-based attention, utilizing magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and deep neural networks (DNN).

Twenty-four participants viewed streams of face and building images in an MEG experiment and an fMRI experiment (Fig. 1A). MEG and fMRI data were acquired at National Taiwan University on a 306-channel Triux system (Elekta Neuromag) and a Siemens 3T Prisma MRI scanner, respectively. Participants were instructed to attend to faces or buildings and respond to pre-defined face or building targets while recording with MEG. They also performed a visual localizer task using the same face and building images and their scramble images in a block design with fMRI.
We conducted an RSA-based fusion analysis to integrate MEG with either fMRI data or DNN data (Fig. 1B). Region-of-interests (ROIs) were defined based on the fMRI contrasts and the Automated Anatomical Labeling template, specifically focusing on fusiform face areas, FFA, and parahippocampal place areas, PPA. Pairwise dissimilarities were calculated across all combinations of conditions and stimuli, linking the multivariate measurements across different modalities. This process yielded representational dissimilarity matrices (RDMs).
Each RDM generated from MEG, fMRI or DNN is symmetrical around the diagonal. The lower triangular part of each RDM was extracted and converted into a vector format. For each ROI, we calculated the Spearman correlation coefficients between the MEG RDMs and fMRI RDMs. This process was repeated separately for each face-selective and building-selective ROI, offering a temporal profile of representational similarities between fMRI and MEG for each ROI. Importantly, Spearman correlation coefficients were also computed between the MEG RDMs and DNN RDMs. RDM similarity estimations were carried out within each DNN RDM for two attention conditions separately. For all analyses, we built a linear model with face and building RDMs as two regressors to predict time-resolved RDMs from the MEG data thereby enabling a detailed exploration of the neural dynamics of object-based attention.

The fusion results showed attentional enhancement of representational similarity within category-selective brain areas. Moreover, we observed that the goal-directed modulation of representational similarity in the category-selective brain areas was influenced by the prefrontal cortex using the Granger causality analysis. Importantly, GLM-based RSA further confirmed the spatiotemporal representational dynamics in the category-selective brain areas for object-based attention (Fig. 2). Finally, the MEG-DNN fusion results revealed a similar attention effect on representational similarity (Fig. 2).

Our fusion results demonstrated the modulation of spatiotemporal patterns of neural responses for object-based attention through the RSA-based MEG-fMRI and MEG-DNN fusion approaches. The study provides novel multivariate evidence of the cognitive operations and neural mechanisms underlying the attentional modulation of object selectivity.

Multivariate Approaches ²

BOLD fMRI

MEG

Attention: Visual ¹

Perception: Visual

Other - Attention; Magnetoencephalography; functional Magnetic Resonance Imaging; Representational similarity analysis; MEG-fMRI-DNN fusion method; face perception; scene perception;