Audiovisual integration during motion adaptation modulates brain activity in the early visual cortex

2494 

Abstract Submission 

Minsun Park¹, Ghootae Kim², Chai-Youn Kim¹

¹Korea University, Seoul, Republic of Korea, ²Cognitive Science Research Group, Korea Brain Research Institute, Seoul, Republic of Korea

Minsun Park  
Korea University
Seoul, Republic of Korea

Ghootae Kim  
Cognitive Science Research Group, Korea Brain Research Institute
Seoul, Republic of Korea

Chai-Youn Kim  
Korea University
Seoul, Republic of Korea

Recent neuroscientific findings challenge the traditional notion of the independence of unimodal sensory cortices (e.g., V1 for vision, A1 for audition) in multisensory integration. Studies, particularly on audio-visual (AV) interactions, demonstrated cross-modal processing, where auditory stimuli alone activated the visual cortex (Poirier et al., 2006), and various auditory information was decoded from the neural patterns in the visual cortex (Rezk et al., 2020; Vetter et al., 2020). Despite potential AV interaction through the visual cortex, a more direct assessment is still required, as sound and vision were not concurrently presented in these studies. In our study, we investigated whether integration of simultaneous AV signals modulate neural activity in the early visual areas. Our previous psychophysical works (Park et al., 2019; under review) utilizing the motion aftereffect (MAE; Mather, 1980) showed an AV direction congruence effect, i.e., enhanced visual MAE when visual adapting motion was accompanied by directionally congruent auditory motion compared to incongruent auditory motion. Using fMRI, we probed the neural basis of such congruence effect by testing whether AV direction congruence in V1 and V2 areas is decoded through multi-voxel pattern analysis (MVPA).

Nineteen naïve individuals (24.09 ± 3.38 years) participated in an fMRI experiment (Siemens, 3T). Across 16 runs of the main experiment, adaptation and MAE phases were presented 8 times per run. During the adaptation phases (36 s initially and 12 s thereafter), the MAE was elicited by the 100%-coherence random-dot kinematograms (RDKs) moving leftward or rightward. Sound motions with the leftward or rightward direction were simulated by modulating white noise intensity presented between binaural channels of noise-canceling headphones. There were four sound conditions: congruent and incongruent conditions based on AV direction congruence, along with stationary and no-sound conditions. In the following 4-s MAE phase, participants reported the MAE duration and direction while viewing stationary RDKs. Using retinotopy and visual direction localizers, we identified a subset of motion-sensitive voxels in V1 and V2 (Figure 1a). Employing multi-class SVM, we performed MVPA to discriminate the four sound conditions for each adaptation and MAE phase. Whole-brain searchlight analysis was conducted to elucidate any regional specificity in patterns based on the sound conditions.

Behavioral results replicated our previous finding, indicating a longer MAE duration in the congruent condition compared to the incongruent condition (Figure 1b). Univariate fMRI results mirrored this congruence effect, showing greater signal changes in V1 and V2 during adaptation in the congruent condition (Figure 1c), whereas no significant differences in signal changes during MAE. Notably, AV direction congruence could be reliably classified from neural patterns in V1 and V2, even in the absence of physical motion (i.e., MAE phase) as well as in the adaptation phase (Figure 1d). Whole-brain searchlight results identified a large part of bilateral superior temporal area including A1, transverse temporal gyrus, and insula during both phases (Figure 2).

Our results suggest neural mechanisms underlying the AV congruence effect observed by MAE duration, demonstrating distinct neural patterns in the early visual cortex depending on AV integration. Searchlight findings suggest that neural signatures in V1 and V2, linked to AV congruence, may originate from feedback influences from higher-order multisensory areas such as STS and insula. The present work adds to the understanding of neural mechanisms governing cross-modal interactions within the retinotopic areas, offering direct examination of AV integration.

Activation (eg. BOLD task-fMRI)

Multivariate Approaches ²

Perception: Auditory/ Vestibular

Perception: Multisensory and Crossmodal ¹

Perception: Visual

Cognition

Cortex

FUNCTIONAL MRI

Machine Learning

Multivariate

Perception

Univariate

Vision

Other - Multisensory integration; Auditory