Decoding visual predictions in stimulus sequences: concurrent EEG/fMRI study

1063 

Abstract Submission 

Ryszard Auksztulewicz¹

¹Freie Universität Berlin, Berlin, Berlin

Ryszard Auksztulewicz  
Freie Universität Berlin
Berlin, Berlin

Mounting evidence suggests that neural processing of the present (perception) is informed by the past (memory) and aimed at the future (prediction). This notion has been formalised in the predictive processing framework [Friston, 2010]. Canonical models posit that predictions are grounded in memories [Baldeweg, 2006], and evidence from animal models suggests that mnemonic and predictive representations co-occur in the sensory neocortex [Cappotto et al., 2022]. However, recent studies in humans suggest that predictive processing may actually interfere with memory formation, and that the same neural structures may switch from encoding prediction errors to predictions at different stages of memory formation [Aitken & Kok, 2022]. Together with colleagues, we have formulated a hypothesis that memory and prediction are simultaneously implemented via distinctive interactions between sensory neocortex and the hippocampus [Barron et al., 2020]. This study is an attempt to replicate and extend our previous findings in animal models [Cappotto et al., 2022], where sensory memory traces could be decoded from sensory (auditory) cortical activity. Crucially, we could simultaneously decode sensory predictions related to upcoming stimuli. Here, we have translated the study from anaesthetised rodents to awake human volunteers; from the auditory modality to the visual modality; and from direct electrophysiological recordings to simultaneous EEG/fMRI. The aim of the study was to test whether we can decode predictions based on EEG and fMRI recordings in healthy human volunteers, and whether fMRI-based prediction decoding relies on the multivariate activity patterns in the sensory neocortex and/or the hippocampus.

N=24 healthy human volunteers were exposed to visual sequences comprising repeated triplets of images (Faces, Houses, Tools). A subset of 10% images was replaced with a visual "impulse" stimulus (concentric grating), aimed at reactivating memory/prediction traces for subsequent decoding [Stokes, 2015]. As a control condition, we presented the same stimuli in a random order (Figure panel A).

While participants were exposed to sequences, their brain activity was recorded using simultaneous EEG/fMRI. In the analysis, first we used multivariate decoding techniques based on cross-validated Mahalanobis distance and representational similarity analysis (RSA) [Cappotto et al., 2022] to decode prediction traces from EEG signals. Second, we used the same techniques to decode prediction traces from fMRI data. Finally, we correlated single-trial EEG-based decoding estimates with fMRI BOLD amplitudes. In the latter analysis, single-trial EEG-based decoding estimates were treated as a regressor in a mass-univariate analysis based on the general linear model. Correction for multiple comparisons across voxels was based on family-wise error rate.

EEG-based decoding showed that predictions of sequence elements replaced by "impulse" stimuli could be decoded in the predictable blocks (based on repeated stimulus triplets) but not in the control/random blocks (paired t-test, p<.05, corrected across time points). This result was linked to EEG latencies approx. 250 ms after impulse onset (Figure panel B, left).

fMRI-based decoding showed the same pattern of results (significant prediction decoding in predictable but not in random blocks; paired t-test, p<.05, cluster-level corrected across voxels). Here, decoding was linked to early visual regions (Figure panel C).

Finally, single-trial EEG-based decoding was found to significantly correlate with BOLD activity in the right hippocampus (Figure panel B, right).

In conclusion, while fMRI-based prediction decoding was linked to early visual regions, EEG-based decoding correlated across trials with BOLD amplitude in the hippocampus. These results provide evidence for a neocortical-hippocampal network subserving predictive processing in stimulus sequences.

Implicit Memory

Learning and Memory Other ¹

Activation (eg. BOLD task-fMRI)

Multivariate Approaches ²

Perception: Visual

Cognition

Electroencephaolography (EEG)

FUNCTIONAL MRI

Learning

Multivariate

Vision

Other - Hippocampus