Consistent representation of affective states between movie watching and recall using hyperalignment

748 

Abstract Submission 

Jongwan Kim¹, Chaery Park²

¹Jeonbuk National University, Jeonju-si, Jeollabuk-do, ²Jeonbuk National University, Jeonju-Si, North Jeolla

Jongwan Kim  
Jeonbuk National University
Jeonju-si, Jeollabuk-do

Chaery Park  
Jeonbuk National University
Jeonju-Si, North Jeolla

Recalling past experiences rekindles emotions similar to the original events, facilitated by memory reconstruction during recall. It serves as a potent method for inducing emotions in experimental studies, with physiological responses indicating similar reactivation of positive and negative emotions. The key distinction between experiences and recall lies in external stimuli during the former and reliance on internal representation during the latter. Recall involves integrating internal information from previous experiences, differing systematically from how information is processed during experiences. Neurological evidence supports distinct neural activation patterns during recall, suggesting its different functioning from experiences, even though it may evoke similar emotions. The aim of this study is to identify common affective representations when experiencing emotions and recalling.

We reanalyzed the fMRI data from Chen et al. (2017) that investigated the representation of shared memories across individuals. In this study, 17 participants watched an episode of the Sherlock in two fMRI sessions. We also used the behavioral data from Kim et al. (2020), where they presented the same stimuli and asked participants to rate affective responses on valence and arousal dimensions.
After preprocessing of fMRI data, we applied hyperalignment (Haxby et al., 2020). Hyperalignment is a method to capture shared information by projecting pattern vectors of neural responses into a common information space. Affine transformations are computed to optimize alignment between trajectories, preserving the geometry of dissimilarities between pattern vectors. This method is known to be advantageous, particularly when individuals exhibit different neuroanatomical structures even when representing a common construct, which benefits cross-participant analyses. We performed hyperalignment between watching and recall datasets for each participant. Initially, we computed a common template between the watching and recall datasets using Procrustes rotation and subsequently aligned each pattern to this common template, generating a new common template. Finally, we aligned each dataset to the mean alignment from the previous iteration. We repeated this procedure for all participants.
We conducted regression-based decoding to confirm if affective states of signed and unsigned valence and arousal can be predicted between watching and recall datasets. The training set was either watching or recall dataset while the testing set was the left one, and then computed regression-based decoding. This procedure was repeated for all subjects. A permutation test was performed for significance testing.

The results revealed significant accuracy in correctly predicting arousal, p<.001. However, predictions of signed and unsigned valence were not significant, ps>.05. We performed repeated measures analyses of variance to compare scene-to-recall and recall-to-scene decoding types and three types of affect (arousal, signed, and unsigned valence). Two decoding types were not significantly different, p=.936, whereas the effect of affect was significant, p=.009. A trend analysis revealed that the linear contrast (signed vs. unsigned valence) was not significant, p=.141, while the quadratic contrast (signed and unsigned valence vs. arousal) was significant, p=.015.

This study investigated how affective states elicited by naturalistic stimuli are represented in the brain. Specifically, we explored if signed and unsigned valence and arousal can be predicted based on neural responses. The hyperalignment technique (Haxby et al., 2020) was employed to transform movie watching and recall datasets into the shared information space. We were able to predict arousal based on neural responses at the whole brain level and representation of arousal is consistent between movie watching and recall.

Emotional Perception ¹

Multivariate Approaches ²

Computational Neuroscience

Emotions

Experimental Design

Multivariate

Other - Hyperalignment; Regression-based decoding; Recall