Mapping idiographic affective appraisals to brain activity using semantic embeddings
Poster No:
727
Submission Type:
Abstract Submission
Authors:
Luke Chang1, Eshin Jolly1, Nir Jacoby1, Younji Choi1, Tor Wager1, Jeremy Manning1
Institutions:
1Dartmouth College, Hanover, NH
First Author:
Co-Author(s):
Introduction:
Emotions are coordinated, multi-system responses to events and situations relevant to survival and well-being. These responses emerge from appraisals of personal meaning that reference one's goals, memories, internal body states, and beliefs about the world [Ashar et al., 2017]. Dysregulation of emotions is central to many brain and body-related disorders, making it of paramount importance to understand the neurobiological mechanisms that govern emotional experiences.
In our prior work, we have observed that the ventromedial prefrontal cortex (vmPFC) involved in processing these affective appraisals appears to be highly idiosyncratic across individuals [Chang et al., 2021]. While discrete patterns of activity within the vmPFC appear to broadly correspond to affective experiences across participants, it remains an open question as to how the vmPFC generates specific appraisals [Roy et al., 2012].
In this study, we develop a novel computational framework to measure an individual's idiosyncratic appraisals that arise in the context of naturalistic movie viewing that combines collaborative filtering with recent advances in natural language processing.
In our prior work, we have observed that the ventromedial prefrontal cortex (vmPFC) involved in processing these affective appraisals appears to be highly idiosyncratic across individuals [Chang et al., 2021]. While discrete patterns of activity within the vmPFC appear to broadly correspond to affective experiences across participants, it remains an open question as to how the vmPFC generates specific appraisals [Roy et al., 2012].
In this study, we develop a novel computational framework to measure an individual's idiosyncratic appraisals that arise in the context of naturalistic movie viewing that combines collaborative filtering with recent advances in natural language processing.
Methods:
Participants (N=122) watched 8 emotionally engaging emotional stories while undergoing fMRI. After participants completed their scanning session, they were presented with a transcript of each story and were asked to mark on the document to indicate what they were thinking about for any time point they could remember. We embedded each thought into a 384-dimensional semantic space using a BERT Sentence Transformer [Reimers and Gurevych, 2019] (Figure 1A). Then for every single embedding dimension, we performed collaborative filtering with a 20-second boxcar kernel to create a dense participant-by-time-by-embedding dimension tensor using the Neighbors toolbox [Jolly et al., 2022] (Figure 1B). For each participant, we used distance correlations [Székely et al., 2007] to map the time-varying multivariate semantic embeddings of their appraisals to their multivariate brain activity separately for each region from a k=50 whole brain parcellation [de la Vega et al., 2016]. We performed a sign permutation test over participants and corrected for multiple comparisons using FDR q < 0.05.
Results:
Overall, we found strong associations between participants' idiosyncratic appraisals and dynamic fluctuations in the vmPFC, dmPFC, ventral striatum, amygdala, thalamus, and visual cortex. These results demonstrate that the vmPFC plays a critical role in generating affective meaning based on subjective interpretations of unfolding narrative events.
·Figure 1. Mapping appraisals onto brain activity using natural language processing. (A) Participants report internal thoughts related to the transcripts after being scanned, which are projected into a
Conclusions:
This study provides a proof of concept of how subjective interpretations of a single individual can be measured and mapped onto brain activity. Similar to work studying how the vmPFC generates idiosyncratic value in the field of decision neuroscience [Plassmann et al., 2007], we find strong evidence that individuals' unique interpretations of the unfolding narratives is encoded in the vmPFC. This finding is particularly striking given that participants Our results have significant implications for translational work for characterizing how patients suffering from psychiatric symptoms may be experiencing mood dysregulation based on maladaptive appraisals of the world.
Emotion, Motivation and Social Neuroscience:
Emotion and Motivation Other 1
Modeling and Analysis Methods:
Multivariate Approaches 2
Keywords:
Data analysis
Emotions
FUNCTIONAL MRI
Multivariate
1|2Indicates the priority used for review
Provide references using author date format
Chang LJ, Jolly E, Cheong JH, Rapuano KM, Greenstein N, Chen P-HA, Manning JR (2021): Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience. Sci Adv 7. http://dx.doi.org/10.1126/sciadv.abf7129.
Jolly E, Farrens M, Greenstein N, Eisenbarth H, Reddan MC, Andrews E, Wager TD, Chang LJ (2022): Recovering Individual Emotional States from Sparse Ratings Using Collaborative Filtering. Affect Sci 3:799–817.
Plassmann H, O’Doherty J, Rangel A (2007): Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J Neurosci 27:9984–9988.
Reimers N, Gurevych I (2019): Sentence-BERT: Sentence embeddings using Siamese BERT-networks. arXiv [cs.CL]. arXiv. https://github.com/UKPLab/.
Roy M, Shohamy D, Wager TD (2012): Ventromedial prefrontal-subcortical systems and the generation of affective meaning. Trends Cogn Sci 16:147–156.
Székely GJ, Rizzo ML, Bakirov NK (2007): Measuring and testing dependence by correlation of distances. Ann Stat 35:2769–2794.
de la Vega A, Chang LJ, Banich MT, Wager TD, Yarkoni T (2016): Large-Scale Meta-Analysis of Human Medial Frontal Cortex Reveals Tripartite Functional Organization. J Neurosci 36:6553–6562.