Historical Feedback Representations in the Brain Robustly Guide Learning

989 

Abstract Submission 

Sangsoo Jin¹, Juhyeon Lee¹, Jong-Hwan Lee¹

¹Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea, Republic of

Sangsoo Jin  
Department of Brain and Cognitive Engineering, Korea University
Seoul, Korea, Republic of

Juhyeon Lee  
Department of Brain and Cognitive Engineering, Korea University
Seoul, Korea, Republic of

Jong-Hwan Lee  
Department of Brain and Cognitive Engineering, Korea University
Seoul, Korea, Republic of

A developing child often needs to guess the internal structure of the environment only relying on received feedback [1]. Similarly, reinforcement learning models can construct representations of learning from the action-feedback loop without any instructions [2]. We identified robust neural representations during feedback-guided learning using a novel real-life task.

We developed a novel feedback-guided learning task called the Photographer paradigm [3] (Fig. 1a). Participants virtually explored five city scenes in random order and captured eight photographs for each city or run. They were instructed to capture scenes with the highest feedback score. Internally, the feedback scores were computed as conceptual similarities between captured scenes and the target context (shared across the cities) via a Contrastive Language-Image Pretraining (CLIP) model [4]. Neither the target context nor internal mechanics of feedback were instructed to participants.

The fMRI data was collected using a 3-T Siemens Tim-Trio MRI scanner with a 12-channel head coil (TR = 2000 ms; TE = 30 ms; 3×3×4 mm3 voxel size). We independently analyzed [5] 32 participants as the Discovery (n = 16; acquired in 2022) and Validation (n = 16; acquired in 2023) groups. We first applied a standard preprocessing pipeline using fMRIPrep 23.0.2 [6]. Run-wise GLMs were fitted using 3dDeconvolve to identify trial-wise Feedback betas while controlling the systematic effect of head motions.

We investigated what kind of feedback information is robustly represented within each run using representational similarity analysis (RSA). Fig. 1b depicts model representation dissimilarity matrices (RDMs) with three levels of historical information. In counterpart, neural RDMs were defined from trial-wise Feedback betas within the 3-voxel radius searchlight spheres [7]. To focus on within-run learning components while minimizing systematic variations between runs, we performed RSA by each run separately and averaged run-wise RSA maps. Group inferences were conducted by a one-sample t-test for each historical model using 3dttest++. We identified robust RSA clusters (voxel-wise p < 0.005 and cluster-level α < 0.05) that were significant in both the Discovery and Validation groups. We further hypothesized that the strength of the within-run feedback representation in a cluster may predict the level of within-run learning [8]. Linear mixed-effect models were fitted to estimate mean feedback scores of the current or following runs from the median RSA strength in each cluster.

Fig. 2a-b represents the model RDMs and associated neural clusters that were significant across the Discovery and Validation groups. Only historical models (i.e., Recent-2/3 Trial) were robustly represented in two groups. Notably, the middle orbital gyrus (MOG) and inferior frontal gyrus encoded historical dynamics during feedback phases. The ventral striatum was associated with the Recent-3 Trial model, suggesting that the dopaminergic learning signal may incorporate the current, one-back, and two-back trials.

We found an association between the neural-model coupling and the level of within-run learning. Fig 2c elucidates the RSA strength of the Recent-3 Trial model in the MOG area predicted the mean feedback score of the following run rather than the current run. This effect remains significant after controlling the previous mean feedback and variability of individuals and runs.

We identified historical feedback representations in the brain during real-life, feedback-guided learning. The ventromedial reward network [9] may encode the current feedback structure even when the goal is unknown, and the learned structure would help future learning. Our study may suggest critical neural signatures of feedback-guided learning that are generalizable for humans [6] and reinforcement learning models [10].

Decision Making

Executive Function, Cognitive Control and Decision Making

Reasoning and Problem Solving ¹

Higher Cognitive Functions Other ²

Learning and Memory Other

Cognition

Computational Neuroscience

FUNCTIONAL MRI

Learning

Modeling

Other - Context learning, Feedback-guided learning, Naturalistic paradigm, Representational Similarity Analysis.