Single-Trial EEG Analysis of Age-Related Differences in Model-Based Learning
Poster No:
904
Submission Type:
Abstract Submission
Authors:
Yangchu Huang1, Shanshan Zhen1
Institutions:
1Department of Social and Behavioural Sciences, City University of Hong Kong, Hong Kong, China
First Author:
Yangchu Huang
Department of Social and Behavioural Sciences, City University of Hong Kong
Hong Kong, China
Department of Social and Behavioural Sciences, City University of Hong Kong
Hong Kong, China
Co-Author:
Shanshan Zhen
Department of Social and Behavioural Sciences, City University of Hong Kong
Hong Kong, China
Department of Social and Behavioural Sciences, City University of Hong Kong
Hong Kong, China
Introduction:
Planning in a mental model of the world to guide goal-directed decisions can be computationally demanding (Kool et al., 2018). Due to the limited cognitive resources, older adults tend to rely more on habitual, model-free learning systems than younger adults (Eppinger et al., 2013). Recent studies suggest that this tendency may stem from age-related deficits in representing a cognitive map (Bolenz et al., 2019; Ruel et al., 2023), the foundation of model-based learning systems (Vikbladh et al., 2019). However, since previous studies used practice and detailed instructions to help participants understand the task structure, it remains unclear whether older adults have difficulty constructing a cognitive map from scratch.
Here, we investigate the reinforcement learning (RL) process of both older and younger adults, aiming to explore whether there are age-related differences in model-based learning, specifically in how the probabilistic task structure is acquired and the corresponding neural representation.
Here, we investigate the reinforcement learning (RL) process of both older and younger adults, aiming to explore whether there are age-related differences in model-based learning, specifically in how the probabilistic task structure is acquired and the corresponding neural representation.
Methods:
57 younger adults (from 18 to 31, M=21.46, SD=2.62) and 62 older adults (from 61 to 83, M=68.94, SD=5.06) performed a sequential two-choice Markov decision task (Fig. 1A, adapted from Gläscher et al., 2010). Among them, concurrent EEG data were recorded from 38 younger adults and 39 older adults. The experiment comprised three sessions (lower panel of Fig. 1A). Both Session 1 and Session 3 consisted of 80 trials each.
We first analyzed second-stage reaction times (RT) to examine whether participants acquired the knowledge of state transitions in Session 1 and 3. Longer reaction times following rare transitions (0.3) in contrast to common transitions (0.7) imply that participants have acquired state knowledge (Seow et al., 2021).
To unfold the learning process, we fitted the choice data of both Session 1 and 3 to the modified HYBRID model built upon prior work (Gläscher et al., 2010; Oguchi et al., 2023) using a hierarchical Bayesian approach. The current model described the mix of model-free and model-based learning using separate model-based weight parameters, w1 for the first half (1-40 trials) and w2 for the second half (41-80 trials) of Session 3. The model also included model-based and model-free learning rate parameters, deciding the learning speed of model-based and model-free learning. Inverse temperature parameters were defined to control the degree of exploitation.
To investigate the underlying neural dynamics, we performed multiple linear regression of single-trial EEG data. The stimuli-locked ERP was expected to covary with trial-by-trial state prediction error (SPE) signals derived from the HYBRID model. In Session 1, we only considered SPE as the predictor, whereas in Session 3, we included both reward prediction error (RPE) signals and SPE.
We first analyzed second-stage reaction times (RT) to examine whether participants acquired the knowledge of state transitions in Session 1 and 3. Longer reaction times following rare transitions (0.3) in contrast to common transitions (0.7) imply that participants have acquired state knowledge (Seow et al., 2021).
To unfold the learning process, we fitted the choice data of both Session 1 and 3 to the modified HYBRID model built upon prior work (Gläscher et al., 2010; Oguchi et al., 2023) using a hierarchical Bayesian approach. The current model described the mix of model-free and model-based learning using separate model-based weight parameters, w1 for the first half (1-40 trials) and w2 for the second half (41-80 trials) of Session 3. The model also included model-based and model-free learning rate parameters, deciding the learning speed of model-based and model-free learning. Inverse temperature parameters were defined to control the degree of exploitation.
To investigate the underlying neural dynamics, we performed multiple linear regression of single-trial EEG data. The stimuli-locked ERP was expected to covary with trial-by-trial state prediction error (SPE) signals derived from the HYBRID model. In Session 1, we only considered SPE as the predictor, whereas in Session 3, we included both reward prediction error (RPE) signals and SPE.
·Fig. 1: A. The task structure (upper panel) and experimental procedure (lower panel). B. Analysis of second-stage reaction times. C. Computational modelling results.
Results:
A hierarchical Bayesian linear regression on second-stage RT revealed no evidence for the transition effect in Session 1 for Old group, but a strong effect for Young group (Fig. 1B), meaning Young group acquired state transition knowledge faster than Old group. Similarly, computational modelling results indicated that younger adults showed faster model-based learning than older adults (MB learning rate in the lower panel of Fig. 1C).
The single-trial EEG regression results exhibited congruent patterns with the observed behavioural results. In Session 1, only Young group's second-stage stimuli-locked ERP covaried with trial-by-trial SPE, whereas Old group's ERP didn't show significant results (Fig. 2A). In Session 3, both groups' second stage (Fig. 2B) and outcome stage (Fig. 2C) ERP showed sensitivity to SPE.
The single-trial EEG regression results exhibited congruent patterns with the observed behavioural results. In Session 1, only Young group's second-stage stimuli-locked ERP covaried with trial-by-trial SPE, whereas Old group's ERP didn't show significant results (Fig. 2A). In Session 3, both groups' second stage (Fig. 2B) and outcome stage (Fig. 2C) ERP showed sensitivity to SPE.
·Fig. 2: Single-trial EEG analysis results.
Conclusions:
This research demonstrates that while older adults may display slower model-based learning compared to younger adults, they are still capable of acquiring knowledge of task structures. Through model-based single-trial EEG analyses, we identified distinctive EEG signal characteristics linked to the probabilistic updates of state transition knowledge.
Higher Cognitive Functions:
Decision Making 1
Lifespan Development:
Aging
Modeling and Analysis Methods:
Bayesian Modeling
EEG/MEG Modeling and Analysis 2
Novel Imaging Acquisition Methods:
EEG
Keywords:
Aging
Electroencephaolography (EEG)
Learning
Other - Reinforcement Learning
1|2Indicates the priority used for review
Provide references using author date format
Eppinger, B., Walter, M., Heekeren, H. R., & Li, S. C. (2013). Of goals and habits: age-related and individual differences in goal-directed decision-making. Frontiers in neuroscience, 7, 253.
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010). States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4), 585-595.
Kool, W., Gershman, S. J., & Cushman, F. A. (2018). Planning complexity registers as a cost in metacontrol. Journal of cognitive neuroscience, 30(10), 1391-1404.
Oguchi, M., Li, Y., Matsumoto, Y., Kiyonari, T., Yamamoto, K., Sugiura, S., & Sakagami, M. (2023). Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task. Scientific Reports, 13(1), 1419.
Ruel, A., Bolenz, F., Li, S. C., Fischer, A., & Eppinger, B. (2023). Neural evidence for age-related deficits in the representation of state spaces. Cerebral Cortex, 33(5), 1768-1781.
Seow, T. X., Benoit, E., Dempsey, C., Jennings, M., Maxwell, A., O'Connell, R., & Gillan, C. M. (2021). Model-based planning deficits in compulsivity are linked to faulty neural representations of task structure. Journal of Neuroscience, 41(30), 6539-6550.
Vikbladh, O. M., Meager, M. R., King, J., Blackmon, K., Devinsky, O., Shohamy, D., ... & Daw, N. D. (2019). Hippocampal contributions to model-based planning and spatial memory. Neuron, 102(3), 683-693.