Active Inference Agents: A Proposal for Structural Learning During Sleep

1945 

Abstract Submission 

David Araya¹, Leandro Torres², Nelson Trujillo-Barreto³, Yunier Pieur-Coloma⁴, Carla Taramasco⁵, WAEL EL-DEREDY⁶

¹Facultad de Ingeniería, Universidad Andrés Bello, Viña del Mar, Región de Valparaiso, ²Universidad de Playa Ancha, Viña del Mar, Región de Valparaiso, ³University of Manchester, Manchester, Manchester, ⁴Universidad de Valparaíso, Valparaíso, Valparaíso, ⁵Universidad Andres Bello, Viña del Mar, Región de Valparaiso, ⁶UNIVERSIDAD DE VALPARAISO, Valparaíso, Valparaíso

David Araya  
Facultad de Ingeniería, Universidad Andrés Bello
Viña del Mar, Región de Valparaiso

Leandro Torres  
Universidad de Playa Ancha
Viña del Mar, Región de Valparaiso

Nelson Trujillo-Barreto  
University of Manchester
Manchester, Manchester

Yunier Prieur-Coloma  
Universidad de Valparaíso
Valparaíso, Valparaíso

Carla Taramasco  
Universidad Andres Bello
Viña del Mar, Región de Valparaiso

WAEL EL-DEREDY  
UNIVERSIDAD DE VALPARAISO
Valparaíso, Valparaíso

Active Inference has emerged as a new theory modeling the brain as a predictive machine composed of hierarchically arranged generative models, which continuously make predictions about inputs from the environment [1]. By minimizing a quantity known as variational free energy, an agent based on Active Inference adjusts the parameters of its generative models, enhancing predictions and aiding the agent's adaptation to the specific environment. Traditionally, this model modification does not include altering the model's structure, which is assumed to be predefined and likely formed during the agent's phylogeny. There is only one prior study that addresses structural changes in these models, focusing on varying the number of models by freeing up slots (pruning) [2]. In our approach, however, we discuss altering the underlying structure of the model itself. Inspired by studies suggesting that the brain, during sleep, engages in internal simulations and cognitive restructuring to optimize environmental understanding [3], we propose that an Active Inference-based agent could optimize its models' structure during sleep, by means of variational free energy minimization.

We implemented an active inference agent based on the partially observable Markov decision process (POMDP) [4]. This agent is designed to identify and classify simple geometric shapes with various rotations in a controlled environment. It uses a repertoire of generative models to represent the geometric shapes in the environment, and parameter estimation is achieved by minimizing the models' free energy. We propose that during night time periods, the agent continues its learning through internal simulations generated by the same generative models, in a cycle of generation and prediction. During this nocturnal phase, the structure of some generative models is modified, employing an evolutionary strategy inspired by neuronal Darwinism [5]. The total number of generative models is kept constant, discarding those models that exhibit the highest variational free energy.

Figure 1 shows changes in free energy corresponding to the agents' simulated behaviour during day and night. During the first day, the free energy decreases, reflecting learning of the models' parameters based on sensory information (geometric figures). During the first night, the free energy remains at the same values, indicating that the sleep simulations are generated using the same models learned during the day, and no further parameters' optimization are made. However, on the second night, a new generative model emerges that simplifies the representation of figure rotations, thereby reducing free energy. The following day, this rotation-invariant model proves its effectiveness in the identification and prediction of geometric figures with various orientations.

In this exploratory study, we propose and test a mechanism by which an active inference agent can modify the structure of its internal generative models. This mechanism is inspired by neurobiological theories about brain simulation during sleep. We hypothesize, that free energy minimization during sleep promotes the creation of generative models with improved generalization capabilities due to a more effective representation of the environment. This hypothesis implies that the brain might be continuously reducing free energy at all times (day and night), even in the presence of minimal sensory information, which could account for the uninterrupted spontaneous activity consistently observed in the brain.

Bayesian Modeling ²

Methods Development ¹

Other Methods

Learning

Modeling

Sleep

Other - Active inference