Communication with Surprise – Computational and Neural Mechanisms of Non-verbal Human Interactions

1627 

Abstract Submission 

Tatia Buidze¹, Jan Gläscher¹, Yuanwei Yao¹

¹Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Germany

Tatia Buidze  
Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf
Germany

Jan Gläscher  
Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf
Germany

Yuanwei Yao  
Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf
Germany

Conventionally, violating expectations signals a need to refine predictions through learning[1]. Yet in communication, surprise can have a different role: guiding the Receiver's focus towards key information. In language-driven communication, attention is captured using standout verbal and prosodic cues[2]. But how do we effectively communicate without a common language in novel situations? Here, we propose that the intentional use of surprise can effectively communicate information by defying expectations.

We explore this in the Tacit Communication Game (TCG)[3], a non-verbal game where the Sender directs the Receiver to a goal on a grid board using "messages" (Fig.1a). The Receiver then deduces his goal based on these messages. We developed a surprise model (SM), for the Sender's message design. SM uses intuitive priors based on principles of movement kinetics(Fig.1h) and goal orientation (Fig.1i) and constructs messages step by step by maximizing surprise at the Receiver's goal state. We compared SM against a belief-based model (BBM)[4], which selects messages through exhaustive search and updates beliefs based on the Receiver's success, but lacks step-by-step predictions. Furthermore, aiming to substantiate the SM, we explored physiological and neural responses to surprise, employing model-based analysis of Pupillary Dilation Responses (PDR) and Electroencephalogram (EEG) data. For analysis of PDR data, we employed a mixed-effect model[5]. In the model, to predict PDR, we included model-derived surprise values as the fixed effects and random intercepts for each participant. For analysis of EEG data, We again used the step-by-step estimated surprise values as a predictor for EEG power in a model-informed EEG analysis. Specifically, for each Receiver, we performed a regression at each electrode and time points, and subsequently we analyzed the regression weights obtained.

The results revealed that The SM accurately reproduces Sender's behavior, while also showed a better fit to the behavioral data compared to BBM (Fig.1e-g). Furthermore, in examining receivers' physiological reactions as they process messages, we found a direct correlation between PDRs and model-derived surprise values (Fig. 2a-b). This indicates that Senders, significantly impact the physiological states of Receivers by creating surprise. An analysis of the neural encoding of surprise via EEG data unveiled two prominent time-space clusters over frontal and frontal-central electrodes. Positioned above the Anterior Cingulate Cortex (ACC), the electrophysiological activity in the observed cluster (Fig.2c, yellow markers) likely reflects the neural processing within this region. This cluster, presumably reflecting ACC activity, can be associated with the Receiver detecting the prediction error, grounded in the anticipated intention of the Sender[6,7]. The second cluster is positioned over the anterior prefrontal cortex (Fig.2c, pink markers), can be linked to action programming[8]. As the senders formulates a message for optimal surprise, they delve into intricate action planning. After both parties align on a successful approach, the receiver possibly fine-tunes their understanding of the sender's planned intent. Temporal aspects of these clusters also hold significance. If, as postulated, the receiver's frontal cluster corresponds to a high-level representation of the sender's strategy, we would anticipate its emergence early in the epoch. This would likely be followed by activations associated with the ACC, which is predominantly implicated in error detection.

In summary, the SM showcases the effectiveness of utilizing surprise in communication without a shared language, a phenomenon explored through the TCG. Not only did the SM exhibit a notable alignment with human sender behaviors, but it also substantiates its influence through both physiological and neural markers, affirming the crucial role of surprise in novel communication.

Social Interaction ²

EEG/MEG Modeling and Analysis ¹

Computational Neuroscience

Electroencephaolography (EEG)

Language

Modeling

Other - Surprise;Tacit Communication Game; Pupil Dilation;