Decoding the 'silent' neural symphony: unraveling the intricacies of lip-reading in the brain

1043 

Abstract Submission 

Joanna Beck^1,2, Jakub Wojciechowski^1,2, Hanna Cygan¹, Martyna Bryłka¹, Tomasz Wolak¹

¹Bioimaging Research Center, Institute of Physiology and Pathology of Hearing, Kajetany, Poland, Warsaw, Poland, ²Nencki Institute of Experimental Biology PAS, Warsaw, Poland

Joanna Beck  
Bioimaging Research Center, Institute of Physiology and Pathology of Hearing, Kajetany, Poland|Nencki Institute of Experimental Biology PAS
Warsaw, Poland|Warsaw, Poland

Jakub Wojciechowski  
Bioimaging Research Center, Institute of Physiology and Pathology of Hearing, Kajetany, Poland|Nencki Institute of Experimental Biology PAS
Warsaw, Poland|Warsaw, Poland

Hanna Cygan  
Bioimaging Research Center, Institute of Physiology and Pathology of Hearing, Kajetany, Poland
Warsaw, Poland

Martyna Bryłka  
Bioimaging Research Center, Institute of Physiology and Pathology of Hearing, Kajetany, Poland
Warsaw, Poland

Tomasz Wolak  
Bioimaging Research Center, Institute of Physiology and Pathology of Hearing, Kajetany, Poland
Warsaw, Poland

Lipreading (LR) is the ability to extract speech information from the movements of a speaker's lips and face. It is an essential skill for individuals with hearing loss (Bernstein et al., 2022) and those who work in noisy or distracting environments (e.g., Erber, 1969; Middelweerd & Plomp, 1987). Visual information from the talker's face helps fill in the missing auditory information.
Articulatory lip movements enable it to recognize visemes (the visual equivalent of phonemes), and supplement degraded auditory information during speech perception. Despite its practical importance, the neural and cognitive mechanisms underlying lipreading still need to be better understood.
Neuroimaging studies have shown that the brain regions involved in LR overlap with those involved in auditory speech processing, suggesting that lipreading relies on neural mechanisms similar to those involved in normal hearing (Calvert, 2001; Skipper et al., 2007). Generally, studies on the correlates of LR point to different areas relevant to this skill; these differences may be due to the different fMRI paradigms used in each study. The goal of our study was to examine
G1. Aspects of language processing during LR in a robust manner
G2. Mechanisms involved in learning LR

To answer question G1, we tested 51 right-handed subjects using a lip-reading task during fMRI. The task used 20-second clips of an actor speaking on various topics (cuisine, style, sports, weather). The paradigm included four conditions: clips with sound, clips without sound, clips backward, and a static face without sound. Half of the blocks presented entire sentences, and the other half presented only words. In the G2 study, 60 healthy right-handed subjects underwent a month-long lip reading course, with three 45-minute sessions of fMRI neurofeedback or sham neurofeedback, with scanning before and after the course. To assess the influence of the factors, a rmANOVA was conducted.

Both G1 and G2 results showed that the lexical vs. non-lexical factor revealed the presence of numerous language-related regions (e.g., STG, MTG, TP). The "words" vs. "sentences" factor produced clusters in the left ATL and bilateral pMTG/TPJ. The effects found in the anterior and inferior temporal poles indicate a differential role for semantic information retrieval (Binder et al., 2011) in reading word and sentence text, likely due to the complexity and difficulty of the linguistic material. Additionally, ATL plays a central role in integrating semantic and syntactic information and is particularly sensitive to meaningful sentences (Visser et al., 2010). In contrast, the TPJ's differential involvement in reading words and sentences may be due to the high cognitive demands during sentence recognition and the involvement of extensive attentional resources in analyzing lip movements.
The results also showed that during lip reading (vs. observation of non-linguistic lip movements), participants were actively engaged in phoneme and lexical encoding. They were also involved in the retrieval of the semantic lexicon. Form G2, longitudinal, pre- and post-scanning analysis showed changes in the mentioned areas, and the pattern of differences was driven by the baseline level of lip-reading ability.

Our study identifies shared and unique neural activity patterns in lip reading, expanding on existing literature. Uncovering task-specific activations that have not previously been reported contributes to a more nuanced understanding of lipreading mechanisms. The cortical regions revealed in our analysis suggest that both cognitive abilities related to language comprehension and language production are engaged in lip reading. This insight holds promise for enhancing support for individuals with hearing disabilities or undergoing cochlear implantation, paving the way for targeted interventions and improved accessibility.

Language Comprehension and Semantics ²

Speech Perception ¹

Activation (eg. BOLD task-fMRI)

Other Methods

FUNCTIONAL MRI

Language