Audiovision: activity in primary visual cortex is driven by musical information.
Poster No:
2491
Submission Type:
Abstract Submission
Authors:
Fernando Bravo1, Emmanuel Stamatakis2, Kristina Herfert1
Institutions:
1University of Tübingen, Tübingen, Germany, 2University of Cambridge, Cambridge, UK
First Author:
Co-Author(s):
Introduction:
The neuroscientific examination of music processing in audiovisual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information.
Although one of the most common harmonic manipulations in film music (e.g., Spielberg's Jaws or Kubrick's The Shining), systematically controlled transformations of musical dissonance have not yet been neuroscientifically studied in regard to affective processing biases in the interpretation of visual contexts. We employed functional magnetic resonance imaging (fMRI) in a naturalistic film-viewing paradigm to investigate the effects of tonal dissonance on the affective processing of visual information.
Although one of the most common harmonic manipulations in film music (e.g., Spielberg's Jaws or Kubrick's The Shining), systematically controlled transformations of musical dissonance have not yet been neuroscientifically studied in regard to affective processing biases in the interpretation of visual contexts. We employed functional magnetic resonance imaging (fMRI) in a naturalistic film-viewing paradigm to investigate the effects of tonal dissonance on the affective processing of visual information.
Methods:
During the present fMRI study, thirty-eight subjects watched the same short-film with systematically controlled consonant or dissonant music, in randomized order. Prior to the start of the audio-visual film clip subjects were instructed to "think about the intentions of the main character in the following film clip". Following the clip participants were asked to rate the valence of the movie character's intentions on a scale ranging from one (good intentions) to four (bad intentions). Two control conditions were included; i) visual alone category (i.e., film clip with no soundtrack but same instruction as above) and, ii) film clip with an instruction to describe the "physical" appearance of the character, to control for multimodal sensory processing, working memory and attentional demands of the task, without cueing subjects to attend specifically to mental states (Figure 1).
Stimulus material: The stimuli employed in the present study were created for, and tested in two previous experiments (Bravo, 2013; Bravo et al., 2019). To construct the two music conditions, a choral piece was made to sound more or less consonant or dissonant by modifying its harmonic structure (i.e., interval content manipulation) producing two otherwise-identical versions of the same musical piece (Figure 2).
Stimulus material: The stimuli employed in the present study were created for, and tested in two previous experiments (Bravo, 2013; Bravo et al., 2019). To construct the two music conditions, a choral piece was made to sound more or less consonant or dissonant by modifying its harmonic structure (i.e., interval content manipulation) producing two otherwise-identical versions of the same musical piece (Figure 2).
·Figure 1. Experimental stimuli and design.
·Figure 2. First measure of the consonant (left) and dissonant (right) music conditions.
Results:
The results showed that increasing levels of dissonance led to more negatively-valenced inferences, displaying the profound emotional impact of musical dissonance. Remarkably, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex response (V1). To our knowledge this is the first report of activations encompassing early visual cortices modulated by musical information.
The function of the primary visual cortex in visual recognition is known to be strongly modulated by multisensory information (van Atteveldt et al., 2014). In particular, auditory feedback signals are the largest contributor (Murray et al., 2016). To further elucidate why dissonance recruited V1 and to examine potential top-down contextual influences influences on early visual processing, we measured functional and effective connectivity. The results showed a significant coupling between the auditory ventral stream (AVS) (Belin et al., 2000) and V1 in response to tonal dissonance, and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1.
The function of the primary visual cortex in visual recognition is known to be strongly modulated by multisensory information (van Atteveldt et al., 2014). In particular, auditory feedback signals are the largest contributor (Murray et al., 2016). To further elucidate why dissonance recruited V1 and to examine potential top-down contextual influences influences on early visual processing, we measured functional and effective connectivity. The results showed a significant coupling between the auditory ventral stream (AVS) (Belin et al., 2000) and V1 in response to tonal dissonance, and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1.
Conclusions:
The present study revealed responses in the primary visual cortex modulated by musical information: tonal dissonance recruited early visual processing via feedback interactions from the auditory ventral pathway to the primary visual cortex. We demonstrated that the auditory "what" ventral stream plays a role in assigning meaning to non-verbal sound cues, such as dissonant music conveying negative emotions, providing an interpretative framework that serves to process the audio-visual experience. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain's sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.
Emotion, Motivation and Social Neuroscience:
Emotional Perception
Social Neuroscience Other 2
Higher Cognitive Functions:
Music
Novel Imaging Acquisition Methods:
BOLD fMRI
Perception, Attention and Motor Behavior:
Perception: Multisensory and Crossmodal 1
Keywords:
Emotions
1|2Indicates the priority used for review
Provide references using author date format
Bravo, F. (2013). The Influence of Music on the Emotional Interpretation of Visual Contexts. Designing Interactive Multimedia Tools for Psychological Research. In M. Aramaki, M. Barthet, R. Kronland-Martinet, & S. Ystad (Eds.), From Sounds to Music and Emotions (Vol. 7900, pp. 366–377). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-41248-6_20
Bravo, F., Cross, I., Hopkins, C., Gonzalez, N., Docampo, J., Bruno, C., & Stamatakis, E. A. (2019). Anterior cingulate and medial prefrontal cortex response to systematically controlled tonal dissonance during passive music listening. Human Brain Mapping, 41(1). https://doi.org/10.1002/hbm.24786
Murray, M. M., Thelen, A., Thut, G., Romei, V., Martuzzi, R., & Matusz, P. J. (2016). The multisensory function of the human primary visual cortex. Neuropsychologia, 83, 161–169. https://doi.org/10.1016/j.neuropsychologia.2015.08.011
van Atteveldt, N., Murray, M. M., Thut, G., & Schroeder, C. E. (2014). Multisensory Integration: Flexible Use of General Operations. Neuron, 81(6), 1240–1253. https://doi.org/10.1016/j.neuron.2014.02.044