A multimodal dataset targeting music processing: neuroimaging, behavior & computational models

983 

Abstract Submission 

Peer Herholz¹, Karim Jerbi², Jean-Baptiste Poline³

¹The Neuro (Montreal Neurological Institute-Hospital), McGill University, Montreal, QC, ²Computational and Cognitive Neuroscience Lab, Department of Psychology, University of Montreal, Montreal, Quebec, ³The Neuro (Montreal Neurological Institute-Hospital), McGill University, Montreal, Quebec

Peer Herholz  
The Neuro (Montreal Neurological Institute-Hospital), McGill University
Montreal, QC

Karim Jerbi  
Computational and Cognitive Neuroscience Lab, Department of Psychology, University of Montreal
Montreal, Quebec

Jean-Baptiste Poline  
The Neuro (Montreal Neurological Institute-Hospital), McGill University
Montreal, Quebec

Due to methodological advancements, the investigation of auditory perception and its neuronal and behavioral correlates has made tremendous progress in recent years. However, a vast amount of previous research work focused on auditory percepts across a broad range of sound categories [1,2], hence missing fine-grained within-category organization, which has been mainly explored for speech [3]. So far, only a very limited amount of studies concentrated on other categories, such as music, which, given its high diversity and dimensionality, poses as an ideal candidate to probe how perceived auditory signals are processed along the hierarchy of the auditory system, combining low level (acoustic) and high level (category/semantic) features in order to achieve behaviorally relevant percepts and how these could be explained by models of varying complexity and sources. The majority of these studies furthermore restricted themselves to a single data modality (e.g. fMRI, behavior or EEG) and did not share the data in a findable, accessible, interoperable and reusable (FAIR) manner.

Aiming to address this gap, we acquired a multimodal dataset targeting the processing of music, which will be openly shared with the community in a standardized and FAIR way. Divided into 5 sub-datasets, it entails sub-datasets spanning 1. fMRI (n=15), 2. EEG (n=12), 3. behavioral data (n=20), and 4. simple acoustic feature, as well as complex computational models that comprise low and high-level features of 5. independently validated (n=20) music stimuli from 20 genres. The genres and examples therein were selected based on prior research and music classification resources. In sub-datasets encompassing neuroimaging, stimuli were presented in a 1-hour long passive listening design, and across all datasets, participants performed a 1-hour behavioral multi-arrangement task. Additionally, a broad range of general demographic and auditory processing-related information was accessed per participant. All datasets will be provided in a version-controlled (via DataLad [4]) and standardized form (BIDS [5]), comprising abundant metadata and derivatives. The latter encompass quality control, preprocessing and statistical modelling to validate the datasets.

Outcomes of the quality control and preprocessing steps indicated that the data was suitable for utilization as no significant artifacts but feasible signal-to-noise ratios were present in the neuroimaging data and the behavioral paradigm yielded reliable responses both within and across participants. Results from the statistical modeling indicated stable responses for all participants in all tested analysis approaches. Here, dataset validity was tested via two commonly applied analyses: stimulus-evoked responses (general linear models in fMRI and event-related potentials in EEG) and encoding models within which the different low and high-level features were employed as predictors. Both indicated spatial and temporal patterns in line with prior research, suggesting not only stimulus-related responses but also that the included features capture aspects of the stimuli that are appropriate for respective modeling approaches. In more detail, the stimuli evoked/the features predicted primary and non-primary regions of the auditory cortex bilaterally (in fMRI) and auditory processing-related ERP components such as P/N1 and P/N2 (in EEG). Moreover, simpler acoustic features predicted primary regions and earlier time points better and vice versa more complex computational models, non-primary regions and later time points.

The obtained datasets provide a FAIR and holistic resource for the multimodal investigation of auditory perception, specifically music processing. It allows to examine respective complex aspects through the integration of diverse data types, including fMRI, EEG, behavioral data and computational models. This furthermore entails its feasibility as a benchmark and validation dataset.

Music ¹

Databasing and Data Sharing ²

Perception: Auditory/ Vestibular

Cognition

Computational Neuroscience

Electroencephaolography (EEG)

FUNCTIONAL MRI

Hearing

Open Data