Identifying the Outliers: Insights from the Reanalysis of Tom et al.(2007) and Botvinik-Nezer (2020)
Poster No:
898
Submission Type:
Abstract Submission
Authors:
Chun-Chia Kung1, Hanshin Jo1, Le-Si Wang1
Institutions:
1National Cheng Kung University, Tainan, Taiwan
First Author:
Co-Author(s):
Introduction:
One of the prominent news of the year 2020's fMRI circle (2019~20) was the NARPS project (and the 2020 Nature paper), inviting 71 labs to co-analyze the same dataset (ds001734), as a registered replication of the original Tom et al. (2007 Science) study (ds005). The results, as was revealed in the Nature2020 paper, were a startling discrepancy: only 1 out of the 9 registered hypotheses was supported by the majority of the analysis teams. While the Nature2020 paper focused on the influences of the varieties of preprocessing pipelines among labs, the reasons for the discrepant fMRI results were still unclear. To find out, in the course of three fMRI graduate courses (2019 Spring, 2020 Fall, and 2023 Summer), the teacher and the students co-analyzed two then-released fMRI raw data from openneuro.org (ds005 and ds001734), with identical analysis pipelines, to dig further into this puzzle.
Methods:
The fMRI preprocessings were done with spm12, and the parametric modulation (both positive and negative) of group-wise GLM was applied by neuroelf under Matlab.
Results:
We successfully replicated two studies with comparable results (qualitative evaluation). Furthermore, the number-matched comparisons (e.g., randomly picking 16 out of the odd-numbered 54 subjects, and comparing with the Tom07 N=16, for 1000 times), also showed that the 4 hypothesized ROIs (Caudate, Insula, Amygdala, and mPFC) were mostly different. Along with the final leave-one-subject-out cross-validation ROI multi-voxel pattern analysis (MVPA), which showed only 6-9 subjects that were more different from the remaining 10-7 subjects.
·Fig. 1, Qualitative evaluations of the replication analysis and the original ones
·Fig. 2, Comparions of the same subject numbers (N=16) between the two studies.
Conclusions:
It was probably not the analysis package difference, nor were all the 16 UCLA subjects, as the primary reason for the resulting discrepancy. Rather, it was due more to a few outliers that resulted in the main difference. Taken together, these findings lend further support to the 'law of small numbers': that extreme results are more likely to happen in small-sample studies.
Higher Cognitive Functions:
Decision Making 1
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI) 2
Multivariate Approaches
Keywords:
ADULTS
Data analysis
FUNCTIONAL MRI
1|2Indicates the priority used for review
Provide references using author date format
Botvinik-Nezer, R, Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J. Johannesson, M. Kirchler, M., et al. 2020. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” Nature 582 (7810): 84–88.