Reconsidering reliability for brain-wide association studies under normative modeling
Poster No:
1696
Submission Type:
Abstract Submission
Authors:
Zi-Xuan Zhou1, Xi-Nian Zuo1
Institutions:
1Beijing Normal University, Beijing, China
First Author:
Co-Author:
Introduction:
Evidence indicated that effect sizes observed in brain-wide association studies (BWAS) are generally very small (Marek et al., 2022), and improving measurement reliability alone is not sufficient to boost effect sizes (thus reduce the required sample size) to meet the real effect sizes and the expectations of current practices.
We aim to re-evaluate the reliability issues in BWAS under normative modeling with age as the predictor variable, i.e., brain charts (Bethlehem et al., 2022). We will dive into the reliability/validity of brain charts, investigate the reliability in cross-sectional and longitudinal BWAS under chart-derived scores, and reveal a hidden space in longitudinal BWAS that could enhance effect sizes by orders of magnitude.
We aim to re-evaluate the reliability issues in BWAS under normative modeling with age as the predictor variable, i.e., brain charts (Bethlehem et al., 2022). We will dive into the reliability/validity of brain charts, investigate the reliability in cross-sectional and longitudinal BWAS under chart-derived scores, and reveal a hidden space in longitudinal BWAS that could enhance effect sizes by orders of magnitude.
Methods:
Mathematical derivations, simulations, and analysis are combined to demonstrate the utility of our methods to the ABCD datasets. The essence of the mathematical derivations will be covered in the Results section, with detailed derivations/explanations provided in the poster/video. For simulations, we established parameter spaces based on the evidence of longitudinal changes and reliabilities of structural MRI phenotypes from the literature (Bethlehem et al., 2022; Biase et al., 2023; Madan et al., 2017; Hedges et al., 2022) and from the analysis of ABCD data. We repeatedly sampled the space and build developmental brain charts to derive variability and bias in the modeling and assessed the impact of measurement noise on chart-based inferences. Finally, the global structural MRI phenotypes of all samples from ABCD Data Release 5.0 that passed quality control were evaluated. To establish benchmarks for their longitudinal changes, in some cases, paired samples (two measurements from an individual) were selected for the evaluations.
Results:
Accurate brain charts require sample sizes in the tens of thousands, appropriate methodologies, and significant population-level longitudinal changes of the phenotypes over the age span. Poor reliability increases observed inter-individual variabilities and reduces relative population-level changes, thus necessitating a larger sample size for accurate models. It also significantly alters the estimated distribution (which is more spread out due to noise) and leads to bias in statistical inferences. Fortunately, the bias can be derived and corrected given the reliability value.
For cross-sectional BWAS under chart-derived scores, the reliability can be reduced as individual differences related to covariates such as age/sex are removed. For longitudinal BWAS, which we defined here as studies linking intra-individual longitudinal brain changes to non-brain measures, this effectively eliminate confounds related to, e.g., massive innate and early-life random factors. Removing population-level changes with chart-derived scores allows better exploration of more subtle person-specific changes. However, despite population-level changes can be accurately mapped with large-scale samples, measured intra-individual changes are drowned by measurement noise that is much larger than the population-level changes. Observed longitudinal changes in chart-derived scores in ABCD data align with the noise-driven changes (Fig. 1) and are very weakly associated with the interscan intervals, suggesting that time-independent noise-like changes dominate the changes and longitudinal BWAS face severe challenges of measurement reliability: noise can be orders of magnitude larger than true person-specific changes, burying the validity of research by sharply reducing effect sizes and reproducibility.
For cross-sectional BWAS under chart-derived scores, the reliability can be reduced as individual differences related to covariates such as age/sex are removed. For longitudinal BWAS, which we defined here as studies linking intra-individual longitudinal brain changes to non-brain measures, this effectively eliminate confounds related to, e.g., massive innate and early-life random factors. Removing population-level changes with chart-derived scores allows better exploration of more subtle person-specific changes. However, despite population-level changes can be accurately mapped with large-scale samples, measured intra-individual changes are drowned by measurement noise that is much larger than the population-level changes. Observed longitudinal changes in chart-derived scores in ABCD data align with the noise-driven changes (Fig. 1) and are very weakly associated with the interscan intervals, suggesting that time-independent noise-like changes dominate the changes and longitudinal BWAS face severe challenges of measurement reliability: noise can be orders of magnitude larger than true person-specific changes, burying the validity of research by sharply reducing effect sizes and reproducibility.
Conclusions:
Our findings suggest a clear path to substantially boosting effect sizes in BWAS, especially longitudinal BWAS, by simply improving reliability under chart-derived scores. This indicates the priority of improving measurement reliability and the prospect of lifting innovative small-scale studies with a priori information from population imaging data, such as lifespan brain charts.
Lifespan Development:
Early life, Adolescence, Aging
Lifespan Development Other
Modeling and Analysis Methods:
Exploratory Modeling and Artifact Removal 1
Methods Development 2
Keywords:
Data analysis
Experimental Design
MRI
STRUCTURAL MRI
Other - measurement reliability; normative modeling; brain chart; brain-wide association studies; longitudinal change; effect size
1|2Indicates the priority used for review
·Noise-driven cumulative distribution curves of the difference in centile scores between two measurements.
Provide references using author date format
Biase, M.A.D. (2023), 'Mapping human brain charts cross-sectionally and longitudinally', Proceedings of the National Academy of Sciences of the United States of America, vol. 120, no. 20, p. e2216798120.
Hedges, E.P. (2022), 'Reliability of structural MRI measurements: The effects of scan session, head tilt, inter-scan interval, acquisition sequence, FreeSurfer version and processing stream', NeuroImage, vol. 246, p. 118751.
Madan, C.R. (2017), 'Test–retest reliability of brain morphology estimates', Brain Informatics, vol. 4, pp. 107-121.
Marek, S. (2022), 'Reproducible brain-wide association studies require thousands of individuals', Nature, vol. 603, pp. 654-660.