Probabilistic prediction of cognitive impairment in continuous time for confidence quantification
Poster No:
1373
Submission Type:
Abstract Submission
Authors:
Bruno Hebling Vieira1, Dorothea Floris2, Camille Elleaume3, Franziskus Liem4, Gaël Varoquaux5, Nicolas Langer1
Institutions:
1University of Zurich, Zurich, Switzerland, 2Methods of Plasticity Research, Department of Psychology, University of Zürich, Zürich, Switzerland, 3University of Zurich, Zurich, Zurich, 4ETHZ, Zurich, Switzerland, 5SODA, Inria, Paris, France
First Author:
Co-Author(s):
Dorothea Floris
Methods of Plasticity Research, Department of Psychology, University of Zürich
Zürich, Switzerland
Methods of Plasticity Research, Department of Psychology, University of Zürich
Zürich, Switzerland
Introduction:
Machine learning models have been employed to great success in the prediction of cognitive impairment from demographic and familial risk-factors and imaging derived phenotypes (IDPs). Using MRI-based markers of brain atrophy, it is possible to obtain accurate continuous time classification of MCI and dementia patients¹, as well as predict subjects' trajectories in different staging and screening instruments²˒³.
To date most models have been trained to give point-predictions in fixed time windows, despite the added benefits of probabilistic prediction and continuous time trajectories, such as the assessment of confidence. This provides adjustable decision thresholds with guarantees for future translation of such models into actionable insights. In this work, we implement continuous time probabilistic prediction of cognitive impairment from brain regional morphometry and covariates.
To date most models have been trained to give point-predictions in fixed time windows, despite the added benefits of probabilistic prediction and continuous time trajectories, such as the assessment of confidence. This provides adjustable decision thresholds with guarantees for future translation of such models into actionable insights. In this work, we implement continuous time probabilistic prediction of cognitive impairment from brain regional morphometry and covariates.
Methods:
We employed T1w MRI data from ADNI (1, 2, GO, 3) for training and validation, while external testing was performed on images from OASIS⁴˒⁵ (3, 4). FreeSurfer 7.3.2 morphometry estimates were used as IDPs (see Figure 1.A).
We predicted Clinical Dementia Rating (CDR) Sum of Box Scores (SOB)⁶ scores from 4 sets of inputs: (1) IDPs (fs2cdr), (2) other CDR session (cdr2cdr), (3) both an IDP session and a different CDR session (fscdr2cdr), (4) covariates only (cov2cdr). Modeling was performed with gradient boosted decision trees from LightGBM (4.0.0), well suited for unstructured tabular data⁷ in Python (3.10.2), predicting the level of six CDR Box Scores. Covariates were included in all models, including patients' current age, sex, APOE2 and APOE4 allele counts, educational attainment, MRI scanner field strength, and the time difference(s) between the input and output time (see Figure 1.A-B).
We compared probabilistic expectations to a regression model that predicts the CDR-SOB directly in the same settings as before, to identify potential performance degradation in the probabilistic model.
We evaluated the calibration of predicting impairment in the future, i.e., p(CDR-SOB>0), to assess if the model is over- or under-confident in identifying abnormal cognition. Biological validity was assessed by the feature importance for the prediction of each CDR Box Score (see Figure 2).
We predicted Clinical Dementia Rating (CDR) Sum of Box Scores (SOB)⁶ scores from 4 sets of inputs: (1) IDPs (fs2cdr), (2) other CDR session (cdr2cdr), (3) both an IDP session and a different CDR session (fscdr2cdr), (4) covariates only (cov2cdr). Modeling was performed with gradient boosted decision trees from LightGBM (4.0.0), well suited for unstructured tabular data⁷ in Python (3.10.2), predicting the level of six CDR Box Scores. Covariates were included in all models, including patients' current age, sex, APOE2 and APOE4 allele counts, educational attainment, MRI scanner field strength, and the time difference(s) between the input and output time (see Figure 1.A-B).
We compared probabilistic expectations to a regression model that predicts the CDR-SOB directly in the same settings as before, to identify potential performance degradation in the probabilistic model.
We evaluated the calibration of predicting impairment in the future, i.e., p(CDR-SOB>0), to assess if the model is over- or under-confident in identifying abnormal cognition. Biological validity was assessed by the feature importance for the prediction of each CDR Box Score (see Figure 2).
·Figure 1
Results:
Probabilistic models maintain good performance for point-predictions, on the same level as direct regression models. In short time-windows, up to 5 years in the future, present CDR provides the best predictions of future CDR according to the observed Pearson correlation between true and predicted CDR-SOB. Approximately 5 years in the future onwards, the IDP-derived model (fs2cdr) surpasses the CDR-only model (cdr2cdr). Including both CDR and IDPs (fscdr2cdr) is optimal at all time scales (see Figure 1.C). We detected underestimation (in red), and overestimation (in blue) of the probability of impairment (see Figure 1.D). This information can be used to recalibrate the underlying model in future work.
The performance of point-predictions from probabilistic models can be explained by biological heterogeneity underlying different CDR Box Score domains⁸. Figure 2.A-B show feature importance for each CDR Box Score. Overall, highlighted regions resemble atrophy patterns observed in Alzheimer's. The left hippocampal volume is highly correlated with verbal episodic memory, while the right hippocampus volume correlates with spatial memory in older subjects⁹. High spatial similarity between importance scores can be explained by the positive correlation between Box Scores, shown in Figure 2.C.
The performance of point-predictions from probabilistic models can be explained by biological heterogeneity underlying different CDR Box Score domains⁸. Figure 2.A-B show feature importance for each CDR Box Score. Overall, highlighted regions resemble atrophy patterns observed in Alzheimer's. The left hippocampal volume is highly correlated with verbal episodic memory, while the right hippocampus volume correlates with spatial memory in older subjects⁹. High spatial similarity between importance scores can be explained by the positive correlation between Box Scores, shown in Figure 2.C.
·Figure 2
Conclusions:
IDPs provide optimal performance to predict cognitive impairment in longer time scales. Probabilistic predictions maintain accuracy of point-predictions with the benefit of confidence quantification. This work represents a significant step towards improving the translation of machine learning models into actionable insights, to enhance research on clinical trials and personalized medicine.
Disorders of the Nervous System:
Neurodegenerative/ Late Life (eg. Parkinson’s, Alzheimer’s) 2
Lifespan Development:
Aging
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Keywords:
ADULTS
Aging
Degenerative Disease
Machine Learning
Morphometrics
MRI
Neurological
STRUCTURAL MRI
1|2Indicates the priority used for review
Provide references using author date format
2. Vieira, B. H. (2022), 'Predicting future cognitive decline from non-brain and multimodal brain imaging data in healthy and pathological aging'. Neurobiology of Aging 118, 55–65.
3. Bhagwat, N. (2018), 'Modeling and prediction of clinical symptom trajectories in Alzheimer’s disease using longitudinal data'. PLoS Computational Biology 14.
4. Koenig, L. N. (2020), 'Select Atrophied Regions in Alzheimer disease (SARA): An improved volumetric model for identifying Alzheimer disease dementia'. NeuroImage Clinical, 26.
5. LaMontagne, P. J. (2019), 'OASIS-3: Longitudinal Neuroimaging, Clinical, and Cognitive Dataset for Normal Aging and Alzheimer Disease'. medRxiv Preprint
6. Cedarbaum, J. M. (2013), 'Rationale for use of the Clinical Dementia Rating Sum of Boxes as a primary outcome measure for Alzheimer’s disease clinical trials'. Alzheimer's Dementia. 9, S45–S55.
7. Grinsztajn, L. (2023), 'Why do tree-based models still outperform deep learning on typical tabular data?', Neurips.
8. Lowe, D. A. (2012), 'Greater Precision when Measuring Dementia Severity: Establishing Item Parameters for the Clinical Dementia Rating Scale'. Dementia and Geriatric Cognitive Disorders, 34, 128–134.
9. Ezzati, A. (2016), 'Differential association of left and right hippocampal volumes with verbal episodic and spatial memory in older adults'. Neuropsychologia 93, 380–385.