A transformer-based meta-matching stacking method improves prediction accuracy of fluid intelligence

1451 

Abstract Submission 

Juhyuk Han¹, Minjae Kim², Won Hee Lee³

¹KyungHee University, Gyeonggi-do, Republic of Korea, Korea, Republic of, ²Kyung Hee University, Yong-in, Yong-in, ³KyungHee University, Yonginsi, Yonginsi

Juhyuk Han  
KyungHee University
Gyeonggi-do, Republic of Korea, Korea, Republic of

Minjae Kim  
Kyung Hee University
Yong-in, Yong-in

Won Hee Lee  
KyungHee University
Yonginsi, Yonginsi

The proliferation of datasets integrating both imaging and non-imagining phenotypes has provided a rich resource for advancing predictive modeling in neuroscience. The meta-matching framework has emerged as a promising avenue, offering the potential to harness models trained on large-scale datasets for predicting non-imaging phenotypes within smaller datasets [1]. Despite its promise, the conventional application of the meta-matching framework to a single modality has revealed limitations, particularly in integrating and exploiting information from diverse modalities. To address this gap, we propose a new stacking approach, termed multimodal transformer stacking, which incorporates self-attention [2] to extract meaningful patterns and relationships across multiple modalities, advancing its potential for accurate predictions of non-imaging phenotypes.

We used 58 non-imaging phenotypes and multimodal imaging features of 750 subjects from the HCP dataset [3]. We computed a new set of structural features derived from the same HCP participants, namely morphometric inverse divergence networks (MIND) [4] and morphometric similarity networks [5]. We used 16 diffusion features derived from diffusion MRI and 6 functional features derived from each task and resting-state fMRI scan. Details about the imaging features derived from T1-weighted, diffusion, and functional MRI scans can be found elsewhere [3]. We divided the data into the training meta-set with 600 participants and 57 phenotypes, and test meta-set with 150 participants and fluid intelligence selected as the target phenotype. We subdivided the test meta-set into a K-shot sample set (n = 100) and a test set (n = 50). We initiated our approach with the advanced stacking method using each imaging feature. In this process, the nodes with the highest coefficient of determination (COD) from the basic deep neural network were selected. A kernel ridge regression was trained on a K-shot sample set using the predictions for the chosen nodes. Across diverse imaging features, we identified those yielding high performance in a test set. These were integrated into the multimodal transformer stacking. Predictions from the five imaging features were concatenated to form a multimodal feature matrix. This matrix entered into a transformer encoder to learn relationships among the phenotypes through self-attention across predicted phenotypes from multiple modalities. The resulting matrix was employed to predict fluid intelligence through a dense layer. The process was repeated 50 times, and the correlations and CODs with the true score were averaged over these 50 repetitions. Our approach was compared against existing methods [1].

Traditional advanced stacking methods resulted in better performance for predicting fluid intelligence, when using MIND (r = 0.09), TBSS axial diffusivity (r = 0.22), tractography axial diffusivity (r = 0.19), resting-state (r= 0.29) and working memory fMRI features (r = 0.37). These were further used as features for the multimodal transformer stacking. Our results demonstrate that multimodal transformer stacking (r = 0.41) outperforms both stacking average and traditional advanced stacking of individual features (r: t = 3.13 – 11.11, p < 0.05; COD: t = 3.12 – 9.77, p < 0.05) (Figure 2). This suggests that the transformer encoder's ability to capture long-range dependencies and relationships among phenotypes from different modalities contributes to improved predictive performance.

We proposed a new stacking strategy for integrating information from diverse imaging modalities to improve the prediction of fluid intelligence. Stacking average averages the predictions from individual models, which may not fully account for the interactions and relationships between the modalities. Our proposed approach has the potential to learn more complex relationships between the features and capture more subtle patterns and interactions for improved prediction of cognitive functioning.

Classification and Predictive Modeling ¹

Connectivity (eg. functional, effective, structural) ²

Other - Meta Matching; non-imaging phenotype; brain imaging data