Activation Likelihood Estimation: How unbalanced dataset-sizes impact meta-analytic contrasts

1896 

Abstract Submission 

Lennart Frahm¹, Simon Eickhoff², Robert Langner³, Veronika Müller³, Theodore Satterthwaite⁴, Peter Fox⁵

¹Forschungszenturm Jülich, Jülich, North-Rhine Westphalia, ²Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf, Düsseldorf, North Rhine–Westphalia Land, ³Inst. Sys. Neurosci., Medical Faculty, HHU Düsseldorf, Düsseldorf, North Rhine-Westphalia, ⁴UPenn, Philadelphia, PA, ⁵The University of Texas Health Science Center at San Antonio, San Antonio, TX

Lennart Frahm  
Forschungszenturm Jülich
Jülich, North-Rhine Westphalia

Simon Eickhoff  
Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf
Düsseldorf, North Rhine–Westphalia Land

Robert Langner  
Inst. Sys. Neurosci., Medical Faculty, HHU Düsseldorf
Düsseldorf, North Rhine-Westphalia

Veronika Müller  
Inst. Sys. Neurosci., Medical Faculty, HHU Düsseldorf
Düsseldorf, North Rhine-Westphalia

Theodore Satterthwaite  
UPenn
Philadelphia, PA

Peter Fox, MD  
The University of Texas Health Science Center at San Antonio
San Antonio, TX

The Activation Likelihood Estimation (ALE) algorithm for neuroimaging meta-analysis has frequently been used to delineate differences in convergence of brain activity between cognitive tasks, domains or groups by virtue of meta-analytic contrasts [1,2,3]. Significance of contrasts is determined by comparing the voxel-wise differences between two ALE maps to an empirical null distribution of ALE score differences. This null distribution is approximated by randomly permuting the experiments between the two datasets 10,000 times and calculating difference scores for each permutation. While being statistically sound and often yielding results with high face validity, there have been concerns that this approach is not suitable when comparing two datasets with very different sizes [4], with contrasts being driven by the larger dataset. To address these concerns, we developed a new, balanced meta-analytic contrast algorithm, in which we calculate 5000 individual contrasts based on same-sized subsamples of the original datasets and then average the results. Such "undersampling" approaches are well established in machine learning literature [5] when dealing with imbalanced class predictions, which are conceptually similar to our use case. Here, we compared the original permutation-based contrasts with the newly developed balanced contrasts, using simulated ALE datasets.

We simulated datasets with two different sizes, including either 150 or 25 experiments. To manipulate the amount of spatial convergence, 0%, 10%, 20%, 30% or 40% of experiments per dataset were selected to feature a coordinate at a "true location" [6,7]. For each dataset size and amount of convergence we created 100 datasets of experiments whose sample size and number of foci included were randomly sampled from normal distributions similar to what is found in empirical datasets according to the BrainMap database [6,8,9]. We then computed both permutation-based and balanced contrasts for each dataset with 150 experiments against a dataset with 25 experiments. Lastly, we examined the proportion of significant differences at the "true location", averaging over all contrasts with the same amount of convergence per dataset.

In general, both contrast-algorithms performed reasonably well, reliably detecting large convergence differences independent of dataset size. Both algorithms slightly favored the larger dataset, reporting more contrasts in its favor even for less strong convergence differences. This effect was slightly stronger for the balanced contrasts. The biggest difference between the two algorithms seems to be that the balanced contrast is generally more sensitive, detecting more significant contrasts even if there is only slightly larger convergence in one of the datasets, whereas the unbalanced permutation contrasts would often yield no significant differences in these cases.

In contrast to previous concerns, we found only little evidence that unbalanced permutation-based contrasts are strongly driven by the larger dataset. This is an important finding, retrospectively validating ALE meta-analyses that employed this methodology. The newly developed balanced subsampling contrast also performed well, reliably detecting differences in convergence between the two datasets. However, its moderately higher sensitivity, relative to the traditional approach, comes at the expense of featuring a somewhat stronger bias toward detecting significant differences in favor of larger datasets. At this moment there is no clearly preferred algorithm choice from our point of view. Future research should look into specific use cases for both contrast algorithm and validate both approaches using a large number of real-life ALE datasets.

Methods Development ¹

Other Methods ²

Other - meta-analysis; activation likelihood estimation