Quantifying the impact of preprocessing strategies on statistical analyses

1951 

Abstract Submission 

Brice Ozenne¹, Martin Norgaard², Cyril Pernet¹, Melanie Ganz²

¹Neurobiology Research Unit, Rigshospitalet, Copenhagen, Denmark, ²University of Copenhagen, Copenhagen, Denmark

Brice Ozenne  
Neurobiology Research Unit, Rigshospitalet
Copenhagen, Denmark

Martin Norgaard  
University of Copenhagen
Copenhagen, Denmark

Cyril Pernet, PhD  
Neurobiology Research Unit, Rigshospitalet
Copenhagen, Denmark

Melanie Ganz  
University of Copenhagen
Copenhagen, Denmark

Preprocessing pipelines have become increasingly complex over the years. This multiverse of analytical pipelines naturally leads to a multitude of results. Here, we provide a conceptual framework and a practical tool to aggregate multiple pipeline results beyond simple averaging. The proposed framework is generic and can be applied to any multiverse scenario, but we illustrate its use based on positron emission tomography (PET) data.

We employ neuroimaging data from a placebo-controlled, double-blinded, clinical study (N=60) [1], measuring cerebral serotonin transporter (SERT) availability following a hormonal treatment. In a previous study [3] multiple valid processing steps (pipelines) were considered to measure SERT availability (see Figure 1). Here we sub-sampled pipelines using the reliability of results in the healthy/placebo arm of [1], while evaluating the results on intervention data (selection independent from testing).

Four methods were used to estimate a global effect across pipelines:
Ψaverage, average: a naive method would be to compute the mean of the estimated association
Ψse, weighted average: the estimated associations are averaged with weight inversely proportional to their standard error
ΨGLS, Generalized least squares (GLS): the estimated associations are average with weights accounting for their standard error and correlations. Highly correlated estimates will be given less weights than independent estimates.
Ψconstrained, constrained GLS: same as GLS but with weights constrained to be at most one to bound (from above) the influence of a single estimate.

We study the behavior of the four statistical estimators to estimate a common effect across pipelines in Figure 2. The dashed vertical line represents the normative value, and all horizontal error bars represent the estimated effect (mean and 95% CI) for a given pipeline and estimator. Across the set of pipelines, 3 of the 8 selected reject the null hypothesis (as indicated by the non-overlapping CI with the normative value) with estimated percent differences between groups ranging between -9% (pipeline 6) and 2.5% (pipeline 1). The Ψaverage, Ψse and Ψconstrained, all fail to reject a common effect across pipelines, whereas the ΨGLS estimator rejects the null hypothesis. However, when inspecting the pipeline weights for the ΨGLS estimator, it assigned a very high weight to four pipelines (i.e. weight above 1 in absolute values) leading to an unreliable estimate. The Ψconstrained estimator did not exhibit this problem.

Meta-analytical tools were developed mostly for independent data/studies, but in the analytical multiverse results are correlated which imposes constraints on estimations. Here we observed that 'usual' estimators can be readily applied, and three of them performed as expected based on the results from individual pipelines. Only the ΨGLS estimator performed differently, rejecting the null hypothesis hinting at an effect across pipelines. This is though, due to the estimator not being able to fit the weights properly in small sample sizes. Since PET neuroimaging studies are rarely beyond sample sizes of n>50, other estimators should be used. We recommend using Ψse when all pipelines are highly correlated or Ψconstrained when pipelines are not similarly related and the sample size is moderate to large.
The framework that we are proposing is not without limitations. The underlying assumption of combining results across pipelines in our analysis is that all pipelines are unbiased. This can however not always be guaranteed. Alternative approaches [4] can be used to reduce the bias of the pooled estimators by assuming a majority of unbiased pipelines or identifying clusters of pipelines and pooling pipeline-specific estimates within clusters.

Motion Correction and Preprocessing ¹

PET Modeling and Analysis

Other Methods ²

Workflows

Computational Neuroscience

Data analysis

Design and Analysis

Informatics

MRI

Neurotransmitter

Positron Emission Tomography (PET)

Psychiatric Disorders

Statistical Methods