Directional Errors with False Discovery Rate on Two-Tailed Tests
Poster No:
1870
Submission Type:
Abstract Submission
Authors:
Anderson Winkler1, Paul Taylor2, Thomas Nichols3, Chris Rorden4
Institutions:
1University of Texas Rio Grande Valley, Brownsville, TX, 2National Institute of Health, Bethesda, MD, 3University of Oxford, Oxford, United Kingdom, 4University of South Carolina, Columbia, SC
First Author:
Co-Author(s):
Introduction:
False discovery rate (FDR) is widely used in neuroimaging to address multiple testing. Many of these analyses also involve two-tailed hypotheses. In general, two-sided tests utilize symmetric thresholds (the same threshold for both tails), such that FDR adjustments would be applied symmetrically. However, in practice, effects being examined can be asymmetric, leading to asymmetric false positive rates that can, by large margin, exceed the nominal test level. It remains unclear how to control such directional errors with FDR. Moreover, most analytic software tools do not provide for the use of asymmetrical thresholds, even for corrected data, which limits the ability to investigate directional effects. Here, we investigate various scenarios of asymmetry in estimated results and strategies for performing FDR adjustments, including a novel approach called "split-tails". We apply these methods to both simulations and to a real FMRI dataset.
Methods:
We considered four strategies, which we termed "canonical", "combined", "two-tailed", and "split-tails", for handling two-tailed tests with the Benjamini-Hochberg (BH) and Benjamini-Krieger-Yekutieli (BKY) procedures in simulations and on real imaging data. Ten scenarios were simulated with varying proportions of true positive/negative effects, balancedness, and test dependencies. False positives were evaluated globally and separately by direction (i.e., positive and negative). Real data came from a gambling task analysis done as part of the Neuroimaging Analysis Replication and Prediction Study (NARPS).
Results:
Simulations demonstrated that, under complete null (no effects), canonical and split-tails controlled false positives at the nominal level when each side was considered separately, but the error rate was doubled when all tests were considered. The combined and two-tailed methods maintained error control globally, but became conservative on each side. With signal in only one side, all strategies were conservative on the side with signal, but two-tailed and combined strategies became invalid in the opposite side, approaching 100% error rate; the canonical and split-tails methods performed well in these cases. Even with balanced effects on both sides, while the strategies controlled the error rate, they leaned towards conservativeness. With unbalanced amounts of signal, the error rate was not controlled for two-tailed and combined strategies in the side opposite to the one with a preponderance of true signal.
On real data (Figure), uncorrected thresholds suggested widespread negative and sparse positive activation related to the gambling task. However, among corrected results, no positive effects remained significant with canonical/split-tails approaches, while combined/two-tailed analyses retained some. In light of the simulation results, these effects on the positive side are likely false positives. Similar results were seen with the BKY procedure, although it was generally more powerful than BH.
On real data (Figure), uncorrected thresholds suggested widespread negative and sparse positive activation related to the gambling task. However, among corrected results, no positive effects remained significant with canonical/split-tails approaches, while combined/two-tailed analyses retained some. In light of the simulation results, these effects on the positive side are likely false positives. Similar results were seen with the BKY procedure, although it was generally more powerful than BH.
·Results of the different correction strategies for directional inference as applied to NARPS data.
Conclusions:
Caution is needed when making directional statements from two-tailed tests with FDR correction, as errors are only controlled globally, not by tail. This is a consequence of FDR lacking localizing power, providing only weak control over the error rate. Asymmetrical thresholds should be used in neuroimaging tools to allow for FDR-based directional analyses even if two-tailed tests are used. Strategies that run FDR separately by side of the original hypotheses or by the sign of the resulting statistical test preserve control over directional error rates, albeit with increased risk of errors under the complete null.
Modeling and Analysis Methods:
Methods Development 1
Univariate Modeling 2
Keywords:
Data analysis
Design and Analysis
Statistical Methods
1|2Indicates the priority used for review
Provide references using author date format
Benjamini, Y., Krieger, A.M., Yekutieli, D. (2006), 'Adaptive linear step-up procedures that control the false discovery rate', Biometrika, vol. 93, pp. 491-507.