Tools and platforms to investigate analytical variability in neuroimaging
Poster No:
2248
Submission Type:
Abstract Submission
Authors:
Jacob Sanz-Robinson1, Jean-Baptiste Poline1, Tristan Glatard2
Institutions:
1McGill University, Montreal, Quebec, 2Concordia University, Montreal, Quebec
First Author:
Co-Author(s):
Introduction:
The reproducibility of neuroimaging studies is often limited by analytical variability [1, 2]. Researchers have access to a multitude of tools, many of which carry out the same tasks but yield different results when applied to the same data [3-8]. Despite the neuroimaging community's array of tools to investigate and address analytical variability, these resources are decentralized and scattered [9]. Consequently, researchers often lack the necessary information and protocols to buttress the reliability of their findings. This review catalogs software tools neuroimaging researchers can use to address result variability arising from computational pipelines and environments. The aim of the review is to promote robust reproducibility practices by highlighting relevant strategies and reducing accessibility barriers.
Methods:
The tools featured in the review were found through an assortment of queries in the NITRC and NIF databases, PubMed, Google Scholar, the background information sections of relevant papers, and suggestions and presentations from the neuroimaging reproducibility community [10]. The inclusion criteria we use are: 1) At least one descriptive publication/DOI exists, 2) Open-source and publicly available, 3) At least one release or repository update in the past 5 years, and 4) At least one published neuroimaging study has used the tool.
We found 30 tools fitting the criteria above and categorized them into four main sections:
• 'Tool variability' stems from analytical choices regarding software-based algorithms, like data processing libraries and pipelines, along with parameters and versions. This section is divided into three subsections. The first subsection describes container technologies and tool repositories that facilitate the distribution and reproducible execution of large collections of tools. It features subdivisions on container-solutions, integrated computational environments, and software collections. Another subsection describes Workflow Engines that provide machine-readable ways of encapsulating and automatedly re-executing sequences of processings. The third subsection is about tools to leverage Continuous Integration by using it to frame a scientific result as an automated "test", keeping track of how a result is affected by changes in the underlying computations.
• 'Environment variability' is caused by the software infrastructure that is used to execute processings. It contains two subsections. One features tools to measure Numerical Stability in algorithms that occur due to varying implementations of floating-point arithmetic on different operating systems and hardware. The other features tools to facilitate running computations across multiple development environments and operating systems.
• The 'Data Provenance' section describes tools that track transformations in data, providing an audit trail of processings.
• The 'Cloud Computing' section features neuroimaging platforms that facilitate performing analyses across many workflows by providing access to computational resources and lowering the technical skill level required to access and execute tools.
We also discuss Open Issues/Limitations of software-based approaches to analytical variability.
We found 30 tools fitting the criteria above and categorized them into four main sections:
• 'Tool variability' stems from analytical choices regarding software-based algorithms, like data processing libraries and pipelines, along with parameters and versions. This section is divided into three subsections. The first subsection describes container technologies and tool repositories that facilitate the distribution and reproducible execution of large collections of tools. It features subdivisions on container-solutions, integrated computational environments, and software collections. Another subsection describes Workflow Engines that provide machine-readable ways of encapsulating and automatedly re-executing sequences of processings. The third subsection is about tools to leverage Continuous Integration by using it to frame a scientific result as an automated "test", keeping track of how a result is affected by changes in the underlying computations.
• 'Environment variability' is caused by the software infrastructure that is used to execute processings. It contains two subsections. One features tools to measure Numerical Stability in algorithms that occur due to varying implementations of floating-point arithmetic on different operating systems and hardware. The other features tools to facilitate running computations across multiple development environments and operating systems.
• The 'Data Provenance' section describes tools that track transformations in data, providing an audit trail of processings.
• The 'Cloud Computing' section features neuroimaging platforms that facilitate performing analyses across many workflows by providing access to computational resources and lowering the technical skill level required to access and execute tools.
We also discuss Open Issues/Limitations of software-based approaches to analytical variability.
Results:
The listing and categorization of the tools is summarized in Fig. 1. Fig. 2 depicts inter-category tool relationships in this study, based on data flow in a neuroimaging pipeline.
·Fig. 1: The structure of the review, listing the tools and their corresponding categorizations.
·Fig. 2 : A flowchart showing the categories of tools to investigate software-based analytical variability in neuroimaging analysis workflows.
Conclusions:
As the magnitude and ubiquity of the reproducibility issues posed by analytical flexibility become clearer, neuroscientists have the responsibility of adapting their computational methods to make sense of the multiverse of analytical approaches available. Despite difficulties in establishing consensus, available tools can aid in creating, organizing, and executing processings that quantify and/or constrain analytical variability, fostering robust scientific conclusions. This study offers descriptions and guidance on the many tools enabling the study of analytical flexibility.
Neuroinformatics and Data Sharing:
Workflows 2
Informatics Other 1
Keywords:
Computational Neuroscience
Computing
Data analysis
Data Organization
Informatics
Open Data
Open-Source Code
Open-Source Software
Workflows
Other - Reproducibility, Analytical Flexibility
1|2Indicates the priority used for review
Provide references using author date format
[2] Schilling, Kurt G., et al. Tractography dissection variability: What happens when 42 groups dissect 14 white matter bundles on the same dataset?. NeuroImage 243 (2021).
[3] Kennedy, D. N. et al. Everything Matters: The ReproNim Perspective on Reproducible Neuroimaging. Front. Neuroinform. 13 (2019).
[4] Carp, J. On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Frontiers in neuroscience 6 (2012).
[5] Bowring, Alexander, et al. Exploring the impact of analysis software on task fMRI results. Human Brain Mapping v.40:11 (2019)
[6] Glatard, Tristan, et al. Reproducibility of neuroimaging analyses across operating systems. Frontiers in Neuroinformatics 9 (2015)
[7] Kiar, Gregory, et al. Comparing perturbation models for evaluating stability of neuroimaging pipelines. The International Journal of High Performance Computing Applications 34:5 (2020)
[8] Salari, Ali, el al. File-based localization of numerical perturbations in data analysis pipelines. Gigascience (2020).
[9] Niso, Guiomar, et al. Open and reproducible neuroimaging: From study inception to publication. NeuroImage 263 (2022)
[10] Repronim, “Repronim Community”, ReproNim: A Center for Reproducible Neuroimaging Computation, https://www.repronim.org/community.html (2023)