Reproducibility and Provenance

#Define what can be repeated

A reproducible computational analysis allows the same data and stated methods to produce consistent results. Terminology varies across fields, so explain what is meant rather than relying on the label. Testing a finding with newly collected data asks a different question: whether the finding holds beyond the original observations.

Provenance is the record of where data and results came from. It connects the original source to cleaning, transformations, analysis and reported outputs. This record helps identify mistakes and explain differences between versions. A workflow that runs successfully can still be biased or based on unsuitable data.

#Record inputs, decisions and versions

Maintain descriptions of data sources, collection conditions, permissions and inclusion rules. Record variable definitions, units and missing-value codes. Preserve an appropriate audit trail for corrections and exclusions. When data are combined, document how records were linked and how duplicate entries or conflicting information were handled.

Analysis records should identify code versions, software dependencies, settings and the exact input version used. For models, record training procedures, fitted parameters and relevant sources of randomness. Link tables and figures to the analyses that generated them. Automated checks can help detect unexpected changes, but they do not replace scientific review.

#Enable checking without compromising privacy

Sharing code, documentation and suitable data can make independent checking easier. Health information may require consent restrictions, controlled access or other safeguards. Removing obvious identifiers does not always eliminate the risk of identifying someone. Explain what can be shared, under which conditions and what remains unavailable.

When access is restricted, a detailed data dictionary, workflow description and appropriate test data may still help others assess the process. Test data cannot establish that results on the original data are correct. Report known limits to repeatability, including hardware or software differences, and preserve clear records of updates rather than silently replacing earlier results.

#Common misunderstandings

Reproducibility does not mean that a finding is necessarily correct. An analysis can produce the same result every time while relying on incomplete data, unsuitable methods or a repeated coding error. Repeating the workflow helps reveal what was done; it does not replace checking whether the approach answers the research question.

Reproducing an analysis using its original data is also different from testing a finding in a new study. New participants, settings or measurement methods can lead to different results without automatically showing that either study was poorly conducted.

Provenance means documenting where data came from and how they changed. It is more than listing sources: exclusions, corrections and transformations can affect the conclusions. Nor does transparency require publishing identifiable health records. Clear documentation, controlled access and carefully designed example datasets can support checking while protecting privacy. However, example data alone cannot establish that the original findings can be reproduced.

#Questions worth asking a clinician

• Can you trace a reported result back to the source data, dataset version and analysis code version used to produce it?
• Where are data cleaning steps, transformations and exclusions documented, including the reasons for excluding records?
• Which software versions and dependencies are needed to rerun the analysis, and how are they preserved?
• What checks address potential biases that could persist even if someone reproduces the analysis exactly?
• How can independent researchers check the analysis while protecting participants’ sensitive information?