Multiple testing and p-hacking: why the analysis search must be visible

Define what a p-value addresses

A p-value summarises how incompatible the observed data are with a specified statistical model through a chosen test statistic. It is not the probability that the null hypothesis is true, and it does not measure effect size. The American Statistical Association statement on p-values emphasises those distinctions and the importance of context. A meaningful report identifies the null model, statistic, sampling assumptions and analysis choices. A small number without those details cannot establish that a finding is important, causal or likely to reproduce.

Count the opportunities for a false positive

Suppose twenty hypothetical tests all concern true null hypotheses, each has a five-percent rejection probability under its null, and the rejection events are independent. The probability of no rejection is 0.95 raised to the twentieth power. Taking the complement gives about 64.15% for at least one rejection. This is a statement about the complete family under explicit assumptions. It does not mean that every collection of twenty real analyses has exactly that probability, since dependence, discrete tests and changing analysis rules can alter the calculation.

Compare families of different sizes

The calculation is 1−0.95ᵐ for m tests. The table illustrates why selecting one result from a larger search is different from conducting one specified test. It is not an empirical estimate of misconduct, research quality or the reliability of a particular scientific field.

Understand a simple family-wise correction

The NIST explanation of the Bonferroni method describes a way to control a family-level error probability. For twenty valid tests and a desired family-wise bound of 0.05, a simple Bonferroni rule compares each p-value with 0.05/20=0.0025. The underlying union bound does not require independence between tests. The method can be conservative, and it is not the only valid multiplicity procedure. The family and scientific question should be defined thoughtfully rather than choosing a correction retrospectively to produce a preferred conclusion.

Distinguish exploration from selective concealment

Exploratory work can be scientifically useful: patterns can suggest new questions, measurement improvements or hypotheses for later study. The problem is presenting an outcome chosen after a broad search as though it were the only prespecified target. Disclose outcomes examined, exclusion rules, transformations and subgroup analyses. Where an analysis plan was recorded before seeing results, retain its date and explain departures. The interpretation of a discovery becomes clearer when readers can see how it was selected, including analyses that did not produce an attention-grabbing result.

Include decisions that change the search space

Multiplicity is not limited to a visible list of hypothesis tests. Repeatedly inspecting accumulating data and deciding when to stop can require a sequential method. Choosing among several plausible models, excluding observations after inspecting results or trying many outcome definitions can create additional selection. Not every analytical decision is improper, but relevant choices should be described. A legitimate adaptation needs a method that matches the adaptation. Repeating a fixed-sample formula after each new observation does not automatically preserve its original error properties.

Report estimates and reproducible reasoning

The National Academies report on reproducibility and replicability distinguishes reproducing calculations from obtaining consistent findings with new data. Share enough of the analysis record to reproduce the reported figures, subject to appropriate data restrictions. Include effect estimates, uncertainty, sample definition and limitations rather than reducing the result to whether one threshold was crossed. An adjusted p-value still does not measure practical importance. Independent data can examine a selected finding, but a new study also needs clear targets and suitable design.

Multiplicity and transparency checklist

• State the scientific question and analysis family.
• Identify prespecified and exploratory analyses.
• Report the number and types of comparisons considered.
• Use a correction or design appropriate to the error target.
• Disclose stopping, exclusion and model-selection rules.
• Present effect sizes and uncertainty with p-values.
• Preserve the complete calculation record.

Readers should be able to tell whether a displayed result was selected because it supported a conclusion or because it answered a question defined in advance. That distinction does not erase the value of exploration. It makes the strength and purpose of the evidence assessable without pretending that a single threshold can settle every scientific judgment.

Sources and verification

• ASA statement release on p-values, 7 March 2016 — American Statistical Association — 2026-10-23
• Bonferroni method — NIST/SEMATECH e-Handbook — 2026-10-23
• Reproducibility and Replicability in Science: summary — National Academies of Sciences, Engineering, and Medicine — 2026-10-23