Sampling bias and missing data: who is absent from the denominator?

Define the population and the selection route

Suppose a fictional organisation wants to estimate the proportion of all members who use a service. It invites one hundred members and receives sixty answers. The respondents are a subset of the invited group, which may itself be a subset of the intended population. The Penn State lesson on bias and random error distinguishes systematic selection problems from sampling fluctuation. More replies can help, but size alone does not establish representativeness if the collection route consistently excludes particular members or encourages only highly engaged people to respond.

Keep missing separate from zero

Among the sixty hypothetical respondents, suppose thirty report using the service. The observed respondent proportion is 30/60=50%. The forty nonrespondents are not automatically nonusers. Columbia University’s missing-data overview explains that incomplete observations can affect both estimates and precision. Coding every missing answer as zero changes the question and can introduce a strong assumption without acknowledging it. Preserve a missing-value indicator and, when known, the reason for nonresponse. A skipped question, an inapplicable question and a recorded negative answer are different states.

Compare explicit nonresponse scenarios

The table is a sensitivity exercise, not four competing measured estimates. It shows what the observed answers leave unresolved. The extremes bound the proportion for this fixed invited group if the thirty observed users are correctly recorded. They do not automatically bound a larger population that the invitation list failed to cover.

Understand missingness labels as assumptions

Missing completely at random describes a model in which missingness does not depend on the relevant observed or unobserved values. Missing at random allows dependence on observed information after conditioning appropriately. Missing not at random involves remaining dependence on unobserved values. These technical labels should not be assigned by visual inspection alone. A file can reveal patterns in observed variables without revealing the missing answers themselves. Explain the substantive reasons for an assumption and examine how conclusions change under plausible alternatives rather than treating a convenient label as a fact established by software.

Use adjustment methods with their limits visible

The UCL Centre for Longitudinal Studies guidance discusses approaches such as weighting and imputation under assumptions. Weighting can give greater influence to observed cases representing underrepresented groups when suitable information exists. Imputation models fill missing values in a way that should account for uncertainty, rather than pretending the replacements were directly observed. Neither procedure magically recovers the unknown truth. Report the variables and assumptions used, and distinguish completed values from collected values in the analysis record.

Avoid changing samples without noticing

Two models can produce different results partly because they use different subsets of complete observations. If one model adds a variable with many missing entries, its apparent improvement or reversal may reflect the changed sample as well as the added variable. Record the number of included cases for every analysis and compare eligibility rules. For grouped or repeated observations, also track how many people, households or devices remain, not merely the number of rows. A missingness report should match the actual unit of inference.

Treat nonresponse investigation as part of design

Before collecting data, identify likely barriers such as inaccessible questionnaires, unavailable contact details or inconvenient response windows. Document follow-up attempts and known reasons for absence without assuming that all refusals mean the same thing. When possible, compare available characteristics of respondents and nonrespondents. Such comparisons can reveal a problem but cannot prove all unmeasured differences have disappeared. The aim is to understand the selection process well enough to limit claims appropriately, not to produce a reassuring response-rate percentage detached from the survey’s purpose.

Sampling and missingness checklist

• Name the target and invited populations.
• Describe how observations entered the dataset.
• Keep missing, zero and inapplicable values distinct.
• Report response counts and changing analysis samples.
• State missingness assumptions explicitly.
• Document weighting or imputation choices.
• Use sensitivity scenarios where the unknowns matter.

The fictional survey supports a clear statement about respondents and a range of possible statements about the invited group. Keeping those claims separate is more informative than choosing a single attractive percentage. A careful report tells readers which conclusion comes directly from observed answers and which conclusion depends on assumptions about people whose answers were not obtained.

Sources and verification

• Bias, confounding and random error — Penn State — 2026-10-23
• Missing data and multiple imputation — Columbia University — 2026-10-23
• Handling missing data — University College London, Centre for Longitudinal Studies — 2026-10-23