Data Preparation: Missing Values, Outliers & Reproducibility

By Mathias Ellmann English version: 22 October 2026

Data & Data Science

Data preparation is part of the analytical method, not a neutral prelude. Filtering, imputation, outlier treatment, type conversion and aggregation can change both the population being analysed and the resulting metrics. The technical task is to make those decisions visible and reproducible.

Short answer

The examination keeps the raw or received data state separate from every derived analysis state. Missing values are classified by their technical origin and meaning before they are imputed, excluded or recoded. Outliers are first identified and investigated; they are not automatically removed merely because they are extreme.

A defensible preparation pipeline records the exact filtering and transformation logic, parameters and input versions. Sensitivity checks show whether the conclusion depends strongly on one preparation choice. The report then separates the preparation finding from the substantive interpretation of the analysis.

1 Define Fix the concrete subject, state and examination boundary.

2 Reconstruct Collect provenance, configuration, processing and other technical evidence.

3 Test Perform reproducible checks and counter-tests against the stated proposition.

4 Report Separate findings, supported inference, uncertainty and legal or normative boundaries.

1. Version raw data and analysis data separately

Preparation changes the evidence basis

The received or extracted dataset should be preserved as an identifiable reference state before preparation begins. Each material preparation stage receives a separate version or reproducible derivation path so that the original observations are not overwritten by cleaning decisions.

The examination records row counts, schema changes, dropped columns, filtered records and other scope changes between versions. This makes it possible to answer which observations were available at the start and which remained in the dataset used for the final analysis.

Where multiple teams or notebooks created different working copies, the relationship between those copies is reconstructed instead of assuming that similarly named files are equivalent.

2. Distinguish missing values by cause and meaning

Not every missing representation has the same technical interpretation

Missing values can arise because a measurement was not taken, a source system did not provide the field, an import failed, a value was not applicable, a user left an optional field blank or a transformation converted an invalid entry to missing. These mechanisms can have different analytical consequences.

The examination identifies technical encodings such as NULL, NaN, blank strings, sentinel values and absent JSON properties and then relates them to the domain meaning documented for the field. Treating every representation identically can create artificial completeness or bias.

Imputation is documented as a transformation with a stated method and scope. It does not recreate an observed fact; it creates an analytically substituted value whose influence should be considered in the result.

3. Mark outliers before removing them

Extreme values may be errors or valid rare observations

NIST guidance on exploratory data analysis treats outliers as observations that appear inconsistent with the rest of the data and emphasises investigation rather than automatic deletion. An extreme value can result from measurement error, data-entry error, unit mismatch or a genuine rare event.

The detection rule is recorded explicitly: statistical test, interquartile range, robust z-score, domain threshold or another criterion. The chosen method and its assumptions matter because different rules identify different observations.

Before exclusion, the examination checks source records, units, timestamps and related fields where available. If a value remains unexplained, retaining it with a sensitivity analysis can be more transparent than silently deleting it.

4. Make filters and transformations reproducible

A preparation rule should be executable or precisely specified

Filtering conditions, type conversions, parsing rules, normalization, joins, deduplication and feature calculations are tied to code, queries or versioned pipeline configuration. Manual spreadsheet edits or ad-hoc notebook steps are documented as evidence gaps if they cannot be reproduced.

Order matters. Removing duplicates before a join can produce a different result from joining first and deduplicating afterwards. The pipeline therefore records the sequence of operations as well as their parameters.

Reproduction uses the same input version and supporting reference data where possible. A successful rerun is compared not only by file existence but also by relevant row counts, summaries, checksums or selected record-level results.

5. Test sensitivity to preparation decisions

Alternative defensible choices can reveal fragile conclusions

The examination asks whether a material result changes when borderline outliers are retained, an alternative missing-value strategy is used, a reasonable threshold is moved or a filter is applied differently. The purpose is not to search indefinitely for every possible pipeline but to test the decisions most capable of changing the conclusion.

Large changes under small, defensible preparation variations indicate that the result depends strongly on the preprocessing choice. That dependence is itself a technical finding and should be visible in the assessment.

Conversely, stability across documented alternatives can support robustness of the result within the tested range, without proving general robustness outside that range.

6. Report preparation separately from the substantive conclusion

The preparation finding should remain reviewable

The final report states which data version was used, which observations or fields changed during preparation and why, which rules were applied and which sensitivity checks were performed. This allows a reader to reconstruct the analytical population.

A technical statement that an outlier rule was applied consistently is different from a domain conclusion that the excluded observations are irrelevant. Likewise, successful imputation does not establish that the imputed values represent the unobserved reality.

Preparation decisions that cannot be reconstructed are stated as limitations. The analytical conclusion is then bounded by the uncertainty about how the examined dataset was produced.

Is the original received data state preserved and identifiable?
Are derived analysis datasets versioned separately?
Are missing-value encodings and their meanings documented?
Is every imputation or exclusion rule reproducible?
Are outliers investigated before removal?
Is the order of filtering and transformations recorded?
Have material results been tested against reasonable alternative preparation choices?
Does the report distinguish a reproducible preparation step from the substantive interpretation of the result?

Constructed example

A transaction analysis excludes values above a fixed threshold as “outliers”. The examination finds that several excluded values correspond to legitimate annual invoices rather than input errors. Re-running the analysis with those records retained changes the mean substantially but has little effect on the median.

The report documents both variants, identifies the business explanation for the extreme values and states that the earlier conclusion based on the mean was sensitive to the exclusion rule. It does not label all extreme values as errors.

The example is entirely constructed and does not describe a real engagement.

Sources and technical basis

NIST statistical handbook NIST/SEMATECH e-Handbook material on outliers, used for the distinction between unusual observations and automatically invalid observations. Open source.
NIST methodology NIST/SEMATECH e-Handbook, Detection of Outliers, including assumptions and formal approaches to outlier detection. Open source.
W3C Recommendation W3C PROV-DM, used to describe derivation and processing steps between data states. Open source.
Author methodology Sources and methodology of the knowledge base; preparation decisions are treated as part of the technical derivation chain.

Critical review and methodological limitation

Data cleaning can make a dataset easier to analyse while simultaneously narrowing the evidentiary basis. The examination therefore treats removals, imputations and transformations as explicit methodological decisions rather than as invisible housekeeping.

Sensitivity testing does not determine the one “correct” preparation method. It shows whether the technical conclusion is stable within a defined set of defensible alternatives.

Missingness: distinguish technical encoding, cause and domain meaning before choosing treatment.
Outliers: detection is not proof of error; investigate source context where possible.
Sensitivity: report when material conclusions depend on a particular preparation choice.

Examination rule: Document exactly which observations and values were transformed or excluded and why. Do not describe an automated cleaning rule as proof that removed data was wrong or that imputed data reflects an observed fact.

Translation and methodological scope

This English article is based on the reviewed German V1 source article. The translation preserves the technical meaning, methodological distinctions, source status and stated limitations. It does not strengthen technical findings into legal conclusions and does not treat technical references as proof of compliance or suitability for a specific system.

Source relationship: The corresponding German article remains the reference baseline during English V1 development. Open the German source article.

Author's technical literature and publications

The following works provide additional technical and methodological context. They do not replace required primary sources, standards, manufacturer documentation or other independent specialist sources. Unpublished manuscripts, drafts and teaching materials are deliberately not presented here as public sources.

Data Science & AI Assessments

Technical examination of data quality, model validity, reproducibility, bias, robustness and documented AI-system behaviour.

Explore this assessment service

Technical Examination of Data Preparation, Missing Values and Outliers