Data as a Subject of Examination: Provenance, Data State and Reproducibility

Data & Data Science

Data is not an abstract input. A technical examination concerns a concrete data state with a traceable origin, a defined acquisition or transfer path and documented transformations. The first task is therefore to determine what dataset was actually examined and how it differs from earlier, later or derived versions.

1. Identify and delimit the concrete dataset

Origin and content must be documented separately

The subject of examination should be described so that another technically competent person can distinguish it from neighbouring data states. Useful properties include source system, export mechanism, file or table identifiers, represented time range, extraction time, encoding, schema version and the number of rows or objects. For large or distributed datasets, manifests and partition inventories can define the examined scope more reliably than a single filename.

The period represented by the data must be separated from the time at which the export was generated and from the date on which the examination takes place. A database extract produced today can describe events from an earlier period, while a historical export can contain records that were updated later in the source system. Mixing these time references can create false chronological conclusions.

Raw data, working copies and analysis datasets should receive different identifiers. An analysis table that has been filtered, joined, deduplicated or enriched is a derived state and should not silently inherit the identity of the original source.

2. Reconstruct provenance and acquisition

The acquisition process belongs to the examination basis

Provenance describes how the examined data came into existence and through which activities or agents it reached the current state. W3C PROV provides a conceptual vocabulary for entities, activities, agents and derivations. In an examination, this can be mapped to source databases, export jobs, API calls, ETL pipelines, analysts, scripts and generated files.

The transfer itself can change the technical evidence. Character encodings, date conversions, CSV quoting, spreadsheet imports, API pagination or database dump options can alter values or omit records. The examination therefore records not only the source but also the method by which the data was acquired.

Where available, export logs, job definitions, API request parameters, database queries, manifests and timestamps support the reconstruction. If only the resulting file exists, the provenance statement must remain correspondingly limited.

3. Secure version, identity and integrity

A filename is not an immutable dataset identifier

A stable technical reference may combine a descriptive identifier with a checksum, size, record count and creation or acquisition timestamp. For file collections, a manifest such as the approach illustrated by RFC 8493 BagIt can document a payload inventory and checksums for the included files.

Checksums are powerful for detecting byte-level changes between two states. They do not establish the origin, completeness, accuracy or business meaning of the content. Two incorrect copies can have identical hashes; a correctly transformed derivative necessarily has a different hash from its input.

Database states require additional care because a live database changes continuously. Snapshots, transaction-consistent exports, database backup identifiers or immutable object-store versions can provide a more defensible reference point than a query executed at an unspecified time.

4. Make transformations reproducible

Transformation must be described as part of the evidence chain

Filtering, joins, type conversions, normalization, aggregation, missing-value handling and feature engineering can materially change the basis of an analysis. Each relevant step should therefore be tied to an executable script, query, notebook, pipeline configuration or an equivalent precise description.

Reproducibility requires more than the presence of source code. Input versions, software dependencies, parameters, locale and time-zone settings, random seeds and external reference tables can affect the output. A script that cannot be associated with the examined data state is weak evidence for how that state was produced.

A practical counter-check is to reproduce selected intermediate and final outputs from the documented inputs. Differences are then investigated as evidence about hidden state, undocumented manual steps or environment dependencies rather than being normalised away.

5. Relate the data state to quality and analytical use

Provenance improves traceability but is not itself a quality result

A well-documented lineage can show where values came from without proving that they are complete, consistent, accurate or sufficiently current. Data quality therefore remains a separate examination layer with its own dimensions, reference values and metrics.

The relevant quality properties also depend on the intended analysis. A dataset can be adequate for a coarse historical count but unsuitable for a per-customer reconciliation if identifiers are missing. The examination should connect the data state to the concrete analytical question instead of assigning a generic quality label.

Derived datasets are assessed with awareness of transformation effects. Deduplication can improve one metric while removing legitimate repeated events; an inner join can simplify analysis while excluding unmatched records. The direction and magnitude of these effects should be visible.

6. Document limitations and statement boundaries

Technical data findings and legal evaluation are separate

The final documentation identifies what is known about source, version, integrity, transformations and reproducibility and explicitly records gaps. Missing export logs, unavailable historical databases or undocumented manual preparation steps limit how strongly the examined state can be tied to an earlier source state.

Technical findings about the existence, content or processing of data do not by themselves establish legal admissibility, ownership, contractual relevance or data-protection compliance. Those questions require their own legal or contractual assessment.

A concise examination record can therefore state the dataset identifier, provenance evidence, acquisition method, integrity markers, transformation chain, quality checks, reproduced results and unresolved uncertainties. This makes the analytical basis reviewable without overstating what the evidence can prove.

Further reading

• Technical Assessment of Data Quality
• Data Provenance, Lineage and Transformations
• Data Preparation, Missing Values and Outliers
• Statistical Analysis, Sampling and Reproducibility

Data Science & AI Assessments

Technical examination of data quality, model validity, reproducibility, bias, robustness and documented AI-system behaviour.

Explore this assessment service