Data & Data Science
Data lineage describes the path from source data through processing steps to later datasets, reports or models. In a technical examination, that path must be grounded in concrete versions, transformations and evidence rather than represented as an idealised architecture diagram.
Short answer
The examination first identifies source systems and data states, then treats transformations as part of the subject of examination. W3C PROV concepts can help structure entities, activities and derivations, while technical evidence such as job definitions, queries, logs, manifests and checksums supports the concrete reconstruction.
A complete-looking lineage graph can still contain unsupported links. Counter-checks and reproduction of selected transformations help distinguish documented derivation from assumed derivation. The final statement therefore records both the reconstructed path and its evidence gaps.
1. Identify source systems and data states
Lineage starts with concrete versions, not generic system names
Each relevant input receives an identifiable state: database snapshot, export, object-store version, API response period or file collection. Without a time and version reference, the same source system can represent many materially different datasets.
The examination records which system produced the data and which interface or extraction mechanism was used. If multiple upstream sources contribute to one output, their respective roles and join points are documented.
Source ownership, technical origin and business meaning are kept distinct. A table hosted in one system may contain values imported from another system, and the hosting location alone does not establish provenance.
2. Treat transformations as part of the examination
A derived dataset is defined by what happened between states
ETL jobs, SQL queries, notebooks, mapping rules, filters, enrichment services and manual edits can all create a new data state. The lineage reconstruction identifies these activities explicitly instead of drawing a direct arrow from source to output.
Parameters and supporting reference data matter. A currency conversion depends on an exchange-rate table; an aggregation depends on grouping keys and time windows; a deduplication step depends on its matching rule. These dependencies belong in the lineage where they affect the result.
Transformation order is recorded because non-commutative steps can produce different outputs. A technically accurate graph therefore describes the sequence needed to reproduce the derivation.
3. Use provenance models and technical evidence together
A conceptual model does not replace case-specific evidence
W3C PROV-DM provides concepts for entities, activities, agents and derivations. The PROV Primer illustrates how these concepts can represent provenance relationships. They are useful for structuring a reconstruction but do not prove that a particular system executed the modelled steps.
Case-specific support comes from pipeline definitions, scheduler history, repository commits, database queries, logs, generated manifests, timestamps and other artefacts that can be tied to the examined period.
Where the evidence consists only of current documentation, the report distinguishes intended lineage from observed or historically supported lineage.
4. Use counter-checks and reproduction
Reproduction tests important edges in the lineage graph
Selected transformations can be rerun from identified inputs to compare intermediate counts, checksums, schema states or record samples. A match supports the documented derivation under the tested environment and parameters.
A mismatch is not automatically evidence of misconduct or error. It can reveal hidden dependencies, changed reference data, environment differences or undocumented manual intervention. The discrepancy becomes a lead for further reconstruction.
Counter-checks can also start from the output and trace selected records backwards to their source attributes. Forward and backward checks reduce the risk of accepting a plausible but unsupported pipeline narrative.
5. State limits of provenance findings
Absence of evidence is not a complete history
Historical lineage may be only partly recoverable when logs have expired, jobs were overwritten or source systems no longer exist. The examination states which edges are directly supported, which are inferred from consistent evidence and which remain unresolved.
A reproducible current pipeline does not prove that the same code and configuration produced a historical dataset. Historical claims require evidence from the relevant period or a justified reconstruction from surviving artefacts.
Lineage also does not establish data quality by itself. It can show where a value came from while the source value remains incorrect or incomplete.
6. Build lineage graphs with control points
A useful graph exposes versions, checks and unresolved edges
The final graph or table names each material data state, transformation and transfer and associates it with evidence. Control points can include checksums, record counts, schema fingerprints, job identifiers, repository commits and timestamps.
The level of detail follows the examination question. A high-level graph may be sufficient to establish an external transfer path, while a disputed numerical result may require field-level transformation lineage.
The documentation should remain reproducible without becoming decorative. Every represented edge should have a defined basis or be marked explicitly as an assumption or unresolved relationship.
Is the technical conclusion sufficiently supported?
- Are source systems tied to concrete dataset versions or time states?
- Are all material transformations represented as separate activities?
- Are parameters and reference datasets included where they affect results?
- Is intended architecture distinguished from technically evidenced lineage?
- Can selected transformations be reproduced from identified inputs?
- Are forward and backward record-level checks used where appropriate?
- Are unsupported or historically unavailable lineage edges marked explicitly?
- Does the result avoid treating provenance as proof of data quality?
Constructed example
A reporting table is said to originate directly from an ERP export. Repository history and scheduler configuration instead show an intermediate ETL job that maps customer identifiers and enriches currency values from a separate reference table. The lineage graph is amended to include both activities and the reference dataset.
The current ETL code reproduces the latest report but cannot reproduce a disputed older report because the historical exchange-rate table version is missing. The report therefore supports the general transformation path while limiting the claim about the exact historical values.
The example is entirely constructed and does not describe a real engagement.
Sources and technical basis
- W3C Recommendation W3C PROV-DM, the conceptual data model of the PROV family for provenance information, entities, activities, agents and derivations. Open source.
- W3C Working Group Note W3C PROV Primer, illustrating core provenance concepts through entities, activities and derivations. Open source.
- Author methodology Sources and methodology of the knowledge base; lineage is treated as an evidence-backed chain of identifiable states and transformations.
Further reading
- Data as a Subject of Examination
- Technical Assessment of Data Quality
- Data Model, Schema and Integrity
- Data Preparation, Missing Values and Outliers
Critical review and methodological limitation
Lineage diagrams are easy to over-trust because they present a coherent story visually. The technical review therefore asks what evidence supports each important edge and which version of each component or data state is actually represented.
Reproduction supports a derivation only under the tested inputs, code and environment. Historical lineage requires historical evidence; a current pipeline should not be projected backwards without support.
- States: identify the concrete input and output versions at each material step.
- Edges: associate transformations with code, queries, jobs, logs or other evidence.
- History: mark where current evidence cannot establish the exact historical execution.
Translation and methodological scope
This English article is based on the reviewed German V1 source article. The translation preserves the technical meaning, methodological distinctions, source status and stated limitations. It does not strengthen technical findings into legal conclusions and does not treat technical references as proof of compliance or suitability for a specific system.
Author's technical literature and publications
The following works provide additional technical and methodological context. They do not replace required primary sources, standards, manufacturer documentation or other independent specialist sources. Unpublished manuscripts, drafts and teaching materials are deliberately not presented here as public sources.
Data Science & AI Assessments
Technical examination of data quality, model validity, reproducibility, bias, robustness and documented AI-system behaviour.
Explore this assessment service