Examining Reproducibility and the Limitations of Technical Findings

A finding is reproducible only in relation to a defined subject of examination and a sufficiently documented method. Repeating the same command on a different version, dataset or environment may answer a different question even when the procedure looks identical.

By Mathias Ellmann English version: 22 October 2026

Courts & Law Firms

A finding is reproducible only in relation to a defined subject of examination and a sufficiently documented method. Repeating the same command on a different version, dataset or environment may answer a different question even when the procedure looks identical.

Short answer

Preserve the system or data state, document tool versions and parameters, state observations before interpretation, test counter-hypotheses and distinguish re-execution of an event from reproducible analysis of preserved evidence. Tie the conclusion to the depth and quality of the evidence.

Reproducibility strengthens reviewability but does not remove uncertainty or decide legal weight. Non-repeatable historical events can still be examined through preserved evidence when the reconstruction method and limitations are explicit.

1. Relate reproducibility to the concrete examination object

Identify the system, data, operating-system, application and configuration state relevant to the finding. A repeat on another version or dataset may be technically valid but not comparable to the original proposition.

Document enough context for a third party to recreate a sufficiently similar starting state rather than merely repeat the same command in a different environment.

The examination separates preserved material, directly observed technical findings, reconstruction and professional inference. Missing historical states or provenance are reported as limitations rather than filled by assumption. A technical expert finding does not decide procedural admissibility, legal responsibility, burden of proof or the court’s evaluation of evidence; those questions remain outside the technical conclusion.

2. Document method and parameters completely

Tool name alone is usually insufficient. Record version, options, filters, queries, time windows and preprocessing where they can affect the result.

Prefer scripts or reproducible queries for automated analysis and document manual intermediate steps so that their influence on the result remains visible.

The examination separates preserved material, directly observed technical findings, reconstruction and professional inference. Missing historical states or provenance are reported as limitations rather than filled by assumption. A technical expert finding does not decide procedural admissibility, legal responsibility, burden of proof or the court’s evaluation of evidence; those questions remain outside the technical conclusion.

3. Separate finding, interpretation and assessment

An observed metric or missing artefact is not identical to its meaning. A missing log entry can reflect no event, disabled logging or lost data.

State the technical observation first, then explain which interpretation the evidence supports and what remains uncertain. This separation makes counter-review possible.

The examination separates preserved material, directly observed technical findings, reconstruction and professional inference. Missing historical states or provenance are reported as limitations rather than filled by assumption. A technical expert finding does not decide procedural admissibility, legal responsibility, burden of proof or the court’s evaluation of evidence; those questions remain outside the technical conclusion.

4. Test counter-hypotheses systematically

Reproducibility is not repetition of the preferred explanation. Use control tests, other data sources or changed parameters to examine relevant alternatives.

If several explanations remain technically compatible, report them and identify what additional evidence would be needed to distinguish them instead of forcing a single cause.

The examination separates preserved material, directly observed technical findings, reconstruction and professional inference. Missing historical states or provenance are reported as limitations rather than filled by assumption. A technical expert finding does not decide procedural admissibility, legal responsibility, burden of proof or the court’s evaluation of evidence; those questions remain outside the technical conclusion.

5. Handle non-repeatable events methodically

Production incidents and volatile states may have disappeared before the examination begins. Reconstruction can then rely on preserved logs, artefacts, snapshots and other time-near evidence.

In such cases reproducibility applies to analysis of the preserved evidence, not necessarily to re-triggering the original event. State that distinction explicitly.

The examination separates preserved material, directly observed technical findings, reconstruction and professional inference. Missing historical states or provenance are reported as limitations rather than filled by assumption. A technical expert finding does not decide procedural admissibility, legal responsibility, burden of proof or the court’s evaluation of evidence; those questions remain outside the technical conclusion.

6. Tie statement limits to evidence and examination depth

A clear limitation states what was examined, what was not and which conclusions cannot be drawn. That boundary is part of a technically sound report rather than a weakness.

Integrity and provenance principles can strengthen reviewability, but they do not replace case-specific reasoning. If a repeat no longer reproduces the original result, examine intervening state changes or possible transience before changing the conclusion.

The examination separates preserved material, directly observed technical findings, reconstruction and professional inference. Missing historical states or provenance are reported as limitations rather than filled by assumption. A technical expert finding does not decide procedural admissibility, legal responsibility, burden of proof or the court’s evaluation of evidence; those questions remain outside the technical conclusion.

Examination checklist

  • Is the exact system or data state identified?
  • Are tool versions, parameters and preprocessing documented?
  • Are findings separated from interpretation?
  • Were relevant counter-hypotheses tested?
  • Are non-repeatable events reconstructed from preserved evidence transparently?
  • Are integrity and provenance of the evidence documented?
  • Are statement limits tied to evidence depth?

Constructed example

A production failure cannot be reproduced weeks later because the service version has changed. The examination reruns the analysis on preserved logs and a build artefact from the incident period, documenting the method and confirming part of the sequence while leaving the original trigger uncertain. The conclusion states that distinction explicitly.

The example is entirely constructed and does not describe a real engagement, client matter or historical case.

Sources and technical basis

  • NIST guidelineNIST SP 800-86 — Guide to Integrating Forensic Techniques into Incident Response. Guidance on technical collection, examination and documentation of digital data. Open source.
  • NIST frameworkNIST Cybersecurity Framework (CSF) 2.0. Framework relevant to documented incident handling, integrity and provenance of collected information. Open source.
  • W3C recommendationW3C PROV-DM. Data model for entities, activities and provenance relationships. Open source.
  • Methodological basisSources & Methodology documents source hierarchy, counter-hypothesis control and statement boundaries for the English knowledge base.

Further reading

Critical review and methodological limitation

A repeatable analytical process does not guarantee that the original event itself can be reproduced. Historical and volatile systems often require reproducible evaluation of preserved evidence rather than re-execution of the incident.

The report should expose where evidence supports a direct finding and where an inference depends on incomplete state, missing logs or alternative explanations. This preserves technical usefulness without overstating certainty.

Translation and methodological scope

This English article is based on the reviewed German V1 source article. The translation preserves the technical meaning, methodological distinctions, source status and stated limitations. It does not strengthen technical findings into legal conclusions and does not treat technical references as proof of compliance or suitability for a specific system.

Source relationship: The corresponding German article remains the reference baseline during English V1 development. Open the German source article.

Author's technical literature and publications

The following works provide additional technical and methodological context. They do not replace required primary sources, standards, manufacturer documentation or other independent specialist sources. Unpublished manuscripts, drafts and teaching materials are deliberately not presented here as public sources.

Mathias Ellmann, IT expert

Technical Expertise for Courts & Law Firms

Neutral technical examination of software, IT projects, data and AI questions in court and legal contexts.

Explore this assessment service