Incident forensics: tracing a production failure to its spec, author, review and test evidence in minutes — from a recorded chain

Last reviewed 2026-08-21 · 8 min read

When something breaks in production, the first hour is spent establishing which change caused it, who or what authored it, whether it was reviewed and tested, and which releases contain it. A recorded development chain answers those questions from the record in minutes — including the regulatory clocks under the CRA — instead of by archaeology under pressure.

What it is

Every incident has two investigations. The first is technical: what is failing, why, how do we stop it. The second is forensic: which change introduced this, who or what authored it, was it reviewed, what tests did it pass, which releases and customers contain it — and, later, can we prove all of that to the people who will ask. The second investigation decides how long the first one takes, and it is the one AI-assisted teams are least equipped for.

In a codebase where agents write a meaningful share of changes, the standard forensic trail breaks in a specific place: git names a human committer for a change an agent actually wrote, review tools hold an approval that may or may not be linkable to the change, and nobody can say quickly whether the change that is now failing in production was reviewed by a person who knew it came from an agent. The answer exists somewhere across five tools and several people’s memories. Under incident pressure, that “somewhere” is the whole problem.

A recorded development chain turns the forensic investigation into a query. Each change carries its actor (human or agent, with tool and model), its spec reference, the review that approved it, the tests that ran and the sealed releases it shipped in — append-only, time-anchored, verifiable. “Trace this failure to its spec, author, review and test evidence” becomes a lookup measured in minutes, and the result is a record you can hand to whoever asks next.

Who it applies to and when

Engineering and operations teams running production software; security teams handling vulnerabilities; and the people downstream of an incident — customers whose own reporting clocks are running, auditors, insurers and, for software products on the EU market, regulators and courts.

The external clocks make forensics a regulated discipline. From 11 September 2026, under the Cyber Resilience Act, a manufacturer that becomes aware of an actively exploited vulnerability in its product must submit an early warning within 24 hours, a fuller notification within 72 hours, and a final report once a corrective or mitigating measure is available; the full regulation applies from 11 December 2027. NIS2 entities run their own 24/72-hour incident-reporting rhythm and need their suppliers to answer fast. And from 9 December 2026 the Product Liability Directive makes the development record of a defective software product disclosable in litigation. Anyone who has run an incident knows what those first hours are actually spent on: working out which products and versions contain the affected component, when it entered the codebase, who changed it last and what the fix path is. That is a provenance query — and the regulation now puts a timer on it.

This topic is not part of our regulatory mapping — it is an operational use of development evidence that touches several acts. The support section uses the mapping’s vocabulary and marks the boundary between the record and the response.

What it asks of your development process

Forensic readiness is mostly a property of the record your process leaves before anything breaks:

Attribution at the source. The actor of every change — human, or agent with tool, model and session — recorded when the change is made. Inferring it afterwards is slow and, for agent-written code, usually impossible.
Spec and review linkage. Each change linked to what it was meant to do (spec reference) and to the human decision that accepted it. The forensic question “was this reviewed, by whom, knowing what” has to be answerable without interviewing people.
Test evidence per change. What ran, what passed — so “did this get tested” is a lookup, and “the tests did not cover this” is a finding the record supports.
Release scope. Which sealed releases contain the change, with their SBOM — the “which versions and customers are affected” question, which is the one the 24-hour clock is really asking.
Integrity. An incident record will be read by people who were not there, sometimes adversarially. A record that could have been edited after the fact is worth less than one that provably was not; a hash chain with external time anchors is the difference.

With those five in place, an incident runbook’s forensic section collapses into a handful of queries: from the failing component to the change; from the change to its author, spec, review and tests; from the change to the releases that contain it; and from the fix back through the same trail to prove remediation. Without them, the same section is a Slack thread and a spreadsheet, assembled under the worst possible time pressure, with a regulator or a customer waiting.

The honest limits: the chain does not find the bug, does not decide severity, does not file the report. It makes the provenance part of the incident fast and provable — which is the part that most often eats the clock.

What OLOBOLO evidence supports

OLOBOLO records each change with its actor, spec reference, review decision and test evidence, seals releases with an SBOM and licence snapshot, anchors the chain with RFC 3161 timestamps, and exposes a searchable chain feed, an export and an open-source verifier. The table uses our mapping vocabulary; this is not a regulatory mapping — incident response, severity, reporting and disclosure are your processes and your counsel’s.

Forensic question	What you need to show	Chain evidence	Support
Which change, authored by whom or what, against which spec?	Attribution and intent per change	`change.authored` (actor type, tool, model, spec reference)	Supports
Was it reviewed and tested before release?	Review decision and test runs linked to the change	`change.reviewed`, `test.evidenced` linked per change	Supports
Which releases — and therefore which customers — contain it?	Release scope with components	`release.sealed` segments with SBOM and licence snapshot	Supports
Can we prove this history to auditors, insurers, regulators or a court?	Contemporaneous, tamper-evident, time-anchored record	Hash chain + RFC 3161 anchors; canonical export; open-source verifier	Supports
When was the fix authored, reviewed, tested and shipped (remediation timeline)?	The fix’s trail with time anchors	The fix’s `change.authored` → `change.reviewed` → `test.evidenced` → `release.sealed`	Partially supports
Fault diagnosis, severity classification, incident reporting to ENISA/CSIRTs/customers, disclosure decisions	Operational and legal incident response	—	Out of scope

“Partially supports” on remediation is honest: the chain evidences the timeline of the fix; the reporting and the operational response around it are yours.

Evidence, not a legal opinion: this maps what the chain supports as documentation — your incident process and your counsel decide what the response requires.

How to start

Record before the incident. Forensic evidence cannot be created afterwards. Install the recorder on every repository behind production and let the chain run from today.
Carry spec references and agent identity in commits. Agents carry tool, model and session; commits carry a spec reference; reviewers’ approvals land as change.reviewed. That is the trail a forensic query walks.
Seal releases with SBOMs. “Which versions contain this component” is the first question on the CRA clock; sealed releases answer it.
Rehearse the query. Pick a past change and trace it end to end from the chain feed — component → change → author, spec, review, tests → releases. If it takes minutes, your forensics are ready; if it takes a meeting, start with step 1.

Evidence can't be reconstructed after the fact — it has to be recorded while you build. Start recording now, free, and every change from this moment is on the record.

Start recording

Frequently asked questions

Does the chain tell us what caused an incident?

It tells you what changed, who or what authored it, what review and tests it passed and which releases contain it — the provenance part of root-cause analysis, in minutes. Diagnosing the actual fault is still your engineers' work; the chain just removes the archaeology from it.

Is this different from our observability and logging?

Yes. Observability shows what the system did; the chain shows how the code that did it came to be. Incident response needs both: logs find the symptom, the chain finds the change — and proves its history to whoever asks afterwards.

Why does tamper-evidence matter in an incident?

Because the record of how the faulty change was made will be read by people who were not there — auditors, insurers, regulators, sometimes a court. A record that could have been edited after the incident is worth less than one that provably was not.

What about the CRA's 24-hour clock?

From 11 September 2026 a manufacturer aware of an actively exploited vulnerability must submit an early warning within 24 hours, a notification within 72 hours and a final report after remediation. The window is mostly spent establishing scope and provenance — which is exactly the query the chain answers.

Sources

Last reviewed 2026-08-21. Dates are cross-checked against our regulatory mapping and radar.

A necessary honesty: OLOBOLO is evidence, not a legal opinion. This guide describes what our chain supports as documentation — Supports / Partially supports / Out of scope — and your counsel decides what compliance requires. No tool can promise the latter.