Core claim: The AI engineering analytics your organization is buying — adoption rates, acceptance percentages, productivity indices — measures how your teams use AI. An audit, a disclosure order or a due diligence team asks what your teams shipped. Those are different questions, and the dashboard cannot answer the second one, because measurement and evidence are built to opposite requirements.

What the dashboard actually knows

The AI-engineering analytics category has settled on a recognizable shape: connect the coding assistants and the repositories, and get back aggregates. Share of suggestions accepted. Share of code estimated to be AI-authored. Time saved, velocity gained, adoption by team and by week. The numbers are genuinely useful for what they were built for — convincing a CFO the licenses are worth renewing, spotting a team that quietly stopped using the tooling, tracking a rollout.

Look at how those numbers are made, though, and three properties stand out. They are estimates — sampled, model-inferred, heuristic; vendors publish methodology notes precisely because the numbers move when the method does. They are aggregates — a percentage across a quarter, a team, a codebase; the individual change disappears into the average. And they are mutable — a dashboard is a query over telemetry the vendor hosts, reprocessed as pipelines improve; last month’s 31% can silently become 28% under a better model. None of this is a scandal. It is what measurement is: an instrument for managing a trend, tuned for insight, not for proof.

What the audit actually asks

Now put the dashboard in front of the questions it will actually meet — the vendor assessment, the technical due diligence, the disclosure order our pillar articles walk through. None of them asks for a trend. Every one of them asks about a specific artifact: this release, this component, this change. Who or what authored it? Under which instruction? Who reviewed it, and is there a record of the decision? Which tests ran against exactly this code, with what result? And the meta-question that decides whether any answer counts: why should we believe your records?

That last question is where the category quietly exits the room. Evidence that survives an adversarial reader has properties measurement was never designed to have: it is contemporaneous (recorded when the work happened, not recomputed later), per-change (about this commit, not a quarter’s average), attributable (naming the actor, human or machine), append-only and tamper-evident (a record you can silently reprocess is a record an auditor can reasonably discount), and independently verifiable (checkable by a third party without trusting whoever produced it — or hosted it). A dashboard fails all five, not because it is badly built, but because nobody asked it to pass. Its numbers live in a vendor’s database, under a methodology the vendor can change, with no integrity chain a skeptic can check. As an answer to “prove it,” a dashboard is a screenshot.

The dangerous comfort of a good number

Here is why this distinction is worth a polemic rather than a footnote: the dashboard’s real risk is not that it is wrong. It is that it feels like enough. An organization that can quote its AI share to one decimal place believes it has the provenance question covered — right up until the question arrives in its serious form, from a buyer’s diligence team or a court’s disclosure order, and the organization discovers it holds a statistic where it needs a record. And by then the record cannot be created, because evidence has the one property statistics never have: it only exists if it was captured while the work happened.

The resolution is not to cancel the dashboard. Measure away — adoption and productivity are legitimate management questions, and the analytics category answers them well. The resolution is to stop asking measurement to moonlight as evidence. They are different disciplines: one optimizes the process, the other proves it. If your AI strategy has a number for everything and a record for nothing, you do not have an evidence problem you can fix later. You have one you could have fixed today.

When the audit comes, nobody will ask how much AI you used. They will ask you to prove what it did — change by change, on a record that does not require trusting you. A dashboard measures; only evidence answers.

This builds on article 1, “Who wrote your code? The provenance gap in AI-built software”, and on “CRA technical documentation: what ‘secure development evidence’ actually looks like” — the five properties above are the same ones the CRA’s documentation duty rewards.

Sources

  • The evidentiary mechanics referenced here (PLD disclosure orders, CRA technical documentation duties) are covered, with CELEX sources and jurist review status, in articles 2-4 — this article deliberately makes no independent legal claims and leans on those pillars.
  • Category description (acceptance rates, AI-share estimates, methodology notes, telemetry reprocessing) is drawn from the public documentation of leading AI-engineering-analytics products, per ✅2 cited as a category, not by name.