Evidence

An auditable review answers four questions a year later.

Audit is not a test of whether you had a process. It is a test of whether the process left traces detailed enough for a stranger to follow without you in the room.

The four questions

What was checked? What was found? Who decided, and on what basis? And can any of it be reproduced? Almost every audit enquiry reduces to those four, and most QA processes can answer only the second.

The gap is usually the first and the last. Teams remember the findings and forget the rule set, so a report cannot be re-derived. Or they remember the decision and forget what it covered, so a signature looks broader than it was.

Designing for audit is mostly a matter of recording four small things at the moment they are true, rather than reconstructing them later from memory and screenshots.

What an evidence set contains

A complete set is small: the version of the rule set, the findings with severities, the score, the scope statement, the decision and the signatory if there was one.

  • Rule set version, so a re-run is reproducible rather than approximate.
  • Findings as records, not prose, so they can be counted and compared.
  • Coverage and truncation, so a clean report cannot be read as a complete one.
  • Decision, including an accepted risk with the name of whoever accepted it.
  • Scope of any human review, plus the case or ticket it belongs to.

The retention problem, honestly stated

Keeping the raw draft forever is a privacy liability. Keeping nothing is an audit failure. The resolution is to decide what the record has to prove and store only that.

In most cases the claim being evidenced is that a gate ran and what it found. Findings plus an excerpt plus a date prove that. The raw file is only needed if someone will later dispute the findings themselves, which is rare and can be handled case by case.

For human reviews, the scope description and the reviewer's change notes carry the evidence. A signed statement that a specialist read the pricing claims is auditable without retaining the client's whole document.

Reproducibility is the strongest position

A deterministic rule set turns an audit question into an experiment. Given the same text and the same rule version, the reviewer can re-derive the findings themselves, which removes the need to trust your logs.

Model-judged reviews cannot offer that. The same input may produce a different verdict, so the only evidence is a stored output and a promise about how it was generated. That is weaker in a way no amount of reporting fixes.

This is the practical argument for keeping the mechanical layer deterministic even if a model is used elsewhere in the pipeline. You want at least one part of the process that behaves like a calculator.

Logs are not reports

An application log records that a check happened. A report records what it said. Auditors ask the second question, and engineering teams often answer the first.

So keep the report with the deliverable, not only in a dashboard. The pull request, the CMS entry, the delivery folder. A report that lives only in a vendor console disappears when the subscription does.

The same logic favours portable formats. A PDF or a pasted block travels. A link to somebody's dashboard is a dependency disguised as evidence.

Preparing for the first audit

Pick three completed pieces of work at random and try to answer the four questions from the records alone, without asking anyone. Whatever you cannot answer is the gap, and it is usually the scope or the rule version.

Then fix the smallest thing that closes it. In practice that is a report attached to the delivery and two extra fields on the approval: what was reviewed, and against which rule set.

Write the policy last. A policy written before the records exist tends to describe a process nobody ran, which is the single most damaging thing to hand an auditor.

Where auditable review changes behaviour

Once people know the record is read, the shape of the process changes. Overrides get written down instead of assumed, scope conversations happen before the file moves, and reviewers stop signing things they did not read.

None of that requires a heavier process. It requires four fields and the habit of keeping the report with the work.

That is also what makes an AI review defensible in a way that a satisfaction survey about AI tools never will be.

What this covers

  • Rule set version

    Reports name the version so a finding can be re-derived later.

  • Findings as data

    Structured records that can be counted, filtered and compared.

  • Scope and coverage

    What ran, what was truncated, and what the rules never see.

  • Decision and owner

    Accepted risk carries the name of the person who accepted it.

  • Portable evidence

    The report travels with the deliverable rather than living only in a dashboard.

Questions this page usually raises

Do we need to keep the AI-generated draft for audit?
Usually not in full. The findings, a short excerpt, the rule version and the decision answer the standard questions. Retain the draft only where a dispute about the findings themselves is plausible.
What is the minimum evidence for a human review?
Reviewer name and discipline, scope description, date, version reviewed, what they changed and the verdict. That set is enough to reconstruct the decision.
Can a model-based review be auditable?
Partly. You can retain inputs and outputs, but a re-run may not reproduce the verdict, so the evidence is a record rather than a reproducible result.
How long should reports be kept?
Long enough to cover the obligations that apply to the deliverable, and no longer. Client agreements and sector rules set the floor; data minimisation sets the ceiling.
Does an audit trail have to include personal data?
Only the minimum needed to answer the question. Names of reviewers and approvers are usually necessary; the contents of a customer dataset are not.

Auditable AI review

Run the checks, then decide who has to sign it.

The automated pass is free, needs no account and uploads nothing. Escalate to a named reviewer only where being wrong is expensive.