Review design

Human-in-the-loop is a design decision, not a checkbox.

Putting a person in the loop does not automatically add safety. If the person has thirty seconds, no context and no authority, the loop is a signature that launders a machine's output.

What the human is actually for

A reviewer is not a slower regex. The value shows up in four places: judgement about coherence, context the tool cannot see, accountability when a decision is questioned, and taste in public-facing work.

Each of those requires something different. Judgement needs time and the argument in front of them. Context needs the brief, the audience and the risk. Accountability needs a name and a record. Taste needs someone who knows the brand rather than the internet average.

If none of those is present, the loop is decoration, and decoration is worse than a documented absence because it looks like assurance.

Route by the cost of being wrong

The routing rule that holds up is economic. Everything mechanical goes to the automated pass. Items where a wrong answer is expensive, public or regulated go to a specialist in that discipline. Everything else ships with a report attached and no human hour.

Expensive means: authentication, payments, deletion, permissions, contractual claims, health and finance statements, anything a customer will follow step by step, anything carrying a company byline.

It is worth writing that list down. Without it, escalation happens when someone feels nervous, which is neither consistent nor correlated with actual risk.

Brief the reviewer like a colleague

A reviewer handed a file and asked to check it will do their own triage, and their hour goes to rediscovering what a rule already found. Hand over the automated report instead, plus four lines: what the work is meant to do, who receives it, what happens if it is wrong, and what the author is unsure about.

That last line is the most valuable and the least commonly provided. Generated work has a characteristic uncertainty, and the author often knows where it is even when they cannot say why.

Scope and rate should be agreed before the file moves. A review that starts before scope is settled ends in a negotiation about hours rather than a verdict about the work.

The rubber-stamp failure and how to prevent it

Four conditions produce stamping: too little time, no access to the source, no authority to block, and no record of what was reviewed. Remove any one and the review becomes real again, usually at the cost of throughput.

One practical mechanism is a written change note. Ask the reviewer to record what they would change, even for accepted work. A reviewer who cannot produce two sentences about the work has not read it, and that becomes visible without anyone having to accuse them.

The second mechanism is disagreement. If a reviewer never disagrees with the automated report, either the rules are perfect or the report is being skimmed. Track the rate; a flat zero is a signal, not a compliment.

Closing the loop back into the pipeline

A review that produces a verdict and nothing else fixes one document. A review that produces a pattern fixes the next fifty.

So the output of the human layer should feed two places: new rules for the classes of defect that can be detected mechanically, and changes to prompts, templates or source material for the rest. Anything recurring that cannot be detected cheaply is a candidate for an explicit checklist item in the brief.

This is the only mechanism by which the human share of the work falls over time. Without it, every document costs the same hour forever.

Measuring whether it works

Count three things: the share of items escalated, the time from request to verdict, and the number of defects found downstream that a reviewer had in front of them. The third number is uncomfortable and the most informative.

Also count the cost. Human review is the most expensive part of any QA process, so it should be spent where the marginal value is highest, which is almost never on a landing page hero.

A process that reports these numbers honestly can defend its budget. One that reports satisfaction scores cannot.

Where the human should not be

Do not use a person to find credentials, unbalanced brackets or unfinished placeholders. Those are settled questions, and paying a specialist to answer them is a waste of the only expensive resource in the process.

Do not use a person to approve volume. If the queue requires a hundred verdicts a day, the process has already failed and the human becomes a rate limiter with a signature.

Use the person where the question has no rule: whether the argument holds, whether the tone fits the moment, and whether anyone should be accountable for shipping it.

What this covers

  • Routing written down

    A list of item classes that require a human, agreed before the queue forms.

  • Report handed over

    The reviewer starts from automated findings rather than repeating them.

  • Brief with the file

    Purpose, audience, consequences and the author's own doubts.

  • Change notes

    Reviewers record what they would change, even when they accept the work.

  • Feedback to rules

    Recurring human findings become mechanical rules or brief checklists.

Questions this page usually raises

Does human-in-the-loop mean someone approves every output?
It should not. Approval on every item turns the reviewer into a bottleneck and encourages stamping. Route by consequence and let the mechanical layer carry the rest.
How much time does a useful review take?
Long enough to read the work and write two sentences about it. Most reviewers publish a typical turnaround; the scoping conversation should happen before the file moves.
Should the reviewer see the automated report first?
Yes. It prevents duplicated effort and it gives them something concrete to disagree with, which is a better use of expertise than a fresh read.
How do we know a reviewer is qualified?
Look for discipline, years in it, a listed rate and a record of completed reviews. A named practitioner you can hold to a scope beats an anonymous pool.
Is a human review required for legal safety?
No process guarantees that. What a named review does is create attributable evidence that someone competent considered the work, which is what a dispute usually asks for.

Human-in-the-loop AI review

Run the checks, then decide who has to sign it.

The automated pass is free, needs no account and uploads nothing. Escalate to a named reviewer only where being wrong is expensive.