Is AI QA the same as prompt testing?

No. Prompt testing asks whether the model follows instructions. QA asks whether the resulting artefact is fit to ship, which includes defects introduced after generation by editing, templating and re-encoding.

How big does a team need to be for a QA function?

One owner is enough to start. The work scales with the number of pipelines, not with headcount, and the mechanical layer costs seconds per item.

Should QA block a release?

Critical classes should block: credentials, unsafe rendering, invented references. Everything else can warn, as long as the warning is read and the override is recorded.

Where does a named reviewer fit?

At the edge of what rules can prove: coherence, audience risk, taste, regulatory judgement. They start from the automated findings so their time buys judgement rather than repetition.

Can the checks run without a server?

Yes. The rule engine runs in the browser against pasted text, which is what makes it cheap enough to run on every item rather than on a sample.

Practice

AI QA is a function with owners, rules and a cadence, not a tool someone bought.

Teams usually start with a scanner and discover six months later that nobody agreed what a failure means. QA for generated work is mostly agreement: which defects matter, who decides, and what happens when the answer is no.

Run a free check Request a human review

QA against a moving target

Classical QA tests a specification. Generative work has no stable specification, so the tests have to target the classes of defect the pipeline keeps producing: invented references, credentials in source, claims without sources, truncated output, stripped metadata.

That reframing is what makes the work tractable. You are not asking whether this particular paragraph is good; you are asking whether the categories of failure you have seen before are present, and whether the process that produced it can be trusted on the rest.

The four things a working function does

A QA function that survives contact with a deadline has four moving parts, and skipping any one of them is what turns QA into theatre.

It defines the defect classes in writing, with severities, so a disagreement is about a rule rather than about taste.
It runs a deterministic gate on every item, cheap enough that nobody routes around it.
It triages by the cost of being wrong, so human attention lands on authentication code and regulated claims rather than on the whole backlog.
It keeps a record: what ran, what was found, who decided, and what changed as a result.

Who owns which decision

The most common failure is a gate with no owner. Engineering owns code risk, editorial owns claims and voice, legal owns regulatory exposure, and product owns the decision to ship anyway. None of those can be delegated to a score.

What can be delegated is the mechanical layer. Nobody needs a committee to agree that a hardcoded key is a defect; that rule is settled. The committee is only needed where a judgement is genuinely contested, and the point of the rule set is to keep that set small.

Cadence: the gate and the sample

Two rhythms do different jobs. The gate runs per item, before the work leaves the machine, and it is where defects get fixed cheaply. The sample runs weekly or monthly over work already shipped, and it is where you learn what the pipeline is doing over time.

The sample matters because gates measure what the gate tests. A defect class that nobody wrote a rule for simply never appears in a gate report, and the only way to find it is to read a random selection of output with a human who knows the domain.

Both rhythms are cheap if the gate is free and instant. Neither works if running a check requires a ticket.

Metrics that are not vanity

Average score is the least useful number in the room, because it goes up when you remove the rule you kept failing. Better measures are duller and more honest.

Finding rate by defect class, which tells you what the pipeline produces most.
Recurrence, meaning how often the same class returns after being fixed once.
Time from finding to fix, which tells you whether the report is being read at all.
Escaped defects, meaning the ones a customer found that a check should have caught.
Share of items escalated to a human, which should be small and deliberate.

Starting from nothing

A first week does not need a platform decision. Pick one pipeline, collect twenty representative outputs, run them through the deterministic checks, and read the findings together. The discussion that follows is the real deliverable, because it produces the defect list and the severity agreement that everything else depends on.

Then wire the gate into the one place the work already passes through: the pull request template, the CMS publish step, the send button in the email tool. A gate that requires a new destination is a gate that gets skipped.

Leave the human layer until you know where it is needed. Escalation volumes are a finding from the first month, not a design assumption.

What this covers

Written defect classes
Each class has a rule, a severity and an owner, so disagreement has a subject.
A gate on the existing path
Checks run where the work already moves, not at a destination someone must remember.
Triage by cost
Human hours go where a wrong answer is expensive, public or regulated.
A weekly sample
Shipped output is read by a person, which is how missing rules are discovered.
Honest metrics
Finding rate, recurrence, fix time and escaped defects rather than average score.

Read the method

Questions this page usually raises

Is AI QA the same as prompt testing?: No. Prompt testing asks whether the model follows instructions. QA asks whether the resulting artefact is fit to ship, which includes defects introduced after generation by editing, templating and re-encoding.
How big does a team need to be for a QA function?: One owner is enough to start. The work scales with the number of pipelines, not with headcount, and the mechanical layer costs seconds per item.
Should QA block a release?: Critical classes should block: credentials, unsafe rendering, invented references. Everything else can warn, as long as the warning is read and the override is recorded.
Where does a named reviewer fit?: At the edge of what rules can prove: coherence, audience risk, taste, regulatory judgement. They start from the automated findings so their time buys judgement rather than repetition.
Can the checks run without a server?: Yes. The rule engine runs in the browser against pasted text, which is what makes it cheap enough to run on every item rather than on a sample.

Run the checks, then decide who has to sign it.

The automated pass is free, needs no account and uploads nothing. Escalate to a named reviewer only where being wrong is expensive.

Run a check Request a specialist