Comparing an AI summary with its source, line by line

Distortion is quieter than invention

Ask a person to lie and they hesitate. Ask a model to summarise and it will compress, round, merge and generalise without any sense that it is doing something risky. It rounds 18 percent to 20 because 20 is a rounder number. It drops the qualifier that made a finding conditional because the qualifier made the sentence longer. It attaches a quotation to the right idea and the wrong speaker and the sentence still scans.

Each of those is a small edit, and each survives a fluent read, because the summary is usually better written than the source it came from. Short sentences, clear structure, no hanging clauses. Nobody re-reads a clean paragraph looking for a lie, which is why summary drift survives review more often than outright fabrication.

Two views catch it, and they catch different things. A word-level diff shows what moved between the two documents: the sentence that grew, the clause that vanished, the number that changed shape. A trace below it ignores phrasing entirely and asks one mechanical question of every figure, quotation and link in the summary: does this string appear anywhere in the source?

The second view is deliberately crude. It cannot be talked out of a finding, it does not have a theory about intent, and it does not care how well the sentence reads. That is exactly why it is worth running before the careful reading, not after.

What the trace catches, and why it is cheap to prove

It catches invented figures: a percentage that appears nowhere in the source, a dollar amount that no page contains, a date that does not exist in the document. These are the failures that are cheapest to prove and most damaging to publish, because anyone with the source can check them in ten seconds.

It catches quotations that are not quotations. A summary that presents a phrase as verbatim, with quotation marks, and then a source that says something adjacent but different. This is the failure that most often reaches a lawyer, because a quotation attributed to a named person is a statement of fact about what they said.

It catches links and citations the source never contained, and placeholder references left in from a template. It also catches the rows of a table where a number is present in the source but attached to a different year or category, at least in the coarse sense that the surrounding string does not match.

What it does not catch is everything subtler. A number can be present and attached to the wrong claim. A paragraph can quote correctly and draw the opposite conclusion. A summary can omit the single sentence that made a finding conditional. The trace is a filter, not a verdict, and a clean result is permission to read closely rather than a certificate.

• A figure with no match anywhere in the source.
• A quoted passage that is not verbatim.
• A citation number or link the source never contains.
• A qualifier present in the source and absent in the summary.

A workflow that survives a second reader

Paste the source first and the summary second. The order matters psychologically: you want the source in front of you when you read the summary, not the other way round, or you will read the summary as the document and the source as its supporting material.

Work the trace before the diff. Resolve each item into one of three decisions: the summary is wrong and gets corrected, the summary is right and the source needs a clearer phrasing, or the flag is a false positive because the summary paraphrased deliberately. Write down which, because a flag you dismissed without a reason is a flag you will meet again.

Then read the diff for the things the trace could not see: dropped caveats, merged findings, a conclusion that is stronger than its evidence. The diff is where meaning changes without any string changing enough to be flagged.

Finally, keep both documents with the final text. The comparison is the evidence that the summary was checked against something, and that evidence is what a client asks for six months later when a figure turns out to be wrong.

Choosing a source you can actually compare

A summary of a PDF is only comparable with the extracted text of that PDF. Paste the text, not a screenshot, and skim it before trusting a clean result: extraction loses column order, drops footnotes and merges table cells in ways that can turn two claims into one.

For transcripts, keep the speaker labels. Attribution errors are the most common failure in meeting summaries and they are invisible if every line arrives as undifferentiated prose. If the transcript has no labels, treat every quotation as unverified until you find it by hand.

For research notes, paste the notes rather than the paper. The summary was written from what was in front of the writer, and comparing it against a document the writer never saw produces findings that are technically true and practically useless.

And when the source is a web page that may change, save it first. A comparison against a live URL is a comparison against whatever the page says today, which may not be what anyone summarised.

Documents that fight the comparison

Scanned PDFs have no text layer, so there is nothing to match a quotation against. The page renders, the words are visible, and the trace finds nothing because the words are pixels. The only route in is OCR, which introduces its own errors, and a clean trace after OCR is a weaker result than it looks.

Extraction mangles columns. A two-column academic paper or a report with a sidebar frequently extracts as interleaved lines, which means a quotation appears as a fractured version of itself and the trace reports it missing. Skim the extracted text before trusting a negative result, and treat a garbled page as unverified rather than wrong.

Transcripts without speaker labels defeat attribution checks entirely. If every line arrives as undifferentiated prose, a summary that attaches a statement to the wrong participant will trace perfectly, because the words are present and nothing records who said them. Keep labels, or verify quotations by hand.

And a live page is not a document. If the source is a URL that may change, save it before you compare. A trace against today's page can confirm a claim that tomorrow's page will contradict, and the moment the page changes your evidence is gone.

Recording the trace in the handoff

A comparison that ends with a feeling is worth very little. The handoff should carry the three things a reader needs to trust the check: the source you compared against, the claims that were traced, and the ones you consciously did not check.

List the traced claims, even when everything passed. Eleven figures confirmed against pages three, seven and twelve is a record; a sentence saying the summary was checked against the source is an assertion. The list is what makes the difference between a review and a recollection visible to somebody else.

Write the scope note in plain language. Anything after section four was not compared, or quotations were checked but figures were not, is the kind of honesty that prevents the most common post-publication argument: the reviewer assumed the whole document was in scope, and so did the client.

Finally, store the source with the comparison. For a file, keep the file. For a page, keep a saved copy and the date you took it. Evidence that cannot be produced in six months is not evidence, and the cost of storing it now is a few megabytes.

Two-document reviews beyond summaries

The same comparison works wherever one document claims to represent another. A proposal against the research it cites. A policy summary against the regulation. A technical abstract against the specification. In each case the question is identical: does the short document contain only things the long one supports?

Contracts are the highest-value case. A quotation in a client-facing summary of the terms, traced back to the definition clause, is one of the few checks that reliably prevents an expensive misunderstanding, and the trace makes it a two-minute job rather than a reading exercise.

The method needs a source document that actually exists in text form. Where the original is a conversation, a video or a person's memory, the comparison cannot run and the review has to happen by other means, which is worth saying out loud rather than substituting a weaker check and calling it the same thing.

Where this leaves the review

A clean trace and a quiet diff mean the summary is supported by the source you pasted. They do not mean the source is right, current or relevant, and they do not mean the summary is good. Three separate questions, three separate checks.

For the substance, the claim matrix gives each figure and name a status and a URL. For the prose, the word highlighter and the inline editor handle how it reads. For anything with consequences — health, money, law, safety, a named individual — a reviewer who works in that field takes the last pass.

The point of the mechanical checks is to make that last pass short and specific. A reviewer who is handed a summary, its source and a list of the eleven claims that were traced is doing a different job from one who is handed a hundred pages and a deadline.