Checking an entire CSV export for duplicate and thin copy

The tell a spreadsheet hides

A thousand product descriptions generated in one pass will each read acceptably in isolation. The problem is collective: the same opening sentence appears ninety times with a different product name in the middle, the same closing benefit appears in every row, and nobody notices because nobody reads four hundred rows in order.

This is what scaled content abuse looks like before anyone calls it that. Not gibberish, not plagiarism, just a template applied so consistently that the catalogue reads as machine output rather than as writing about products.

Row-level review cannot find it, because every row passes. The finding only exists across rows, which means the check has to be comparative and it has to run on the whole file.

The same is true in the other direction for thin content: a description of eleven words is not a defect you notice while scrolling, and it is obvious the moment you filter by word count.

How near-duplicates are detected

Each row is reduced to its overlapping four-word sequences — shingles — and compared against the rows before it. Two descriptions that share eighty percent or more of those sequences are reported as a near-duplicate pair, with the percentage and both row numbers.

Four words is the right window for this job. Shorter and every row about a similar product looks duplicated; longer and only exact copies are caught. Four-word overlap catches the same sentence with a noun swapped, which is the failure mode that matters.

The pairwise comparison is why the tool slows down on very large files. It is quadratic in the number of rows, and a hundred thousand rows will take longer than a coffee break. Splitting the export by category is faster and produces a more useful report anyway, since duplication usually clusters within a category.

The second view counts five-word phrases that turn up in two or more rows. A handful is normal boilerplate; the same phrase in forty rows is a cohort that will read as programmatic to any human and to any helpful-content review.

• Near-duplicate pairs: 80%+ shared four-word phrasing.
• Reused phrases: five-word sequences appearing in two or more rows.
• Thin rows: below the word floor you set for that column.
• Empty values: the column is blank for a row that needs it.

Thin is a count, so the floor is yours

Thin content here means a word count below a threshold, and nothing else. That definition is deliberately narrow, because the alternative — a tool with an opinion about what thin means — would be wrong for most of its users.

A 155-character meta description is thin by any product-description standard and exactly right for its slot. A category introduction of forty words may be doing its job. A product description of nine words is a placeholder that shipped.

Set the floor per column rather than per file, and set it low enough that the report is actionable. A threshold that flags forty percent of rows produces a review nobody finishes.

What the check cannot decide is whether the pages should exist at all. A thousand near-identical product pages is a content strategy problem, and no word count fixes a decision that was made before the copy was written.

Reading the report against the plan

Start with duplicate pairs, because they are the most expensive finding. Each pair is either a copy that needs rewriting or two rows that should be one page, and the second is usually the better answer.

Then the reused phrases. If one phrase appears in half the rows, it is probably structural: the sentence that describes the benefit in every listing. Rewriting it in ten variants is worth doing; rewriting it in four hundred is not, and the honest move is to remove it from the template and let the product specifics carry the row.

Then thin rows, sorted by word count. Fix the bottom of the list first: the rows with four words are unpublished work, while the rows with 120 words are a judgement call about a floor you set yourself.

And keep the report with the export. When a search performance problem appears six months later, the duplicate analysis is the first thing anyone asks for, and regenerating it for a file that has since changed tells you about the new file.

Splitting a large export

The pairwise comparison grows with the square of the row count, so a hundred thousand rows is not a bigger version of a thousand rows, it is a different job. Splitting by category, collection or product family is faster and produces a report about something a person can act on.

Splitting also improves the findings. Duplication clusters: the same sentence appears across the rows that were generated together, which is almost always within one category. A single file hides that structure; twenty files reveal it immediately.

Keep the header row in every split. A file without headers parses as data with the first row as column names, and the report then refers to columns by product names, which is confusing enough to be a bug in your workflow rather than in the tool.

And keep a manifest of the splits: which rows went where, and the order they were checked. When a finding refers to row 412 of a file you split, the manifest is the difference between five minutes and an afternoon.

Fixing a cohort without rewriting every row

When forty rows share a sentence, the fix is not forty rewrites. It is a template change and a regeneration of that cohort, followed by a check that the new rows are not identical to each other, which is the failure that regeneration reliably produces.

Where the copy is genuinely per-product, rewrite the rows that carry a finding and leave the rest alone. The tool's job was to tell you which rows are affected, and eighty percent of a catalogue being fine is a normal result.

Delete rows rather than repairing them when the underlying page has no reason to exist. A thin row about a variant that nobody searches for is a decision about the catalogue, and rewriting it to three hundred words produces a longer version of the same mistake.

Then re-run the check on the changed cohort only. A second full pass over the file tells you nothing new about the rows you did not touch, and it doubles the time the fix takes.

A word floor per market, not per file

Word counts are not comparable across languages. A German compound carries what English needs four words to say, a Japanese description of the same length is not measuring the same thing, and a single global floor will flag one locale as thin and pass another that says less.

Set the floor per column and per locale, and set it from three examples of copy the business is happy with rather than from an industry benchmark. The floor is a proxy for enough to be useful, and enough is a property of the catalogue rather than of the language.

Record the floor in the export's notes. A report that flags eighty rows with no stated threshold is a report that will be argued with, and the argument will be about the threshold rather than the copy.

The report a merchandiser can use

A report full of percentages is not actionable for somebody who owns a catalogue. Group the findings by collection, name the rows, and put the count next to the collection rather than next to the file.

Where the fix is a template change, say so at the top. A merchandiser who reads eighty findings will assume eighty rewrites; one who reads a template change affecting eighty rows knows it is an afternoon.

Finish with the rows that need no work. A short line saying the remaining seven hundred rows are unique and above the floor is the sentence that stops the report being re-run next week out of doubt.

What bulk checking leaves out

Truth. A description can be unique, well-sized and entirely inaccurate about the product, and nothing in this check will notice. Product accuracy needs the specification, not the copy.

Search competition. Two rows can be completely different and still cannibalise each other for the same query, which is a keyword strategy question rather than a duplication one.

And quality in the sense a customer means it. The tool measures repetition and length because those are measurable at scale; it cannot measure whether a sentence makes someone want the thing.

The practical role is triage. Four hundred rows reduced to thirty findings is a brief somebody can work through, and a brief is what a scaled-content problem needs in order to stop being one.