Engineering

Generated code has a signature set of bugs, and they cluster.

It is not random. Assistant-written code fails in repeatable ways: the happy path only, the invented method, the swallowed error, the credential in the example that was supposed to be illustrative.

The clusters worth predicting

Five patterns account for most of what reviewers find in generated changes, and each is detectable at a different stage.

  • Happy-path logic: no handling for empty input, timeouts, partial responses or retries.
  • Invented APIs: methods and options that read plausibly and do not exist on the installed version.
  • Swallowed failures: empty catch blocks, defaults returned instead of errors, logging that goes nowhere.
  • Unsafe data flow: interpolation into markup, string-built SQL, user input reaching a shell.
  • Residue and drift: debug statements, duplicated helpers, a test suite that was never written.

What static detection can find

Anything visible in the source is fair game: credentials and key-shaped strings, eval and the Function constructor, innerHTML assignment, SQL concatenation, empty catch blocks, unresolved imports, syntax errors and duplicated blocks.

That covers the clusters where the defect is a pattern in the text, which is more than half of them and the half that fails silently in production.

Static analysis is also the only kind that runs before the code executes anywhere, which is the moment a fix costs least.

What it cannot find, and why that matters

It cannot know that a method exists on your pinned version, that a null is possible at that line, that a promise rejects on the error path, or that the route above is missing an authorisation check.

Those need types, tests and a reader who knows the system. Generated code makes this more urgent rather than less, because it looks conventional enough to be trusted.

It also cannot judge design. A function can be free of every finding and still be the wrong abstraction, and no scanning tool will ever say so.

Testing is the complement, not the alternative

The single most reliable predictor of generated-code quality is whether tests were written for it. Generated changes usually come without them, because the prompt asked for the feature.

Write the tests for the failure modes the code lacks: empty input, error responses, unauthorised calls, duplicate submission. Those are the paths the generation did not consider.

For anything touching money, permissions or deletion, add an integration check that exercises the real dependency. A unit test with a mock will happily confirm an invented API.

Triaging a finding list

Order by consequence, not by count. A credential is an incident; an unbalanced bracket is a build failure; forty loose-equality warnings are a style decision.

Apply severity weighting, fix what blocks in the same session, and record the rest with an owner. A findings list with no owner is a document nobody reads twice.

Where a rule keeps firing on legitimate code, narrow the rule rather than adding an exemption. Exemptions accumulate and hide the one finding that mattered.

The gate, and the person behind it

Run the checks on the paste, again before the pull request, and once more in CI for the repository and its history. Three placements, three different classes of coverage.

Attach the report to the pull request so reviewers start from findings rather than from a fresh skim. That is what keeps a review focused on design and integration.

For authentication, payments, permissions and deletion, require a named engineer to state the scope they reviewed. A report proves known defect classes are absent; only a person can say the change is fit for purpose.

Measuring whether detection is working

Count escaped defects: the ones found in production that a check should have caught. Each is a missing rule or a missing test, and both are cheap to add once identified.

Count time from paste to merge as well. If adding checks made the team slower without reducing escapes, the gate is testing the wrong things.

The healthy pattern is a falling escape rate with a flat review time, which means the mechanical layer is absorbing work the reviewers used to do.

What this covers

  • Error paths

    Empty catches, defaults in place of failures, unobservable errors.

  • Real dependencies

    Imports resolve and methods exist on the pinned version.

  • Unsafe flow

    Interpolation into markup, string-built SQL, input reaching a shell.

  • Tests for the gaps

    Empty input, error responses, unauthorised calls, duplicate submission.

  • Escaped defects tracked

    Production bugs that a check should have caught, feeding new rules.

Questions this page usually raises

Can a tool find logic bugs in generated code?
Not reliably. Pattern-based detection finds structural and security defects. Logic errors need tests that exercise the failure paths the generation skipped.
Why does generated code fail on error paths so often?
Because prompts describe the successful behaviour. The model produces what was asked for, and the unasked-for failure handling is what goes missing.
Is static analysis enough before merging?
For routine changes it covers the mechanical risk. For anything touching access, payments or deletion, add tests and a named reviewer with a stated scope.
How do I stop the same bug recurring?
Add a rule if it is a pattern, add a test if it is a behaviour, and add a line to the brief if it is a habit. Recurrence is a signal about the process, not the model.
Does the check need my source uploaded?
No. The rules run in your browser against the pasted code, which is what makes it usable on proprietary repositories.

AI bug detection

Run the checks, then decide who has to sign it.

The automated pass is free, needs no account and uploads nothing. Escalate to a named reviewer only where being wrong is expensive.