Review AI-written code with contracts and counterexamples

Read the branch that disagrees with the explanation

In the operator's MaybeTomorrow source, an outbox comment said that a pending delete would not be downgraded by an upsert. The executable body updated the existing row to an upsert, and an inline explanation said that the latest user intent was authoritative. Summarizing only the first comment would have produced the wrong account of the implementation.

We found that disagreement while preparing a source-backed case study, not during a reported production incident. It offers a useful starting point for reviewing AI-written code: the explanation is a claim about the code, and the code must still be inspected. A fluent summary, whether written by a person or an assistant, cannot substitute for following the branch that matters.

This guide uses the actual outbox inspection and our executed Imgobot source-grouping fixtures to develop a review method. It does not claim that the inspected code was entirely AI-written or that a comprehensive security audit was performed. The transferable subject is how to review a proposed implementation without letting its accompanying explanation define what counts as correct.

Original TrendsWhat review guide. A passing check is useful only when it covers the behavior the change is supposed to provide.

Put the contract beside the code

Before reviewing a function, write the behavior that the caller expects. For an outbox removal operation, the contract might be: remove only the version acknowledged by the server, leaving a newer edit pending. For a source-grouping function, it might be: keep readable fragments under the correct heading path while omitting page furniture.

These statements are more useful than “the code should be clean.” They define what a counterexample would look like. If an old acknowledgement removes a newer edit, the first contract fails. If a table under a nested heading moves to the next chapter, the second fails. A reviewer can now look for those mechanisms rather than judge indentation or naming first.

The contract should come from the task and the caller, not solely from the proposed implementation. Otherwise, a generated function can quietly redefine the requirement around what it already does. When the requirement is ambiguous, identify the decision that needs clarification and continue reviewing independent parts rather than assuming the most convenient behavior.

Follow one value across the boundary

Choose a consequential value and trace it from input to result. In the outbox example, the expected update timestamp travels into the deletion condition. The condition compares it with the stored row. That is the mechanism intended to prevent a stale acknowledgement from removing a newer mutation.

In the Imgobot grouping example, the heading category determines a level, the heading path is adjusted, and later fragments inherit the resulting path. The test fixture then checks the complete returned unit. Following that path explains more than a statement that the function “handles headings.” It identifies how the behavior is implemented and where an assumption could fail.

For an AI-written change, ask the assistant to identify this path, then inspect it yourself. A useful explanation names the input, transformation, state check, and output. A vague assurance that the code is robust is not a substitute for those locations. If the explanation references a helper, read the helper when its behavior determines the contract.

Construct the smallest counterexample

A counterexample is an input or sequence that would reveal a wrong implementation. It should be small enough that the expected result is clear. The goal is not to invent a huge test suite before understanding the change. It is to challenge the assumption most likely to make the code fail its task.

For a version guard, the sequence is an original write, a newer edit, and a response to the original write. For document grouping, a nested heading followed by a new top-level heading checks whether the old path resets. For a date validator, a date-shaped string that is not a real calendar date checks whether syntax is being confused with meaning.

Our output-validation lab executes that last distinction with February 30. The value passes the shape pattern and fails the calendar rule. The example is effective because it isolates one missing layer. A random collection of unusual strings would be less helpful if the reviewer could not explain which assumption each one tests.

Distinguish tests that describe from tests that prove

A test file describes an intended check. An executed result provides evidence about the current implementation under the tested conditions. Keep those statements separate. In the outbox case study, we read the test definitions but did not run the iOS suite. In the Imgobot case, we executed an exact function snapshot against three synthetic fixtures and saved the results.

That difference should appear in the review report. “A regression test exists” is not the same as “the regression test passed in this checkout.” Likewise, a passing unit test does not establish a full user flow. A function can behave correctly while its caller supplies the wrong state or ignores the returned decision.

Inspect what the assertion actually checks. A test that only confirms the function returns something can pass while the important field is wrong. A test generated from the implementation can also repeat its mistaken assumption. Expected behavior should be grounded in the contract, with a counterexample capable of failing a plausible wrong implementation.

Review ordinary code around the AI call

Many important defects occur before or after model inference. Input parsing can drop data. Normalization can turn missing values into defaults. A persistence operation can overwrite a newer state. An interface can show completion before the authoritative write is known to have succeeded.

The operator's StockNote approval flow is useful context: extraction produces a draft, and a separate operation commits reviewed fields. A code review of that boundary should inspect which values the commit actually uses, whether the draft belongs to the caller, and how repeated processing is handled. Those checks are distinct from whether the assistant's prose sounds helpful.

Do not let the presence of an AI API call make the rest of the program seem routine. The surrounding software determines what the model sees, what its output means, and what actions follow. A small deterministic guard may contribute more to the required behavior than another paragraph in the prompt.

A worked review note

Consider this fictional review observation: “The delete operation uses only the row ID, so an acknowledgement for an older version can remove a newer edit stored under that ID. Include the expected revision in the condition and verify the stale-response sequence.” This is an illustrative review note, not a claim that this defect exists in the inspected outbox, which already includes the timestamp guard.

The note has four useful parts: the triggering sequence, the current mechanism, the consequence, and the proposed verification. It avoids vague labels such as “race condition risk” without explanation. Another developer can understand why the issue matters and what evidence would show it is fixed.

Use that shape for feedback on AI-written code. Keep the scope tight and distinguish a demonstrated defect from an untested possibility. If you cannot explain the trigger or the consequence, investigate further before presenting the concern as a finding. Review quality improves when each comment can lead to a concrete action.

Copy this contract-first review sheet

This sheet is a thinking aid, not a requirement to write tests for every cosmetic edit. Match the review effort to the change. A reversible wording correction may need a visual check. A state mutation or parsing change deserves a check that exercises the relevant boundary. Repeating unrelated suites does not make a narrow claim more precise.

Use tools to answer specific questions

Search tools can locate callers, definitions, and related state names. Type checking can reveal interface mismatches. A focused test can exercise a counterexample. A browser can show whether the result reaches the user. Each tool answers a different question, so choose it based on the uncertainty you need to resolve.

In the Imgobot test, comparing the source snapshot hash with the inspected function established that the tested snapshot matched the local implementation at preparation time. The three fixture results then established behavior for those inputs. Neither fact establishes a production deployment, parser accuracy, or learning outcomes. The chain is useful because each claim stays attached to the evidence that supports it.

An assistant can help orchestrate these checks, but it should not collapse their meanings into “verified.” Ask for the result and the scope. A concise report naming the exact behavior tested is more useful than a long list of commands without an explanation of what they covered.

Stop adding guards that change the contract

A defensive-looking fallback can hide a defect. Returning an empty array when required data is missing may make a page look successful while concealing an upstream failure. Replacing an unknown date with today may produce valid output that no longer reflects the user's request.

When reviewing such code, ask whether the fallback is an explicit product decision. If it is, document and test the behavior. If it is not, preserve the error or unresolved state so the caller can handle it appropriately. “The code does not crash” is not always the same as “the code does the right thing.”

This is especially relevant to generated patches, which may optimize for a passing check by adding broad defaults. The review should return to the intended behavior and determine whether the patch makes that behavior more true. A smaller change that exposes a real problem can be better than a larger one that hides it.

What to carry into the next review

Begin with the contract, trace a consequential value, and construct a counterexample that challenges the implementation's central assumption. Read comments as explanations to verify, not as the final authority. Report executed checks separately from tests you merely found.

The project inspections behind this guide show why those habits matter. A comment can disagree with a body, a field name can imply more than a query does, and a small fixture can make a hidden boundary visible. None of that requires dismissing AI-generated code. It requires reviewing it with the same concrete standards as any other proposed change.

Sources, materials, and limits

• Outbox case study: actual comment/body discrepancy and inspected version guard.
• Imgobot grouping case: executed exact-source snapshot and three fixtures.
• Output validation experiment: syntax versus calendar meaning.
• Original review sheet and fictional feedback example. No claim about who authored all inspected code, no comprehensive security audit, and no unexecuted test presented as passing.