What to Save Before Rerunning an AI Bug

A useful AI bug report lets another engineer recreate the conditions that violated a requirement. Save the failing request, the relevant system state, and the assertion before rerunning. A screenshot of a wrong answer is evidence of the symptom; it rarely supplies the setup.

This Saturday field note turns the recent QA series into a handoff checklist. The example below is synthetic, with invented documents and results. It is a proposed exercise, not an incident report or a claim about any employer system.

Start with the requirement

Consider a document assistant answering, "Who approves an equipment request?" The current fixture says the team manager approves it. An older fixture says Finance approves it. The assistant returns Finance.

Write the bug title around the failed requirement: "Assistant cites superseded approval policy after document replacement." Then record the expected result: the answer must identify the team manager and cite the current document revision. Avoid an expectation such as "answer should be better," which leaves the next engineer to invent the test.

This connects to building a golden set: an expected outcome needs a versioned fixture and an explicit grading rule. The bug report is where those ingredients first become concrete.

Capture the setup before the second attempt

In the synthetic exercise, upload revision A, ask the question, replace it with revision B, wait for the application to report that indexing completed, and ask again in the same conversation. Save both requests in order.

Now compare that sequence with opening a fresh conversation and uploading only revision B. They are different tests. If the second passes, it has not disproved the first failure. The prior conversation, cached answer, or retrieval index may still matter.

The pytest guidance on flaky tests identifies uncontrolled system state and inadequate isolation as common causes of intermittent behavior. Apply that lesson before attributing every inconsistent AI result to the model. In this exercise, deliberately preserve state inside the request sequence and reset it between independent reproductions.

Put six things in the handoff

I would use the following report template. Its purpose is to make missing evidence visible, not to demand an exhaustive production data dump.

Keep the actual synthetic fixture files alongside the report. A document name alone cannot distinguish two revisions with the same name. A content hash can help identify a file, but the hash cannot reconstruct its contents. The engineer still needs access to the fixture.

Record unknown fields as unknown. If the provider exposes only a model alias, save that alias and the request timestamp. Do not invent a model snapshot or suggest that an alias guarantees identical behavior later.

Use the trace to locate the next experiment

OpenTelemetry describes a trace as the path of a request, with spans representing operations along that path in its observability primer. For this assistant, useful instrumented operations would include cache lookup, retrieval, prompt assembly, and generation.

Those observations guide different investigations. A cache hit with no retrieval points toward the cached-answer path. Retrieval returning revision A points toward document selection or index state. Retrieval returning B while the assembled prompt still contains A points toward context assembly. A prompt containing only B followed by a Finance answer directs attention to generation and the assertion.

These are hypotheses to test, not root-cause verdicts. A missing retrieval span might also mean missing instrumentation. The earlier trace checklist explains why the observation path itself needs verification. Preserve the evidence behind the hypothesis rather than turning a dashboard gap into a confident diagnosis.

Separate reproduction from frequency

First, ask whether a controlled sequence can reproduce the violation. Then, if needed, design repeated trials to estimate how often it occurs. Record the number of attempts, reset procedure, and each outcome. Preserve the initial failure even when subsequent attempts pass.

For example, a proposed batch of ten fresh-sequence trials is a debugging budget, not a reliability guarantee. A result from one continuing conversation should not be pooled silently with results from ten clean conversations. Their starting conditions differ.

When reducing the reproduction, change one suspected condition at a time: retain or clear the conversation, bypass the answer cache, or pin retrieval to the known fixture. Record which intervention removes the failure. Keep a full-path test as well; a passing stubbed test only covers the dependencies it actually exercises.

Make the final handoff executable

A teammate should be able to obtain the synthetic fixtures, run the ordered setup, and evaluate the assertion without asking what "wrong" meant. Include a command if the test exists; otherwise label the steps as a manual reproduction. Do not present an unimplemented test as passing coverage.

Close the report with the smallest next experiment and the remaining unknown. For this example: "Repeat with cache bypassed while preserving the conversation; index revision at failure was not captured." That sentence gives the next engineer a concrete starting point without pretending the investigation is complete.