Replay the Decision, Not the Answer

An agent test that compares only the final answer is testing the last line of a program while ignoring the branches that produced it.

That is acceptable for a simple text transformation. It is weak coverage for an agent that can retrieve documents, call tools, write files, or refuse an action. Two runs can return the same sentence while taking materially different paths. One path may be safe and reproducible. The other may have skipped a permission check and happened to reach the same conclusion.

The useful artifact is not only the answer. It is the decision trace.

What an agent actually decides

An agent run contains more decisions than most evaluation dashboards display:

1. Which context to retrieve.
2. Which tool to call, if any.
3. Which arguments to send.
4. Whether to ask for clarification or continue.
5. Whether an action is allowed under the current policy.
6. How to handle a timeout, malformed result, or partial failure.
7. What evidence is sufficient to produce the final response.

The final answer is downstream of all seven. If the test stores only input, output, and a scalar score, it cannot tell whether a regression occurred in the middle of the path.

This is the same reason a trading system needs more than its end-of-day P&L. The return can be unchanged while the position path, order count, or risk boundary has changed. A good result does not prove a good transition.

Replay needs a contract

Replay is not the claim that a probabilistic model will emit identical tokens forever. It is a narrower engineering promise: given a recorded scenario and controlled dependencies, the system should make the same important decisions, or explain why it did not.

That requires a trace schema. For each run, I would save the scenario ID, model and prompt versions, retrieved document IDs, tool names, normalized arguments, policy decisions, dependency responses, retry count, and final output. A timestamp is useful, but it is not enough. The test needs the inputs that influenced the branch.

External tools should be replayed from fixtures before the model is evaluated. If a search endpoint returns a different document set during every test run, a changed answer is not evidence of a model regression. It may only be evidence that the fixture was never fixed.

The fixture should include failures, not just happy responses. A tool timeout, empty retrieval result, permission denial, and malformed payload each exercise a different decision boundary. If the harness only records successful calls, it trains the system and the reviewers to ignore the states where agent behavior matters most.

Compare decisions at the right level

Exact trace equality is often too strict. Retrieval order can change without changing the usable evidence. An agent may choose one of two equivalent tools. A response can differ in wording while preserving the same factual claims.

The test should compare normalized invariants instead:

• Did the agent call an approved tool?
• Were required arguments present and within bounds?
• Did it use the retrieved evidence that the rubric required?
• Did it refuse when the fixture represented a forbidden action?
• Did it retry only the failures marked retryable?
• Did it cite the source records that support the answer?

Those checks are more valuable than token-level similarity because they describe the contract. The wording can vary. The authority boundary should not.

This also makes failures easier to classify. A changed answer with an unchanged decision trace may be a language-quality issue. A changed tool call is a routing regression. A missing refusal is a policy regression. A second side-effecting call after a timeout is an idempotency regression. One output score cannot express those differences.

The test must observe side effects

The highest-risk agent tests are not the ones with the most impressive prompts. They are the ones connected to real state. A test that can send an email, modify a record, place an order, or publish content needs a write boundary around the action.

In a test environment, that boundary can be an allowlist and a fake adapter. The adapter should record the proposed action, return a deterministic result, and expose whether the call happened once or more than once. The assertion then checks both the decision and the postcondition.

For example, after a simulated transport timeout, the correct result may be a reconciliation lookup rather than a second write. The final answer might be identical either way: the agent can report that the action is unresolved. The trace distinguishes a careful recovery from a duplicate side effect.

What I would put in the release gate

I would keep two layers. The first is a small golden set with fixed fixtures and explicit expected decisions. It runs on every prompt, model, tool, or policy change. The second is a broader set scored for answer quality, latency, and cost. It runs less frequently and is used for investigation.

The release gate should fail on a changed safety boundary even when the answer score improves. It should also preserve the failing trace, not just print a red number. A reviewer needs to see which input, retrieved record, tool argument, or policy check changed.

The practical rule is simple: replay the decision that made the answer possible. If the test cannot show why the agent selected a tool, accepted evidence, retried a failure, or refused an action, it is not yet testing the agent. It is testing a text sample produced by the agent.

That distinction matters most when the system is working. Failures are obvious when the answer is wrong. The dangerous regressions are the ones that keep the answer plausible while quietly changing the path underneath it.