The Tool-Call Checklist for QA Engineers

The fastest way to find an agent bug is to inspect the tool call, not the final sentence.

An agent may answer with perfect grammar after selecting the wrong tool, inventing an argument, or attempting an action that the user never authorized. A text assertion can miss all three failures because the final answer is only a summary of a decision that already happened.

For QA, the useful boundary is the tool invocation: selected tool, arguments, authority, result, and user-visible claim. Test that boundary directly.

A small synthetic example

Imagine a support agent with three tools:

• search_orders(customer_id, date_range) reads order history.
• create_return(order_id, reason) starts a return request.
• cancel_order(order_id) cancels an order.

The user says, "Find my recent order and tell me whether I can return it." The expected behavior is a search followed by an explanation. A return should not be created. A cancellation is outside the request entirely.

The agent can fail in several ways while still sounding helpful. It can call create_return because the phrase "can I return" was treated as an instruction. It can call search_orders with a different customer ID from the authenticated session. It can call cancel_order after a retry replays an earlier intent. It can return a confident answer after the search failed and no order data was available.

Those are not writing defects. They are decision and authorization defects.

The five assertions

Start with five assertions for every tool-capable workflow.

1. Tool selection

Assert that the selected tool belongs to the set allowed for the user intent. For the synthetic request, search_orders is allowed and the two mutation tools are not. This is more precise than asserting that the answer contains the word "return."

2. Argument integrity

Assert arguments against trusted request state, not only against the model output. The customer ID should come from the authenticated principal. A date range should have a declared default and a bounded format. Reject unknown fields, missing required fields, and values outside the permitted scope.

A useful negative test is to place a second customer ID in retrieved text and verify that the agent cannot substitute it into the tool call. Retrieval is evidence for an answer, not authority to change identity.

3. Authorization before execution

A policy check should run after argument construction and before the side effect. Test that an unauthorized call is refused without contacting the mutation tool. The trace should preserve the attempted tool, policy decision, and refusal reason.

Do not treat a refusal as a failed test. If the request is outside the agent authority, refusal is the passing outcome.

4. Idempotency under retry

Inject a timeout after a mutation is submitted but before the client receives the result. Then retry the workflow. The test should verify one durable intent, not two returns or two cancellations. This requires an idempotency key or an equivalent reconciliation step.

Also test a timeout before submission. The expected behavior is different. A system that cannot distinguish those states cannot safely decide whether retry is allowed.

5. Claim accuracy

The final response must match the observed tool result. If the tool returned no order, the answer must not cite an order number. If the mutation was refused, the answer must not say it completed. If the source is stale, freshness must not be implied.

This is where final-answer evaluation belongs, but it should be the last assertion in the chain rather than the only one.

A compact trace contract

For each decision, record a trace that a test can inspect without parsing prose. A minimal event can contain:

trace_id, intent_id, actor_id, tool_name, arguments, policy_result, attempt, dependency_result, side_effect_id, and claim_state.

The values should make the state transition observable. For example, policy_result=deny should pair with no side_effect_id. A submitted mutation with an unknown transport result should pair with dependency_result=unknown, not failed, until reconciliation completes.

Do not log secrets or complete payment details. Redact sensitive values while retaining enough structure to compare the request with the permitted scope.

Saturday checklist

Before shipping an agent tool, run these cases:

1. Happy path: correct tool, trusted arguments, permitted result.
2. Wrong intent: mutation wording appears inside a question.
3. Prompt injection: retrieved text asks the agent to use a different identity or tool.
4. Malformed arguments: missing, extra, or out-of-scope fields.
5. Authorization refusal: verify no side effect occurred.
6. Timeout before submission and timeout after submission.
7. Empty or stale evidence: verify the answer exposes the limitation.
8. Replay: same intent arrives twice and produces one mutation.

Branch coverage tells you that each path ran. The trace contract tells you whether each path preserved meaning. For agent QA, that difference is the work.