Test Agent Cancellation After the Tool Commits
Suneet Malhotra
Oct 15, 2026
Cancelling an agent request does not prove that its tool action was undone. A useful QA test checks three things separately: whether the interface stopped waiting, whether the worker stopped executing, and whether the destination already changed.
This follows the tool-call checklist with a narrower failure case: the user presses Stop while a tool is creating a ticket. The exercise below uses a fictional ticket service and synthetic events. It describes a test design, not results from an employer system.
Where cancellation sits
An agent application may send a tool request through Model Context Protocol, or MCP, to a server that calls another service. That gives us several boundaries: the interface, the MCP client, the tool handler, and the ticket database. A cancellation signal has to reach work that is still interruptible to prevent a change.
Pin the protocol version in your fixture. In the MCP 2025-11-25 cancellation specification, ordinary in-progress requests use a cancellation notification referencing the request ID. Receivers should stop processing and release resources, but may ignore cancellation when processing has completed or the request cannot be cancelled. Task-augmented requests use a separate mechanism. These are version-specific rules, not a universal recipe for every SDK.
The practical implication is that sending the signal is insufficient evidence for a rollback. A successful database commit can precede the signal even if the response has not reached the client. That timing window is the target of this exercise.
Define the observable contract first
Give the fictional create-ticket operation a durable operation ID, such as qa-ticket-042. Keep it distinct from the request ID used to correlate a protocol exchange. In this proposed application contract, multiple attempts for the same operation must resolve to the same ticket, and reuse with different ticket contents must be rejected.
This follows the general approach in the Amazon Builders' Library discussion of idempotent APIs: caller-provided identifiers let a service distinguish a retry from a new intent. The article also explains the ambiguity of a missing response and the need to handle an identifier reused with different parameters. MCP request correlation alone does not implement this application behavior.
For the fixture, expose a read-only lookup by operation ID. Define three user-visible outcomes: cancelled before creation, ticket created, and outcome unconfirmed. The last state matters when the client cannot reach the lookup service. Do not let the interface translate missing evidence into a claim that nothing happened.
Place barriers around the commit
Avoid relying on a short sleep to reproduce the race. Add test-only synchronization barriers to the fake service. One pauses immediately before the write transaction; another pauses after commit but before returning the response. The test controller releases each barrier explicitly.
Run these cases independently, resetting the fixture between them:
| Controlled interruption | Expected destination state | Expected application behavior |
|---|---|---|
| Cancel before the pre-write barrier is released | No ticket, when the fixture honors cancellation | Report cancellation after verifying no creation |
| Cancel after commit, before response delivery | Exactly one ticket | Reconcile and report the existing ticket |
| Lose the response and make lookup unavailable | State cannot yet be confirmed by the client | Report uncertainty and retain the operation ID |
| Repeat the same operation after a lost response | Exactly one ticket | Return the existing result under the fixture contract |
| Reuse the operation ID with different contents | Original ticket unchanged | Reject the conflicting request |
These expectations belong to this synthetic service contract. A third-party tool without lookup or idempotency support cannot be assumed to provide them. For that integration, document the unresolved outcome and prohibit an automatic retry that could create another ticket.
Assert the destination, then the message
The strongest assertion in the post-commit case reads the ticket store and counts records for the operation ID. The count should be one. Next, assert that the agent did not send a second create operation with a fresh identity. Finally, inspect the user-facing status: it must agree with the reconciled destination state.
Keep conversation rendering separate from reconciliation. A response arriving after cancellation need not resume the abandoned conversation. A dedicated status check can still establish whether the requested action happened and update an activity record.
Capture the operation ID, protocol request ID, protocol revision, barrier reached, cancellation observation, commit outcome, lookup outcome, and final displayed status. Use logical event ordering from the fixture rather than comparing wall clocks across machines. Connect these records to the trace continuity test, so missing telemetry becomes an explicit test failure.
Prove the test can catch the defect
Add a negative control: make the fake service ignore cancellation after commit and make the client immediately display "Nothing was created." The destination assertion should still find one ticket, and the message assertion should fail. If the suite stays green, it only tested that the Stop button changed appearance.
Then remove the deliberately incorrect client behavior and rerun the same controlled ordering. Require a recorded reconciliation result or an explicit unconfirmed state. A green result now has a concrete meaning: this fixture detects a false cancellation claim.
Cancellation also does not grant permission to delete a ticket that already exists. Reversal is another state-changing operation with its own authorization and failure cases. Keep that workflow outside this test unless it is explicitly part of the product contract.
For the next tool integration you test, identify the last point where work can stop without changing the destination. What evidence will your interface show when the user presses Stop just after that point?
Share this post
You Might Also Like
90% Judge Agreement Can Miss Every Defect
An LLM judge can agree with reviewers on 90% of answers while missing every defect. Use a confusion matrix, Cohen's kappa, and explicit false-pass counts to test it.
AI & AutomationBuild a Golden Set That Can Explain a Regression
A golden set needs frozen fixtures, explicit expectations, and case-level comparisons. Build a synthetic QA suite that catches stale evidence and explains changed results.
Career & Best PracticesThe Cheapest Test Is the One You Run Before You Build
The most expensive engineering mistakes in my systems were not hard bugs. They were reasonable implementations of questions nobody had defined.
Quantitative TradingA Signal Is Not a Decision
A signal says that conditions are interesting. It does not say that the portfolio should trade. The missing layer is the decision record between detection and execution.
Latest Blog Posts
90% Judge Agreement Can Miss Every Defect
An LLM judge can agree with reviewers on 90% of answers while missing every defect. Use a confusion matrix, Cohen's kappa, and explicit false-pass counts to test it.
Build a Golden Set That Can Explain a Regression
A golden set needs frozen fixtures, explicit expectations, and case-level comparisons. Build a synthetic QA suite that catches stale evidence and explains changed results.
Test AI Latency from First Token to Finished Answer
A fast first token can hide a late or invalid answer. Test queueing, streaming, and completion with a synthetic workload before accepting a model-serving change.
Related Tools & Demos
The QA Field Manual to Language Models
A free 24-chapter book. Start at βwhat is AI, really?β and finish with a small language model you built yourself β one that reads a failing Playwright test and proposes a fix you can run. Read it in your browser, or download the PDF or the Mac app.
View Source Code βMulti-Model LLM Harness
One interface to call any AI model β capability routing, fallback chains, budgets, circuit breakers, and a quality feedback loop. A practical architecture pattern write-up.
Automated Trading System
Multi-engine trading platform with real-time risk management, regime-based strategy selection, and automated order execution.
View Source Code β
Stay in the Loop
Get weekly insights on AI-driven QA, engineering leadership, and automation strategies.
No spam, ever. Unsubscribe anytime.