A Fallback That Looks Healthy Can Still Be Wrong

A fallback is not correct merely because it prevents an error from reaching the user.

In an AI workflow, a failed retrieval call might fall back to a cached answer. A model timeout might select a smaller model. A tool authorization failure might return a read-only explanation. Each behavior can be sensible. Each can also produce a response that looks healthy while silently changing what the system was able to know or do.

That is the failure mode worth testing: graceful degradation that preserves availability but loses meaning.

The response code is not the result

Traditional service tests often stop at a stable boundary. The request returned a 200 response. The JSON parsed. The latency stayed below the threshold. Those checks are useful, but an agent can pass all three while violating the product contract.

Imagine an assistant that answers questions about an internal runbook. The primary path retrieves the current document, adds its revision ID to the prompt, and produces an answer. When retrieval fails, the fallback asks the model to answer from its general training data. The endpoint still returns 200. The answer may be fluent and even plausible. It is no longer an answer grounded in the runbook.

The defect is not that the fallback exists. The defect is that the system has hidden the loss of evidence.

For QA, this means the assertion cannot be only status == 200. The test needs to inspect the decision trace and the user-visible contract. Which source was available? Which model ran? Did the system claim freshness it did not verify? Was a write action refused, or did the fallback pretend it completed?

Model the degraded path explicitly

I use a small state table before writing the test. It makes the fallback's obligations concrete.

The important row is not the happy path. It is the boundary between an answer the system is authorized to give and one it merely knows how to phrase.

A fallback should have a declared semantic budget. It may preserve formatting, but not freshness. It may preserve explanation, but not execution. It may preserve a best-effort answer, but only if uncertainty is visible. If those tradeoffs are not written down, the implementation will choose them accidentally.

A practical test with synthetic evidence

Here is a compact scenario for an automated regression suite. Create two synthetic documents:

• runbook-v1: "The service uses port 8443."
• runbook-v2: "The service uses port 9443."

Make the primary retriever return runbook-v2, then ask, "Which port does the current runbook specify?" Assert the answer contains 9443 and records the revision as v2.

Next, inject a timeout into retrieval. The expected result depends on the product contract, but it must be explicit. A strict assistant should refuse to claim the current port. A stale-cache design may answer 8443 while labeling the cache as v1 and its age. What should fail is an unqualified answer that says 9443, because the test has removed the evidence needed to support it.

Then inject a timeout into the primary model after retrieval succeeds. If the smaller fallback model is allowed to use the same context, the answer can remain valid, but the trace should identify the fallback model and the reason for the switch. If the fallback cannot accept the retrieved context, it must not silently answer as though it did.

The assertions should cover at least four surfaces:

1. The answer: factual content and uncertainty language.
2. The trace: retrieval status, source revision, model, attempts, and fallback reason.
3. The action boundary: whether any tool call or write was attempted.
4. The telemetry: a counter or event that lets an operator distinguish primary success from degraded success.

This is not testing the model's prose alone. It is testing whether the system preserved the meaning of its answer while its dependencies changed.

Observability must expose the downgrade

Many dashboards collapse primary and fallback responses into one success metric. That makes the system look more reliable as dependencies become less reliable. Availability rises on paper while evidence quality falls.

Track fallback rate separately from request success. Break it down by cause: timeout, empty retrieval, rate limit, authorization, and schema failure. Record the selected model and whether the response depended on cached or live context. Sample the decision trace with care for privacy, but do not remove the fields needed to explain the transition.

The useful alert is not always "fallback happened." Some fallbacks are expected. The useful alert may be "fallback answered freshness-sensitive questions" or "write-capable requests reached a read-only model without a refusal." Those alerts are tied to semantic risk rather than plumbing noise.

Test the boundary, not just the branch

There is a subtle difference between branch coverage and contract coverage. A unit test can execute the timeout handler and confirm that it returns a response. That proves the branch is reachable. It does not prove that the branch preserves the rules that matter to a user.

For each fallback, add at least one invariant that must remain true and one capability that must become false. For example, a cached retrieval may preserve the document's last known text, but it must lose the claim that the text is current. A smaller model may preserve a classification schema, but it may lose permission to call tools. A read-only mode may preserve explanations, but it must not create an order, update a ticket, or send a message.

These paired assertions are easier to maintain than a large collection of example answers. The invariant describes the contract. The lost capability describes the safety boundary. The wording can change without making the test meaningless.

Also test recovery. After the dependency returns, the next request should use the primary path again, unless the system deliberately has a circuit breaker with a documented reset policy. A common bug is a sticky fallback: one transient timeout changes a process-level client, and every later request quietly uses the degraded path. The response remains valid enough to avoid an obvious outage while quality and freshness drift for hours.

The recovery test should therefore send requests in sequence: primary success, injected failure, fallback response, dependency recovery, primary response. Assert the route taken for every request. A fallback is part of a state machine, not an isolated exception handler.

The checklist

Before shipping a fallback, ask:

• What capability is lost when this path runs?
• What claims must the fallback refuse to make?
• Can the user tell that the system degraded?
• Does the trace identify the cause, model, source, and attempt count?
• Are side effects disabled or reconciled when execution becomes uncertain?
• Does monitoring separate degraded success from primary success?
• Is there a test that removes the evidence behind the most dangerous claim?

A healthy-looking response is an observation about transport. Correctness is an observation about the contract. Agent QA needs both. The fallback earns its place when it keeps the system useful without pretending that an unavailable dependency was available.