Test the Trace Before You Trust the Dashboard

An AI request is easier to debug when you can follow the same operation from the application through retrieval, model access, and tool execution. Before relying on that view, test that the connections survive each boundary. A trace full of individually healthy spans can still leave out the step that explains a failure.

This Saturday field note adds one exercise to The Tool-Call Checklist for QA Engineers: deliberately break trace propagation in a synthetic environment, then check whether your observability test catches the break. The exercise below is a proposed test design, not a report of production measurements.

Three terms worth separating

A trace groups related work. A span describes one operation within that work, such as an HTTP request or a database lookup. Context propagation carries the identifiers needed to connect operations across execution boundaries. OpenTelemetry explains these relationships in its trace documentation and context propagation guide.

For a simple synchronous request, imagine a support assistant calling an application-owned retrieval service and then an application-owned model gateway. The application creates a root span. Each outgoing call and receiving service contributes connected spans. The resulting view lets a tester inspect the route taken through the system.

Define the boundary of that view explicitly. A client span around an external model API measures the call from your application. It does not reveal the internal queue, hardware, or execution stages of the provider. Label that interval as an external call rather than inventing an explanation for its duration.

Write the expected shape first

Use a fixed synthetic request: "Find the refund policy for fixture order QA-104." Stub the retrieval response and model response so that this exercise tests instrumentation rather than answer variability. Do not use a customer conversation as the fixture.

Before running it, record the operations your application must expose:

• One root operation for the assistant request.
• A retrieval client call connected to the receiving retrieval service.
• A model-gateway client call connected to the receiving gateway.
• A completion outcome on the application operation.

For this fixture, require the application-owned synchronous operations to share a trace identifier and have the expected parent relationships. Avoid asserting an exact total span count: library instrumentation may legitimately add spans. Assert the required operations and connections instead.

These are proposed acceptance criteria for this topology. They are not universal rules for every distributed system. A background workflow may intentionally start a separate trace and link it to its trigger. OpenTelemetry supports span links for causal relationships beyond a simple parent-child tree; the expected graph should reflect the architecture you actually chose. See the trace documentation on span links.

Break one boundary on purpose

First run the fixture with propagation enabled and save the exported spans. Then configure the test proxy between the application and retrieval service to remove the trace-context headers. Keep request and response bodies unchanged.

The answer may still be correct because retrieval still works. Your instrumentation assertion should now fail: the receiving retrieval span no longer belongs to the expected connected request. That is the useful separation. The functional test asks whether the feature returned the fixture answer. The observability test asks whether the evidence needed to investigate that answer survived.

If both tests remain green, inspect the observability assertion. Does it merely count spans with a matching operation name? Does it find an old successful trace instead of the current run? Give each fixture invocation a unique, non-sensitive test-run attribute and restrict the query to that invocation.

Restore propagation and rerun. The same assertion should pass again. A failing negative control followed by a passing restored run is stronger evidence than a screenshot of a populated tracing dashboard.

Make missing evidence a visible result

Run this exercise with recording and export configured to retain every fixture trace in the isolated test environment. Set a bounded wait for export before querying. Record whether the expected telemetry arrived; do not let a query that returns zero rows pass because there were no explicit errors to inspect.

Treat a collector outage or exporter failure as a failed evidence check requiring diagnosis. It does not automatically mean the assistant returned a wrong answer. Keep functional outcome and telemetry availability as separate fields in the test result so that a broken observation pipeline cannot masquerade as application success.

Add a second synthetic case in which the retrieval stub returns a controlled error. Require the exported evidence to distinguish the failed dependency attempt from the final application outcome. If the application falls back successfully, retain both facts in the expected result. A recovered request should remain explainable.

Keep the fixture small enough to maintain

A useful first version needs one happy path, one stripped-context path, and one dependency-error path. Store the expected relationships alongside the fixture, and review changes to them like changes to any other test oracle. Otherwise an instrumentation regression can be hidden by updating the expected graph after seeing what the system produced.

Capture operation names, service names, trace relationships, outcomes, and a synthetic run identifier. Avoid adding raw prompts or credentials merely to make debugging convenient. The fixture should prove that correlation works without requiring sensitive payloads.

The release question is concrete: when this request fails at a known boundary, can a tester locate that boundary using the exported evidence? Make that answer executable before expanding the dashboard.