Test Trace Sampling Before Trusting AI Failure Rates

A trace dashboard can help explain an AI failure without telling you how often that failure happens. If your pipeline keeps every error but only some successful requests, the displayed error percentage describes retained traces. It does not describe all requests. QA should test that distinction before using a dashboard as a release gate.

This lesson builds on testing trace propagation. That exercise checked whether evidence stayed connected. Today we check which evidence survives selection. The workload, counts, and acceptance criteria below are synthetic test designs, not production measurements or employer data.

Where sampling sits

Imagine a support assistant with an application service, a retrieval service, and a model gateway. Each request produces spans describing operations. A sampling decision determines which trace evidence the system records or retains.

Head sampling decides early, before the request outcome is known. Tail sampling can use later information, including errors and latency, to decide what to retain. OpenTelemetry explains these tradeoffs in its sampling guide. Keeping unusual failures is useful for diagnosis, but selecting by outcome changes the population you see.

For this exercise, draw the path as: synthetic harness -> assistant -> collector -> trace store. Keep a separate result ledger in the harness. That ledger records one final outcome per logical request, independently of trace retention. It is the reference for this controlled experiment, not an assumption that any production logging pipeline is complete.

A correct collector can produce a misleading percentage

Construct 1,000 synthetic requests: 980 successful answers and 20 application failures. Configure an isolated retention policy to keep all failures and a selected tenth of successes. For a deterministic fixture, choose exactly 98 successful trace IDs; a random sampler would fluctuate around that count.

Nothing in this arithmetic requires a broken model or a broken collector. The retained set deliberately overrepresents failures. A panel labeled "request failure rate" would be wrong if it divided failed retained traces by all retained traces and presented that result as population-wide.

Make the panel name part of the assertion. "Failure fraction among retained traces" is accurate for the second row. "Application failure rate" should use a separately validated request counter or another complete, clearly bounded measurement source. A trace-derived estimate needs a justified sampling correction; multiplying every retained row by ten would also multiply the failures that were already retained in full.

Separate three kinds of failure

For the synthetic assistant, define application failure as failure to deliver the required answer within the fixture deadline. Track dependency failures separately. A retrieval timeout followed by a successful fallback is one failed dependency attempt and one successful application request.

Add a third outcome: answer quality. A model can return a fluent, incorrect answer while its HTTP request completes successfully. In the fixture, label this case using a deterministic expected fact, such as the refund period in a synthetic policy document. Do not assume transport-error sampling will preserve semantic defects.

This is a proposed instrumentation contract: request outcome, dependency outcome, and evaluation outcome have separate fields. Define when the evaluation result becomes available. If a judge runs after a trace-retention decision, its verdict cannot influence that earlier decision unless you have explicitly designed a later retention mechanism.

Test the policy with controlled cases

Start with retention disabled in the isolated environment so every fixture trace should arrive. Check the complete ledger against distinct request identifiers in the trace store after a bounded export wait. Then enable the intended policy and rerun with new identifiers.

Use four fixtures: ordinary success, application error, successful fallback after a dependency error, and an incorrect answer with successful transport. Write the expected keep-or-drop decision for each before running. For this policy test, use deterministic selection for ordinary successes; test probabilistic behavior separately with a predefined tolerance and sufficient repetitions.

Assert both membership and counts. Finding twenty error traces is insufficient if they are stale results from a previous invocation. Every retained trace must belong to the current run, and every required fixture must be identifiable. Store the collector version and policy configuration alongside the result.

Now introduce a negative control: configure the upstream SDK to drop the designated error fixture before export. The downstream retention assertion should fail. A tail sampler cannot recover evidence it never received. Restore the upstream configuration and require the same assertion to pass.

Exercise the collector boundary

The OpenTelemetry Collector tail-sampling processor documentation describes decision timing, trace buffering, and handling of late spans. It also requires spans for a trace to reach the same processor instance for effective tail sampling. Check the documentation matching your deployed version before choosing configuration values.

Add one delayed error span and one controlled burst to the test plan. Verify whether the error arrives before the configured decision and whether the burst causes evidence loss. Record missing telemetry separately from application success. Do not turn a missing trace into a passing outcome because the query returned no errors.

For a small release check, require the fixture ledger to reconcile, required diagnostic cases to survive the documented policy, and every dashboard denominator to name its source. Save a screenshot only after these assertions pass; the executable result is what makes the screenshot trustworthy.

The practical question for your next dashboard review is specific: does this percentage describe all requests, evaluated answers, or only the traces we chose to keep?