Test AI Latency from First Token to Finished Answer

A fast first token does not prove that an AI request finished on time. For QA, measure when useful output begins, when the answer completes, and whether the completed answer passes its checks. Those are separate acceptance conditions.

This extends AI Infrastructure 101 for QA Engineers into model serving: the software that schedules requests and turns model execution into responses. The exercise below is a proposed synthetic test, not a production benchmark or a claim about any employer system.

Where the waiting happens

For a conventional autoregressive language model, prefill processes the input context and decoding generates output tokens. Before execution, a request may wait for the serving scheduler. vLLM exposes distinct measurements for queue time, prefill time, decode time, time to first token, and end-to-end request latency. Its metrics documentation defines these server-side measurements.

Think of the request path as arrival, waiting, input processing, output generation, and completion. This is a diagnostic sketch, not a guarantee that every implementation executes five isolated stages. The application and network add their own work outside the model server.

For a client test, define three timestamps on one monotonic clock: request start, first nonempty answer content, and successful stream completion. Subtract start from the second timestamp for client-observed time to first content. Subtract start from the third for total completion time. These client boundaries differ from server token timestamps. Do not silently label the two measurements as equivalent.

A response header, keepalive, or empty streaming event should not satisfy the first-content assertion. For a structured-output workflow, even the first visible text may be unusable until the full object arrives and passes validation. Write down what the product considers useful before measuring it.

Why serving configuration belongs in regression testing

Scheduling can trade one latency objective against another. vLLM documents chunked prefill, which divides input processing into smaller chunks and batches it with decode work. Its optimization guide explains how the token budget can affect inter-token latency and time to first token. The right setting depends on the workload and hardware; the guide is not a promise that one value improves every request.

The QA implication is a test-design recommendation: include both short interactions and long-context tasks when reviewing a serving change. A dashboard average cannot tell you whether your particular workflow still meets its contract. Keep the model fixed while changing the serving configuration so that the comparison has a clear subject.

Build a small, explicit workload

Use an isolated test environment and four synthetic request classes. None of the following sizes is a universal recommendation. They are starting points for an experiment that should fit within your model context limit and test budget.

• Short input, short output: approximately 256 input tokens and an output cap of 64 tokens.
• Long input, short output: approximately 4,096 input tokens and the same output cap.
• Short input, longer output: approximately 256 input tokens and an output cap of 512 tokens.
• Long input, longer output: approximately 4,096 input tokens and the larger output cap.

Generate harmless fixtures such as numbered facts and requests for summaries. Count input tokens with the serving tokenizer. Record actual output lengths: an output cap is a maximum, not a guarantee of how much the model generates. An early stop can otherwise make a run appear faster merely because it did less work.

Pin the model revision, tokenizer, prompt fixtures, decoding parameters, serving version, hardware allocation, and cache policy. Record cache state where observable. Treat warmup separately from measured traffic and keep the warmup procedure identical across configurations.

Run each class alone first, then a documented mixture. Try bounded concurrency levels of 1, 4, and 8 as an initial diagnostic sweep. Save the request count, duration, and exact mixture for each run. These small experiments help locate differences; they do not establish production capacity.

Define the verdict before the run

Here is an illustrative product contract: first answer content within two seconds, completion within twelve seconds, and an output that passes the fixture-specific correctness check. The numbers are invented for this exercise. Replace them with agreed requirements before treating the test as a release gate.

Apply all three conditions to each request. A request that starts streaming at one second but finishes at fifteen seconds fails completion. A request that finishes at six seconds with an invalid result fails correctness. A timeout remains a failed request even when other completed requests are fast.

Report the fraction passing all conditions for each request class. Alongside that fraction, show latency distributions for completed requests, output-token distributions, error counts, and timeout counts. State the sample size. Avoid presenting a high percentile from a tiny sample as a stable property of the system.

For a configuration comparison, repeat the same workload and alternate the order of the configurations. If the apparent improvement disappears across repetitions, preserve that uncertainty in the verdict. Do not select the fastest repetition and call it the result.

Use the failure to choose the next experiment

When first content slows, compare the client observation with server queue and prefill measurements. When completion slows after a similar start, inspect output length, decode timing, and stream delivery. These are investigation paths, not automatic root-cause assignments. Without server telemetry, report the boundary you measured and leave the internal cause unresolved.

Follow Test the Trace Before You Trust the Dashboard to check that request identifiers connect the client evidence to the relevant server work. Preserve the failed output as well as its timing.

The release artifact should make one decision reviewable: under this declared workload, did the serving change preserve timely, correct completion for every required request class? That is a useful QA question even before anyone starts tuning the GPU.