Test Model Readiness Before Sending User Traffic

A model server is ready when it can accept the workload its service promises. A running process, an open port, and a completed answer are different observations. QA should test the transition between them before a deployment sends user traffic to a new replica.

This Monday infrastructure note follows testing AI latency. That article separates the first token from a finished answer. Here, the question comes earlier: what evidence allows a replica to receive requests at all?

The exercise below is synthetic. Its timings, fixtures, and acceptance rules are proposed test inputs, not production measurements or claims about an employer system.

Three probes, three decisions

Kubernetes distinguishes startup, readiness, and liveness probes. A startup probe gives initialization time to complete before readiness and liveness checks begin. Readiness determines whether a Pod should receive traffic through matching Services. Repeated liveness failures can trigger a container restart. Those are different control decisions, even when an application exposes similarly named endpoints. See the Kubernetes probe documentation.

For an AI service, define what each decision means in your architecture. The HTTP frontend might start before the inference worker finishes initialization. A wrapper might depend on a separate model process. These are possible designs to inspect, not assumptions to make about every serving framework.

Write a small state contract: initializing, eligible for traffic, temporarily unavailable, and irrecoverably stuck. For each state, name the observable signal and the action it permits. The important review question is whether restarting the process could plausibly repair the condition being reported.

Read the endpoint contract

vLLM documents a health endpoint and a model-list endpoint in its online serving documentation. Their existence does not establish the acceptance contract for your entire application. Record the installed version, inspect its behavior, and test the wrapper that your deployment actually probes.

A service can have additional requirements: a particular model revision, an adapter, a prompt template, or an authenticated route. Decide which prerequisites belong in readiness and which belong in a deployment smoke test. Do not silently promote a generic health response into proof that all these conditions hold.

Use a cheap, deterministic readiness check where possible. Run a bounded inference smoke separately to exercise the serving path. Making every frequent probe generate a long response would add workload to the system being measured; the test design should account for that cost.

Build a controllable startup exercise

Create a test deployment with two distinguishable replicas: an existing ready replica and a new candidate. Use synthetic prompts with no private data. Add a test-only barrier that holds candidate initialization before its declared readiness prerequisites complete.

Keep a request stream running through the same gateway and Service path used by clients. Include a request ID and record the replica identity in trusted telemetry. Sending requests directly to the candidate is useful for diagnosis, but it bypasses the routing behavior this exercise must verify.

During the hold, require the candidate to remain ineligible for normal traffic. Verify that the existing replica continues answering. Then release the barrier, wait for the readiness transition and routing propagation, and verify that requests reach the candidate and complete successfully.

Do not require an instantaneous transition. Choose a bounded observation window before running the test, based on your deployment configuration. Preserve the timestamps for barrier release, readiness change, routing visibility, and first completed candidate response. If the bound is exceeded, report which transition stalled.

Add a negative control

In an isolated test environment, deliberately replace the readiness decision with a check that only confirms the frontend is listening. Keep the initialization barrier closed. The same test should detect traffic reaching a candidate that cannot satisfy the request contract.

If it still passes, investigate the test before trusting the deployment. Perhaps traffic never reached the candidate, replica identity was missing, or the assertion only checked HTTP status. Restore the real readiness implementation and repeat the sequence.

This negative control is essential because a green run can mean that routing protected users, or simply that the test never exercised the dangerous interval. Save evidence that distinguishes those outcomes.

Test recovery without inventing a restart cure

Next, introduce a temporary failure in a required dependency after startup. State the intended policy first: should the replica stop accepting work, return a bounded error, or use a documented fallback?

Observe readiness, restart count, request outcomes, and recovery after the dependency returns. For this exercise, require no restart when the process remains healthy and restarting cannot repair the dependency. That is a chosen application requirement, not a universal Kubernetes rule.

Also test an intentionally stuck worker for which a restart is the expected recovery. These paired cases prevent a single generic unhealthy signal from hiding two different operational needs. Kubernetes warns that incorrect liveness configuration can cause cascading failures under load; see its probe configuration guidance.

Keep a compact release record

For each run, preserve the model and server revisions, deployment configuration, injected condition, ordered transition timestamps, request outcomes, and restart counts. Record missing telemetry as missing evidence, rather than treating an empty error chart as success.

Before accepting the change, answer four questions:

• Did normal traffic avoid the candidate while required initialization was blocked?
• Did the candidate receive and complete requests after becoming eligible?
• Did the deliberately weak readiness check make the test fail?
• Did temporary dependency failure and a stuck worker produce their separately specified recovery behavior?

A completed smoke response proves one serving path worked at one moment. The transition test adds something operationally different: evidence that deployment control decisions track the conditions users actually depend on.