The Fast Answer That Belonged to Another Tenant

An AI answer cache must check who is asking before it reuses an answer. Two customers can submit the same question and require different responses. A fast, fluent cache hit can therefore be a failed isolation test.

This synthetic case study follows the series on AI latency. It uses two fictional tenants and invented policy documents. The request sequence is a proposed regression exercise, not a report of a production incident or a measured performance improvement.

Where the cache changes the request path

Imagine a support assistant that retrieves a customer policy and asks a model to explain it. To avoid repeating that work, the application saves complete answers. On a later request, it checks the answer cache before retrieval and generation.

An exact-match cache looks up a key derived from the question. A semantic cache can also match paraphrases using vector similarity. Redis documents this distinction and supports metadata filters alongside similarity search in its semantic caching guide. This article concerns saved answers, not the internal attention state used by model-serving systems.

The important architectural change is the shortcut. If authorization previously happened only during document retrieval, a cache hit can skip the very step that restricted the answer. The test needs to exercise both branches.

Microsoft makes the responsibility explicit for a shared Azure Managed Redis instance: the application enforces tenant separation. Adding a tenant prefix is useful only when every relevant read and write uses the trusted tenant identity.

Construct a failure with two documents

Create tenant Amber and tenant Birch. Each has a document named "Return policy," but Amber allows returns within 14 days and Birch allows 30 days. These numbers are arbitrary fixture values chosen to make a wrong answer visible. Give each document an opaque identifier and a revision.

Use the identical question for both users: "How long do I have to return an item?" Stub generation initially so that the output faithfully reflects the retrieved fixture. This isolates the cache behavior from model variability.

In the deliberately broken implementation, the answer key contains only the normalized question. Start with an empty cache, authenticate as Amber, and ask the question. The miss retrieves Amber's document, returns 14 days, and stores that answer. Now authenticate as Birch and ask again. The key is identical. The faulty cache returns 14 days without consulting Birch's document.

That outcome follows from the fixture and key definition; it is not an empirical benchmark. Both requests could receive HTTP 200, and the second could avoid a model call. Neither observation establishes correctness.

Turn the sequence into assertions

Keep the cache alive across requests within each test. Clearing it before every request removes the condition you are trying to detect. Reset it between independent scenarios, and repeat the sequence in reverse order.

Check the answer, source identity, cache decision, and generation call count separately. A system that disables caching entirely can pass the isolation rows, so retain the repeated same-tenant request as a positive control. A system that merely rewrites "14" to "30" must still fail if it cites Amber's source.

For semantic caching, inject a deterministic similarity result in the component test so that the forbidden candidate would otherwise qualify. Then run an integration test against the actual index and embedding configuration. A paraphrase that never matches anything proves little about filtering.

Repair the eligibility rule

For this exercise, derive the tenant from authenticated server context. Restrict candidate selection to the caller's authorized scope before accepting a match. Apply the same scope when storing an answer, and reject entries whose required metadata is missing. Treat similarity as a relevance signal after eligibility has been established.

Tenant identity alone is insufficient when two users inside a tenant have different permissions. Extend the fixture with a restricted document and an ordinary user. Specify which authorization context makes an entry reusable, then assert that the ordinary user cannot receive an answer derived from that document.

Also define freshness. Warm the cache, revoke document access, and repeat the request before its expiry time. The expected result is no disclosure of the revoked content. A time-to-live only sets an age limit; in this fixture it does not represent current permission. The design might recheck access, invalidate dependent entries, or bypass caching for restricted answers. Test the chosen contract explicitly.

Make the regression diagnosable

Record a request identifier, an opaque authorization-scope identifier, the cache outcome, an entry identifier, source revisions, and the reason an entry was accepted or rejected. Keep fixture content out of general-purpose production logs. The trace-testing lesson applies here: assert that the cache branch actually emits the evidence needed to explain the result.

Finally, remove the scope filter in a controlled test build. Require the cross-tenant test to fail, restore the filter, and require both isolation and legitimate reuse to pass. This negative control shows that the assertion detects the intended defect.

The limitation is scope: this exercise tests answer reuse, not every storage layer, race condition, or side channel. Expand coverage to concurrent requests and permission changes once the deterministic case works. Which existing test would fail if your answer cache stopped checking the caller's scope?