Test RAG Document Deletion Across Answer Paths

An AI document assistant should stop using a deleted document once the product says that deletion has taken effect. Test that promise across retrieval, generated answers, citations, and conversation history. Removing a file from the upload screen proves only that one screen changed.

This Tuesday QA exercise follows what to save before rerunning an AI bug. That field note preserved a document replacement failure. Here we turn document removal into a repeatable regression test with explicit evidence at each boundary.

Everything below is a synthetic test design. The documents, facts, and acceptance rules are invented fixtures, not production measurements or descriptions of an employer system.

Trace the copies that can answer

For this exercise, assume an assistant stores an uploaded file, divides its text into chunks, indexes those chunks, retrieves relevant passages, and sends selected passages to a model. It also keeps conversation history and an answer cache. Your application may have fewer layers; draw the actual path before writing assertions.

The test must distinguish an accepted deletion request from a completed deletion operation. Search visibility can lag a write. Elasticsearch, for example, documents refresh as the step that makes recent index operations visible to search. Its near real-time search documentation describes this boundary. That is an example of a storage contract, not a universal timing guarantee for an AI application.

Choose a product-level completion signal and a bounded deadline. The signal should describe what is complete: source removal, retrieval exclusion, or the whole assistant workflow. A generic successful HTTP response is insufficient unless its documented contract includes the state you intend to assert.

Plant a recognizable synthetic fact

Create document A with an invented instruction: "The synthetic Zephyr equipment request requires approval code KESTREL-482." Create unrelated document B about office hours. Give both stable document IDs and record their revisions and chunk IDs.

Ask for the Zephyr approval code before deletion. Require the answer to contain KESTREL-482 and cite A. Also inspect the retrieved passages and the context actually sent to the model. This positive control establishes that A is searchable and relevant. If it fails, the deletion test has not reached its starting condition.

The unusual code makes stale content easier to identify. It does not establish that the model could never produce the same string accidentally. The decisive evidence is the combination of source identity, supplied context, and the final response.

Specify what happens to old conversations

A fresh conversation and an existing conversation are separate cases. In the existing conversation, the approval code is already present in the transcript. Removing chunks from search does not, by itself, remove text that the application resubmits as conversation history.

For this synthetic product, define a strict rule: after deletion completes, the assistant must neither supply A as evidence nor repeat its approval code, including in an existing conversation. It should explain that the source is unavailable. This rule requires the application to handle retained history and cached answers as well as retrieval.

Another product may intentionally preserve historical conversations. If so, write that policy down and test historical labeling and current-answer behavior separately. Do not let the test silently invent a retention policy. Also keep this exercise separate from proving physical erasure across backups; it tests what the assistant can use and expose through these paths.

Run the sequence without resetting away the bug

Upload A and B, complete indexing, and run the positive control. Keep the same account and application state. Delete A through the normal user-facing path, then observe the declared completion signal within the chosen deadline.

After completion, ask the same question in a fresh conversation and in the original conversation. Include a paraphrase so an exact-query cache miss cannot become the only reason the test passes. Query B too: deleting every document would satisfy the absence check while breaking the application.

Use this acceptance matrix for the synthetic contract:

Preserve state within each sequence, then create clean fixtures for the next independent sequence. In pytest, parameterize the conversation mode and question wording rather than maintaining copied test bodies. The pytest parametrization guide documents how to run one test against multiple input sets. Keep each case identifiable in the report.

Make a broken deletion fail on purpose

In an isolated test deployment, deliberately leave one A chunk retrievable while marking the deletion operation complete. Require the retrieval assertion to fail even if the model happens to omit the code. This negative control proves that the test checks stale evidence independently of answer wording.

Next, restore chunk removal but deliberately retain a cached answer. The answer assertion should fail. These are proposed fault injections, not claims that a particular database or model behaves this way.

Do not repeatedly ask until one answer passes. Save every attempted request, the completion timestamp, retrieved IDs, context snapshot, cache outcome, and assertion result. If visibility never reaches the required state by the deadline, report a bounded failure with the last observed state.

Keep the release decision narrow

A useful result says which paths were exercised and which deletion contract they satisfied. It does not claim that one absent phrase proves complete data removal. Missing context telemetry should appear as an evidence gap, not a green check.

The practical release question is concrete: after the application declares deletion complete, can any tested answer path still use the removed source?