AI Infrastructure 101 for QA Engineers: From a Prompt to a Test Result

Short answer: AI infrastructure is the chain from the data center and accelerator hardware to the model-serving software that produces a response. For QA engineers, understanding that chain makes it easier to distinguish an application defect from a compute, memory, networking, serving, or evaluation problem.

A QA engineer does not need to become a semiconductor designer to test an AI system well. But it helps to understand the path between a user's prompt and the answer that eventually appears on screen.

That path is a stack of dependencies: electricity reaches a data center; the data center powers servers; servers connect CPUs, GPUs, memory, storage, and networking; software schedules the workload; and an AI model turns the request into tokens. The final response is only the visible end of that chain.

The short version of the stack

Think of an AI request as moving through seven layers:

1. Power and cooling: The facility supplies electricity and removes heat from dense equipment.
2. Compute: CPUs handle general orchestration while GPUs, TPUs, or other accelerators perform large parallel operations.
3. Memory: High-bandwidth memory keeps model weights and active data close to the accelerator. Slow memory access can limit performance even when compute capacity is available.
4. Networking: High-speed links move data between accelerators, servers, storage, and users.
5. Runtime software: Drivers, kernels, schedulers, batching, caching, and serving systems decide how the hardware is used.
6. Model execution: The system tokenizes the request, runs prefill and decoding, and returns output incrementally.
7. Evaluation and application behavior: The application decides whether the answer is useful, safe, and correct for the task.

The QA mistake is to test only layer seven. A slow response may be an application defect, but it may also be a memory bottleneck, a saturated network, inefficient batching, or a cold model replica. A wrong answer may reflect retrieval, tool use, prompt construction, model behavior, or an evaluator that silently accepted a false result.

What this changes in QA automation

The test harness should preserve evidence across the stack. For each test case, record the input, model and prompt version, retrieved context, tools invoked, output, latency, token counts, error state, and evaluation result. This does not require exposing proprietary infrastructure. It requires making the path observable enough to explain a failure.

The same harness should be able to compare more than one model. Teams may use frontier models, smaller open-source models, or task-specific components depending on the job. The important design choice is a common contract: the same test cases, rubric, trace format, and release gates should apply across providers. That reduces vendor dependence and makes model changes easier to audit and roll back. A practical harness can route the same scenario through multiple providers—including open-source models where appropriate—while preserving comparable traces and evaluation rules.

LLM-as-a-judge is another test component

An LLM judge can help scale evaluation, but its score should not automatically be treated as truth. Compare it with independent human labels and inspect disagreements. For categorical judgments, Cohen's kappa can measure agreement beyond chance. The number is meaningful only when the team also explains the rubric, sample, label balance, reviewer process, and disagreement cases.

A useful regression dashboard therefore tracks task success, human agreement, false positives, false negatives, hallucination rate, latency, cost, and critical-scenario failures. A model has improved only when the important measures move in the right direction together. A higher average score can hide a serious regression in a narrow but important workflow.

Frequently asked questions

What is the difference between AI training and inference?

Training updates model parameters using large datasets. Inference uses an already-trained model to process a request and generate an output. Most application QA work focuses on inference: latency, tool calls, retrieval, safety, correctness, and behavior under realistic inputs.

Why does memory matter for AI systems?

Model weights and active requests must be moved through memory efficiently. High-bandwidth memory can help accelerators keep data close to computation, while memory pressure, transfers, or contention can increase latency and reduce throughput.

How should QA teams test an LLM application?

Start with a representative golden set, define a rubric before testing, capture the prompt and retrieved context, record tool calls and output, and compare results across controlled changes. Review important failures with humans instead of relying only on a model-generated score.

What is an LLM-as-a-judge?

It is an evaluator model that scores or classifies another model's output against a rubric. It can make evaluation scalable, but it should be calibrated against human labels and audited for systematic disagreement. Cohen's kappa is one useful measure for categorical agreement beyond chance.

Sources and further reading

• Suneet Malhotra's QA and AI automation work
• Suneet Malhotra on GitHub

The practitioner takeaway

Start with one small golden set of representative QA scenarios. Run it against the current system and save the complete traces. Then change one thing—model, prompt, retrieval, tool, or serving configuration—and run the same set again. Require an explanation for every changed result.

That is the beginning of an evaluation layer: not a single score, but a repeatable record of what the system saw, what it did, what humans judged, and where it failed. For QA engineers, this is the bridge between infrastructure literacy and reliable AI automation.

This is Day 1 of a practical series on AI infrastructure and QA automation. I will build from the physical stack toward model evaluation, observability, and production testing.