Welcome

The best way to understand a system is to build a small version of it yourself. That is what this book is: a guided build of a local, QA-specialized language model — one that reads a failing Playwright test and proposes a grounded fix — from first principles to a green re-run, entirely on your own machine.

You will not need a cloud account, an API key, or anyone's permission. CI logs are proprietary, so everything here runs on-device: the reading, the data, the fine-tune, and the test execution. When you finish, you will have three things — an understanding of how a language model actually works, a small model you shaped yourself, and at least one real automation test that your model helped you diagnose and repair.

The book is linear on purpose, and the ramp is deliberate. Part I teaches the AI itself, from zero — what a model is, how it reads, how it learns — using everyday examples before any test-automation ones; it assumes no AI background. Part II turns the lens on QA: failures, evidence, grounding. Part III you build — data, fine-tune, chat. Part IV you apply and lead — repair a real test, then roll the practice out to a team. Field exercises along the way turn reading into practice. Use the arrows above, or ← and → on your keyboard.

Foreword: from the author's chair

I have spent more than twenty years in quality engineering — running test organizations at places like Tinder, Amazon's Ring, Spokeo, and Live Nation, and today as a Sr. Manager of Test Engineering at Motorola Solutions. In that time I have watched three waves of tooling promise to end flaky tests forever. Record-and-playback didn't. Codeless automation didn't. The current wave — language models — actually might change the daily texture of this job, and that is exactly why I refuse to treat it as magic.

Every durable skill I've seen in great QA engineers comes down to the same move: open the box. Read the stack trace instead of re-running the suite. Read the DOM instead of blaming the framework. This book is that move, applied to the model itself. We will open the box far enough that when a model hands you a fix, you'll know what it saw, what it couldn't see, and whether to trust it — because you'll have built a small one with your own hands.

Three readers are welcome here, and each gets a different payoff. If you're new to QA — or new to AI, read linearly: Part I assumes no AI background and almost no automation background, and every concept arrives with an everyday example before a QA one. If you're a seasoned SDET, the build chapters are yours — real data, real fine-tune, real red-to-green loop, all on your laptop. If you lead a quality org, watch for the "Leader's lens" notes threaded through every chapter: they translate each concept into the decisions you actually own — budgets, gates, trust, and rollout order.

One promise, carried over from how I try to run teams: the numbers in this book are real, and where they are not yet human-validated, the text says so out loud. A benchmark you can't interrogate is a rumor. We don't ship rumors.

What is AI, really?

Strip away the branding and the doom headlines, and the "AI" everyone suddenly works next to is one specific thing: a language model — a program that reads some text and predicts what text comes next. That's it. Not a mind, not a database, not a search engine. A prediction machine for words, grown very large.

You already use a tiny one every day: the word suggestions above your phone keyboard. Type "I'll be there in five" and your phone offers minutes — not because it understands lateness, but because in the mountains of text it learned from, that's what usually comes next. A modern model like the ones in this book is that same idea with billions of learned settings instead of a few thousand. Try being the model yourself:

Three terms, sorted once so the rest of the book never has to: AI is the umbrella marketing word for machines doing things that look smart. Machine learning is the actual method — programs that improve from data instead of hand-written rules. A large language model (LLM) is a machine-learning program trained on text to do next-word prediction. ChatGPT, Claude, Gemini, and the small local models in this book are all LLMs — different brands of the same species.

Two consequences follow from "it predicts words," and they explain almost every AI behavior that will ever puzzle you. First, the model's knowledge is whatever was in the text it learned from — impressively broad, invisibly dated, and silent about your private world. Second, it always produces plausible text, which is not the same as true text. A prediction machine asked a question it can't answer doesn't say "error" — it predicts what an answer would look like. Fluent, confident, and sometimes wrong: you now understand "hallucination" better than most of the industry's marketing departments.

The red pipeline

Not your code. Not the app. A test — the one that clicks the checkout button on saucedemo. You didn't touch it. Nobody touched it. And yet:

1) [chromium] › checkout.spec.ts:42:5 › completes purchase
   Error: locator.click: Timeout 30000ms exceeded.
   Call log:
     - waiting for locator('.btn.primary')
       - locator resolved to 0 elements
   at checkout.spec.ts:42:14

You stare at it. Thirty seconds of waiting, then nothing. The button is right there when you open the site by hand. Somewhere in your team's chat, someone is already typing "can you look at the flaky checkout test again."

This book exists because a language model can read that failure and, in a few seconds, tell you what a tired human takes twenty minutes to see: the front-end renamed a class, the selector went stale, and the one-line fix is to target the data-test attribute instead. But it can only do that reliably if you understand what the model is — what it sees, what it can't see, and when it's guessing.

Under all the ceremony, a language model does exactly one thing: given some text, it predicts what token comes next. That's the whole trick. Every diagnosis, every suggested fix, every confident paragraph is that one move, repeated. You played this game with everyday words in the opening chapter; here is the same machine mid-way through writing a Playwright line, deciding what follows await page.

Everything you learned in Part I — tokenization, embeddings, attention, sampling — exists to make that one prediction good. And "good," for us, has a narrow, testable meaning: given a failing test's error and the relevant evidence, the next tokens should form a grounded diagnosis with a citation, or the honest admission that there isn't enough evidence to give one.

What is a QA model?

A general-purpose LLM will happily write you a sonnet, a SQL query, and a wedding toast. A QA model, as this book uses the term, is narrower and more useful: a model shaped to do one job — given a failing automation test's error output and the relevant docs, produce a diagnosis and a fix that are grounded in evidence, with a citation — and to abstain when the evidence doesn't support an answer.

That last clause is not a footnote; it is the design center. In test automation, a confident wrong fix is worse than no fix. You apply it, push, and wait through a full CI cycle before learning it was a guess. A model that says INSUFFICIENT_EVIDENCE and asks for the trace costs you one sentence. A model that invents a plausible cause costs you the afternoon.

Intent is the piece engineers forget. "Completes purchase" is a title, not a specification. If you don't say that .btn.primary used to be the green Checkout button in the cart footer, the model is guessing. Good models guess well. But a diagnosis grounded in your intent beats a clever guess every time.

So the model we build will be judged on four axes, and you'll see all four again in the evaluation chapter: answer correctness, citation precision (does it point at real evidence?), abstention accuracy (does it refuse when it should?), and hallucination risk (how often does it invent?). Capability is nice. Trustworthiness is the product.

Before transformers

Next-token prediction is not a new idea. The oldest usable version is the n-gram model: count, in a big pile of text, how often each word follows the previous one or two, and predict by lookup. No neurons, no training loop — just counting. And on narrow text, counting gets you surprisingly far. Try it: the widget below has learned from a small corpus of CI log lines, and nothing else.

The wall is obvious once you've hit it. A CI failure's meaning routinely spans forty lines — the selector on line 3, the DOM evidence on line 41. An n-gram model with a two-word window can never connect them, and widening the window explodes the table: every extra word of context multiplies the rows you need to count. By the 1990s the field's answer was the recurrent neural network — read tokens one at a time, left to right, carrying a compressed summary of everything so far. In principle, unlimited memory. In practice, a game of telephone: by the time an RNN has read a long trace, the beginning has been squeezed through so many updates that it has effectively evaporated. Long-range links — this timeout relates to that selector — are exactly what got lost.

The 2017 transformer removed the bottleneck with a blunt idea: stop compressing. Let every token look directly at every other token and decide for itself what matters — the attention mechanism you'll dissect over the next two chapters — and the reason a model can bind an error line to evidence forty lines away. The cost is quadratic work as the window grows; the win is that distance stopped mattering. Almost everything since — bigger windows, faster attention — is engineering against that cost.

Tokenizing a CI log

Time to open the box. The first surprise inside: models don't see characters, and they don't see words. They see tokens — sub-word chunks, typically three to five characters of English, learned by a compression algorithm from mountains of text. Common words get to be a single token. Rare ones get split. And CI logs are full of rare ones: selectors, hex ids, millisecond counts, file paths.

Paste a line from a real failure below and watch how it breaks apart. Each colored chip is one token — one unit of the model's attention.

Why should a QA engineer care? Because of the context window. A small local model might fit 8,000 tokens — and a single failed run with full browser traces can be tens of thousands. Everything you paste competes for the model's attention. Paste the whole log and the one line that matters drowns.

Embeddings

Once tokenization is done, the model holds a list of integer identifiers — and an identifier is just a name, a numbered drawer. "Token 5089" isn't 5089 of anything. You can't add two drawer numbers and get anything meaningful.

The model needs a more useful form than a bare id, so it looks each one up in a table and gets back a list of numbers: a vector. The model keeps one big table with a row for every token in its vocabulary. To embed a token, you take its identifier and grab that row.

Each row is a point in a space with as many axes as the vector has numbers. We use 50 here; a production model uses hundreds or thousands. Before training, the whole table is noise — "timeout" sits no closer to "wait" than to "Tuesday." Training slowly drags related tokens together, because tokens that behave alike in real logs help predict each other.

Anatomy of a transformer

You now have a sequence of vectors — one per token of the failure you pasted. A transformer is a stack of identical layers that repeatedly refine those vectors until the last one is good enough to predict what comes next. Two operations alternate, dozens of times.

Attention is the interesting one. Each token's vector gets to look at every other token's vector and pull in what's relevant. When the model processes the word exceeded in your error line, attention is how that position reaches back to Timeout and 30000ms and binds them into one idea — this was a wait that ran out. Attention is why the model can connect a selector on line 3 with a DOM snippet you pasted forty lines later.

The feed-forward block is the other. After attention gathers context, each position is pushed through a small neural network that transforms it — this is where the model's stored knowledge lives, the patterns it absorbed from everything it read. Attention moves information between positions; the feed-forward block thinks within each one.

Our base model, Qwen2.5-0.5B, is this exact architecture at pocket size: half a billion parameters, small enough to fine-tune on a laptop in minutes with Apple MLX, big enough to follow instructions. We will not pre-train from scratch — that takes millions of GPU-hours. We adapt a capable small base. That's the honest, practical path for a specialist model.

Attention, up close

The anatomy chapter waved at attention; this chapter slows it down, because attention is the single piece of the machine that most changes how you should use the model as a QA engineer. Every token position asks one question of every other position: how relevant are you to what I'm trying to figure out here? — and then blends in information from the positions that answer loudest.

The mechanics are three lookups per token, learned during training: a query (what am I looking for?), a key (what do I contain?), and a value (what will I contribute if chosen?). Relevance is the match between one token's query and another's key; the output is a weighted blend of values. Click any token below and watch where it looks.

Two practical consequences. First, attention is why what you paste matters more than what you ask. The model can only attend to evidence that is physically in the window — a brilliant question about a trace you didn't include attends to nothing. Second, attention degrades gracefully but not uniformly: bury one relevant line in ten thousand irrelevant ones and the relevant line still gets attention, just less reliably. Triage isn't a courtesy to the model; it is how you sharpen its focus.

Sampling: how answers get chosen

The model's forward pass ends in a probability for every token in the vocabulary — and then something has to actually pick one. That picker is the sampler, it sits outside the neural network, and it is the most underrated setting in any AI-assisted QA tool, because it decides whether the same failure gets the same diagnosis twice.

You met temperature in the opening chapter — it reshapes the distribution before the draw. Two more dials matter in practice. Top-k keeps only the k most likely candidates and deletes the rest. Top-p keeps the smallest set of candidates whose probabilities add up to p — an adaptive cut that keeps one candidate when the model is sure and many when it isn't. Play with all three against the same next-token distribution:

Here is the QA translation. Greedy decoding (always take the top token) gives you reproducibility — the property your whole discipline is built on. Loose sampling gives you diversity — genuinely useful when you want ten different hypotheses about a gnarly flake. The failure mode is using one setting for both jobs: a diagnosis pipeline that samples creatively will "flake" exactly like the tests it's meant to fix, and you will burn a week discovering that your AI tool's nondeterminism was a config value.

Training data: your QA corpus

Most tutorials teach on toy data, and it shows. This book's corpus is different, and it is the single biggest reason to trust what comes later: the failures you'll train and evaluate on were harvested from real test runs — 260 executions across two live sites: the author's production portfolio suite and the public, reproducible SwagLabs fixture at saucedemo.com.

Look at that distribution for a moment, because it should change how you think about "AI for testing." The glamorous failure — a deep race condition, a genuine product bug — is the minority. The bulk is selector ambiguity: the app changed a class, an id, a structure, and the test's grip slipped. This is exactly the kind of failure a language model is suited to: it's a reading problem, not a running problem.

From this corpus we build the actual training set: each failure paired with a grounded gold answer — diagnosis, fix, citation of the evidence — plus, crucially, abstention negatives: questions whose correct answer is INSUFFICIENT_EVIDENCE, so the model learns that refusing is sometimes the right output. Data quality is the ceiling on everything downstream. Garbage in, garbage out is the oldest law in the field and the most often ignored.

How models learn

Training is disappointment, quantified and repeated. Show the model a failure from the training set, let it predict the gold answer one token at a time, and measure how surprised it was by each correct token. That surprise is the loss. Then nudge every weight a tiny step in the direction that would have made it less surprised, and do it again. A few thousand times.

The chart below is not a sketch. It is the loss curve from the actual MLX LoRA run this book's model comes from — Qwen2.5-0.5B, 100 iterations, 70 supervised examples built from a corpus of real test failures — Part II introduces it properly — trained on a laptop.

Two numbers matter on any curve like this. Train loss falling means the model fits the examples it sees. Validation loss — measured on held-out examples it never trains on — falling with it means the model is learning the behavior, not memorizing the answers. When train keeps dropping but validation turns upward, you've crossed into memorization, and more training makes the model worse at its job.

Pretraining, at toy scale

Everything we fine-tune later stands on a base model that someone already pretrained: shown a mountain of raw text and asked, trillions of times, to predict the next token. No labels, no gold answers — the text itself is the answer key. That's the entire recipe, and it is why base models feel weirdly worldly: to get good at next-token prediction on everything, you're forced to absorb grammar, code, idiom, and a million facts as a side effect.

Frontier pretraining costs millions of GPU-hours, which puts it out of reach of every QA team on earth — but the mechanism fits in your browser. The widget below pretrains a genuinely real (and genuinely tiny) character-level model on a few kilobytes of CI-log text, live, when you press the button. Watch the loss fall and the samples sharpen, epoch by epoch.

Two things transfer directly from this toy to the real thing. First, the corpus is the personality: our tiny model writes timeouts and selectors because that's all it ever read, and a frontier base model "knows" Playwright for exactly the same reason — public docs, issues, and code were in the mountain. Second, pretraining teaches likelihood, not correctness. The model learns what usually follows, which is why a base model will cheerfully continue a wrong diagnosis in perfect prose. Nothing in the pretraining objective ever rewarded being right — that discipline comes later, from us.

Teaching behavior: from imitation to preference

A pretrained base is a brilliant improviser with no professional standards. The build chapters ahead install those standards with supervised fine-tuning — curated example answers the model learns to imitate. But imitation has a ceiling: an SFT example says "here is a good answer," and stays silent about the answer the model almost gave instead. For the judgment calls that matter most in QA — grounded-but-modest versus confident-but-unsourced — what you really want to teach is a ranking.

That is preference training. Instead of one gold answer, each training item is a pair: two responses to the same failure, one chosen, one rejected. Methods like direct preference optimization (DPO) push the model's probabilities toward the chosen answer and away from the rejected one — teaching taste, not just format. Build a few preference pairs yourself; your picks below are exactly the raw material DPO consumes:

Why this matters to a QA org even if you never run DPO yourself: every hosted model you buy has been through preference training, and its preferences are someone else's. General-purpose assistants are tuned to be maximally helpful — which quietly means always answer. That is precisely the wrong prior for test diagnosis, where "not enough evidence" beats a plausible invention. Our SFT data fights back with abstention examples; a preference dataset would fight harder, by explicitly ranking refusal above confident fabrication whenever evidence is thin.

Sizing it up

Model sizes get thrown around like sports stats — 0.5B, 4B, 70B, a trillion — and it's worth thirty seconds to make the numbers mean something, because sizing is a decision you will own: what runs in CI, what runs on the team's machines, what has to leave the building to an API. A parameter is one learned number. Count them, multiply by the bytes each one occupies, and you know — before any benchmark — whether a model can physically live where you need it.

Now the result this book already handed you, which the arithmetic makes less surprising: on the EdgeQA leaderboard in the chat chapter, the 4B model beat both 7–8B models on root-cause agreement over real failures. Parameter count is capacity, not competence. Past the size where the model can represent your problem, wins come from the match between its training diet and your domain — and from your data discipline — not from another billion parameters of general knowledge about cooking and cricket.

Grounding, and knowing when to shut up

Every fix a model hands you is a bet on your time. You apply it, push, and wait eight minutes for the pipeline. If the fix was right, great. If it was a confident guess dressed up as an answer, you burned a CI cycle, a context switch, and a little of your trust in the tool.

So the trait you want is not cleverness. It's grounding — every claim tied to something the model can point at — plus the discipline to say "I don't have enough to tell you" instead of inventing a plausible cause. Grounding looks like citations: if the model says the button lost its class, it should quote the DOM line that proves it. A claim without receipts is decoration.

Here's a grounded answer done right. The test fails at the password step; the prompt includes both the failure and the live DOM. Run the model and watch every sentence tie back to a fact you supplied.

// was
await page.fill('#pass', 'secret_sauce');
// use the test contract already on the field
await page.getByTestId('password').fill('secret_sauce');

Now the harder skill — the one that separates a tool you can trust from one you can't. Sometimes the honest answer is I can't tell yet. Here's a flaky failure with a real gap in the evidence: no screenshot, no trace, no network log. The right move is to refuse to guess.

Project setup

From here on you're building. The toolchain is deliberately boring and entirely on-device, because the logs you'll eventually feed it are proprietary and because a laptop is genuinely enough for a 0.5B-parameter model.

• Ollama runs the chat model locally — ollama run gemma3:4b and you have an inference server on localhost:11434.
• Apple MLX (via mlx-lm) does the fine-tuning — LoRA training of Qwen2.5-0.5B on Apple Silicon, minutes not hours.
• Playwright is the execution harness — the thing that turns a suggested fix into a verified green run.

# the whole stack
ollama run gemma3:4b                     # local inference
python3.12 -m venv sidecar/.venv
sidecar/.venv/bin/pip install mlx-lm fastapi uvicorn
npm run sidecar                          # fine-tune server on 127.0.0.1:8765
npx playwright install chromium          # real browser execution

Nothing installed? Everything in this book still works — the recorded runs you're reading are from this exact pipeline, clearly labeled. The design rule, in the app and in this book, is graceful degradation: live when the runtime is present, honest precomputed results when it isn't, and never an unlabeled blur between the two.

Fine-tuning data

Fine-tuning data is behavioral. We are not teaching the base model English, or code, or what a timeout is — it knows. We are teaching it a discipline: the exact shape of a good QA answer. Each training example is a conversation — a failing test's evidence in, the gold response out — and the gold response always has the same skeleton: diagnosis → fix → citations, or abstention.

And roughly one example in five is an abstention negative: a question that cannot be answered from the given evidence, whose gold answer is exactly INSUFFICIENT_EVIDENCE. Leave these out and you train a people-pleaser — a model that has learned that every question deserves a confident paragraph. Include them and refusal becomes a legitimate, learned output.

"prompt": "Which proprietary test data record caused the failure?"
// evidence contains nothing about test data records…
"gold":   "INSUFFICIENT_EVIDENCE"

Seventy examples sounds small, and it is — deliberately. This is a demonstration-scale dataset built from the seed benchmark, sized so the full pipeline runs on a laptop in minutes and every artifact fits in this book. The pipeline, not the size, is the lesson; scaling the data up is engineering, not new ideas.

Supervised fine-tuning

Time to train. LoRA — low-rank adaptation — is the trick that makes this a laptop job: instead of updating all half-billion weights, we freeze the base model and train small adapter matrices alongside a fraction of a percent of the parameters. The base keeps its fluency; the adapters learn your discipline.

A falling curve is nice; changed behavior is the point. Below is the same prompt answered by the base model and by the tuned adapter — both outputs verbatim from the real run.

Chat with your model

Here is the loop you now own end to end: a failure comes in, you triage the evidence (tokenization taught you why), the model reads it (embeddings and attention taught you how), and answers in the shape the fine-tune drilled: diagnosis, fix, citations — or a refusal.

// brittle — couples to styling
await page.locator('.btn.primary').click();
// stable — couples to a test contract
await page.getByTestId('checkout').click();

So how good is the local stack, really? Measured, on the real failures from chapter seven — root-cause agreement of on-device models against the rule-based classifier across the 17 real failure artifacts:

Write, run, and repair a real test

Reading about broken tests is one thing. Fixing one while it fails in front of you is where it sticks. Below is a real Playwright test against saucedemo.com — deliberately broken: someone (fine, us) coupled it to a selector that no longer exists, the same class of mistake from the Introduction. Run it. Watch it go red. Read the diagnosis. Apply the fix. Run it green.

import { test, expect } from '@playwright/test';

test('user can log in', async ({ page }) => {
  await page.goto('/');
  await page.locator('[data-test="username"]').fill('standard_user');
  await page.locator('[data-test="password"]').fill('secret_sauce');
  // This selector no longer matches the saucedemo login button:
  await page.locator('#signin-button').click();
  await expect(page).toHaveURL(/inventory/);
});

That loop — write → run → read → repair → re-run — is the actual job, and now you own both ends of it. The model is fastest at the read: what broke, what to change. It is useless at the verify: only your run proves the fix. Keep both. Let it diagnose; make green the judge.

Rolling it out to your team

Everything before this chapter you can do alone. This chapter is for the moment it works — your model diagnosed a real failure, the re-run went green, and someone in standup says "can we get that for the whole team?" Twenty years of watching tooling rollouts succeed and fail says: the technology is now the easy half. The rollout is a trust problem, and trust is built the same way we build it for tests — incrementally, with evidence, and with the ability to revoke it.

Three habits make the ladder real. Own an eval. The seed benchmark in this book is small, but it is yours — failures from your suites, gold answers your seniors wrote. Vendor benchmarks tell you what a model can do; only your eval tells you what it does on your pain. Re-run it on every model update, exactly like a regression suite, because model updates are regressions until proven otherwise. Log the abstentions. The refusals are your roadmap — each "insufficient evidence" is a missing artifact (a trace flag, a screenshot-on-failure) that one config change would supply. Keep the human red-green loop. The model reads; the run judges. The day a diagnosis merges without a green re-run behind it, you've replaced flaky tests with flaky fixes.

And the people part, which is the part that actually decides adoption: introduce the model as a colleague being evaluated, not a mandate. Publish its scorecard where the team can see it — the same leaderboard treatment this book gave gemma3:4b. Engineers extend trust to things they can audit and revoke; they resist things imposed on faith. You now have the vocabulary to run that conversation honestly — window and grounding, sampling and abstention — because you built the small version yourself.

Acknowledgements

This book is the companion text to the Scholarly QA Model Builder app, and its pedagogy — narrative prose with the interactive pieces embedded right in the reading — is inspired by Language Model Builder (languagemodelbuilder.com), reimagined here for the software-test-automation domain.

The numbers are real. The failure corpus was harvested from 260 test executions on two live sites — the author's production portfolio suite and the public SwagLabs fixture at saucedemo.com. The fine-tune artifacts come from an actual MLX LoRA run of Qwen2.5-0.5B on Apple Silicon. The evaluation figures are from the automation-qa-eval benchmark (EdgeQA-Bench v0). Where a number is not yet human-validated, the text says so.

Built on Apple MLX, Ollama, Playwright, and the patience of everyone who ever re-ran a flaky suite "just to check."

About the author. Suneet Malhotra is a Sr. Manager of Test Engineering at Motorola Solutions, with more than twenty years leading quality and test automation across Tinder, Amazon's Ring, Spokeo, and Live Nation. He writes and builds at the intersection of AI and quality engineering — this book, its app, and every artifact in it were produced on his own machine, from his own test runs. More at suneetmalhotra.com.

The QA Field Manual to Language Models · Suneet Malhotra · 2026 · fully offline — this file is the whole book