I Let AI Agents Run My QA Pipeline for 30 Days — Here's What Happened

Thirty days ago, I made a decision that made a few of my colleagues nervous: I was going to let AI agents handle the bulk of my QA pipeline autonomously. Test generation, failure triage, coverage gap analysis, even the daily reporting — all of it offloaded to agents I'd built and configured.

The context: I'm a Sr. Manager of Test Engineering at Motorola Solutions, with 20+ years in QA. I've seen every wave of automation tooling. When I say agentic AI is different, I don't mean it lightly.

Here's the honest account of what worked, what failed, and what I'd do differently.

The Setup

The pipeline had four AI agents running in sequence:

Agent 1: PR Analyzer — On every pull request, this agent read the diff, mapped changes to existing test coverage, and generated candidate Playwright tests for uncovered paths. Prompt-engineered with project context, it produced surprisingly idiomatic test code about 70% of the time.

Agent 2: Failure Triage Bot — When CI runs failed, instead of a raw stack trace in Slack, this agent analyzed the failure, classified it (flaky vs. real regression vs. environment issue), and wrote a 3-sentence summary with a suggested fix or next step.

Agent 3: Coverage Sentinel — Nightly, this agent scanned the codebase for new components and routes with no test coverage and filed GitHub issues with context on what needed testing.

Agent 4: Weekly Report Writer — Every Friday at 4 PM, it pulled metrics from our CI dashboard, wrote a quality health summary, and sent it to the engineering leadership Slack channel.

What Actually Worked

The failure triage agent was the biggest win — and the one I least expected. Engineers hate reading CI logs. A two-sentence "this looks like a real regression in the auth flow, here's the relevant line" message changed team behavior fast. People started fixing failures the same day instead of letting them sit for days.

The PR analyzer saved real time. Not perfectly — maybe 60% of its generated tests were usable with minor edits, another 20% needed significant rework. But even 60% usable is time I'm not spending on manual test scripting for boilerplate cases.

The weekly reports were... fine. They were accurate and saved me 30 minutes of report preparation. But they were also somewhat bland. Leadership noticed they read slightly differently week over week. I eventually added a human review step before send.

Where It Broke Down

The coverage sentinel generated noise. Without context about which parts of the codebase were actually risky versus internal utilities, it filed issues for things that genuinely didn't need tests. I spent time triaging the agent's own output — which partially defeated the purpose.

The fix: I needed to inject domain knowledge. After teaching the agent which directories mattered (API routes, auth flows, payment components) versus which were safe to ignore (i18n strings, static assets), the signal-to-noise ratio improved dramatically.

The PR analyzer also had a hallucination problem on complex state management code. It would generate tests that looked syntactically correct but tested subtly wrong things — testing that a function was called rather than that the right outcome happened. These are the kinds of errors that give false confidence. I added a human review gate for any agent-generated tests touching payment or auth.

The Real Lesson: Agents Need Guardrails, Not Babysitting

The biggest mental shift was realizing agents aren't juniors you supervise constantly — they're systems you design well upfront and then monitor. The 30 days taught me that most agent failures traced back to one of three root causes:

1. Insufficient context — the agent didn't know enough about the domain
2. Missing constraints — I hadn't told it what NOT to do
3. No feedback loop — it had no way to learn from bad outputs

Once I addressed those three things — injecting richer context, adding explicit constraint prompts, and building a lightweight feedback mechanism — reliability improved substantially.

What I'd Tell Other QA Engineers

Agentic AI in QA isn't coming. It's here. The engineers I see struggling with it are the ones treating it like another tool to learn. The ones thriving are treating it like a new team member to onboard.

You wouldn't hand a new hire a task without context, constraints, and a feedback channel. Same rule applies.

If you're building toward Suneet Malhotra QA automation territory — AI-driven test generation, self-healing pipelines, agentic coverage analysis — start small. Pick one pain point (failure triage is a great first one). Build the agent, measure the outcome, iterate.

The next 18 months of QA engineering will be defined by who figured out how to deploy agents reliably. That's where the interesting work is.

---

Suneet Malhotra is Sr. Manager of Test Engineering at Motorola Solutions, with 20+ years in QA automation and AI-driven testing.