How I Automated Our Entire QA Triage Process With AI Agents — And Why Our Team Isn't Drowning in Bugs Anymore

Six months ago, our QA triage process looked like this: a Slack channel called #build-failures where engineers would frantically post CI screenshots at 10 AM, pointing out which test had failed and whether it was "probably flaky." Someone would dig through logs. Someone else would open a GitHub issue. A third person would triage it to "needs investigation." The same test might fail identically three days later and nobody would remember what we'd already learned.

It was chaos. Every sprint, our triage overhead grew. Engineers stopped trusting the test suite. We had 12+ hours of pure busywork every sprint that could have gone toward actually building features.

So I spent a week building an AI-powered QA triage agent. After four weeks in production, our flakiness rate dropped 68%, our mean time to diagnosis fell from 45 minutes to 6 minutes, and we recovered 12 hours per sprint that used to vanish into triage blackholes.

This is how it works, why it matters, and why every QA team should be building something like this right now.

The Problem We Were Drowning In

By March 2026, our Playwright test suite had grown to 840 tests across four services. That sounds healthy until you realize: not all tests are created equal. Some are rock-solid — they've passed 10,000+ times. Others are notoriously flaky, failing intermittently due to timing assumptions, test order dependencies, or environment-specific weirdness.

When CI would fail, here's what happened:

Engineer discovers failure at 10 AM: Slack message: "build-failures: LoginFlow failed again"

Fifteen minutes of investigation: Is this a real regression or a known flaky test? Does it pass on retry? Did something actually break or is the network slow today?

GitHub issue creation (maybe): If it seemed real, create an issue. But what priority? What are the reproduction steps? Nobody wrote them down.

Duplication: The same test would fail the next day and we'd repeat the entire cycle because nobody had documented the root cause.

Developer frustration: Engineers learned to distrust flaky tests, so they'd just re-run the whole suite when things failed. Net result: slower feedback loops and false confidence when tests happened to pass.

This was a QA problem dressed up as a test automation problem. We didn't need more tests. We needed better triaging and classification of failures.

The Solution: An AI Triage Agent

I built an automated pipeline that runs on every CI failure:

CI Failure → Webhook → AI Agent → Analysis → Classification → Action

Here's what the agent does:

Step 1: Context Collection (30 seconds)

The agent collects: the full CI log, the test output, the recent git diff, and the test file itself. It also queries a local database of known flaky tests and their history.

Step 2: Root Cause Analysis (20 seconds)

Using Claude's API with a specialized prompt, the agent analyzes the failure and classifies it:

• Real regression — Code change likely caused the failure
• Known flaky test — We've seen this failure pattern before
• Environment issue — Network timeout, disk full, etc.
• Test quality issue — Test is fragile, needs refactoring
• Intermittent/unclear — Needs human investigation

Step 3: Context-Aware Action (15 seconds)

Based on classification, the agent:

• Files a GitHub issue with reproduction steps and confidence level
• Tags it appropriately (flaky, regression, environmental)
• Posts to Slack with a one-line summary instead of a raw log
• Opens a draft PR with a suggested fix if it's a known pattern
• Escalates to engineer on-call if it looks like a critical regression

Step 4: Learning and Feedback

When an engineer reviews a triaged failure, they can provide feedback: "yes, this was flaky" or "no, this is a real bug." That feedback gets logged and feeds back into the agent's knowledge base.

The Results: Four Weeks In

Flakiness dropped 68% — Mostly because we finally had visibility into which tests were actually problematic. We prioritized refactoring the worst offenders and added better test isolation.

Mean time to diagnosis: 45 min → 6 min — Instead of manual log reading, the agent provides a structured diagnosis immediately.

False positives: <5% — The agent's classification is surprisingly accurate. The few misclassifications are usually edge cases.

Time recovered: 12 hours/sprint — Engineers stopped context-switching between triage tickets and actual work.

Best unexpected benefit: Institutional memory. The agent's database of past failures means we rarely re-diagnose the same issue twice. New team members can see "oh, this test fails every Tuesday morning when the integration server restarts" rather than spending three days learning that lesson themselves.

How to Build Your Own (It's Easier Than You Think)

You don't need a startup to do this. Here's the minimal viable agent:

1. Webhook receiver (30 lines of Node.js)

When CI fails, capture the logs and metadata:

app.post('/ci-failure', async (req) => {
  const { buildId, logs, testName, diff } = req.body;
  await triageAgent.analyze({ buildId, logs, testName, diff });
});

2. AI analysis prompt

The magic is in a good prompt:

You are a QA engineer with 15 years of experience. Analyze this test failure:

Test: ${testName}
Error: ${testError}
Recent changes: ${diff}
Test history: ${failureHistory}

Classify as: REAL_REGRESSION | FLAKY | ENV_ISSUE | TEST_QUALITY | UNCLEAR
Confidence: 0-100
Root cause: [one sentence]
Suggested action: [fix suggestion or investigation step]

3. Conditional action routing

Based on classification, take different actions:

if (classification === 'FLAKY') {
  createIssue('flaky-test', testName);
} else if (classification === 'REAL_REGRESSION') {
  escalateToOnCall();
} else if (classification === 'ENV_ISSUE') {
  logMetricAndIgnore();
}

That's it. The rest is tuning the prompt and building feedback loops.

What Actually Surprised Me

The agent caught patterns I missed. After analyzing 200+ failures, it flagged that one test was failing specifically on Thursday afternoons. Turns out the QA environment gets reset every Thursday 2 PM and there's a 10-second race condition during startup. We fixed one bug and saved ourselves a year of debugging.

Team behavior shifted immediately. When engineers saw structured triage instead of raw logs, they started trusting the process. They'd actually investigate an issue if the agent flagged it as "likely regression" because the context was there.

Feedback loops matter enormously. The first week, the agent misclassified about 15% of failures. By week three, after collecting feedback, it was down to 5%. The agent learns from corrections.

The Limitations (Be Honest)

The agent is genuinely good at pattern recognition and classification. It is not good at novel, complex debugging. When an engineer encounters a failure that doesn't fit any known pattern, the agent flags it as "UNCLEAR" and routes it for human investigation. That's the right call.

Also: the agent never makes automatic fixes to your codebase. It can suggest fixes and open PRs, but a human has to approve and merge. Safety first.

For QA Engineers Reading This

This is the future of QA automation. Not writing more tests, but writing smarter systems to triage the tests you have. The engineers who build these kinds of agent-powered quality systems will be in insanely high demand.

Start small. Pick one pain point — mine was triage chaos. Build an agent that handles just that. Measure the impact. Expand from there.

The compounding payoff is enormous: less busywork, better visibility, faster feedback, and most importantly, developers who trust the test suite again.

Fight On. ✌️

---

Suneet Malhotra is Sr. Manager of Test Engineering at Motorola Solutions with 20+ years in QA automation and AI-driven testing. Building the future of quality, one agent at a time.