I Watched GPT-5, Gemini 2.0, and Claude 4 Battle It Out — Here's Who Actually Won

Let me be honest with you: I went into this comparison fully expecting to write a post that validated whatever I already believed. That's not what happened.

Over two weeks, I ran identical engineering and QA tasks across the major frontier models — GPT-5, Gemini 2.0 Flash, Gemini 2.0 Pro, and Claude Sonnet 4. Same prompts. Same context. Same evaluation criteria. I'm a Sr. Manager of Test Engineering with 20+ years in the industry, so my lens is practical: which model actually helps me ship better software faster?

The results were nuanced, occasionally counterintuitive, and in at least one case, genuinely surprising.

The Task Suite I Used

I didn't test these models on contrived benchmarks. I used real work:

• Test generation: Given a TypeScript React component, write comprehensive Playwright end-to-end tests
• Failure triage: Given a CI log with a failing test, classify the failure and suggest a fix
• Code review: Spot bugs and security issues in a Node.js API route
• Documentation: Write a technical spec from a rough bullet-point outline
• Reasoning under ambiguity: Given incomplete requirements, identify the most likely edge cases to test

Each task ran three times per model, with the same system prompt framing Suneet Malhotra QA automation context, and I scored outputs on accuracy, usability, and how much editing they required before I'd actually commit them.

Test Generation: Claude Pulls Ahead

This was the closest category, and Claude Sonnet 4 edged out the competition — but not by much. What separated it was consistency. The Playwright tests it generated were syntactically correct every single time, used proper async/await patterns, and organized assertions in a way that matched our team's conventions without being told.

GPT-5 generated great tests too, but it had a tendency to over-engineer: adding abstraction layers and helper utilities that felt clever but added friction. For greenfield projects, that might be fine. For dropping tests into an existing codebase, it created unnecessary overhead.

Gemini 2.0 Pro surprised me — its test coverage instincts were genuinely strong. It consistently identified edge cases I hadn't specified. Where it fell down was in occasionally inventing API methods that don't exist in the Playwright spec, which meant every output needed a lint pass before use.

Failure Triage: GPT-5's Reasoning Shines

GPT-5 was the clear winner here, and it wasn't close. When I handed it a gnarly CI log — 400 lines of noise with a real regression buried in the middle — it not only identified the root cause but traced it back to a likely upstream change in a way that felt like a senior engineer had actually looked at it.

Claude was solid but more conservative. It tended to say "this looks like it could be X or Y" where GPT-5 committed to a diagnosis with a confidence level. For triage workflows where engineers need to act fast, a confident wrong answer is sometimes more useful than a hedged correct one — as long as the reasoning is transparent.

Gemini Flash was fastest by a significant margin, but its triage summaries on complex failures were too shallow. Good for quick classification, not for root cause analysis.

The Surprise: Gemini 2.0 Pro on Documentation

I expected this to go to Claude. Claude has always been my go-to for writing tasks. But Gemini 2.0 Pro's documentation output genuinely impressed me.

It restructured my bullet-point outline in a way that made logical sense I hadn't considered, added context sections I hadn't asked for, and wrote in a voice that was authoritative without being stiff. The spec it produced needed less editing than any other model's output in any category across the entire two weeks.

I think what's happening is that Gemini's training data — particularly its deep integration with Google's internal documentation tooling — has given it an edge on structured technical writing that the other models haven't fully matched yet.

Where They All Struggled

Every model tripped on the same thing: ambiguous requirements. When I gave incomplete specs and asked each model to identify edge cases, they all defaulted to generating exhaustive lists of generic scenarios rather than reasoning about what was actually risky in this specific context.

This is the fundamental gap between current LLMs and a strong human QA engineer. A good tester reads a requirements doc and immediately asks "what's the worst thing that could go wrong here?" — with domain intuition. The models generate comprehensive but often undifferentiated lists. The prioritization still lives with the human.

My Actual Recommendation

After two weeks of this, here's my honest take:

Use Claude for code generation and test scripting — the consistency and adherence to existing patterns is a real productivity multiplier.

Use GPT-5 for debugging and triage — its reasoning on failure analysis is the best I've seen in a production context.

Use Gemini 2.0 Pro for documentation and specs — it may have overtaken Claude for structured technical writing in ways that will surprise you.

Don't pick one and ignore the others. The engineers on my team who've gotten the most leverage from AI are the ones who've learned which model to reach for based on the task at hand — not the ones who picked a favorite and stopped there.

The AI model landscape in 2026 isn't about which one "won." It's about knowing your tools well enough to use the right one at the right moment. That's the same thing I've been saying about test automation frameworks for 20 years, and it turns out it applies here too.

Fight On, and keep shipping. ✌️