What AI Actually Changes About Testing (and What It Doesn't)

Every few months a new tool promises to "automate QA with AI." I've tried a handful of them by now — some generating test cases from user stories, others watching an app and writing Playwright scripts by themselves, a couple doing visual diffing with a model instead of pixel-matching. Some of it is genuinely useful. Some of it just moves the work somewhere less visible.

Here's roughly how I've come to think about it.

Where AI genuinely helps

The best use I've found is at the first draft stage — turning a rough idea into something concrete fast. Give a model a user story or an API spec and ask for edge cases, and it will reliably surface things a tired human skips on a Friday afternoon: empty arrays, null fields, boundary values, unicode in a name field, what happens on the second submit instead of the first. It won't catch everything, but it catches more than a blank page does, and it catches it in ten seconds instead of twenty minutes.

It's also good at the tedious middle of automation — turning a manual test case into a Playwright or REST Assured skeleton, generating realistic-looking test data, or explaining what an unfamiliar chunk of test framework code is actually doing. None of that requires judgment. It requires speed, and AI is fast.

Self-healing locators are the other place I've seen real value. A button gets a new data-testid, the old selector breaks, and instead of a red pipeline and a Teams message, the framework quietly finds the element by a different signal and keeps going. That saves real maintenance hours on a large suite.

Where it quietly shifts the risk instead of removing it

What AI is not good at — at least not yet, in anything I've tested — is deciding which test actually matters for this release. It doesn't know that the payments team broke something adjacent last sprint, that a particular API has flaked under load before, or that the one edge case worth obsessing over is the one that maps to an actual support ticket from a real user. That context lives in the team, not in the model.

So when a generated test suite looks complete, there's a temptation to treat "complete" as "sufficient." Those aren't the same thing. A hundred AI-generated assertions that all check toBeVisible() on the happy path give you a false sense of coverage — lots of green checkmarks, very little actual risk reduction. I've started treating a large AI-generated suite the same way I'd treat a large PR from someone new to the codebase: promising, worth reviewing carefully, not worth merging on trust.

The other risk is subtler. AI-written tests tend to assert what the code currently does, not what it's supposed to do. If there's already a bug in the implementation, a model watching the app or reading the code will often happily encode that bug into a passing test — which then actively defends the bug against being fixed later. A human who understands the requirement is still the only reliable check against that.

Where I've landed

I use AI the way I'd use a fast, tireless junior tester: great for generating volume and catching the obvious misses, bad at judgment calls about priority and intent. The generation gets faster. The thinking about what's actually worth testing — and why — still has to be mine.

That's probably the honest shape of "AI-assisted testing" for a while: less about replacing the judgment part of QA, more about clearing enough of the repetitive work that there's actually time left to use that judgment.