2d • AI&QA
AI Test Automation Mistakes: 3 Problems With AI-Generated Tests (and Fixes)
AI coding agents in 2026 can write UI and API tests very fast, with minimal direction.
But there are 3 problems. Let's look at what they are and how to fix or avoid them.
────────────────────────────────────────
🔴 𝟏. 𝐓𝐞𝐬𝐭 𝐝𝐞𝐬𝐢𝐠𝐧 𝐝𝐨𝐞𝐬𝐧'𝐭 𝐦𝐚𝐭𝐜𝐡 𝐭𝐡𝐞 𝐟𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤
Before writing a single line of code, an agent scans your framework for a pattern to "learn" how to write the tests.
If the framework itself is inconsistent (a few ways to structure a page object, different assertion styles, mixed naming conventions), the agent won't see one "right way" to write code. It will end up doing one of two things:
- Picks whichever example it happened to see most recently and copies that
- Decides there is no standard and invents its own
The long-term fix:
Make the framework itself consistent. This isn't really an AI problem. It's a framework hygiene problem that AI just makes visible faster than a human reviewer would.
- One way to structure a test.
- One way to name things.
- One place patterns live.
The short-term fix (while the framework catches up):
Write the design you actually want into `AGENTS.md`, or a standalone design doc. Feed it to the agent at the start of the task.
Then use that same doc again after the task is done, as a second pass, to verify the output actually matches the design.
────────────────────────────────────────
🔴 𝟐. 𝐓𝐞𝐬𝐭𝐬 𝐝𝐨𝐧'𝐭 𝐮𝐬𝐞 𝐝𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐢𝐬𝐭𝐢𝐜 𝐥𝐨𝐜𝐚𝐭𝐨𝐫𝐬
A strange but common AI agent habit: instead of picking one reliable locator, the agent picks a few and writes code that loops through them to see which one exists or loads on the page.
- `locator1 or locator2 or locator3`, "just in case"
- Fallback chains that quietly hide which locator the test actually relies on
This makes a test non-deterministic before it has even run once. You don't know what it's actually checking against.
Fix:
State the rule explicitly. Only 1 locator per element. No fallback chains. If the agent can't find a stable locator, that's a signal to fix the app's markup or test IDs.
────────────────────────────────────────
🔴 𝟑. 𝐅𝐥𝐚𝐤𝐲 𝐭𝐞𝐬𝐭𝐬
This is the hardest one. There is no shortcut here. A flaky test fails intermittently, and a single run of logs almost never shows why. The agent needs a pattern to find the cause, and a pattern needs history.
Fix:
Run the test multiple times. Save the logs from every run, pass and fail. Feed all of it to the agent at once, not just the latest failure. Only with enough runs to compare can it actually spot what was different between the passes and the failures, instead of guessing at a fix.
────────────────────────────────────────
📌 Want a repeatable setup for this in your own repo: rules files, design docs, and AI test generation workflows that actually hold up?
Join the AI AutoTest Live Workshop: hands-on practice with AI coding agents, Playwright CLI, and real test automation workflows.
👉 Click here to read more: https://www.skool.com/qa-automation-career-hub/welcome-to-qa-automation-roadmap-lab-start-here
3
0 comments
AI Test Automation Mistakes: 3 Problems With AI-Generated Tests (and Fixes)
skool.com/qa-automation-career-hub
From QA Automation to AI-Powered SDET. Join AI & QA Accelerator.
Powered by