The most expensive gap in software
GitHub's 2025 Octoverse report found that the average engineering team deploys 4.7 times per day. Stripe deploys 5,000 times per day. Shopify averages 80 production deploys daily. Deploy velocity has become a competitive advantage — the faster you ship, the faster you learn.
But testing hasn't kept up. According to Sauce Labs' 2025 State of Testing survey, 62% of teams still rely on manual QA for end-to-end flow verification. Only 23% run automated E2E tests on every deploy. The rest ship and hope.
The result is predictable. Sentry's 2025 data shows that 1 in 7 deploys introduces a user-facing regression. Not a crash — a broken flow. The signup form that no longer submits. The dashboard widget that renders blank. The checkout button that disappeared because someone refactored the billing component. These bugs pass unit tests. They pass integration tests. They even pass Playwright suites — because the button's data-testid still exists, even though it's now off-screen behind an overflow:hidden div.
The only test that catches these is the test nobody does: open the app, click through the main flows, look at the screen, and check that everything works. That's manual QA. And manual QA doesn't scale to 4.7 deploys per day.
AI QA does.
What makes AI QA different from Playwright
This is not "Playwright but with AI." The approach is fundamentally different.
Playwright, Cypress, and Selenium navigate your app by finding elements in the DOM — CSS selectors, XPath expressions, data-testid attributes. When the DOM changes, the test breaks. Engineering teams spend 30–50% of their test automation effort maintaining selectors (Mabl's 2025 Test Maintenance Report). That's not testing. That's babysitting infrastructure.
AI QA navigates your app the way a human does: by looking at the screen. Claude's vision model takes a screenshot, identifies buttons, forms, text, and navigation elements visually, and decides what to click. It doesn't know or care about the DOM. When a designer moves the login button from the top-right to a centered card layout, the agent finds it in the new location. Zero test maintenance. Zero selector updates.
Selector-based testing (Playwright/Cypress)
- Breaks every UI redesign, every component refactor
- 30–50% of test automation effort is maintenance
- Only engineers can write and fix tests
- Tests pass even when the page looks broken
- Adding 1 new flow = 30–60 minutes of code
- Flaky tests train engineers to ignore failures
Vision-based testing (AI QA)
- Adapts to UI changes automatically
- Zero maintenance — no selectors to rot
- Anyone can write flows in English
- Sees the screen — catches visual regressions
- Adding 1 new flow = 5 minutes of writing
- Screenshots on every step = proof of what happened
A modal that renders behind the page content. A button with white text on a white background. A form that works but is clipped at 50% width. An error message that says "undefined" instead of the actual error. Playwright sees none of these because the elements exist in the DOM. The AI agent sees all of them because it's looking at the screen.
What the report looks like
After every deploy, the AI agent posts a structured report to Slack (or your channel of choice). It takes 3 minutes. The engineering team sees it before they've moved on to the next task.
4/4 flows passed (187 seconds)
- Signup flow — PASS. Account created, redirected to dashboard. [screenshot]
- Login + dashboard — PASS. All widgets loaded. Settings page accessible. [screenshot]
- Billing + checkout — PASS. Plan selector rendered, Stripe widget loaded. [screenshot]
- Mobile viewport — PASS. Hamburger menu functional. Login form usable at 375px. [screenshot]
When something fails, the report shows exactly what went wrong, with a screenshot of the failure state:
3/4 flows passed, 1 FAILED (193 seconds)
- Signup flow — PASS.
- Login + dashboard — PASS.
- Billing + checkout — FAIL at step 5. "The 'Upgrade' button is present but clicking it shows an error toast: 'Failed to load plans.' The pricing table did not render." [screenshot of error state]
- Mobile viewport — PASS.
That failure screenshot is worth more than a hundred passing unit tests. The engineer sees the exact error toast, the exact state of the page, and can reproduce the issue immediately. No "well it works on my machine." The screenshot is the evidence.
The economics: why this is a no-brainer
Let's do the math for a team that deploys to staging 5 times a day.
Compare this to the alternatives:
| QA approach | Monthly cost | Coverage | Maintenance |
|---|---|---|---|
| Manual QA engineer (dedicated) | $8,000–12,000 | Tests 1–2x per week, maybe | N/A (human judgment) |
| Playwright/Cypress suite (engineer-maintained) | $1,500–3,000 (20% of one engineer's time) | Every deploy, if tests aren't too flaky | 30–50% of the testing effort |
| Visual testing SaaS (Percy, Chromatic) | $500–2,000 | Pixel diffs only — no flow testing | Low (but high false positive rate) |
| AI vision QA | $17 | Every deploy, every flow, visual + functional | Zero selectors, zero maintenance |
This isn't a marginal improvement. It's a category shift. The cost of comprehensive QA drops from thousands of dollars per month to lunch money. The question stops being "can we afford to test every deploy?" and becomes "why wouldn't we?"
Who writes the tests? Everyone.
The biggest change isn't technical. It's organizational. Test definitions are English, not code.
"Navigate to /billing. Click 'Upgrade.' Verify that the pricing table shows three plans. Select the Pro plan. Verify that the checkout form loads."
A product manager can write that. A designer can write that. A customer support lead who knows exactly which flows break the most — they can write that. You don't need an SDET or a QA automation specialist. You need someone who can describe what a working product looks like.
This changes who owns quality. QA is no longer a bottleneck gated by the team's one test automation engineer. Every team member can contribute test flows for the features they own. The PM adds a flow for the new onboarding wizard. The designer adds a flow for the redesigned settings page. Support adds a flow for the password reset that broke three times last quarter.
The selector rot problem is worse than you think
If you've maintained a Playwright or Cypress test suite, you know the cycle:
- Engineer writes 50 E2E tests over 2 months. High coverage. Team feels good.
- Designer ships a UI refresh. 15 tests break. Selectors changed.
- Engineer spends 2 days fixing selectors. Tests pass again.
- Product adds a new feature. 8 tests break because the page structure changed.
- Engineer is busy on a deadline. Fixes 3 tests, marks 5 as
test.skip(). - Three months later, 20 of the 50 tests are skipped. Nobody remembers why.
- CI is "green" but only because the failing tests are disabled.
- Management says "we have automated testing" but nobody trusts it.
Mabl's data puts numbers on this: 68% of test suites have at least one permanently-skipped test. 41% of E2E test failures are false positives caused by selector changes, not actual bugs. Engineers learn to ignore test failures, which means they also ignore real failures. The test suite becomes a liability instead of a safety net.
AI vision QA doesn't have this problem because there are no selectors. The agent finds elements by looking at them. A button labeled "Submit" is a button labeled "Submit" regardless of whether it's a <button>, an <a>, a <div role="button">, or a Radix UI primitive with 14 wrapper divs. The class name can change. The component library can change. The entire frontend framework can change. The test still works.
Parallel by default: 4 flows, 4 browsers, 3 minutes
Each user flow runs in its own browser container — a real Chrome instance in its own isolated cloud container. Four flows run simultaneously in four browsers. The wall clock time is the duration of the slowest flow, not the sum of all flows.
A typical suite of 4–6 critical flows completes in 2–4 minutes. The agent creates the browsers, navigates the flows, takes screenshots, and compiles the report. You can scale to 20 flows running in 20 parallel browsers without changing the architecture — just add more flow definitions.
Each browser is isolated. Flow A can't interfere with Flow B. If the billing flow crashes, the signup, login, and mobile flows still complete. Results arrive per-flow, so the report shows exactly which flow failed and which ones passed.
It works for desktop applications too
Not every product is a web app. If your product is an Electron app, a Java GUI, or any desktop application, the AI agent tests it the same way — but in a desktop container instead of a browser container. Full XFCE desktop with VNC access. The agent opens the installed application, navigates the GUI with vision, and screenshots every step.
This is particularly valuable for teams that have zero automated testing for their desktop product because Playwright doesn't support it. The AI agent is the first automation they've ever had.
What it doesn't replace
Be clear about what AI QA is and isn't:
- It doesn't replace unit tests. Your Jest/pytest/Go test suite catches logic bugs, edge cases, and API contract violations. AI QA catches user-visible regressions that slip through unit tests.
- It doesn't replace Playwright for component-level checks. If you need to verify that a React component renders correctly with 15 prop combinations, Playwright is faster and more precise. AI QA is for end-to-end flows.
- It doesn't replace security testing. Use Snyk, CodeQL, and penetration testing for security. AI QA tests user flows, not attack surfaces.
- It doesn't replace human judgment for UX. The agent verifies that the page works. A human designer judges whether the page is good. Different skills.
The AI QA engineer sits in the gap between automated tests (fast, narrow, brittle) and manual testing (slow, broad, expensive). It gives you the breadth of manual testing at the speed and cost of automation.
Getting started this afternoon
- List your 5 most critical user flows. Signup, login, core workflow, billing, and the one flow that breaks the most.
- Describe each flow in 5–8 English sentences. What pages to visit, what to click, what success looks like.
- Run the QA agent against your staging environment. Review the screenshots. Tune the flow descriptions where the agent gets confused.
- Wire it to your deploy pipeline. One webhook. After every staging deploy, the agent runs and posts to Slack.
- Expand. Once the 5 critical flows are stable, add more. Each new flow takes 5 minutes to write.
You can have AI-powered QA running on every deploy by end of day. Not end of quarter. Not after a 6-month procurement cycle. Today.
What's next
Test every deploy. Automatically.
Parallel browsers. Vision-based navigation. Screenshots on every step. A report before you've opened Slack. $0.15.
Get Started Free