Hawkeye: Why We Stopped Maintaining UI Tests and Built an AI Tester Instead
You must have heard the joke.
A QA tester walks into a bar. He orders one beer. Then two. Then zero.
Then 99,999,999.
He tries every combination he can think of.
Nothing breaks.
The QA tester smiles. “Testing complete.”
A customer walks into the bar.
“Excuse me,” she says. “Where’s the bathroom?”
The bar catches fire.
Turns out there is wisdom in this old joke, and it is about human testers.
What actually happens when code outpaces tests
AI-assisted development changed how fast a frontend changes. It did not change how fast a scripted test suite can be updated.
That gap compounds. A team shipping UI changes with agents can rewrite a flow in an afternoon: rename a field, reorder a step, split one screen into two. Every scripted test touching that flow breaks. The product still works. The script described the old screen.
Someone has to fix those. To keep a scripted suite green across several UI repositories, all moving at agent speed, you need a QA function larger than the development teams generating the changes, because the work is proportional to the rate of change, and the rate of change is now bounded by inference cost instead of typing speed.
The obvious move is to buy your way out, so that is what we tried. There is a category of vendor built for exactly this: point it at your codebase, and it generates Playwright scripts for your flows, heals them when selectors drift, runs them on every change, and hands you reports. On paper it removes the maintenance burden entirely.
It did not work for us, and not because the tool was weak. A very large team worked on it from the vendor side and the gap still did not close. Healing keeps a script running. It does not tell you whether the script still describes a flow the product has. The suite stayed green while drifting away from the product, and when it went red, nobody could say why. Ownership sat with people who had not built the feature and had no way to know which of the fifty failures that morning were real.
The bottleneck was never the cost of writing test code. It was that generated test code is nobody’s to own.
Why scripted suites fall short
The second problem survived every tool we tried.
Generated Playwright code tests what a script can see: selectors, DOM nodes, network responses. A human tester looks at the screen. They notice the button is present but rendered behind the modal. They notice the success toast says the right thing in the wrong language. They notice the page is white for three seconds before anything appears.
Back in the days, this was somebody’s craft. Excellent QA people, the kind who always found the problems, would open the app and break it in ways no script was ever written for. They were not running assertions. They were being suspicious about a product, in a way that only comes from using it every day.
We do not have that any more. No QA team, nobody dedicated to manual testing. Developers test their own work by hand, conscientiously, and that is the one kind of testing everybody is worst at: nobody is suspicious about the thing they just built.
So a scripted suite asserted that an element with a given selector existed, and there was nobody left whose job was to assert that the product worked. Only one of those is what we ship.
That left two problems: test code nobody could maintain, and test code that did not test the thing that mattered. Adding more test code was not going to fix either.
What we do instead
The bet was simple: what if a test case was a description of what to do and what should happen, written in plain English, and something else did the clicking?
Give an agent a browser. Give it the test case in prose. Let it decide how to get there.
That changes the shape of the maintenance problem. “Click the login button” does not break when the button moves, changes colour, or gets a new label, because the instruction was never coupled to the implementation. It breaks when logging in stops working, which is when we want it to break.
What a test case looks like
This is a suite from our repository, with the host replaced:
Client Portal: Login
The login page authenticates organisation users to the client portal.
Target
App: https://app.example.internal/login
Credentials: Use $TEST_EMAIL and $TEST_PASSWORD
Test Cases
Successful login
Navigate to https://app.example.internal/login
→ Login page loads with email and password fields
→ "Anmelden" (Login) button is visible
Enter email from $TEST_EMAIL
Enter password from $TEST_PASSWORD
Click "Anmelden" (Login) button
→ OTP verification screen appears ("Identität prüfen") with 6 individual numeric input boxes
Enter OTP code (use $TEST_OTP or the shared test OTP)
→ An organisation selector dialog appears
Click the first organisation in the list
→ User is redirected to the home page (/v2/home)
→ Welcome message "Willkommen, [Name]" is displayed
→ No error messages are shown
If the UI is not in German, switch language to Deutsch
→ All sidebar labels are in German (Anfragen, Dokumente, Monatsabschluss, Verbindungen, Einstellungen)Actions on the left, expectations after the arrow. That is the entire format, and the suite carries on that way through the cases you would expect: wrong password, an account with no organisations, a session that has already expired.
A human can read it. A product person can read this suite, notice that we never check what happens when the OTP expires, and add three lines. No selectors to look up, no test helper to import, no local setup. Contributing a test case costs a sentence.
Credentials never live in the suite. They are referenced as environment variables and injected into the agent as secrets at run time.
How it runs
Two triggers.
Nightly, the agent runs everything. Every flow, across the apps. This is the safety net.
Per pull request, running everything is wasteful and slow. The agent gets the diff first and decides which suites relate to what changed. A change to the invoice table does not need the login suite re-verified from scratch. A change to the auth provider does.
Then it opens a browser and behaves like a tester: navigates, types, scrolls, waits, reads the screen, and compares what it sees against the expectations in the suite. When something disagrees, it reports what it expected, what it saw, and a screenshot of the moment.
Who owns the suites
Plain English suites solved maintenance cost. They did not solve maintenance ownership. A suite still drifts when a flow legitimately changes and nobody updates the prose.
So we automated that too. When a UI repository changes, an automation opens a pull request against the Hawkeye suites with the update, and tags the people who made the change as reviewers.
This changed behaviour more than anything else. The person who moved the OTP step reviews the sentence describing the OTP step, in the same week, while it is still in their head. Test ownership moved to the feature builders, where it always belonged and never stayed, because previously it meant owning test code in a language and repository they did not work in. Reviewing a paragraph is a different ask.
What we learned
Plain text carried the whole thing. The suites became the only artifact that engineers and product people read equally fluently. Arguments about whether the test was wrong turned into edits to a sentence.
Coupling is the whole problem. Scripted UI tests are expensive to maintain because they encode implementation details that were never the point. Describe intent and most of the churn disappears.
Testing like a user catches a different class of bug. The things that were green in CI and broken on screen are the failures a customer would have reported.
Give the agent eyes, or it will write scripts. An agent asked to verify a UI without the ability to look at one falls back to generating test code and trusting it. That is the problem you started with, rebuilt at greater expense.
What’s still rough
Non-determinism. An agent tester is flaky in a different way than a script. A script fails identically every time. An agent can take a different path, phrase a failure differently, or occasionally disagree with itself on an ambiguous expectation. Telling a broken product from a badly worded expectation is still a judgement call.
Relevance selection is a bet. Picking suites from the diff is what makes per-PR runs fast, and it is also how you miss the one suite that would have caught the regression. The nightly full run is the backstop, which means some classes of bug are caught the next morning instead of at review time.
Review fatigue. Automatically opened suite pull requests only transfer ownership if people read them. The moment they become noise to be approved on autopilot, we are back to drifting tests with extra steps.
Result
Hawkeye does not replace the great testers of back in the days. It gets their habit back: open the app, use it like somebody who does not know how it was built, and say whether it works. The bar still catches fire in the one way nobody scripted. What changed is that describing a new suspicion costs a sentence instead of a sprint.
This post is part of the SDLC2 series. It applies the idea from Agents as Code, that the useful artifact is often a plain English document written together, to our tests. For the same problem one layer down, where synthetic companies are manufactured to test the backend, see There’s no fixture for a company.