Building Prooftrace: an AI tester that has to prove itself

For two years, my job was to make AI systems believe things that were not true. In the lab at Tennessee Tech I ran reflective attacks on LiDAR, put adversarial camouflage in front of cameras, and spoofed GPS, all to see what an autonomous vehicle's perception stack would accept. It usually accepted more than it should have. The lesson I carried out of that work is short: never take a model at its word. A confident answer is not evidence.

That lesson followed me into software engineering, where I now build GenAI assistants for a living. Every feature I have shipped came with a promise attached: its acceptance criteria. And I have watched what happens to those promises after the sprint ends.

The gap nobody budgets for

Acceptance criteria are the most precise thing a team writes about a feature, and the least automated. Three kinds of work follow them, all by hand.

Someone clicks through the app and checks each criterion by eye. Then the same person, or a different one, does it again in dev, again in UAT and again in production, after every deployment. Then a third person turns those checks into automated tests, one selector at a time, and the tests drift as the app changes, until a red build stops meaning anything.

The numbers are not dramatic per story, which is why nobody budgets for them. Fifteen minutes to check an eight-criterion story in one environment, times three environments, times ten releases, is seven and a half hours for a single story. A team shipping twenty such stories a sprint spends a working week each sprint re-checking things that already worked.

I wanted an AI that would do the clicking. But I wanted it held to the standard of a good tester, which is not "I did it" but "here is what I saw."

What a run looks like

Prooftrace takes a user story and one command.

prooftrace run --story stories/create-project.md

The story is a Markdown file, or a GitHub issue, with acceptance criteria under a heading. The run goes through seven stages, and each stage opens and closes with an event, so the terminal, the console and the stored record all see the same thing.

The run in the video on my Prooftrace page is the real thing: "Create a project from the projects dashboard", eight criteria, 43 steps, 8 of 8 proven, 29.7 seconds, $0.028 in model cost. Nine model calls in total: one to extract field values, one to link criteria to steps, six to review sections, and one to review the evidence. The model never chose a click.

The model reads text, not pixels

The first design decision was what the model gets to see. Screenshots are what a person wants. For a model they are slow to produce, expensive to send and surprisingly easy to misread, so I gave it something closer to what a screen reader gets: a text snapshot built from the accessibility tree, one line per control with a stable reference, plus the page's visible text and table rows.

- heading "Projects" [ref=e11]
- button "New project" [ref=e12]
- row "Field Service Scheduling AWK-342 | Adventure Works | Draft | 2026-07-20"
  - link "Field Service Scheduling" [ref=e84]
- text "Status Draft"
- textbox "Password" value="***"

Three things make this workable. A reference stays the same for as long as the element stays on the page, so the model acts on exactly what it saw. After each action the model gets a diff, not the whole page again, and every six turns the older turns fold into one-line summaries, which keeps the prompt prefix stable enough to cache. And a screenshot reaches the model only when it asks to look, which in practice is rare. Password fields are masked either way.

Who acts on a step

Most steps never reach the model at all. A workflow is a YAML recipe that anyone can write for their own app: the steps of a flow, each targeting an element by role and accessible name, with an optional sentence describing what should be true when a section is done. When a recipe matches the story, the executor runs it directly. That is the "seed" layer, and it costs nothing.

The model steps in for three things only. When an element is not found after retries, a fallback call sees the step, the failure and the current snapshot, and picks one action so the recipe can carry on. After each section, a validation call reviews the page against the criteria and reports passed, warning or failed; this is advisory, and it never marks a criterion proven. And when a story has no recipe, the model explores: one decision per turn, with a guard that ends the run on a failed action, a cap on turns, and an optional cost ceiling checked before every call.

Whatever the layer, a model decision is one of ten structured actions: click, fill, select, press, navigate, assert, wait, look, done or give up. Each comes back as JSON against a schema and is checked again in code before anything happens in the browser. There is no free-text "do whatever you think" path.

No proof, no pass

The rule that shaped Prooftrace came from a run that lied to me.

An early version marked a sign-in criterion as verified. The evidence was a replay of a sign-in that had failed. The steps themselves had not thrown, the page had loaded, and the model, asked whether the user was signed in, said yes with the confidence models have. Nothing in the system was designed to disagree, and that was my mistake, not the model's.

Now there is. A criterion is verified only when an assertion linked to it passed in the generated test. Actions prove nothing on their own. A separate model call reviews the evidence for each criterion in a fresh prompt, sees only what the linked assertions showed, and has to quote that evidence exactly; the quote is checked in code, and text that appears on every page is refused as evidence. If any included criterion is not proven, the run's status is failed, whatever else went well.

The third run on my page shows what that looks like from the outside. The story asks the dashboard to show each project's budget. There is no budget column. The run drives the page, proves three of four criteria, and stops with a red banner that names the fourth: "None of the table rows or headers mention budget." Thirteen seconds, and a truthful answer.

The tests you keep

A run leaves four files behind, and the emitter that writes them uses no model at all, so the code is deterministic and every piece of run text is encoded for where it lands.

tests/features/create-project.feature      Gherkin, each criterion beside the steps that prove it
tests/steps/create-project.steps.ts        step definitions for playwright-bdd
tests/pages/northwind-projects.page.ts     a page object: signIn() and one method per step
tests/specs/create-project.spec.ts         a Playwright test with one test.step per run step

The spec is plain Playwright. The step labels carry the criteria they prove, so a failure names the promise it broke.

await test.step('st02 Click "New project" [c1]', () => app.clickNewProject());
await test.step('st05 Assert the project name is required [c2]',
  () => app.assertTheProjectNameIsRequired());

Credentials are read from environment variables at run time through a helper that fails loudly when one is unset, and the code uses relative paths, so pointing Playwright's base URL at dev, UAT or production reruns the same test there. A rerun needs no model and costs nothing. The same recipe with the AI switched off passes 8 of 8 in 16.3 seconds, which is the path a CI pipeline runs on every push.

One stream, every view

Everything the engine learns about a run is an event: run started, criteria extracted, plan ready, step finished with its screenshot, section validated, artifact emitted, verify finished, run finished. Each is wrapped in an envelope with the run id, a sequence number and a timestamp, and validated against a schema at every trust boundary.

Redaction happens once, before an event becomes an envelope, so no listener ever sees a raw secret: sign-in credentials become placeholders the executor resolves later, and anything else that matches a secret becomes three stars. The terminal renders the stream as it arrives. A SQLite store keeps every envelope. The console opens a WebSocket per run and resumes from the last sequence number it delivered. One reducer turns events into run state for all of them, which is why a live run, a stored run and a replay always agree.

That last part is also how the console on my website works. It is the real console, in replay mode, reading a recording exported from the store, with no server behind it.

Keeping the model on a leash

An agent that drives a browser is a new kind of attack surface, and I spent my research years on the other side of that fence. So every change to Prooftrace was attacked before it was merged. Nine issues came out of that review, and each is now a guard.

• A page that asks the model to type the password somewhere else gets nothing: the password placeholder fills only password fields, only on the app's own origins.
• A page that steers the model to another site gets nothing: navigation stays on the app's origins.
• Page text that poses as instructions is treated as data. The prompts say so, and the structured actions leave no room to comply anyway.
• A story title that smuggles code into the generated tests fails: every string is encoded, and locators are parsed, never pasted. The probe that found this used a Unicode line separator to break out of a comment in the generated spec.
• Generated tests cannot read your API keys: the test process gets an allowlisted environment, with only the two credential variables it needs.
• A website that rebinds its domain to your local server, or opens a WebSocket to it from another origin, is refused: the server answers only to its own host names and its own origin.

None of this is exotic. It is the boring set of guards that anything holding your credentials and driving your browser should have on day one, and most of it took an afternoon once the probe had shown the hole.

What running the real model taught me

The offline tests, more than three hundred of them, caught the mechanical bugs. The real model caught the design mistakes. Three changes came directly from watching it work.

The snapshot gained page text. Early on the model saw only controls, and it could not answer questions about what a page showed. Exploration gained assertions: a run that only acted could not prove anything, so the explore loop learned to assert what it saw and to refuse an assertion that matched only a button or a table header. And the verification rule became strict, for the reason above.

One smaller change made the biggest difference to speed. Instead of fixed waits after each action, a step settles as soon as the page has been quiet for 80 milliseconds, and it keeps waiting only while a real loading indicator is on screen. That is most of the difference between a run that takes half a minute and one that takes three.

Bring your own model

Prooftrace is not tied to a vendor. The engine calls one interface and asks it only for structured output. A provider is an adapter package chosen by name in the config, and the key is an environment variable the config names, never a value it holds.

llm:
  provider: anthropic        # or openai, ollama
  model: claude-sonnet-5
  apiKeyEnv: ANTHROPIC_API_KEY
  tiers:                     # a model per purpose
    extract: claude-haiku-4-5
    decide: claude-sonnet-5

OpenAI and Anthropic adapters ship today. An Ollama adapter for local models exists and is marked experimental until it has run against a live server. Azure OpenAI and Gemini are one adapter each behind the same interface. Story sources work the same way: Markdown files and GitHub issues today, with Azure DevOps and Jira next, each a single package that implements the story-source interface and registers its name. Nothing in the engine changes.

How it was built

I designed and built Prooftrace on my own time, and I held the process to the same standard I was asking of the product. Every change started as a written brief with acceptance checks. It was implemented, reviewed against the brief, attacked with probes, and only then merged. Eighteen tasks went through that loop, along with more than 320 automated tests and the nine security findings above. It was slower than just writing code. It is also the reason I trust what the tool tells me.

Try it

You can watch a real run on this site and click any step to see what the browser saw. If you want to read the code, there is a request form on the same page; I grant read-only access for seven days. And if you spend your Fridays clicking through acceptance criteria for the third environment that week, I would like to hear about it: abhijeet.solanki@outlook.com.