There's no fixture for a company
Our product runs on branching journeys that take a real company weeks to complete, and AI agents work inside them. No test fixture can represent that. This is how we solved it: by building synthetic companies.
Most testing advice assumes the state you need fits in a fixture: seed a user, seed a record, run the test. Our app doesn’t work like that. A client moves through branching journeys that take weeks, and every branch point multiplies the states a company can be in: how it was onboarded, which setup path it took, how far each month’s close has progressed, what got approved, rejected or left waiting along the way. No two companies arrive at the same state the same way, and we need to test all of it. When something broke deep inside one of those paths, reproducing it meant assembling a test organization by hand, and by the time you had one, the conditions had usually changed underneath you.
This post is about how we fixed that. The product team was closest to the problem, so we built our own app: a test suite that manufactures complete synthetic companies, drives them through the real APIs to any point in any user journey, and then wakes the AI agents that process the data, through the same events that wake them in live operation. Those agents book real client months today, with a person reviewing every proposal before it touches the books, and most proposals now go through without an edit. Getting there took a place to reproduce the runs that went wrong, and that is what the suite provides.
The problem
Our product runs on long, multi-actor journeys. Onboarding a company is a proper mandate setup: the client and the tax advisor take turns submitting, approving and signing, and the path branches depending on how the mandate is structured. The month-end close moves through preparation, review and rework, operated from two seats with different permissions. And accounting state only makes sense as a sequence of months: whether a month looks realistic depends on the months that came before it.
None of this fits a fixture file. There is no seed script for “onboarding approved but not yet signed” or “a close in review with changes requested.” Staging data drifts, shared test organizations decay, and the most interesting edge cases live in states that are expensive to rebuild. AI agents raise the bar further. They read documents, bank transactions and prior bookings, so their behavior depends on the entire history of a company. Every run is reviewed by a person before anything touches the books, so clients were never exposed, but acting on those review findings was slow: without a way to recreate the exact company behind a finding, each improvement took longer to validate than it should have.
Three decisions
The suite is built on three decisions:
- Seed a company, not a row. Every test starts from a complete synthetic organization, created and driven through the same APIs and events the product itself uses. Nothing writes to the database directly.
- Generate data that is statistically realistic. Every company, document and transaction in the suite is generated from scratch, shaped to behave the way real-world accounting data behaves: realistic distributions and frequencies, never a record, a name or a document.
- Treat the agents as actors like any other. Triggering an agent run is a button. It publishes the same events the product publishes, and the full run is inspectable afterward.
Here is what that looks like for a month-end close.
1. A target state becomes a seeded company
A run starts with a generated organization: an invented German company name with realistic legal-form weighting, bank accounts, a configurable volume of generated transactions, and PDF invoices rendered from scratch and uploaded through the same endpoint clients use. Every synthetic write is flagged as test data at the API layer, and none of it bypasses the backend.
Journeys are encoded as graphs, and any node can be selected as a target: the suite drives the organization there and stops. “Approved but not yet signed” is a selectable state, which makes the signing step itself directly testable. For the month-end close, the suite encodes the production state machine’s allowed transitions and finds a legal path to the target state by search, executing each transition from the correct seat.
Because realistic state is a sequence of months, the seeder also builds history: closed months almost fully reconciled, the previous month mostly done, the current month still open. That gradient is what a company under active bookkeeping looks like, and it is what the agents, the reviewer views, and the reporting all read from.
2. Synthetic data that behaves like the real world
Early versions of the test data had a familiar flaw: every invoice had a due date, every transaction had a purpose line, every total added up. Real-world data looks nothing like that, and an agent that performs well on clean data tells you little about how it will perform on messy data.
One constraint was fixed before any design work started: the suite never touches client data. We handle sensitive financial information, and keeping test tooling fully separate from it was non-negotiable. That meant the generators had to close the realism gap entirely on their own. They encode the statistical shape one would expect of documents and bank transactions in a production environment, distributions and frequencies only. Every invoice, transaction and vendor the suite produces is invented, but it behaves like the real thing: invoice-numbering schemes at realistic frequencies, filenames with mangled umlauts and email-forward artifacts, truncated card descriptors, subscriptions that repeat the same odd-cent price every month, and wrong totals that resemble plausible OCR errors instead of patterns a matcher could learn to detect.
Calibration also has to work in the other direction. During review we caught a seeded document that exposed information an agent could only have gotten from the seed data itself, effectively handing it the answer, and guarding against that became a permanent part of the generator’s job.
3. The agents run against the seeded company
Once a company exists in the target state, the suite publishes the same trigger event the product publishes when a close becomes ready for booking. The agents pick the work up exactly as they always do, process the seeded documents and transactions, and write their proposals for a human to review.
Every run carries a trace ID, and the suite uses it to reconstruct the run afterward: every tool call and reasoning step rendered as a timeline, with the artifacts downloadable as a bundle. A finding from live operation can now be recreated against a fresh synthetic company and inspected in full, as many times as the investigation needs.
Deciding where to simulate
The hardest design decisions were not about what to seed but about which parts of the real system to run.
The suite deliberately bypasses real document extraction for seeded invoices, injecting synthetic extractions with a controlled mix of complete, incomplete and low-confidence results. Running the real extractor over thousands of seeded PDFs would make every run slower, non-deterministic and coupled to a model version, and it would test extraction inside a suite meant to test everything that happens after extraction. A separate test bed covers the opposite question, observing the real pipeline end to end.
The second decision has a name in the codebase: seed quiescence. Background pipelines race the seeder, and early on their churn surfaced later as failures that looked like the agents’ fault. The suite now asserts, before any agent is triggered, that the seeded state is still exactly what was seeded. A violation fails the run as a seeding error, never as an agent error, which keeps the suite’s own noise out of the agents’ results.
Where this led
The same machinery now carries our eval harness: fresh disposable organizations, fixtures with known expected output, and production incidents pinned as regression cases. We’ll cover that loop in a separate post.
Our engineering team built our internal app platform so that the people closest to the product problems could build production-grade tooling for those problems themselves. That is how a product manager ended up shipping the tool the whole team now tests, builds and demos with. The states that mattered were obvious from the product seat, because we work inside these journeys with clients and tax operations every week, and the platform meant that knowledge was enough to build the tool. The suite has since outgrown its original job. Engineers seed companies to build against while a feature is still taking shape, and our product demos run on synthetic companies too, polished ones instead of doctored test accounts. And it is part of how every agent change ships: seed the states, run the agents, read the timelines, and only then let them near a real client’s books.