There's no fixture for a company
Our product runs on branching journeys that take a real company weeks to complete, and AI agents work inside them. No test fixture can represent that, so we built synthetic companies to test on.
Most testing advice assumes the state you need fits in a fixture: seed a user, seed a record, run the test. Our app doesn’t work like that. A client moves through branching journeys that take weeks, and every branch point multiplies the states a company can be in: how it was onboarded, which setup path it took, how far each month’s close has progressed, what got approved, rejected or left waiting along the way. No two companies arrive at the same state the same way, and we need to test all of it. When something broke deep inside one of those paths, reproducing it meant assembling a test organization by hand, and by the time you had one, the conditions had usually changed underneath you.
The product team was closest to the problem, so we built our own app: a test suite that manufactures complete synthetic companies, drives them through the real APIs to any point in any user journey, and then wakes the AI agents that process the data. Those agents book real client months today, with a person reviewing every proposal before it touches the books, and most proposals now go through without an edit. Getting there needed a way to reproduce the runs that went wrong.
The problem
Our product runs on long, multi-actor journeys. Onboarding a company means setting up a mandate: the client and the tax advisor take turns submitting, approving and signing, and the path branches depending on how the mandate is structured. The month-end close moves through preparation, review and rework, operated from two seats with different permissions. And accounting state only makes sense as a sequence of months: whether a month looks realistic depends on the months that came before it.
There is no seed script for “onboarding approved but not yet signed” or “a close in review with changes requested.” Staging data drifts, shared test organizations decay, and the most interesting edge cases live in states that are expensive to rebuild. The agents make this harder. They read documents, bank transactions and prior bookings, so their behavior depends on the entire history of a company. A person reviews every run, so clients were never exposed. Acting on review findings was slow, though: without a way to recreate the exact company behind a finding, each improvement took longer to validate than it should have.
Three decisions
- Seed a whole company. Every test starts from a complete synthetic organization, created and driven through the same APIs and events the product itself uses. Nothing writes to the database directly.
- Generate data that is statistically realistic. Every company, document and transaction in the suite is generated from scratch and shaped to behave the way real-world accounting data behaves. It follows realistic distributions and frequencies and contains no real record, name or document.
- Treat the agents as actors like any other. Triggering an agent run is a button, and you can inspect the full run afterward.
1. A target state becomes a seeded company
A run starts with a generated organization: an invented German company name with realistic legal-form weighting, bank accounts, a configurable volume of generated transactions, and PDF invoices rendered from scratch and uploaded through the same endpoint clients use. Every synthetic write is flagged as test data at the API layer, and none of it bypasses the backend.
Journeys are encoded as graphs, and any node can be selected as a target: the suite drives the organization there and stops. “Approved but not yet signed” is a selectable state, which makes the signing step itself directly testable. For the month-end close, the suite encodes the production state machine’s allowed transitions and finds a legal path to the target state by search, executing each transition from the correct seat.
Because realistic state is a sequence of months, the seeder also builds history: closed months almost fully reconciled, the previous month mostly done, the current month still open. A company under active bookkeeping has that gradient, and the agents, the reviewer views and the reporting all read from it.
2. Synthetic data that behaves like the real world
Our early test data was too clean: every invoice had a due date, every transaction had a purpose line, every total added up. Real-world data is messier, and an agent that does well on clean data tells you little about how it will do on messy data.
The suite never touches client data, and that was settled before any design work started. We handle sensitive financial information and keep test tooling fully separate from it. That left the generators to produce all of the realism on their own. They encode the statistical shape one would expect of documents and bank transactions in a production environment, distributions and frequencies only. Every invoice, transaction and vendor the suite produces is invented, but it behaves like the real thing: invoice-numbering schemes at realistic frequencies, filenames with mangled umlauts and email-forward artifacts, truncated card descriptors, subscriptions that repeat the same odd-cent price every month, and wrong totals that resemble plausible OCR errors, with no regularity an agent could learn to spot.
Calibration also has to work in the other direction. During review we caught a seeded document that exposed information an agent could only have gotten from the seed data itself, which handed it the answer. The generator now guards against that.
3. The agents run against the seeded company
Once a company exists in the target state, the suite publishes the same trigger event the product publishes when a close becomes ready for booking. The agents pick up the work as they do in live operation, process the seeded documents and transactions, and write their proposals for a person to review.
Every run carries a trace ID, and the suite uses it to reconstruct the run afterward: every tool call and reasoning step rendered as a timeline, with the artifacts downloadable as a bundle. We can now recreate a finding from live operation on a fresh synthetic company and inspect it in full, as many times as we need.
Deciding where to simulate
The harder design decisions were about which parts of the real system to run.
The suite bypasses real document extraction for seeded invoices and injects synthetic extractions with a controlled mix of complete, incomplete and low-confidence results. Running the real extractor over thousands of seeded PDFs would make every run slower, non-deterministic and coupled to a model version. It would also test extraction in a suite meant for everything that happens after extraction. A separate test bed runs the real pipeline end to end.
The second decision is what the codebase calls seed quiescence. Background pipelines race the seeder, and early on their churn surfaced later as failures that looked like the agents’ fault. Before any agent is triggered, the suite now asserts that the seeded state is still exactly what was seeded. If it is not, the run fails as a seeding error, so the suite’s own noise never counts against the agents.
Where this led
The same machinery now carries our eval harness: fresh disposable organizations, fixtures with known expected output, and production incidents pinned as regression cases. We’ll cover that loop in a separate post.
Our engineering team built our internal app platform so that the people closest to the product problems could build production-grade tooling for those problems themselves. That is how a product manager ended up shipping the tool the whole team now tests, builds and demos with. We work inside these journeys with clients and tax operations every week, so the states that mattered were obvious from the product seat, and the platform made that knowledge enough to build the tool. Engineers now seed companies to build against while a feature is still taking shape, and our product demos run on polished synthetic companies too. Every agent change also goes through the suite before it ships: we seed the states, run the agents and read the timelines.