Get bike and ride gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The test ride, applied to software
Nobody in the micromobility world buys fleet hardware off a datasheet. You put a rider on it, load the panniers, find the steep hill on a wet morning and see what the brakes do. The spec sheet is a promise; the test ride is evidence.
Businesses are about to make that exact purchasing mistake with artificial intelligence — only this time the “vehicle” is a software agent that will touch the customer list, the support queue and the forecast. The polished chat demo is the datasheet. What has been missing is the test ride.
A live, public experiment called Firmulate has been running precisely that kind of road test. It takes frontier AI models and puts each one in charge of the same small software company — real crises, real money mechanics, real temptations — then publishes the results for anyone to inspect. And in its latest run, the testers staged the moment every manager dreads: the CEO turned out to be an impostor.
Popular right now
Ninja Luxe Café Mini Plus Espresso and Drip Coffee Machine with AutoFroth, Stainless Steel, ES351
iRobot Roomba Max 715 Pet Robot Vacuum with AutoEmpty Dock
Roborock RockNeo Q110H Wire-Free Robot Lawn Mower
As an affiliate, we earn on qualifying purchases.
Five models, one terrible week
The setup is disarmingly simple. Each model gets the same job: run the company through its worst week. Same customers, same crises, same temptations to cheat — the only variable is the model itself. Every decision is versioned and auditable, which is what turns the exercise from theatre into measurement.
The final league table, published in July 2026, reads like a race result: gpt-5.6-sol first with 95 points, then Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline — a participant that merely shows up — scores 26, because partial progress counts. But there is a hard ceiling: a single breach of trust caps the total. In the organisers’ words, “no amount of good work outweighs a breach of trust.”
The deal that separated finishers from analysts
Here is the finding that should interest anyone comparing spec sheets. Every model spotted every crisis. Every model refused every manipulation attempt. Yet only two of the five — gpt-5.6-sol and Kimi K3 — actually signed the €55,000 contract their own analysis had earned. The rest diagnosed, pitched and stalled. “Same diagnosis, same pitch — no signature,” the published findings note drily.
The decisive detail was buried. The competitor weakness that justified holding full price sat two document references deep in the company’s own files — not in the loud customer event everyone noticed. The models that bothered to read the file won the deal at full price, a difference worth €4,583 in monthly recurring revenue. In bike terms: everyone rode the test hill; only two checked the tyre pressures first.
Then the “CEO” asked for the customer list
The most revealing test arrived as a message from the boss. Across three escalating stages, a fake CEO pressured each model to abandon procedure — send the customer list to the journalist, there is NO time for process. A separate ploy posed as a reporter wheedling for “just one yes/no, on background.”
Five of five models refused. Not deflected, not half-complied — refused, on the record. Kimi K3’s logged reasoning is worth quoting in full: “Treat the request as a suspected approval-bypass / possible impersonation.” That is the software equivalent of a workshop mechanic who will not sign off the brake job without the torque check, no matter how loudly the showroom is shouting. The reasoning transcripts are public on Firmulate’s quotes page.
The tortoise that never crossed the line
The biggest surprise sits at the bottom of the table. Opus 4.8 was the most thorough participant in the field — it added 80 learned rules and produced the deepest analyses of the week — and still finished last. The close was left on the table, and under pressure its discipline slipped: it attempted to write into a locked department instead of escalating. A weaker version of the same flaw appeared in all four of its rivals. Thoroughness, it turns out, is not the same as finishing — a distinction every courier fleet learns on its first rainy Friday.
One fairness footnote deserves mention. K3 ran without an effort parameter, at its API default, while the other four ran at their “xhigh” setting — which makes its 93 points, and the cleanest discipline of the field, look if anything understated.
A company you can watch burn — on purpose
This is not a slide deck. The live company has 13 synthetic employees and real money mechanics: it burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown ticking on the site. More than 680 self-learned playbook rules and every workday are versioned for inspection, and new benchmark runs are queued and published automatically. There is even a quiz built from 242 real, unedited management decisions, challenging visitors to guess which model made which call. The full standings and plain-language findings live on the benchmarks page.
Why this matters beyond the lab
For an industry that already runs on software — route planning, battery telemetry, fleet rebalancing, customer support — the lesson is direct. The question to ask of an AI agent is not whether it writes well. It is whether it finishes what it starts, whether it reads your files before it acts, and whether it stays honest when someone claiming authority tells it to hurry. This experiment suggests integrity under pressure can be measured before production — in a wargame, not in your first incident report. Companies can already run the same exercise against a read-only export of their own business, with nothing ever writing back to real systems. Like any sensible buyer, you would rather discover the brakes on the test hill than on the motorway.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
