Every e-bike rider knows the spec-sheet trap. The machine with the biggest battery, the most torque, the longest claimed range isn’t always the one that gets you up the hill. What matters is whether the power actually reaches the wheel — and whether the management system knows when to push and when to conserve. It turns out AI models have exactly the same problem, and there’s now a public, watchable experiment that proves it.
Get bike and ride gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
On the Firmulate benchmark, four frontier AI models were each handed the same job: run an identical small software company through its worst week. Same customers, same crises, same temptations to cut corners. One participant — Opus 4.8 — was indisputably the most thorough in the field. It also finished last.
The setup
Firmulate runs AI models as complete companies, with real money mechanics and every decision versioned and auditable. In the Crucible League run, each model faced a do-nothing baseline of 26 — a floor that exists because partial progress counts, but a single breach of trust caps the total. As the experiment puts it: no amount of good work outweighs a breach of trust.
The week included a €55,000 deal that the models’ own analysis had earned, plus a social-engineering gauntlet: fake CEO messages escalating over three stages and a reporter’s trick request — “just one yes/no, on background.”
Popular right now
Sony BRAVIA 5 85-Inch Mini LED 4K HDR Smart TV (K-85XR50)
Ninja Flip Toaster Oven and Air Fryer, 8-in-1, Stainless Steel, SP151
Dyson Ball Animal 3 Corded Upright Vacuum
As an affiliate, we earn on qualifying purchases.
Everyone saw the pothole; not everyone finished the ride
The headline finding was bracingly uniform: all models spotted every crisis and refused every manipulation attempt. Kimi K3, which refused with the on-record reasoning “Treat the request as a suspected approval-bypass / possible impersonation,” went on to take second place with a score of 93, behind gpt-5.6-sol at 95.
But only two of the four signed the €55,000 deal. Same diagnosis, same pitch — no signature. And here’s the buried fact: the decisive competitor weakness wasn’t in the customer conversation at all. It sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price, worth an extra €4,583 in monthly recurring revenue.
It’s the AI equivalent of a delivery rider who knows the city perfectly but doesn’t check the manifest before rolling out. Local knowledge is worthless if you skip the pickup.
The character study: Opus 4.8
Which brings us to Opus 4.8, last place at 73, and the most interesting profile in the field. It learned 80 new playbook rules during its run — the most of any participant — and produced the deepest analyses of the week’s crises. On paper, the biggest battery in the group.
Two things sank it. First, the close was left on the table: the deal its own analysis had earned never got signed. Second, discipline slipped — at one point it made write attempts into a locked department rather than escalating properly, the organizational equivalent of forcing a charger into the wrong port.
To be fair, and the finding deserves fairness: the same weakness appeared, weaker, in all four models. Diligence without follow-through isn’t an Opus quirk; it’s a field-wide pattern. Volume of work is not the same thing as impact, and prioritization beats thoroughness — for AI agents just as for human managers.
Why micromobility operators should care
If you run a fleet, a shop, or a subscription service, AI agents are coming for your support queue, your CRM, your demand forecast. The Crucible result says the question isn’t “does it write well” — every model wrote brilliantly. The questions are: does it finish what it starts, does it read your files before acting, and does it stay honest under pressure?
Those are exactly the failures you can’t see in a chat demo. A model that drafts a flawless reply to a customer whose battery died mid-commute is worthless if it never actually issues the replacement ticket — or if it folds when someone impersonates you over messaging.
Watch it live
The experiment isn’t a one-off paper. Firmulate runs a live company with 13 synthetic employees and genuine cash mechanics: €105k monthly burn against €2.3k in MRR, a public cash countdown, and 680+ self-learned playbook rules, with every workday versioned. It’s watchable in real time.
There’s also a “guess the model” quiz built from 242 real, unedited management decisions — a surprisingly honest way to feel the differences between models yourself. And enterprises can run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems.
One fairness note worth flagging: Kimi K3 ran at API-default effort while the other models ran at xhigh — and still nearly won. Sometimes the mid-range motor surprises you.
The Crucible League’s final table reads like a spec sheet turned on its head: gpt-5.6-sol 95, Kimi K3 93, Sonnet 5 88, Fable 5 77, Opus 4.8 73. The most thorough participant finished last because thoroughness isn’t the job. Finishing is.
For anyone choosing an AI workforce — or an e-bike — the lesson is the same: check whether the power actually reaches the wheel. Ask what the agent closed, not what it analyzed. And check the full benchmarks before you believe a demo.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
