OmnisBench: LLM routing efficiency

Every routing benchmark I looked at had the same problem. It measured the wrong thing, on data the models had already read.

The pitch for LLM routing is simple. You've got a spread of models, from a cheap small one to an expensive frontier one, and most requests don't need the expensive one. Send each request to the cheapest model that can handle it and you keep the quality while cutting the bill. The question is how much you save, and nobody selling a router wants to answer it with numbers you can check.

OmnisBench answers it with numbers you can check. It's an open benchmark of routing efficiency: how close a routing policy gets to the best possible quality per dollar on a fixed pool of models. Apache-2.0, every result re-grades offline, and the whole thing runs from a config file and a pair of API keys. It's the data behind OmnisRouter, and it's built so a sceptic can rebuild it.

The mistake I nearly shipped

The first run looked great. Ideal routing hit 99.7% task success at a fraction of the cost of always calling the frontier model, and I wrote it up as the headline. Then a reader pointed out the obvious. The benchmark was HumanEval and GSM8K, two sets so old and so republished that every model in the pool has almost certainly trained on them. The models weren't solving the problems. They were reciting them.

When every model already knows every answer, they all score near 100%, routing looks like it saves you a fortune for nothing, and the number is a lie. So I threw the headline out and rebuilt around a split the models can't have seen.

Fresh problems, where routing has to earn it

The fix is data with a date on it. OmnisBench pulls LiveCodeBench problems published after the pool's training cutoff, keeps the ones graded by running real test cases, and scores the policies on those alone. No memorised answers.

On the fresh split, ideal routing reaches 93.3% task success against the frontier model's 86.7%, at roughly 60% lower cost than always calling the frontier. Two things fall out of that. Routing beats the single best model on quality, because no one model solves every fresh problem and picking the right model per task wins. And it does it for well under half the price. The contaminated split, run alongside, still pins every policy near 100%, and I keep it on the page next to the real number so you can see the gap for yourself.

The perturbation-gap probe

The part I'm most pleased with turns "trust me, the split is fresh" into a measurement, and then holds itself to what the measurement will bear.

A contamination label is a judgement about release dates. It doesn't tell you how much a given model leaned on having seen the exact problem before. The perturbation gap does. You score each model twice: once on the original problem, and once on a reworded copy that keeps the answer and the test cases byte-for-byte identical, changing only the wording, the variable names and the surface story. The grader never changes, so any drop in the score comes from the wording alone.

gap = accuracy(original) - accuracy(reworded)

A model that understood the problem scores the same both ways. A model that pattern-matched the exact phrasing it saw in training drops when you change it. A large positive gap is a memorisation signal for that model on that set. A gap near zero means the score was earned.

There's a trap in doing this the naive way. If one model does the rewording, it can favour its own family's phrasing and flatter the result. The pool spans two providers, so no single rewriter is neutral to all of it. The probe rewords each model with a model from the other provider, and a validation pass throws out any rewrite no model in the pool can solve, so a broken paraphrase can't pose as memorisation.

So I ran it on 24 fresh problems, reworded once by claude-sonnet-5 and once by gpt-5, and dropped any rewrite no model in the pool could solve. The points each model gives up, under each rewriter:

gap when reworded by       sonnet-5    gpt-5
claude-opus-5                +20.0      +5.9
claude-haiku-4-5             +40.0      +5.9
gpt-5                        +20.0     +23.5
gpt-5-nano                   +20.0     +52.9
                             n=15       n=17

One number holds across both rewriters: gpt-5 gives up about 20 points whoever rewords it. That's a solid sign its score on this set leaned on the exact phrasing. The rest swing with who did the rewording. The two Anthropic models barely move under gpt-5's rewrites and fall hard under sonnet's, and gpt-5-nano does the reverse, so at 15 to 17 problems the choice of rewriter and the surviving task set move the number more than the model does. I won't rank the pool by memorisation off that, and the probe is built to make me admit it rather than dress it up.

What the numbers do show cleanly is that every model loses ground when you reword a fresh problem, by at least 6 points and often far more. A post-cutoff date keeps the exact problem out of training, but the phrasing, the structure and the usual tricks are still in there, and the models lean on them. The date label is necessary and it isn't sufficient, and now there's a measurement that says so. The next step is more problems and more rewriters per model, so the model's fingerprint separates from the rewriter's. Until then the probe has earned its place: it's the thing that stopped me publishing a tidy story the data wouldn't back.

You can rebuild every number

omnisbench verify re-runs the graders against the stored model responses and rebuilds the whole thing, the leaderboard, the per-split boards, the release-date margins and the perturbation gaps, with zero API calls. A faked number fails verification the same way a tampered answer does. The perturbations are generated once, offline, and committed, so verify re-grades what's published and never quietly rewrites the test.

What's small about it, said plainly

The fresh sample is a pilot, and I'll label it as one wherever it appears. It shows the method and the direction, not a final verdict, and widening it is the roadmap. oracle is the ceiling of routing, chosen after the fact per task, not a router you can ship, so the distance between a real router and oracle is the real scorecard. Both of those are written on the page, not buried in a footnote. A benchmark you can't inspect the internals of isn't worth publishing, which is the whole reason this one exists.