Every number here was read from the production database.

What that means in the data

Averaged benchmarks grade a model alone. Nothing runs alone.

The common approach is to score a model once, on a benchmark, in isolation — then route every request in a whole capability category to whatever won that average. But a model in production is never by itself. It is holding a specific tool, a specific slice of context, inside a specific agent, answering a specific prompt. Change any one of those and the ranking changes with it. So we score the combination, and keep a separate earned quality signal for each layer rather than one leaderboard.

Six independent scoreboards, updated from graded outcomes. A model that is excellent at code and poor at extraction is not one number here.

Grading in context also means blaming in context

Scoring each layer separately is only honest if a failure is charged to the layer that caused it. A single execution error quarantines the one node that failed. A 401 or 403 disables every node sharing that dead credential, because the credential is what died, not the model. A 429 temporarily suppresses the nodes on that key rather than the model itself. A provider 5xx spares the node entirely and cools off the model. Four failure classes, four different blast radii — so an outage at one host never quietly demotes a model that was working fine everywhere else.

And every layer is sized to the prompt, not to a default

Selection decides which; sizing decides how much. The catalogue is banded into 98 nano, 163 small and 116 frontier models, and the tier is resolved per task at run time rather than set per customer or per route. Context is graded the same way — a chunk’s token count puts it in one of four tiers, and a short chunk never pays for a 4,096-dimension embedding. When a tier has no reachable model the chunk is summarised down a rung rather than silently truncated. Token budgets come from the resolved model’s own catalogue entry, so nothing is capped by a constant.

The field

None of them decompose the prompt

We reviewed 13 routing and gateway products in September 2026. 5 of them route a whole prompt to a single model — one prompt, one choice. The other 8 are gateways, where routing is a configuration and fallback rule rather than anything learned; there, the customer writes the routing rules, maintains that mapping as models change, and carries the blame when it drifts. The product moves the request; the fitting is homework.

0 of 13 split one prompt into separately sized subtasks and price each on its own merits. That gap is the reason this platform exists. We are describing our own review rather than publishing a scorecard on named products — if you want to walk through it, ask.

Component speed

Where the milliseconds actually go

Measured per stage rather than per finished run, which is the only way to see whether orchestration itself costs anything. Logarithmic scale, 1 ms to 3 s.

Agent selection is the median of the selection step itself, over 2,575 selections. Embedding figures cover 965,062 successful calls: 262 ms median, 439 ms at p90, 1.44 s at p99. Scoring this finely is only worth doing if it is cheap enough to run on every request — the per-prompt decision costs roughly a five-hundredth of the model call it is choosing.

The model network

What served the traffic

Every request is served by a model node in the network. 736,029 of 738,248 jobs completed successfully across the network’s lifetime — 99.7% on a distributed pool where any single node can drop out mid-request. It has processed 227.2 million tokens over 3.2 million individually traced dispatches.

58 distinct models served production traffic, drawn from a catalogue of 377 across 10 providers, running on 203 model nodes online at the time of reading. Those are three different counts and we keep them apart deliberately.

That gap is maintained rather than left to rot. A background health cycle decommissions the nodes behind any model the catalogue has flagged deprecated, and separately cross-checks the catalogue against what each provider will still serve — where a call the provider actually rejects counts for more than an absence from a published list. The retirements hold: a node taken out for a model that no longer exists does not quietly reappear on the next sync, and its run history stays on the record.

Where the tokens went

Top models by production token volume, input plus output. Routing concentrates on small and mid-tier models and escalates only when a task earns it.

Self-learning

What the platform wrote for itself, and threw away

The platform generates its own prompt templates and then benchmarks them against the incumbent. 738 of 1,529 candidates — 48.3% — failed that gate and never reached a user. A learning loop you cannot reject from is not a learning loop; it is drift.

That gate is a trial, not a review. A candidate is put in front of a slice of live traffic beside the incumbent it would replace, graded on those runs by the same automated grader, and it carries nothing more until it beats what is already there. Self-governed means only that, on our own production traffic: an agent earns its place from graded outcomes and loses it the same way.

Of runs that reached a terminal state, 95.8% finished clean at a mean automated grade of 0.855, graded on every run rather than sampled afterwards. 77% of 1,332 rated answers came back positive, and each rating moves a per-layer quality score that changes what gets picked next time. Feedback here is not a survey; it is the routing signal.

The estate also watches itself

A learning loop decays quietly if nobody is watching it: one model’s quality score drifting down, more work falling back to a larger tier than it used to need, graded signal thinning out, the pool running short of nodes. Those patterns are scanned for on live telemetry and raised as alerts against the layer they belong to. The scan reports; it does not remediate. Repair is the failure classification described further up, kept as a separate mechanism on purpose — an alert that silently fixes things is an alert nobody can audit afterwards.

The same discipline governs what gets kept. A chain of tasks is only recorded as proven once it has been graded and settled, after which it can be recalled for a similar goal instead of being planned from scratch again. Grounding content is written to each embedding space a tenant can actually serve rather than to a single one, and anything held without real embedding capacity is re-embedded on a later pass once that capacity returns. Draining a retired embedding space is a deliberate migration rather than something the platform does to your data on its own.

Third-party validation

Benchmarked the same way we route: nothing pinned, nothing self-graded

Every question runs through the full production selection funnel — agent, template, task graph, model and miner are bound dynamically at request time, exactly as for a paying user. Requests carry a dynamic routing alias, never a model id. Runs authenticate as a dedicated evaluation account, the only principal permitted to tag a request as a benchmark run.

That tag is enforced server-side and cannot be overridden by the caller. It no-ops every learning write in both directions for the tagged run: quality-EMA and feedback updates, adaptive persistence, proven-chain recall AND storage, recipe capture, and community reinforcement — a benchmark run can neither learn from platform memory nor add to it.

Each benchmark is graded by its own upstream scorer, run verbatim from the vendored upstream source. The platform never grades its own answers; where a benchmark's own methodology calls for an LLM judge, that judge is an external model, not the miner pool being measured. Monthly, via a self-gated internal job — one sweep per benchmark family per calendar month. Each published row pairs a live run (current policy state) against a frozen-baseline run for the same family, bracketed by a policy-state snapshot read before and after the series. If that snapshot moved mid-series the row is stamped policyMoved: the scores still stand, but the row cannot be used for month-over-month attribution.

Benchmark families

Capability benchmarks first, per our own sequencing; agentic benchmarks are a later iteration.

Published runs

First published run: pending. No sweep has been run against these families yet. Once the first monthly sweep completes, this section shows the live score next to the frozen-baseline score for that family, the upstream release and harness commit it ran under, and a link to the raw scored JSONL — not a placeholder number.

Raw scored output for every published run lands at /downloads/benchmarks/, one JSONL file per sweep, so a reader can re-derive any figure above from the per-question record rather than trust the summary line.