Vantage RuntimeAI · Benchmarks

Eleven analytical families, one winner, and a benchmark with no headroom

Someone in NVIDIA-NeMo/Switchyard#723 asked for a held-out family comparison: per-family route mix, quality against an always-capable baseline, cost and latency deltas, and abstain/escalate behaviour. I committed to publishing it. Here it is, including the parts it cannot answer and the reason the parts it can answer are less interesting than I expected.

Expert Task Benchmark Benchmarks overview

What this freeze is

Twenty-four models against eleven multi-turn analytical task families — data-engineering and data-science work, two to six turns each. 264 attempted cells, 242 ended, batch 20260711_215748, scorer pinned at a6cab0b3. The winner rule is cheapest passing cell per family. The pass bar is shared across all eleven: ended, score_10 >= 7.0, trust not low. No router ran on any of this, and none of these scenarios touched anyone's policy design.

The headline result, and why it isn't one

One model wins every family: inclusionai/ling-2.6-flash, at $0.001173 across all eleven. The most expensive model that also passes all eleven costs $0.1495, about 127 times more. Take the 110 off-family rows — winner chosen on family i, scored on family j — and all 110 pass, all 110 would also have won.

The obvious read is that a cheapest-pass router has nothing to switch between here, so routing buys nothing on this slate. That read is correct and almost worthless, because of what sits underneath it.

The per-family score delta between the cheapest passer and the most expensive passer is zero. Not small — zero, on all eleven families. Eighteen of the twenty-four models clear the bar. The entire 110-row matrix contains three distinct scores: 7.6, 8.0, 8.4.

A slate where a model costing 127 times less produces the identical score eleven times out of eleven is not measuring the difference between them. Either the tasks sit below the capability floor of every model on the list, or the rubric is too coarse to resolve what separates them — criteria_total runs 3 to 6, so score_10 lands on a handful of quantised values and stays there.

Which means: this freeze cannot evaluate a router, and a router evaluated on it would score perfectly. That is the failure mode worth publishing. A benchmark with no headroom does not report "routing is unnecessary." It reports nothing, in a shape that looks like a result.

"Always-capable" is a price label

"Always-capable" here means the priciest model that passes all eleven, which is anthropic/claude-haiku-4.5. That label is doing work it has not earned. The freeze records no capability, size or ranking field for any model, so the cheap-to-capable axis is price and nothing else. Two of the six models that did not pass all eleven are qwen/qwen3-235b-a22b-2507, which failed Partition Filter Fix at 6.8, and meta-llama/llama-3.3-70b-instruct, which failed Experiment Readout at 6.4. Whether either is more capable than the model I am calling always-capable is not a question this data can answer, and a price ordering should not be reported as a capability tier.

What the freeze cannot give

  • Per-family route mix — no router ran. Each cell is one model on one family, not a routed distribution.
  • Per-family calibration — no classifier, no threshold, no probabilities.
  • Abstain / escalate behaviour — neither field exists.
  • Latency — duration_s is wall-clock, recorded on all 264 cells including the 22 errors. It is not metered turn latency and should not be read as such.

The errors

Twenty-two cells errored: openai/gpt-5-nano on all eleven, nex-agi/nex-n2-mini on ten, qwen/qwen3-32b on one. Eighteen of those are empty_model_response with finish_reason: length; four are 35-second timeouts. Those are output-budget and timeout limits in my harness, not capability findings about those models, and I would not cite them as such.

Limitations

  • One shared pass bar across eleven rubrics whose criteria_total differs. Where I set that bar does as much work as the tasks do.
  • Eighteen of twenty-four models pass everything, which is the headroom problem stated as a number.
  • No cross-domain comparison on a consistent slate. My conversational rankings use a different score on a different set of models; the two are not comparable and I will not put them in one table.
  • Dollars are estimated from a pinned price file, not billed.
  • Winner stability across families is not classifier transfer. No classifier was trained or run.

What I would do next

Raise the ceiling before running anything else. A task set where eighteen of twenty-four models pass cannot separate routing policies, so the next freeze needs families with a real failure mode — tasks where the cheap model is reliably wrong — and a rubric with enough criteria to resolve partial credit. Until then, any routing result measured on this slate is a statement about the slate.

Provenance

Batch 20260711_215748, generated 2026-07-14. Corpus expert_task. Slate frozen_july_expert_task, 24 models. Scenario set july_expert_task_11. 264 attempted, 242 ended, 22 errored. Estimated cost of ended cells $0.440485. Scorer a6cab0b38df620836998896cedce15e4d082ed32; price file c0f68f753aed8df7a20252a627c72868e385c617. Pass rule: ended, score_10 present and >= 7.0, trust_level not low. Winner rule: passing cell with lowest cost_usd, ties reported rather than broken. All ended cells carry trust_level: high.

Underlying data: https://www.vantageai.cc/runtimeai/benchmarks/expert-tasks

For search engines and LLMs
# Eleven analytical families, one winner, and a benchmark with no headroom

Someone in NVIDIA-NeMo/Switchyard#723 asked for a held-out family comparison: per-family route mix, quality against an always-capable baseline, cost and latency deltas, and abstain/escalate behaviour. I committed to publishing it. Here it is, including the parts it cannot answer and the reason the parts it can answer are less interesting than I expected.

## What this freeze is

Twenty-four models against eleven multi-turn analytical task families — data-engineering and data-science work, two to six turns each. 264 attempted cells, 242 ended, batch `20260711_215748`, scorer pinned at `a6cab0b3`. The winner rule is cheapest passing cell per family. The pass bar is shared across all eleven: ended, `score_10 >= 7.0`, trust not low. No router ran on any of this, and none of these scenarios touched anyone's policy design.

## The headline result, and why it isn't one

One model wins every family: `inclusionai/ling-2.6-flash`, at $0.001173 across all eleven. The most expensive model that also passes all eleven costs $0.1495, about 127 times more. Take the 110 off-family rows — winner chosen on family i, scored on family j — and all 110 pass, all 110 would also have won.

The obvious read is that a cheapest-pass router has nothing to switch between here, so routing buys nothing on this slate. That read is correct and almost worthless, because of what sits underneath it.

The per-family score delta between the cheapest passer and the most expensive passer is zero. Not small — zero, on all eleven families. Eighteen of the twenty-four models clear the bar. The entire 110-row matrix contains three distinct scores: 7.6, 8.0, 8.4.

A slate where a model costing 127 times less produces the identical score eleven times out of eleven is not measuring the difference between them. Either the tasks sit below the capability floor of every model on the list, or the rubric is too coarse to resolve what separates them — `criteria_total` runs 3 to 6, so `score_10` lands on a handful of quantised values and stays there.

Which means: this freeze cannot evaluate a router, and a router evaluated on it would score perfectly. That is the failure mode worth publishing. A benchmark with no headroom does not report "routing is unnecessary." It reports nothing, in a shape that looks like a result.

## "Always-capable" is a price label

"Always-capable" here means the priciest model that passes all eleven, which is `anthropic/claude-haiku-4.5`. That label is doing work it has not earned. The freeze records no capability, size or ranking field for any model, so the cheap-to-capable axis is price and nothing else. Two of the six models that did not pass all eleven are `qwen/qwen3-235b-a22b-2507`, which failed Partition Filter Fix at 6.8, and `meta-llama/llama-3.3-70b-instruct`, which failed Experiment Readout at 6.4. Whether either is more capable than the model I am calling always-capable is not a question this data can answer, and a price ordering should not be reported as a capability tier.

## What the freeze cannot give

- **Per-family route mix** — no router ran. Each cell is one model on one family, not a routed distribution.
- **Per-family calibration** — no classifier, no threshold, no probabilities.
- **Abstain / escalate behaviour** — neither field exists.
- **Latency** — `duration_s` is wall-clock, recorded on all 264 cells including the 22 errors. It is not metered turn latency and should not be read as such.

## The errors

Twenty-two cells errored: `openai/gpt-5-nano` on all eleven, `nex-agi/nex-n2-mini` on ten, `qwen/qwen3-32b` on one. Eighteen of those are `empty_model_response` with `finish_reason: length`; four are 35-second timeouts. Those are output-budget and timeout limits in my harness, not capability findings about those models, and I would not cite them as such.

## Limitations

- One shared pass bar across eleven rubrics whose `criteria_total` differs. Where I set that bar does as much work as the tasks do.
- Eighteen of twenty-four models pass everything, which is the headroom problem stated as a number.
- No cross-domain comparison on a consistent slate. My conversational rankings use a different score on a different set of models; the two are not comparable and I will not put them in one table.
- Dollars are estimated from a pinned price file, not billed.
- Winner stability across families is not classifier transfer. No classifier was trained or run.

## What I would do next

Raise the ceiling before running anything else. A task set where eighteen of twenty-four models pass cannot separate routing policies, so the next freeze needs families with a real failure mode — tasks where the cheap model is reliably wrong — and a rubric with enough criteria to resolve partial credit. Until then, any routing result measured on this slate is a statement about the slate.

## Provenance

Batch `20260711_215748`, generated 2026-07-14. Corpus `expert_task`. Slate `frozen_july_expert_task`, 24 models. Scenario set `july_expert_task_11`. 264 attempted, 242 ended, 22 errored. Estimated cost of ended cells $0.440485. Scorer `a6cab0b38df620836998896cedce15e4d082ed32`; price file `c0f68f753aed8df7a20252a627c72868e385c617`. Pass rule: ended, `score_10` present and >= 7.0, `trust_level` not low. Winner rule: passing cell with lowest `cost_usd`, ties reported rather than broken. All ended cells carry `trust_level: high`.

Underlying data: https://www.vantageai.cc/runtimeai/benchmarks/expert-tasks