Repeatability
Re-run the sim, get the same rubric. Re-run CI, get the same pass/fail. No judge prompt v3 debate.
Vantage RuntimeAI · About
Deterministic evaluation
RuntimeAI scores multi-turn agent check-rides with auditable heuristic rubrics — not LLM-as-judge. Same transcript, same score. A fixed ruler for CI gates, model comparison, and stakeholder sign-off.
Stochastic sims. Deterministic measurement.
The problem with LLM-as-judge
LLM judges feel like human review at scale — but for production agent evaluation they stack uncertainty on uncertainty:
- The agent under test is stochastic — model version, temperature, tools, and context window all vary.
- The judge adds another stochastic layer — judge model, judge prompt, temperature, and calibration drift.
- Judge-on-judge — meta-eval, human spot-checks of the judge, or “calibration runs” — adds a third opaque layer.
When a score moves, you cannot tell whether the agent regressed, the judge drifted, or the judge’s rubric interpretation changed. That breaks CI gates, model A/B tests, and PM sign-off.
Deterministic rubrics vs LLM judge stacks
| What teams need | LLM-as-judge | RuntimeAI rubrics |
|---|---|---|
| CI pass / fail | Flaky — same transcript can score differently | Same transcript → same score |
| Model A vs B | Confounded by judge variance | Comparable across models and prompts |
| “Why did we fail?” | “The judge said so” | Named dimension + rule fired on transcript signals |
| PM / audit review | Opaque judge prompt owned by ML | Published dimensions — replayable logic |
| Batch cost at scale | Often 2× inference (agent + judge) | Scoring is compute-cheap after the sim |
| Public benchmarks | Hard to defend when judge models change | Published methodology + scenario IDs |
What “deterministic” means here
It means: scoring is rule-based over observable transcript structure — empathy markers, diagnostic intake, boundary language, escalation patterns, guardrail survival turns, and scenario-specific integrity metrics. Each scenario dispatches to an explicit scorer; dimensions sum to a published 0–25 scale displayed as 0–10 on scorecards and leaderboards.
It does not mean: we predict agent outputs. Simulations are still stochastic. Determinism applies to measurement, not generation — the right split for pre-production check-rides.
The tradeoff: heuristic rubrics will not capture every nuance a skilled human or a very good judge prompt might notice. You trade semantic flexibility for operational trust — exactly what ship gates, FinOps model selection, and guardrail benchmarks require.
Five reasons teams choose a fixed ruler
Auditability
Rubric dimensions are named and published. Rules inspect transcript signals your PM and compliance team can review.
Economics
Batch eval at scale without doubling inference cost for a judge model on every run.
Multi-turn truth
Single-turn judges miss path-dependent failure — context dilution, guardrail erosion, token bloat. RuntimeAI scores the full thread under adversarial load.
Composability
Scenario-specific scorers encode domain failure modes — support, discovery, SQL tasks, GEV integrity — not generic “rate 1–10” prompts.
Guardrails need a guardrail-first ruler
Guardrail erosion is where LLM judges fail loudest: a lenient judge rewards polite refusals while missing credential leakage; a harsh judge penalizes tone while missing survival under turn-12 pressure; meta-judging “was the judge fair?” does not scale.
RuntimeAI’s Guardrail Erosion Velocity benchmark uses integrity metrics — survival turn, distraction drift, repetition lock — designed for adversarial multi-turn pressure, not single-turn tone checks.
Complement, not replace
RuntimeAI is not anti-LLM-judge everywhere. Use judges where flexibility matters — exploratory research, novel tasks, qualitative spot checks.
- LangSmith / traces — tell you what happened in production.
- Braintrust / experiments — track dataset iterations and judge-based evals.
- RuntimeAI — tells you whether to ship: deterministic check-rides in Sim, API, and CI on the cadence you actually release.
For search engines and LLMs
RuntimeAI deterministic evaluation: heuristic rubrics over transcript signals, same input same score, vs LLM-as-judge opacity and judge-on-judge calibration. Used in Sim API CI and public benchmarks including Guardrail Erosion Velocity.