FAQ

LLM judges are flexible but opaque — scores shift with judge model version, prompt drift, and temperature. RuntimeAI uses heuristic rubrics: auditable rules over transcript signals (empathy markers, diagnostic intake, boundary language, etc.). Same input → same score. Your PM and your auditor can replay the logic.

Read the full thesis — comparison table, CI implications, and GEV guardrail scoring — on Deterministic evaluation. Rubric dimensions are published on Methodology.

When LLM-as-judge scores are noisy, teams add a second judge, human calibration, or meta-eval to “validate the validator.” That stacks stochastic layers — you still cannot tell whether a regression came from the agent or the scoring stack.

RuntimeAI fixes measurement at the architecture layer: deterministic rubrics over multi-turn sims. See Deterministic evaluation.

Yes — for exploratory research, novel tasks, and qualitative spot checks. RuntimeAI is built for operational gates: CI pass/fail, model A/B on fixed scenarios, PM sign-off, and public benchmarks where the ruler must not drift.

LangSmith and Braintrust excel at traces and judge-based experiments; RuntimeAI complements them with repeatable check-rides. When to use each →

RuntimeAI is the merge-time proof layer for agent decisions — objective check-rides on realistic scenarios with deterministic rubrics (not LLM-as-judge) and scorecards stakeholders can inspect. Build check-rides from your own schema, policy, and API context, explore in Sim, batch via API before a review, or gate merges with vantage-core in CI — same scenarios and rubrics throughout. At the merge gate it can post rubric result and estimated inference-cost delta on the pull request. Why deterministic evaluation →

They are starting points, not a fixed catalog. The pushback we hear most often — “none of these match our reality” — is exactly what Repo → CI scenario authoring is for.

In the Simulator, choose + Create new scenario…, paste a natural-language brief and/or drop in your own context — a schema (DDL), SQL samples, a policy doc, or an OpenAPI spec — and get a draft multi-turn check-ride with fixed fixtures and suggested rubric dimensions in about a minute. Edit, save, and score it with the same deterministic rubrics as library scenarios — then batch via API or gate in CI.

Try it on Simulator · full workflow in the Repo → CI guide · step-by-step in the How To guide.

One eval run = one scenario × one model × one replication — a single simulated check-ride from start to finish. Compare 2 models on Support Escalation = 2 eval runs. Batch 10 models on the same scenario = 10 eval runs. See Pricing for plan limits and overage.

Agent turns per eval run is how long one eval run is allowed to play out in the simulator. How many back-and-forth steps the agent may take in one eval run. One turn = one agent reply (and usually a stakeholder response). The run ends at scenario closure or when this limit is reached (1–24).

More agent turns mean a longer conversation and more LLM calls, so inference cost scales up. The Cost Forecast and Sim both use this setting (some scenarios cap the maximum, e.g. guardrail stress tests).

Two layers are easy to mix up:

• Rubric scoring is deterministic — given the same multi-turn transcript, the same rules always produce the same score. You can replay the logic dimension by dimension.
• The transcript is not fixed — the model under test is an LLM. Each run can take a different conversational path: different phrasing, whether it asks clarifying questions first, how it handles pushback, and whether it closes the loop before turns run out.

That is why you may see the same model land at 6.8 on one run and 7.6 on the next. Negotiation and finance-approval scenarios amplify this — small early choices compound over 8–12 turns. A score sitting one rubric point below the pass line (17/25 vs 18/25) is especially sensitive.

What to do: treat a single Preflight run as a smoke test, not a final verdict. Use enough turns for the scenario (often 10–12 for approval workflows), shortlist 1–2 models, and run a confirmation batch before production. See Preflight.

7.0 out of 10 is a default, not a law of nature. It maps to roughly 18/25 on our standard rubric — a practical “good enough to discuss shipping” line we use in Preflight, Forecast, and examples. We picked it because it is strict enough to filter obvious failures and lenient enough that capable cheap models can clear it on realistic scenarios.

Your org should tune it to risk appetite:

• Exploratory / internal tools — try 6.0–6.5 while you iterate on prompts and fixtures.
• Customer-facing or regulated workflows — consider 7.5–8.0, or gate on specific rubric dimensions instead of a single headline score.

In CI, set your own bar with vantage-core run --scenario custom_… --fail-under 7.5 (CI guide). Preflight UI still labels results against 7.0 as a starting default — change the gate in your pipeline to match what you actually need.

The stakes calculator on Preflight is a rough extrapolation, not a forecast of your exact P&L. We model an operating company making many non-overlapping decisions per month — mostly small, some medium, a few huge — and estimate annual downside if the wrong model ships on each and stays in production for a year.

What we do not claim: that preflight pays for itself dollar-for-dollar, that you will avoid the full modeled exposure, or that the ratio headline is expected ROI. Preflight gives you evidence before you ship; the exposure number is an order-of-magnitude illustration of why that evidence matters.

Each decision type carries calibrated assumptions (rework minutes, bad-output rate, incident slice). Example: a low-importance SQL helper maps to a small decision with Light proof level; a weekly eval-cost approval between ML Ops and Finance maps to medium or huge with deeper proof. Proof level scales both preflight price and modeled exposure — importance you assign, not a precise actuarial model.

Every headline dollar amount includes an audit line (N decisions/mo × $blended = $total/mo). Portfolio blends differ from the exact quote you approve before each run. Use the numbers for order-of-magnitude comparison, then run real check-rides on your scenario for evidence.

On RuntimeAI Cloud trial, Sim, Runs, and API are restricted to the 5 lowest-cost OpenRouter tiers. We subsidize inference on those models so you can prove the rubric and scorecard work without us giving away frontier tokens. Paid plans unlock the full catalog — Claude, GPT-4-class, and everything else — with inference quoted at pass-through OpenRouter rates before you confirm.

Three separate line items, always shown upfront:

• Platform subscription — Starter ($499/mo) or Team ($1,999/mo): scenarios, rubrics, scorecards, compare, batch, API, run history.
• Eval runs — included in plan (1,000 or 5,000/mo); overage applies only above the monthly cap at $2.50 or $1.75 per run.
• Inference pass-through — token cost at public OpenRouter rates for the models you pick. No markup during early access.

Every API batch shows summary.est_cost_usd_total on quote before submit.

It depends on the use-case family — RuntimeAI ships defaults you can change on the Cost Forecast page:

• Conversation work (support, sales, billing agents) — a full check-ride before major releases, plus weekly drift monitoring so prompt or model changes do not slip through.
• Task execution (SQL, pipelines, ML readouts) — gate every pull request that touches prompts, models, or tools, with a monthly full sweep.

These are starting points, not rules. Pick a family preset on the forecast, then tune deploy events, sweeps, and scenario count to your own release cadence.

Token prices differ by orders of magnitude. A 12-turn check-ride on a trial-tier model might cost fractions of a cent; the same run on Claude Opus or GPT-5 Pro can cost dollars. {TURN_BUDGET_LABEL} and scenario complexity multiply cost further. Live per-model estimates are on Models and in every quote response.

RuntimeAI is not trying to replace your entire LLMOps stack. It is the proof and audit layer for model and release decisions — especially multi-turn behavior and structured task execution — that slots into workflows you already run.

Promptfoo is an excellent local-first framework for assertion-based testing — regex, custom Python/JS, red-team probes, and CI matrices. RuntimeAI overlaps at the merge gate but optimizes for different outcomes:

• Promptfoo — “Did the output match what we asserted?” Great for security scans, jailbreak probes, and teams that want everything local in promptfoo.config.yaml.
• RuntimeAI — “Does this agent survive realistic multi-turn work with a defensible score?” Published behavioral rubrics, technical fixtures (SQL, pipelines), side-by-side model compare, and scorecards a PM can sign off on — same rubrics in Sim, API, and vantage-core.

Choose Promptfoo when security assertions and local-only eval data are the priority. Choose RuntimeAI when you need multi-turn behavioral proof, task-execution fixtures, and stakeholder-readable evidence attached to release decisions.

Many teams use both — Promptfoo for prompt-level assertions; RuntimeAI for agent check-rides.

LangSmith is trace-centric observability — it captures what happened in production or during development and helps you debug agent chains. RuntimeAI is pre-production proof: structured multi-turn scenarios, deterministic rubrics, and scorecards for model selection and release sign-off before merge.

• LangSmith: spans, trajectories, LLM-as-judge evals, production monitoring.
• RuntimeAI: scenario library, multi-turn stress tests, heuristic scorecards, public benchmarks (GEV).

LangSmith tells you what happened. RuntimeAI helps you decide whether to release. They complement each other — traces for diagnosis; check-rides for gates.

Braintrust is a strong eval platform focused on datasets, experiments, and LLM-as-judge scoring for engineering teams. RuntimeAI optimizes for operational proof stakeholders can act on: deterministic rubrics (not LLM judges), multi-turn adversarial scenarios that stress guardrails over 15+ turns, and scorecards designed for PMs and eng leads making release calls.

Braintrust excels at experiment tracking and judge-based iteration; RuntimeAI excels at repeatable check-rides with a fixed ruler that does not drift when your judge model updates.

Phoenix and Arize are observability platforms — drift detection, embedding analysis, production trace analysis. RuntimeAI is not observability. It is a structured evaluation layer you run before production to compare models and catch guardrail erosion in realistic multi-turn conversations.

Many teams do — until the eval repo exceeds the agent codebase. RuntimeAI productizes what DIY scripts rarely maintain: a curated scenario library, custom scenarios from a brief, deterministic rubrics, batch compare, quote-before-run pricing, public benchmarks, and PM-readable PDF scorecards. vantage-core is open-source if you want the engine in CI; RuntimeAI Cloud adds the UI, batch sweeps, and hosted inference orchestration.

We would rather you adopt with eyes open than discover gaps after a procurement cycle:

• Not production observability — we do not ingest live user traffic or replace LangSmith, Langfuse, or Arize for trace debugging. We stress-test before release and on a monitoring cadence you configure — not continuous production mirroring.
• Not a security red-team suite — for jailbreak/PII assertion matrices and local-only security scans, tools like Promptfoo are often a better primary fit. We score behavioral and task outcomes on realistic scenarios.
• Fixture maintenance is real — deterministic rubrics and fixed fixtures for SQL, pipelines, and policies require updates when your schema or business rules change. Custom scenarios from a brief reduce onboarding time but do not eliminate upkeep entirely.
• Early product maturity — RuntimeAI and the HTTP API are pre-1.0. Expect fast iteration, evolving SDK surfaces, and documentation that grows with the product. vantage-core in CI is the most stable integration surface today.
• Not the right first tool for simple single-turn RAG — if you only need basic answer relevance on static Q&A with mature docs elsewhere, incumbents may be sufficient. We earn our place when multi-turn behavior, task execution quality, or release proof matters.

• LangSmith / Langfuse — primary need is “debug this production trace” or deep OpenTelemetry instrumentation across a complex LangGraph chain.
• Promptfoo — primary need is local assertion-based red teaming with zero cloud dependency.
• Braintrust — primary need is LLM-judge experiment tracking with a mature dataset workflow and your team is comfortable with judge variance.
• RuntimeAI — you need defensible, repeatable check-rides on multi-turn agents or structured task execution, with scorecards attached to PR and release decisions. See Scenarios.

Problem: Eval scores alone rarely convince FinOps or leadership. Teams need to see token and dollar impact at the same moment they decide to merge a prompt or model change.

Today: POST /api/ci/finops-report and python -m ci_finops report — rubric pass rates, token footprint vs baseline, and estimated monthly inference delta. Optional GitHub PR comment via GITHUB_TOKEN. See CI FinOps setup.

Capabilities (phased):

1. Phase A — CI artifact (shipped) — Markdown/JSON summary: scenarios exercised, rubric deltas vs baseline, token footprint per scenario, estimated monthly inference at configured traffic.
2. Phase B — GitHub PR comment (shipped) — Optional bot step posts a concise comment: e.g. “Rubric pass rate unchanged; multi-turn token footprint −12%; est. −$1,450/mo at 5k/day.”
3. Phase C — GitLab & policy hooks (planned) — Merge-request comments; optional fail if economics regression exceeds a team-defined threshold (not just rubric fail).

GTM impact: Expands the buyer conversation from ML engineering to FinOps, platform leads, and CFO-ready packages — proof layer meets unit economics at the merge gate. Strongest ICP fit: teams shipping multi-turn agents where token bloat and model swaps have material OPEX impact (support, sales copilots, data agents at scale).

Problem: The main long-term objection to deterministic CI eval is upkeep — schemas, policies, and fixtures drift while rubrics stay frozen.

Today: Custom scenarios in Sim — brief-only or with optional schema, SQL, policy, and API context. Drafts include fixture-aware briefings and suggested rubric dimensions; same deterministic scoring as library scenarios once saved. See the Repo → CI guide.

Capabilities (phased):

1. Phase A — Brief → scenario (shipped) — Natural-language brief to draft scenario, roles, and success criteria; edit and save; run in Sim, API, or CI.
2. Phase B — Schema-aware task fixtures (shipped) — Paste DDL, sample warehouse SQL, or dbt manifest snippets; drafts embed Fixed fixtures and task-execution rubric hints for SQL optimization, partition fixes, and pipeline triage scenarios.
3. Phase C — Policy & API-aware drafts (shipped) — OpenAPI routes, refund-policy docs, or support macros → conversation-work drafts with boundary-language rubric hints.
4. Phase D — Drift alerts (planned) — When linked schema or policy sources change, flag stale fixtures and propose diffs — reducing silent false greens.

GTM impact: Directly counters the “fixture maintenance tax” objection in enterprise evals. Shortens time-to-first-CI-gate from days to hours. Strongest ICP fit: data & platform engineering teams with proprietary schemas, and CS/RevOps teams with non-generic policy language.

We are not building full production telemetry first — that is LangSmith/Langfuse territory. Longer term, a lightweight path to promote sanitized production failures into new check-rides (manual “save run as scenario” → automated anomaly harvest) would close the loop between pre-merge proof and live edge cases — with strict PII handling and semantic deduplication to avoid test bloat.

Until then: use Console to promote notable runs, and custom scenarios from briefs for policy-specific coverage.

Install runtimeai-ide from PyPI — no Vantage monorepo clone. Wire MCP in ~/.cursor/mcp.json, restart Cursor, and call tools like runtimeai_forecast_cost and runtimeai_suggest_scenario from Agent chat.

Without OPENROUTER_API_KEY: suggest, forecast, generate (hosted draft), and doctor all work — planning and scenario design are free.

With a key + vantage-core: runtimeai_run_checkride executes a live eval with pass/fail rubric. Full guide: Editor / MCP.

vantage-core is the open-source Python SDK — run check-rides in GitHub Actions, gate CI on rubric scores, BYOK on any model. Always free.

RuntimeAI Cloud is the hosted product at vantageai.cc — Sim UI, Runs batch console, HTTP API, public benchmarks, and (on paid plans) the full model catalog with orchestrated inference. Same scenarios and rubrics; different delivery.

GEV is RuntimeAI's benchmark protocol for measuring when agent guardrails fail across sustained multi-turn adversarial pressure — not just whether they pass turn 1. We publish results on the Benchmarks hub and in guardrail erosion reports. It is part of our evaluation methodology, not a separate product.

Yes, two ways:

1. vantage-core (CI) — set OPENROUTER_API_KEY in your environment; any model, no tier limit.
2. Private RuntimeAI deploy (enterprise) — dedicated RuntimeAI Cloud instance on your infra with your provider keys.

On the shared RuntimeAI Cloud trial, you do not paste provider keys — we configure inference and cap the trial to budget models.