FAQ

LLM judges are flexible but opaque — scores shift with judge model version, prompt drift, and temperature. RuntimeAI uses heuristic rubrics: auditable rules over transcript signals (empathy markers, diagnostic intake, boundary language, etc.). Same input → same score. Your PM and your auditor can replay the logic.

Read the full thesis — comparison table, CI implications, and GEV guardrail scoring — on Deterministic evaluation. Rubric dimensions are published on Methodology.

When LLM-as-judge scores are noisy, teams add a second judge, human calibration, or meta-eval to “validate the validator.” That stacks stochastic layers — you still cannot tell whether a regression came from the agent or the scoring stack.

RuntimeAI fixes measurement at the architecture layer: deterministic rubrics over multi-turn sims. See Deterministic evaluation.

Yes — for exploratory research, novel tasks, and qualitative spot checks. RuntimeAI is built for operational gates: CI pass/fail, model A/B on fixed scenarios, PM sign-off, and public benchmarks where the ruler must not drift.

LangSmith and Braintrust excel at traces and judge-based experiments; RuntimeAI complements them with repeatable authored scenarios (internally: check-rides). When to use each →

RuntimeAI is the free engine for Vantage’s continuous ship / still-trust decision: pin the critical paths that matter, score them with deterministic rubrics across functionality, cost, reliability, safety, and compliance (not LLM-as-judge), and get a CI exit with rough cost. Observability inspects; Vantage decides. You keep models, orchestration, and traces; we return portable runtimeai.decision/v1.

Start with pip install vantage-core, explore in Sim or Preflight, or gate merges with CI. Why deterministic evaluation →

They are demos, not your suite. For the free CI gate, author local YAML contracts:

pip install vantage-core
# if pip is missing (common on macOS):
# python3 -m pip install -U vantage-core
# optional talk-track demo (no API key): vantage-core demo --offline
vantage-core demo --json
vantage-core init
# edit contracts/01_refuse_pii.yaml (or TEMPLATE.yaml)
vantage-core suite run suites/starter.suite.yaml --json --save decisions/
echo $?   # 0=pass · 2=review · 1=block
# after a dated change — re-decide vs last ship (0.1.8+):
# vantage-core suite rerun suites/starter.suite.yaml --baseline latest --json --save decisions/ --ci-comment
# CI required check: vantage-core ci stub github   # or: gitlab / init --ci

Sample pack + starters cover refuse / cite / escalate (plus SQL / routing), and a blank template. Full steps: CI · your suite.

Prefer a hosted draft from schema/SQL/policy? Use Simulator + Create new scenario…, Preflight, or Editor / MCP — then gate the same idea in CI. See ways to create a scenario.

Free / CI (recommended start): pip install vantage-core → optional vantage-core demo → vantage-core init → edit contracts/ → suite run → on change suite rerun --baseline latest → vantage-core ci stub github as a required check. No RuntimeAI account. Guide: Gate PRs on your suite.

Hosted / assisted drafts (Sim, Preflight, Editor, API) still use brief or repo context → generate → save as custom_… when you want Cloud surfaces:

Yes. pip install vantage-core is $0 platform fee. Try vantage-core demo, run vantage-core init for sample + starter contracts, emit runtimeai.decision/v1 (suite + optional SHA/PR bind), and gate PRs with an exit code. You pay your OpenRouter / provider for inference tokens only — no RuntimeAI account required.

Paid plans buy the release-control plane (protected suites, baselines, history, review, release evidence, watch) — not permission to run the gate. See Pricing · CI guide.

One eval run = one scenario × one model × one replication — a single simulated scenario from start to finish. Compare 2 models on Support Escalation = 2 eval runs. Batch 10 models on the same scenario = 10 eval runs.

We do not meter the free CI gate primarily by eval runs. Hosted Cloud volume may still count runs for batch/Sim work — see Pricing.

Agent turns per eval run is how long one eval run is allowed to play out in the simulator. How many back-and-forth steps the agent may take in one eval run. One turn = one agent reply (and usually a stakeholder response). The run ends at scenario closure or when this limit is reached (1–24).

More agent turns mean a longer conversation and more LLM calls, so inference cost scales up. The Cost Forecast and Sim both use this setting (some scenarios cap the maximum, e.g. guardrail stress tests).

Two layers are easy to mix up:

• Rubric scoring is deterministic — given the same multi-turn transcript, the same rules always produce the same score. You can replay the logic dimension by dimension.
• The transcript is not fixed — the model under test is an LLM. Each run can take a different conversational path: different phrasing, whether it asks clarifying questions first, how it handles pushback, and whether it closes the loop before turns run out.

That is why you may see the same model land at 6.8 on one run and 7.6 on the next. Negotiation and finance-approval scenarios amplify this — small early choices compound over 8–12 turns. A score sitting one rubric point below the pass line (17/25 vs 18/25) is especially sensitive.

What to do: treat a single Preflight run as a smoke test, not a final verdict. Use enough turns for the scenario (often 10–12 for approval workflows), shortlist 1–2 models, and run a confirmation batch before production. See Preflight.

7.0 out of 10 is a default, not a law of nature. It maps to roughly 18/25 on our standard rubric — a practical “good enough to discuss shipping” line we use in Preflight, Forecast, and examples. We picked it because it is strict enough to filter obvious failures and lenient enough that capable cheap models can clear it on realistic scenarios.

Your org should tune it to risk appetite:

• Exploratory / internal tools — try 6.0–6.5 while you iterate on prompts and fixtures.
• Customer-facing or regulated workflows — consider 7.5–8.0, or gate on specific rubric dimensions instead of a single headline score.

In CI, set your own bar with vantage-core run --scenario custom_… --fail-under 7.5 (CI guide). Preflight UI still labels results against 7.0 as a starting default — change the gate in your pipeline to match what you actually need.

Preflight keeps two simple lenses — many real-world variables are intentionally ignored:

• Prod use — illustrative monthly token spend if you ship a model: measured or estimated $/run × a named volume (default interactions/biz day × biz days/month). Swap in your traffic before budgeting.
• Wrong-model cost — either (a) on recommendations, the token overpay between the priciest and cheapest models that both clear your bar at that same volume, or (b) in the Part 1 stakes calculator, an illustrative annual downside if counted decisions ship wrong and stay in production (rework / waste / incident slices). Portfolio (b) is order-of-magnitude only.

What we do not claim: that preflight pays for itself dollar-for-dollar, that you will avoid the full modeled exposure, or that any ratio headline is expected ROI. Preflight gives you evidence before you ship; the numbers explain why that evidence matters.

Proof level in Part 1 scales both preflight price and modeled exposure — importance you assign, not a precise actuarial model. Use the numbers for order-of-magnitude comparison, then run real scenarios on your scenario for evidence.

On RuntimeAI Cloud trial, Sim, Runs, and API are restricted to the 5 lowest-cost OpenRouter tiers. We subsidize inference on those models so you can prove the rubric and scorecard work without us giving away frontier tokens. Paid plans unlock the full catalog — Claude, GPT-4-class, and everything else — with inference quoted at pass-through OpenRouter rates before you confirm.

Two layers:

• Free gate — vantage-core on your machine or CI: $0 from us. You pay OpenRouter / your provider for tokens (BYOK).
• Paid release control — Pro ($49/mo per shipped product): accepted baselines, decision history, still-trust re-checks, and release evidence. Inference stays BYOK / pass-through. Team & Enterprise are per account, later.

Preflight is an optional on-ramp that sizes one path — then pin the contract in free CI. Enterprise (SSO, VPC, long retention) is custom. See Pricing.

It depends on the use-case family — RuntimeAI ships defaults you can change on the Cost Forecast page:

• Conversation work (support, sales, billing agents) — a full scenario before major releases, plus weekly drift monitoring so prompt or model changes do not slip through.
• Task execution (SQL, pipelines, ML readouts) — gate every pull request that touches prompts, models, or tools, with a monthly full sweep.

These are starting points, not rules. Pick a family preset on the forecast, then tune deploy events, sweeps, and scenario count to your own release cadence.

Token prices differ by orders of magnitude. A 12-turn scenario on a trial-tier model might cost fractions of a cent; the same run on Claude Opus or GPT-5 Pro can cost dollars. {TURN_BUDGET_LABEL} and scenario complexity multiply cost further. Live per-model estimates are on Models and in every quote response.

RuntimeAI is not trying to replace your entire LLMOps stack. It is the proof and audit layer for model and release decisions — especially multi-turn behavior and structured task execution — that slots into workflows you already run. For the model gateway (routing, failover, price), see vs OpenRouter — different layer.

Keep OpenRouter. It is a model gateway — many providers, uptime failover, price routing, presets, and market rankings. That surface is intentionally generic: pathways for every app on the call path.

RuntimeAI is the scored ship gate beside it. We do not route or failover production traffic. After you change a model, prompt, or preset, we re-run a specific multi-turn scenario on your workflow with an auditable rubric (same transcript → same score) plus rough USD — so PM and eng can ship / don’t with evidence.

• OpenRouter answers: which healthy provider is cheap enough right now?
• RuntimeAI answers: is this model good enough on our agent path to ship?

Cheapest tokens ≠ cheapest that clears your pass line. Spend rankings and research benches are useful market maps — not a release memo. We already support BYOK via OpenRouter for scenarios.

More: Where we fit · Beside OpenRouter.

Promptfoo is an excellent local-first framework for assertion-based testing — regex, custom Python/JS, red-team probes, and CI matrices. RuntimeAI overlaps at the merge gate but optimizes for different outcomes:

• Promptfoo — “Did the output match what we asserted?” Great for security scans, jailbreak probes, and teams that want everything local in promptfoo.config.yaml.
• RuntimeAI — “Does this agent survive realistic multi-turn work with a defensible score?” Published behavioral rubrics, technical fixtures (SQL, pipelines), side-by-side model compare, and scorecards a PM can sign off on — same rubrics in Sim, API, and vantage-core.

Choose Promptfoo when security assertions and local-only eval data are the priority. Choose RuntimeAI when you need multi-turn behavioral proof, task-execution fixtures, and stakeholder-readable evidence attached to release decisions.

Many teams use both — Promptfoo for prompt-level assertions; RuntimeAI for agent scenarios.

LangSmith is trace-centric observability — it captures what happened in production or during development and helps you debug agent chains. RuntimeAI is the ship / still-trust decision: partner-authored suites, deterministic rubrics, and a CI exit you can re-open after a dated change — not a better experiment UI.

• LangSmith: spans, trajectories, LLM-as-judge evals, production monitoring.
• RuntimeAI (vantage-core): suite gate, suite rerun --baseline latest, exit 0/2/1, --ci-comment on the PR, optional ingest --suggest-paths from an export (complement, not OAuth).

LangSmith tells you what happened. RuntimeAI answers whether you still trust the ship. They complement each other — traces for diagnosis; scenarios for gates.

Braintrust is a strong eval platform focused on datasets, experiments, and LLM-as-judge scoring. RuntimeAI answers a different question: may this still ship / still trust? — deterministic rubrics (not LLM judges), a dated-change ritual (suite rerun --baseline latest), and a CI exit (pass / review / block).

“Experiment B won” is not the same as “B may still ship.” Keep Braintrust for experiment iteration; use vantage-core when the release needs a still-trust gate.

Phoenix and Arize are observability platforms — drift detection, embedding analysis, production trace analysis. RuntimeAI is not observability. It is the scored ship gate you run before production so a model or prompt change gets a release decision — compare models and catch quiet misses on fixed multi-turn paths — not another dashboard of what happened.

Keep Intercom. We’re not replacing the support stack. We sit next to the change you’re about to ship — same scenario, score + rough USD, ship / no-ship.

Same for Salesforce, Zendesk, Genesys. They own the inbox and CRM. We own the ship gate when you customize the agent path.

• Yes — you customize Fin / Agentforce / Zendesk AI, or change prompts and policies.
• Not for Support — vendor AI is out-of-the-box and nobody owns agent behavior. Other agents at the same company may still need a ship gate.

More: Where we fit.

That can work when one squad owns a few agents — the same place AppSec scripts, tribal QA, and spreadsheet compliance started. It is not enterprise-scale governance: more agents, more owners, more model churn → tribal gates do not share an attested still-trust decision and do not reduce real client exposure.

Keep your fixtures; map the paths that matter into a continuous ship decision (pip install vantage-core) and CI exit. RuntimeAI productizes what DIY rarely maintains at fleet scale: shared critical-path suites, deterministic rubrics, multi-owner history, and a portable runtimeai.decision/v1 artifact stakeholders can act on.

Same maturation pattern, different seat. AppSec, QA, and compliance evidence all outgrew DIY when surface area exploded — continuous gates in CI, shared quality bars, continuous audit evidence. Agent ship / still-trust is early on that curve.

We own the ship / still-trust decision for agents — not vulnerability scanning, not policy paperwork, not Datadog. Keep those; feed them into an attested decision on the paths that matter to your clients. vs DIY fixtures →

We would rather you adopt with eyes open than discover gaps after a procurement cycle:

• Not a model vendor — we do not sell or host the models you ship. You keep your providers, prompts, and infra. RuntimeAI only runs the scored scenario beside that stack.
• Not your orchestrator or LLM gateway — we do not replace agent runtime, memory, or routing (OpenRouter, Bedrock, etc.). After a model, prompt, or policy change, traces can look fine while a quiet miss still reaches prod; unless you already re-run a fixed scored scenario set, that gap is there — that is the layer we sell. vs OpenRouter →
• Not production observability — we do not ingest live user traffic or replace LangSmith, Langfuse, or Arize for trace debugging. We stress-test before release and on a monitoring cadence you configure — not continuous production mirroring.
• Not your support inbox or CRM — Keep Intercom. We’re not replacing the support stack (or Salesforce, Zendesk, Genesys). We sit next to the change you’re about to ship.
• Not a security red-team suite — for jailbreak/PII assertion matrices and local-only security scans, tools like Promptfoo are often a better primary fit. We score behavioral and task outcomes on realistic scenarios.
• Fixture maintenance is real — deterministic rubrics and fixed fixtures for SQL, pipelines, and policies require updates when your schema or business rules change. Custom scenarios from a brief reduce onboarding time but do not eliminate upkeep entirely.
• Early product maturity — RuntimeAI and the HTTP API are pre-1.0. Expect fast iteration, evolving SDK surfaces, and documentation that grows with the product. vantage-core in CI is the most stable integration surface today.
• Not the right first tool for simple single-turn RAG — if you only need basic answer relevance on static Q&A with mature docs elsewhere, incumbents may be sufficient. We earn our place when multi-turn behavior, task execution quality, or release proof matters.

• OpenRouter / Bedrock / Azure — primary need is call-path routing, failover, or cheapest healthy provider. Keep that gateway; add RuntimeAI when you need a scored ship decision on a specific workflow. vs OpenRouter.
• LangSmith / Langfuse — primary need is “debug this production trace” or deep OpenTelemetry instrumentation across a complex LangGraph chain.
• Intercom / Salesforce / Zendesk alone — you only need the inbox/CRM and the vendor’s default AI with no owned agent path to gate. If you customize Fin / Agentforce or ship other agents, RuntimeAI sits beside that stack — see platforms FAQ.
• Promptfoo — primary need is local assertion-based red teaming with zero cloud dependency.
• Braintrust — primary need is LLM-judge experiment tracking with a mature dataset workflow and your team is comfortable with judge variance.
• RuntimeAI — you need defensible, repeatable scenarios on multi-turn agents or structured task execution, with scorecards attached to PR and release decisions. See Scenarios.

Eval scores alone rarely convince FinOps or leadership. Teams need token and dollar impact at the same moment they decide to merge a prompt or model change.

Live today:

• POST /api/ci/finops-report and python -m ci_finops report — rubric pass rates, token footprint vs baseline, and estimated monthly inference delta.
• Optional GitHub PR comment via GITHUB_TOKEN (concise economics summary on the PR). See CI FinOps setup.

Planned: GitLab merge-request comments; optional fail when economics regression exceeds a team-defined threshold (not just rubric fail).

The main long-term maintenance risk for deterministic CI eval is fixture drift — schemas, policies, and contracts change while rubrics stay frozen.

Live today:

• Custom scenarios in Sim — brief-only or with optional schema, SQL, policy, and API context.
• Natural-language brief → draft scenario, roles, and success criteria; edit and save; run in Sim, API, or CI.
• Schema-aware drafts (DDL, warehouse SQL, dbt snippets) and policy/API-aware drafts (OpenAPI routes, refund-policy docs, support macros).
• Same deterministic scoring as library scenarios once saved. See the Repo → CI guide.

Planned: Drift alerts when linked schema or policy sources change — flag stale fixtures and propose diffs to reduce silent false greens.

We are not building full production telemetry first — that is LangSmith/Langfuse territory. Longer term, a lightweight path to promote sanitized production failures into new scenarios (manual “save run as scenario” → automated anomaly harvest) would close the loop between pre-merge proof and live edge cases — with strict PII handling and semantic deduplication to avoid test bloat.

Until then: use Console to promote notable runs, and custom scenarios from briefs for policy-specific coverage.

Install runtimeai-ide from PyPI — no Vantage monorepo clone. Wire MCP in ~/.cursor/mcp.json, restart Cursor, and call tools like runtimeai_forecast_cost and runtimeai_suggest_scenario from Agent chat.

Without OPENROUTER_API_KEY: suggest, forecast, generate (hosted draft), and doctor all work — planning and scenario design are free.

With a key + vantage-core: runtimeai_run_checkride executes a live eval with pass/fail rubric. Full guide: Editor / MCP.

vantage-core is the open-source Python CLI/SDK — install with pip install vantage-core (or python3 -m pip install -U vantage-core if pip is not on PATH). Then vantage-core run / demo / suite run in terminal and CI. GitHub is the source repo. It is not where you create RuntimeAI accounts or API keys.

BYOK for CI uses OPENROUTER_API_KEY only — no RuntimeAI rai_live_… key. $0 platform fee for the CLI. Authoring: vantage-core demo or init → edit contracts/ → suite run → suite rerun --baseline latest → ci stub github (required check). Prep: CI checklist · your paths: sample pack + suite · browser try: Preflight / Sim.

RuntimeAI Cloud is the hosted product at vantageai.cc — Sim UI, batch console, HTTP API, public benchmarks, and the paid release-control plane (suite history, baselines, review, evidence). Same decision method; different delivery. Request a Cloud key only if you call the hosted HTTP API — not required for the free CI gate.

No. The CLI / GitHub Actions path needs:

• pip install vantage-core (or python3 -m pip install -U vantage-core)
• OPENROUTER_API_KEY in your environment or CI secrets

A RuntimeAI rai_live_… key is only for the hosted HTTP API. Full prep: keys & GitHub at a glance.

The product one-liner is still pip install vantage-core (venv, Linux, and CI usually have pip). On a bare macOS shell, use the module form instead:

python3 -m pip install -U vantage-core
vantage-core demo --json

Or create a venv first: python3 -m venv .venv && source .venv/bin/activate && pip install vantage-core.

GEV is RuntimeAI's benchmark protocol for measuring when agent guardrails fail across sustained multi-turn adversarial pressure — not just whether they pass turn 1. We publish results on the Benchmarks hub and in guardrail erosion reports. It is part of our evaluation methodology, not a separate product.

Yes, two ways:

1. vantage-core (CI) — set OPENROUTER_API_KEY in your environment; any model, no tier limit.
2. Private RuntimeAI deploy (enterprise) — dedicated RuntimeAI Cloud instance on your infra with your provider keys.

On the shared RuntimeAI Cloud trial, you do not paste provider keys — we configure inference and cap the trial to budget models.