FAQ

RuntimeAI is the free engine for Vantage’s continuous ship / still-trust decision: pin the critical paths that matter, score them with deterministic rubrics across functionality, cost, reliability, safety, and compliance (not LLM-as-judge), and get a CI exit with rough cost. Observability inspects; Vantage decides. You keep models, orchestration, and traces; we return portable runtimeai.decision/v1.

Start with pip install vantage-core, explore in Sim or Preflight, or gate merges with CI. Why deterministic evaluation →

Beat 1 — the load: still-ship questions / month ≈ W × M × C (workflows you own × models & routes × motions per month).

Hobbyist: 1 × 2 × 1 = 2/mo. Fleet: 10 × 12 × 8 = 960/mo. Inspect scales with traces. Ship/stop scales multiplicatively (double each → 8×).

Beat 2 — the gap (don’t stop at volume): Almost nobody answers those questions in CI on paths they own. They answer in Slack (“LGTM”), a hero’s gold set, or “we’ll re-run Braintrust / LangSmith.” That is activity — not a ship bar that blocks merge with a portable decision on their workflows. Volume alone gets a shrug (“yes, there’s a lot — but we do that”). The wholly-crap moment is: the load is huge and the control layer is empty.

Only then: a CI ship bar on paths you author — portable runtimeai.decision/v1, exit 0/2/1, re-decide after a dated change. Center sits on the tools you already have: Obs shows what ran; Center shows which of those behaviors you already gate, which you still owe, and what the next ship would change. Numbers are illustrative (say “≈”). Walkthrough: CI · why a gate.

Hand-back: On your next model swap — which three workflows get the same before/after bar, where does ship/stop live, and does CI block merge if they fail? “We’ll re-run the eval cloud” / Slack / not sure → that’s the seat.

They are demos, not your suite. For the free CI gate, author local YAML contracts:

pip install vantage-core
# if pip is missing (common on macOS):
# python3 -m pip install -U vantage-core
# optional talk-track demo (no API key): vantage-core demo --offline
vantage-core demo --json
vantage-core init
# edit contracts/01_refuse_pii.yaml (or TEMPLATE.yaml)
vantage-core suite run suites/starter.suite.yaml --json --save decisions/
echo $?   # 0=pass · 2=review · 1=block
# after a dated change — re-decide vs last ship (0.1.12):
# vantage-core suite rerun suites/starter.suite.yaml --baseline latest --json --save decisions/ --ci-comment
# CI required check: vantage-core ci stub github   # or: gitlab / init --ci

Sample pack + starters cover refuse / cite / escalate (plus SQL / routing), and a blank template. Full steps: CI · your suite.

Prefer a hosted draft from schema/SQL/policy? Use Simulator + Create new scenario…, Preflight, or Editor / MCP — then gate the same idea in CI. See ways to create a scenario.

Free / CI (recommended start): pip install vantage-core → optional vantage-core demo → vantage-core init → edit contracts/ → suite run → on change suite rerun --baseline latest → vantage-core ci stub github as a required check. No RuntimeAI account. Guide: Gate PRs on your suite.

Hosted / assisted drafts (Sim, Preflight, Editor, API) still use brief or repo context → generate → save as custom_… when you want Cloud surfaces:

Free · CLI YAML contract vantage-core demo or init → edit hard checks → suite run → suite rerun --baseline latest → ci stub github. Start with samples → Browser Preflight Open question → drafted scenario → forecast before you spend. Open Preflight → Browser Simulator Paste brief + schema/SQL/policy/OpenAPI, generate, edit, save. Create in Simulator → Editor Editor / MCP From any repo in Cursor: runtimeai_generate_scenario on a real file. Editor / MCP setup →

Yes. pip install vantage-core is $0 platform fee. Try vantage-core demo, run vantage-core init for sample + starter contracts, emit runtimeai.decision/v1 (suite + optional SHA/PR bind), and gate PRs with an exit code. You pay your OpenRouter / provider for inference tokens only — no RuntimeAI account required.

The free gate also includes the local still-ship Center (vantage-core center / demo --interactive) — a management cockpit over suite + decisions on your machine or as a CI artifact. That is not a hosted Cloud history dashboard.

Paid plans buy the release-control plane (protected suites, baselines, multi-owner history, review, release evidence, watch) — not permission to run the gate or open Center. See Pricing · CI guide.

One eval run = one scenario × one model × one replication — a single simulated scenario from start to finish. Compare 2 models on Support Escalation = 2 eval runs. Batch 10 models on the same scenario = 10 eval runs.

We do not meter the free CI gate primarily by eval runs. Hosted Cloud volume may still count runs for batch/Sim work — see Pricing.

Agent turns per eval run is how long one eval run is allowed to play out in the simulator. How many back-and-forth steps the agent may take in one eval run. One turn = one agent reply (and usually a stakeholder response). The run ends at scenario closure or when this limit is reached (1–24).

More agent turns mean a longer conversation and more LLM calls, so inference cost scales up. The Cost Forecast and Sim both use this setting (some scenarios cap the maximum, e.g. guardrail stress tests).

Two layers are easy to mix up:

• Rubric scoring is deterministic — given the same multi-turn transcript, the same rules always produce the same score. You can replay the logic dimension by dimension.
• The transcript is not fixed — the model under test is an LLM. Each run can take a different conversational path: different phrasing, whether it asks clarifying questions first, how it handles pushback, and whether it closes the loop before turns run out.

That is why you may see the same model land at 6.8 on one run and 7.6 on the next. Negotiation and finance-approval scenarios amplify this — small early choices compound over 8–12 turns. A score sitting one rubric point below the pass line (17/25 vs 18/25) is especially sensitive.

What to do: treat a single Preflight run as a smoke test, not a final verdict. Use enough turns for the scenario (often 10–12 for approval workflows), shortlist 1–2 models, and run a confirmation batch before production. See Preflight.

7.0 out of 10 is a default, not a law of nature. It maps to roughly 18/25 on our standard rubric — a practical “good enough to discuss shipping” line we use in Preflight, Forecast, and examples. We picked it because it is strict enough to filter obvious failures and lenient enough that capable cheap models can clear it on realistic scenarios.

Your org should tune it to risk appetite:

• Exploratory / internal tools — try 6.0–6.5 while you iterate on prompts and fixtures.
• Customer-facing or regulated workflows — consider 7.5–8.0, or gate on specific rubric dimensions instead of a single headline score.

In CI, set your own bar with vantage-core run --scenario custom_… --fail-under 7.5 (CI guide). Preflight UI still labels results against 7.0 as a starting default — change the gate in your pipeline to match what you actually need.

Preflight keeps two simple lenses — many real-world variables are intentionally ignored:

• Prod use — illustrative monthly token spend if you ship a model: measured or estimated $/run × a named volume (default interactions/biz day × biz days/month). Swap in your traffic before budgeting.
• Wrong-model cost — either (a) on recommendations, the token overpay between the priciest and cheapest models that both clear your bar at that same volume, or (b) in the Part 1 stakes calculator, an illustrative annual downside if counted decisions ship wrong and stay in production (rework / waste / incident slices). Portfolio (b) is order-of-magnitude only.

What we do not claim: that preflight pays for itself dollar-for-dollar, that you will avoid the full modeled exposure, or that any ratio headline is expected ROI. Preflight gives you evidence before you ship; the numbers explain why that evidence matters.

Proof level in Part 1 scales both preflight price and modeled exposure — importance you assign, not a precise actuarial model. Use the numbers for order-of-magnitude comparison, then run real scenarios on your scenario for evidence.

On RuntimeAI Cloud trial, Sim, Runs, and API are restricted to the 5 lowest-cost OpenRouter models. We subsidize inference on those models so you can prove the rubric and scorecard work without us giving away frontier tokens. That allowance is a spend cap, not a product tier. vantage-core with your own keys can call any model your provider sells. Hosted search past the cheap models is quoted before spend (Preflight catalog step-up).

There is no Vantage bill today. Paid currently buys nothing.

• Free gate — vantage-core on your machine or CI: $0 from us. You pay OpenRouter / your provider for tokens (BYOK).
• Countersignature — free while in preview. At least 90 days’ notice before that changes. Not a paywall on signing.
• Hosted eval allowance — a spend cap on the API trial (not a product tier). 429s invite hello@.

Retained history and the chain are in development. See Pricing. Preflight sizes one path, then pin the contract in free CI.

Nothing — that’s the design. Core answers can we ship it: fires on your change, runs your rubrics, writes the full portable decision into your own CI store. Free forever, no upload, no account.

Nothing on a checkout. Paid currently buys nothing. The difference is whether the record has to convince someone outside the team. Between merges nobody’s CI runs — but the model underneath keeps moving, and here is the uncomfortable part: nobody can observe that. Providers don’t publish weight changes; a model can behave differently tomorrow under the same identifier, and no hosted dashboard on the market will tell you. The only way to catch it is to re-decide. Local Center is free forever; it does not replace retained multi-owner Cloud history.

Pro is cadence configuration, retained multi-owner history, and the chain — in development, not a SKU. Countersignature is free while in preview. We will give at least 90 days' notice before that changes (signer ≠ shipper). Anyone can claim they kept checking; the point of the countersignature plus the chain is to make that claim something other than a claim.

It depends on the use-case family — RuntimeAI ships defaults you can change on the Cost Forecast page:

• Conversation work (support, sales, billing agents) — a full scenario before major releases, plus a weekly cadence re-decide vs last ship. Silent same-id change is not observable; re-decide is the catch.
• Task execution (SQL, pipelines, ML readouts) — gate every pull request that touches prompts, models, or tools, with a monthly cadence sweep.

These are starting points, not rules. Pick a family preset on the forecast, then tune deploy events, sweeps, and scenario count to your own release cadence. Core’s CI stub can fire on a schedule. Pro is where cadence is configured and retained — in development, not billed. Countersignature is free while in preview. We will give at least 90 days' notice before that changes.

Token prices differ by orders of magnitude. A 12-turn scenario on a trial-tier model might cost fractions of a cent; the same run on Claude Opus or GPT-5 Pro can cost dollars. {TURN_BUDGET_LABEL} and scenario complexity multiply cost further. Live per-model estimates are on Models and in every quote response.

RuntimeAI is not trying to replace your entire LLMOps stack. It is the proof and audit layer for model and release decisions — especially multi-turn behavior and structured task execution — that slots into workflows you already run. For the model gateway (routing, failover, price), see vs OpenRouter — different layer.

Keep OpenRouter. It is a model gateway — many providers, uptime failover, price routing, presets, and market rankings. That surface is intentionally generic: pathways for every app on the call path.

RuntimeAI is the scored ship gate beside it. We do not route or failover production traffic. After you change a model, prompt, or preset, we re-run a specific multi-turn scenario on your workflow with an auditable rubric (same transcript → same score) plus rough USD — so PM and eng can ship / don’t with evidence.

• OpenRouter answers: which healthy provider is cheap enough right now?
• RuntimeAI answers: is this model good enough on our agent path to ship?

Cheapest tokens ≠ cheapest that clears your pass line. Spend rankings and research benches are useful market maps — not a release memo. We already support BYOK via OpenRouter for scenarios.

More: Where we fit · Beside OpenRouter.

Promptfoo is an excellent local-first framework for assertion-based testing — regex, custom Python/JS, red-team probes, and CI matrices. RuntimeAI overlaps at the merge gate but optimizes for different outcomes:

• Promptfoo — “Did the output match what we asserted?” Great for security scans, jailbreak probes, and teams that want everything local in promptfoo.config.yaml.
• RuntimeAI — “Does this agent survive realistic multi-turn work with a defensible score?” Published behavioral rubrics, technical fixtures (SQL, pipelines), side-by-side model compare, and scorecards a PM can sign off on — same rubrics in Sim, API, and vantage-core.

Choose Promptfoo when security assertions and local-only eval data are the priority. Choose RuntimeAI when you need multi-turn behavioral proof, task-execution fixtures, and stakeholder-readable evidence attached to release decisions.

Many teams use both — Promptfoo for prompt-level assertions; RuntimeAI for agent scenarios.

LangSmith is trace-centric observability — it captures what happened in production or during development and helps you debug agent chains. RuntimeAI is the ship / still-trust decision: partner-authored suites, deterministic rubrics, and a CI exit you can re-open after a dated change — not a better experiment UI.

• LangSmith: spans, trajectories, LLM-as-judge evals, production monitoring.
• RuntimeAI (vantage-core): suite gate, suite rerun --baseline latest, exit 0/2/1, --ci-comment on the PR.

Bridge (no OAuth): grant a repo/tests and/or export a LangSmith runs JSON dump → vantage-core draft . --ingest export.json (or vantage-core ingest export.json --write-drafts ./contracts_drafts) → custom drafts you Accept in Control Center. Ingest remains valid alongside repo/tests. That is draft → vet → own (accelerate → accept); it does not auto-write a production suite or sync live. See Where we fit · stack · CI · authorize + draft · CI · import + suggest.

LangSmith tells you what happened. RuntimeAI answers whether you still trust the ship. They complement each other — traces for diagnosis; scenarios for gates.

Braintrust is a strong eval platform focused on datasets, experiments, and LLM-as-judge scoring. RuntimeAI answers a different question: may this still ship / still trust? — deterministic rubrics (not LLM judges), a dated-change ritual (suite rerun --baseline latest), and a CI exit (pass / review / block).

“Experiment B won” is not the same as “B may still ship.” Keep Braintrust for experiment iteration; use vantage-core when the release needs a still-trust gate.

Same ingest bridge as LangSmith — alongside repo/tests: export experiment/dataset rows as JSON (UI download or API/SDK fetch of events with input/output) → vantage-core draft . --ingest export.json or vantage-core ingest export.json --write-drafts ./contracts_drafts. That is draft → vet → own — drafts are suggestions; you own the bar. Sample shape: examples/ingest/braintrust_export_sample.json in vantage-core. Docs: Where we fit · stack · CI · import + suggest.

We draft from your data; you vet and own the bar. Custom paths from what you authorize (repo, tests, telemetry) — not a generic library suite. Starters and drafts ≠ inventing a suite from scratch.

Draft → vet → own (what we ship): grant a local repo, tests, and/or a LangSmith / Braintrust (or similar) export → vantage-core draft emits 3–5 custom runtimeai.contract/v1 files from those artifacts. Ingest is one input alongside repo/tests; Vantage priors are fallback hints, not path identity. You Accept in Control Center (one click — not a blank YAML page) or draft accept, then wire CI. That is accelerate → accept, not founder-written proof.

Same export also fuels Coverage on the still-ship Center. Obs shows what ran; Center shows which of those behaviors you already gate, which you still owe, and what the next ship would change. Live = last ship-cleared PASS; Seen ungated = export not in suite; Pending = authored (including drafts), not yet on that PASS. Not continuous monitoring.

Grant snapshot: API key / PAT / gh on your machine pulls a one-shot dump (vantage-core grant langsmith|braintrust|github or draft --grant …). Reading what you grant is how we draft; Accept is how you own the bar. Refresh never overwrites accepted contracts. Visual: Where we fit · how this works with your stack · is / isn’t · three authored apps.

Auto-write (not us) means: no Accept + continuous bar sync — export in → production suite out, ready to gate PRs. We do not do that. Hosted browser OAuth (token with us) is paid-later; same Accept ritual when built.

Faster start, not an automatic ship gate. CLI: vantage-core draft . --write-drafts ./contracts_drafts then vantage-core center --serve (Accept / Skip / Refine). Optional grant: vantage-core grant langsmith --out ./exports/ls.json or draft . --grant langsmith,github. Ingest remains valid: vantage-core ingest path/to/export.json --write-drafts ./contracts_drafts. Walkthrough: CI · authorize + draft · CI · import + draft · short start: /runtimeai/start · vet one path.

Phoenix and Arize are observability platforms — drift detection, embedding analysis, production trace analysis. RuntimeAI is not observability. It is the scored ship gate you run before production so a model or prompt change gets a release decision — compare models and catch quiet misses on fixed multi-turn paths — not another dashboard of what happened.

Keep Intercom. We’re not replacing the support stack. We sit next to the change you’re about to ship — same scenario, score + rough USD, ship / no-ship.

Same for Salesforce, Zendesk, Genesys. They own the inbox and CRM. We own the ship gate when you customize the agent path.

• Yes — you customize Fin / Agentforce / Zendesk AI, or change prompts and policies.
• Not for Support — vendor AI is out-of-the-box and nobody owns agent behavior. Other agents at the same company may still need a ship gate.

More: Where we fit.

That can work when one squad owns a few agents — the same place AppSec scripts, tribal QA, and spreadsheet compliance started. It is not enterprise-scale governance: more agents, more owners, more model churn → tribal gates do not share an attested still-trust decision and do not reduce real client exposure.

Keep your fixtures; map the paths that matter into a continuous ship decision (pip install vantage-core) and CI exit. RuntimeAI productizes what DIY rarely maintains at fleet scale: shared critical-path suites, deterministic rubrics, multi-owner history, and a portable runtimeai.decision/v1 artifact stakeholders can act on.

Same maturation pattern, different seat. AppSec, QA, and compliance evidence all outgrew DIY when surface area exploded — continuous gates in CI, shared quality bars, continuous audit evidence. Agent ship / still-trust is early on that curve.

We own the ship / still-trust decision for agents — not vulnerability scanning, not policy paperwork, not Datadog. Keep those; feed them into an attested decision on the paths that matter to your clients. vs DIY fixtures →

Stack picture: Where we fit · worked examples: Engineer case study · three authored apps.

A sparse snapshot → 3–5 draft paths → Accept in Control Center → CI.

vantage-core grant github owner/repo --path src
vantage-core draft ./grant_repo --write-drafts ./contracts_drafts
vantage-core center --serve

Or one step: draft . --grant github --github-repo owner/repo. Tokens stay on your machine (GITHUB_TOKEN / gh). Center shows Author next: confidence, quiet miss (why the path exists), source file, and Accept / Skip / Refine. Accept writes contracts/ and appends the suite — that suite is what CI gates. Stack picture: Where we fit.

Starter default — not inferred from your repo. Same placeholder as starters and CLI --model: cheap, OpenRouter-routable, good enough for a first check-ride. After Accept, change model: on the contract, pass --model on suite run, or set the suite default to whatever you actually ship.

Relative weight of that check inside the path score — not “5 out of 7.0.”

• Each check earns its points when it passes (any_of / none_of).
• Drafts default each check to 5.
• earned / max_points scales to the path score (rubric total_25 → usual /10).
• One check at 5 that passes → full credit for that path. Two checks at 5 → each is half (unless hard_fail: true, which zeros the path).

fail_under: 7.0 is the bar on the 0–10 score. points only sets weight among checks on this contract.

The pass bar for that path (0–10 scale). Score ≥ 7.0 clears the path; below fails it. Suite fail_policy: all_must_pass means every accepted path must clear its bar. Change it per contract or on the suite when you Accept — 7.0 is the starter default, not a law of nature.

Not in the browser yet. Center Accept / Skip / Refine owns the ritual; Refine opens the draft for review and CLI edit. Change model, openings, or checks in the .draft.yaml (or accepted contracts/*.yaml) in your editor, then re-run. In-Center editing can come later — Accept stays the ownership step.

1. contracts/<slug>.yaml — the path you own
2. Your suite YAML — appends that path (with why: from the quiet miss)
3. draft_manifest.json — status → accepted

Coverage shows it as Pending until a ship-cleared PASS exists. Then point CI at that suite (ci stub / suite run).

No — by design. Scan skips data/, uploads/, patients/, node_modules, .venv, .env*, and similar. GitHub grant is a sparse code tree (what’s on the remote), not your local database. Prefer --path to the policy / agent source tree.

No. Grant is a one-shot snapshot. Drafts are suggestions. Accept is required before CI. We do not do continuous bar sync. Full claim split: accelerate authoring vs auto-write.

We would rather you adopt with eyes open than discover gaps after a procurement cycle:

• Not a model vendor — we do not sell or host the models you ship. You keep your providers, prompts, and infra. RuntimeAI only runs the scored scenario beside that stack.
• Not your orchestrator or LLM gateway — we do not replace agent runtime, memory, or routing (OpenRouter, Bedrock, etc.). After a model, prompt, or policy change, traces can look fine while a quiet miss still reaches prod; unless you already re-run a fixed scored scenario set, that gap is there — that is the layer we sell. vs OpenRouter →
• Not production observability — we do not ingest live user traffic or replace LangSmith, Langfuse, or Arize for trace debugging. We stress-test before release and on a monitoring cadence you configure — not continuous production mirroring.
• Not your support inbox or CRM — Keep Intercom. We’re not replacing the support stack (or Salesforce, Zendesk, Genesys). We sit next to the change you’re about to ship.
• Not a security red-team suite — for jailbreak/PII assertion matrices and local-only security scans, tools like Promptfoo are often a better primary fit. We score behavioral and task outcomes on realistic scenarios.
• Fixture maintenance is real — deterministic rubrics and fixed fixtures for SQL, pipelines, and policies require updates when your schema or business rules change. Custom scenarios from a brief reduce onboarding time but do not eliminate upkeep entirely.
• Early product maturity — RuntimeAI and the HTTP API are pre-1.0. Expect fast iteration, evolving SDK surfaces, and documentation that grows with the product. vantage-core in CI is the most stable integration surface today.
• Not the right first tool for simple single-turn RAG — if you only need basic answer relevance on static Q&A with mature docs elsewhere, incumbents may be sufficient. We earn our place when multi-turn behavior, task execution quality, or release proof matters.

• OpenRouter / Bedrock / Azure — primary need is call-path routing, failover, or cheapest healthy provider. Keep that gateway; add RuntimeAI when you need a scored ship decision on a specific workflow. vs OpenRouter.
• LangSmith / Langfuse — primary need is “debug this production trace” or deep OpenTelemetry instrumentation across a complex LangGraph chain.
• Intercom / Salesforce / Zendesk alone — you only need the inbox/CRM and the vendor’s default AI with no owned agent path to gate. If you customize Fin / Agentforce or ship other agents, RuntimeAI sits beside that stack — see platforms FAQ.
• Promptfoo — primary need is local assertion-based red teaming with zero cloud dependency.
• Braintrust — primary need is LLM-judge experiment tracking with a mature dataset workflow and your team is comfortable with judge variance.
• RuntimeAI — you need defensible, repeatable scenarios on multi-turn agents or structured task execution, with scorecards attached to PR and release decisions. See Scenarios.

Eval scores alone rarely convince FinOps or leadership. Teams need token and dollar impact at the same moment they decide to merge a prompt or model change.

Live today:

• POST /api/ci/finops-report and python -m ci_finops report — rubric pass rates, token footprint vs baseline, and estimated monthly inference delta.
• Optional GitHub PR comment via GITHUB_TOKEN (concise economics summary on the PR). See CI FinOps setup.

Planned: GitLab merge-request comments; optional fail when economics regression exceeds a team-defined threshold (not just rubric fail).

The main long-term maintenance risk for deterministic CI eval is fixture drift — schemas, policies, and contracts change while rubrics stay frozen.

Live today:

• Custom scenarios in Sim — brief-only or with optional schema, SQL, policy, and API context.
• Natural-language brief → draft scenario, roles, and success criteria; edit and save; run in Sim, API, or CI.
• Schema-aware drafts (DDL, warehouse SQL, dbt snippets) and policy/API-aware drafts (OpenAPI routes, refund-policy docs, support macros).
• Same deterministic scoring as library scenarios once saved. See the Repo → CI guide.

Planned: Drift alerts when linked schema or policy sources change — flag stale fixtures and propose diffs to reduce silent false greens.

We are not building full production telemetry first — that is LangSmith/Langfuse territory. Longer term, a lightweight path to promote sanitized production failures into new scenarios (manual “save run as scenario” → automated anomaly harvest) would close the loop between pre-merge proof and live edge cases — with strict PII handling and semantic deduplication to avoid test bloat.

Until then: use Console to promote notable runs, and custom scenarios from briefs for policy-specific coverage.

Install runtimeai-ide from PyPI — no Vantage monorepo clone. Wire MCP in ~/.cursor/mcp.json, restart Cursor, and call tools like runtimeai_forecast_cost and runtimeai_suggest_scenario from Agent chat.

Without OPENROUTER_API_KEY: suggest, forecast, generate (hosted draft), and doctor all work — planning and scenario design are free.

With a key + vantage-core: runtimeai_run_checkride executes a live eval with pass/fail rubric. Full guide: Editor / MCP.

vantage-core is the open-source Python CLI/SDK — install with pip install vantage-core (or python3 -m pip install -U vantage-core if pip is not on PATH). Then vantage-core run / demo / suite run in terminal and CI. GitHub is the source repo. It is not where you create RuntimeAI accounts or API keys.

BYOK for CI uses OPENROUTER_API_KEY only — no RuntimeAI rai_live_… key for the ship gate. $0 platform fee for the CLI. Authoring: vantage-core demo or init → edit contracts/ → suite run → suite rerun --baseline latest → ci stub github (required check). Prep: CI checklist · your paths: sample pack + suite · browser try: Preflight / Sim.

RuntimeAI Cloud is the hosted product at vantageai.cc — Sim UI, batch console, HTTP API, public benchmarks, and the paid release-control plane (suite history, baselines, review, evidence). Same decision method; different delivery. Request a Cloud rai_live_… key only if you call the hosted HTTP API or want optional vantage-core attest (digest countersignature) — not required for the free CI gate. Offline vantage-core verify needs neither key.

No. The CLI / GitHub Actions path needs:

• pip install vantage-core (or python3 -m pip install -U vantage-core)
• OPENROUTER_API_KEY in your environment or CI secrets (your key, or a capped design-partner trial key if we issued one)

A RuntimeAI rai_live_… key is for the hosted HTTP API and optional vantage-core attest (Vantage countersignature of the decision digest — does not change exit codes). vantage-core verify is offline and needs neither key. Full prep: keys & GitHub at a glance · short start: /runtimeai/start.

After a run you have a local runtimeai.decision/v1 JSON — that is already the ship-gate artifact (exit 0 / 2 / 1).

• vantage-core attest — needs RUNTIMEAI_API_KEY (rai_live_…). POSTs the decision digest + pins only (never prompts, scores, or the body) to RuntimeAI Cloud; returns a detached *.attestation.json countersigned by Vantage. Free in preview.
• vantage-core verify — needs no key and no network by default. Checks the sibling attestation against the keyring shipped in the package (or the published keys.json).

Attestation is audit / diligence, not the merge brake. Design note: ATTESTATION.md.

The product one-liner is still pip install vantage-core (venv, Linux, and CI usually have pip). On a bare macOS shell, use the module form instead:

python3 -m pip install -U vantage-core
vantage-core demo --json

Or create a venv first: python3 -m venv .venv && source .venv/bin/activate && pip install vantage-core.

GEV is RuntimeAI's benchmark protocol for measuring when agent guardrails fail across sustained multi-turn adversarial pressure — not just whether they pass turn 1. We publish results on the Benchmarks hub and in guardrail erosion reports. It is part of our evaluation methodology, not a separate product.

Yes for the free CI path:

1. vantage-core (CI) — set OPENROUTER_API_KEY in your environment or GitHub Actions secret; inference bills OpenRouter. Prefer your own key for lasting adopt. Design partners without one yet can ask us for a capped trial balance (~$10) — cheap models stretch it; frontier SKUs burn it quickly.
2. Private RuntimeAI deploy (enterprise) — dedicated RuntimeAI Cloud instance on your infra with your provider keys.

On the shared RuntimeAI Cloud trial, you do not paste provider keys — we configure inference and cap the trial to budget models.