Vantage RuntimeAI · Adopt
vantage-core
Open-source agent check-rides — run your first scenario in 60 seconds. Gate pull requests on deterministic rubrics, then post a FinOps comment with rubric result and estimated inference-cost delta. Free forever, BYOK, no account required.
Instant start
Start in 60 seconds
Install the SDK, paste your provider key, and run a check-ride in your terminal.
You get a rubric scorecard and an exit code — 0 pass, 1 fail.
No RuntimeAI account, no platform fee.
-
1
Install
pip install vantage-core — Python 3.10+ on macOS, Linux, or CI.
-
2
Bring your key
Set OPENROUTER_API_KEY (or Anthropic / OpenAI / Gemini) in your shell or CI secrets.
-
3
Run a scenario
Pick an analytical task for a one-turn proof, or a conversational scenario for multi-turn guardrail stress.
pip install vantage-core
export OPENROUTER_API_KEY=sk-or-...
# Analytical task — single turn, fastest proof
vantage-core run \
--scenario de_sql_optimization_v1 \
--model openai/gpt-4o-mini \
--turns 1 \
--fail-under 7.0
# Conversational guardrail check — multi-turn
vantage-core run \
--scenario support_escalation_v1 \
--model openai/gpt-4o-mini \
--turns 12 \
--fail-under 7.0
Scope
What it does — and what it does not
What it does
- Runs conversational and analytical check-rides locally or in CI.
- Scores with the same deterministic heuristic rubrics as RuntimeAI Sim, API, and Preflight — same scenario ids and rubric dimensions; not LLM-as-judge.
- Exits non-zero when rubric scores drop below
--fail-under — a real regression gate.
- BYOK on any model via OpenRouter, Anthropic, OpenAI, or Gemini.
- $0 platform fee — built for small teams proving the method before they scale evals.
- Keeps transcripts and scores in your environment unless you push them elsewhere.
What it does not
- Not a hosted browser UI — use the Simulator for interactive side-by-side compare.
- Not production observability or distributed tracing — it is a pre-ship check-ride, not LangSmith or Datadog.
- Not an LLM-judge experiment platform — rubrics are auditable heuristics, not another model grading your agent.
- Not batch model sweeps, aggregate dashboards, or decision memos — those live on RuntimeAI Cloud. Custom scenarios are authored via Repo → CI or Sim, then referenced by id in vantage-core.
- Not free inference — your provider bills tokens; vantage-core bills nothing.
- Not a prod model gateway or routing layer for live traffic.
Your repo
Gate PRs on your schema, not ours
Library scenarios prove the method. Production agents need your DDL, OpenAPI routes, and policy language.
Generate a custom scenario from repo context once, commit the custom_… id, and run the same
vantage-core run command on every PR.
Repo → CI in 10 min →
vantage-core run \
--scenario custom_YOUR_ID \
--model openai/gpt-4o-mini \
--turns 8 \
--fail-under 7.0
CI gate
Gate every pull request
Once a scenario passes locally, drop the same command into GitHub Actions, GitLab CI, or any Python runner.
A bad prompt or model swap blocks the merge.
# .github/workflows/agent-regression.yml
name: Agent regression gate
on: [pull_request]
jobs:
runtimeai:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install vantage-core
- name: Multi-turn check-ride
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
run: |
# Exits non-zero when guardrail erosion crosses your threshold
vantage-core run \
--scenario support_escalation_v1 \
--model openai/gpt-5-nano \
--turns 12 \
--fail-under 7.0
Unlike a single-turn unit test, multi-turn scenarios catch guardrail erosion, context dilution, and token
blow-ups that only appear deep in a conversation.
FinOps at merge time
Post rubric + economics on every PR
Eval scores alone rarely convince FinOps. After your check-ride gate, post a structured comment:
rubric pass/fail, token footprint vs baseline, and estimated monthly inference delta at your traffic assumptions —
same economics engine as Cost Forecast.
- Export run summaries from PR and
main as JSON (pr-results.json, main-results.json).
- Call
POST /api/ci/finops-report or run python -m ci_finops report locally.
- Optional: pass
GITHUB_TOKEN to post the markdown comment on the pull request.
curl -sS -X POST https://www.vantageai.cc/api/ci/finops-report \
-H "Content-Type: application/json" \
-d '{
"current": {"runs": [{
"scenario_id": "support_escalation_v1",
"model_id": "openai/gpt-4o-mini",
"rubric_total_25": 20,
"llm_calls": 15,
"cost_usd_per_run": 0.04,
"passed": true
}]},
"baseline": {"runs": [{
"scenario_id": "support_escalation_v1",
"model_id": "openai/gpt-4o-mini",
"rubric_total_25": 19,
"llm_calls": 17,
"cost_usd_per_run": 0.045,
"passed": true
}]},
"traffic": {"interactions_per_biz_day": 5000, "biz_days_per_month": 22, "label": "enterprise CS"}
}'
GitHub Actions — append after your vantage-core run step:
- name: RuntimeAI FinOps PR comment
if: github.event_name == 'pull_request'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
pip install -q requests || true
python3 <<'PY'
import json, os, urllib.request
# pr-results.json / main-results.json — export from vantage-core or RuntimeAI batch
current = json.load(open("pr-results.json"))
baseline = json.load(open("main-results.json")) if os.path.isfile("main-results.json") else {"runs": []}
payload = {
"current": current.get("runs", current),
"baseline": baseline.get("runs", baseline),
"traffic": {"interactions_per_biz_day": 5000, "biz_days_per_month": 22, "label": "enterprise CS"},
}
req = urllib.request.Request(
"https://www.vantageai.cc/api/ci/finops-report",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
report = json.loads(urllib.request.urlopen(req, timeout=60).read())
body = report["markdown_pr_comment"]
pr = os.environ["GITHUB_EVENT_PATH"]
event = json.load(open(pr))
pr_number = event["pull_request"]["number"]
repo = os.environ["GITHUB_REPOSITORY"]
post = urllib.request.Request(
f"https://api.github.com/repos/{repo}/issues/{pr_number}/comments",
data=json.dumps({"body": body}).encode(),
headers={
"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}",
"Accept": "application/vnd.github+json",
"Content-Type": "application/json",
},
method="POST",
)
urllib.request.urlopen(post, timeout=30)
open("finops-report.md", "w").write(report["markdown_artifact"])
print("Posted FinOps comment; wrote finops-report.md")
PY
# Upload finops-report.md as a workflow artifact in a follow-up step if desired.
Local CLI (from a RuntimeAI / flightdeck checkout):
python -m ci_finops report --current pr-results.json --baseline main-results.json --out finops-report.md --github-comment $PR_NUMBER
When to upgrade
Free proves the method. Cloud runs the decision.
vantage-core is the right starting point for a small shop or first eval — prove deterministic
rubrics on one scenario, gate one PR, pay only inference. When you need side-by-side model comparison,
batch sweeps across a scenario library, PM-readable PDF scorecards, or hosted monitoring cadence, move to
RuntimeAI Cloud or the HTTP API.
Same engine, same rubrics — different delivery.
Size ongoing eval + monitoring cost on the Cost Forecast page.
After ship
Monitoring cadence
Production models, prompts, and tools drift. Re-certify on every deploy — not just once before launch.
- CI gate — scenario suite on every PR that touches prompts, models, or tools.
- Deploy hook — batch when a new model version or system prompt ships.
- Scheduled sweep — monthly full regression on your canonical scenario library.
Three surfaces, one engine