Vantage RuntimeAI · Forecast

Cost Forecast

Estimate monthly eval runs in three steps — model bake-off, then post-ship monitoring. One scenario sets scorecard costs; scenario counts in steps 1 and 3 scale the total.

Estimates use OpenRouter public pricing, agent turns per eval run, and — where available — aggregated results from extensive benchmark runs on this platform. Provider rates can change without notice; actual inference cost depends on your execution (turn depth, prompt size, retries, and model choice).

1. What are you evaluating?

Pick one representative scenario for model costs and scorecards. Most teams test several scenarios before ship — set Scenarios in bake-off below; step 3 covers your monitored production suite.

Representative scenario Scorecard and inference costs load from this scenario. Scenarios in bake-off How many library scenarios you run in the bake-off — same models × each scenario. Step 1 picks one for scorecard pricing. Agent turns per eval run How many back-and-forth steps the agent may take in one eval run. One turn = one agent reply (and usually a stakeholder response). The run ends at scenario closure or when this limit is reached (1–24).

2. Model selection bake-off

Choose up to 3 models. We sort by EFF (efficiency index from scorecards — quality × speed × cost). Higher % is better value.

What is EFF?

Efficiency index (% of leader) — quality × speed × cost from benchmark scorecards on this scenario. Higher % is better value.

raw = sim score ÷ (latency × cost factor)

Sim score — automated scorecard mean on 0–10 (same as scoreboards).
Speed — mean seconds per agent turn from scored runs.
Cost factor — Low ×1, Medium ×2, High ×3.5 (catalog tier).

EFF = raw ÷ best raw in this scenario (100% = leader). Same metric as scoreboards and Rankings.

Replications How many times each selected model runs the scenario in one batch — more reps tighten scorecard averages. Batches / month How many full model bake-offs you run per month while choosing a production model (e.g. weekly during selection).

3. Post-ship monitoring

Regression checks after you pick a production model — via CI or Cloud API.

Include monitoring after deploy

Scenarios monitored How many different scenarios stay in your post-ship regression suite. Deploy events / mo Eval triggers on production deploys — each deploy × scenarios × reps counts as eval runs. Full sweeps / mo Scheduled full-suite reruns (e.g. weekly) on top of per-deploy checks.

Plan

Trial $0/mo 20 included runs · 5 budget tiers Starter $499/mo 1,000 included runs · full catalog Team $1,999/mo 5,000 included runs · lower overage

Monthly breakdown

Production inference at your provider is separate — see pricing for unit economics.

For search engines and LLMs

RuntimeAI Quick Cost Forecast: simplified three-step monthly estimate for model selection eval runs and post-ship monitoring.