1. Product Outline: What is SimOps?
Vantage SimOps is a continuous behavior evaluation platform for customer-facing teams. It is built for leaders whose hiring, coaching, and launch-readiness workflows have outgrown ad-hoc roleplays and subjective manager opinions, and who need recorded evidence of how people perform under pressure in realistic threads.
Core Capabilities
- Human-in-the-loop check-rides: Teammates and candidates type as the protagonist (sales rep, PM, support agent, etc.) while an AI counterpart drives the other side of a live Slack-style thread.
- Scenario libraries: Pre-built multi-turn templates for discovery, escalation, board prep, backlog triage, and more — plus custom scenario authoring from a natural-language brief.
- Deterministic heuristic rubrics: Scorecards rely on auditable keyword and structural heuristics — not an LLM judging another LLM — for repeatable debriefs in under 15 minutes.
- Screening safeguards: Optional paste-blocking and tab-focus tracking for candidate loops, with PDF scorecards and transcript exports for hiring committees.
- Agent mode (optional): Run fully automated agent-vs-agent simulations and side-by-side model comparisons when you want to benchmark models before putting humans in the seat.
Scope Boundaries
- What it is: An operational decision and coaching platform for simulated performance evidence.
- What it isn’t: An LMS, a live whisper-coaching bot, a CRM, or soft-skills entertainment roleplay — it records structured check-rides for debrief and calibration.
2. Step-by-Step Testing Guide
This sandbox is fully decoupled and accessible via the production URL. Follow these four workflows to stress-test the core engine.
-
Execute a Human Check-Ride
Action
- Navigate to the live simulation terminal: /simops/sim
- Set mode to You · Human (default). Pick a scenario such as Data Drop, Discovery Call, Support Escalation, Board Prep, or Backlog Triage.
- Optionally enter participant details when screening is enabled (name, email for scorecard delivery).
- Click Start Sim and type as the protagonist while the AI counterpart replies in the thread.
What to evaluate
Does the human flow feel like a real 15-minute check-ride? Can managers trust the transcript and rubric for a structured debrief?
-
Run a Side-by-Side Model Comparison (Agent mode)
Action
- Switch mode to AI Agent and enable Compare 2 models.
- Select two OpenRouter models and the same scenario framework.
- Start the sim and review dual-column transcripts and comparative scorecards.
What to evaluate
Useful for benchmarking counterpart quality before human candidates use the same scenario in screening.
-
Grade the Heuristic Scorecard Output
Action
- Complete a run or click End Sim.
- Review the automated scorecard (five scenario-specific dimensions, 0–5 each, scaled to /10) and transcript highlights.
- Download or email the PDF scorecard when configured.
What to evaluate
Whether rubric dimensions match what your managers actually coach on (de-escalation, discovery depth, evidence discipline, etc.).
-
Explore Administrative Operations
Action
- Open the admin console: /simops/admin
- Review completed runs, use Ask about this run, and inspect the Feedback tab.
- Explore Model Costs / Batch Run and Model Test Results for multi-model sweeps.
- Try + Create new scenario… from the sim toolbar for custom authoring.
What to evaluate
Does the admin shell give enough visibility for hiring calibration, coaching programs, and cost-justified rollout decisions?
3. Initial Feedback Benchmarks
As you test the sandbox, we are looking for your perspective on:
- Rubric Usefulness: Do scenario-specific dimensions capture what you debrief on with reps, support leads, and PMs?
- Human vs. Agent Workflow: Is Human mode the right default for your hiring/coaching use case? What is missing for programmatic screening at scale?
- Custom Authoring: Does the scenario generator produce system prompts with the right vertical boundaries for your business?
4. Share your feedback
Use the form below after testing the sandbox, or email simon@vantageai.cc directly.
Prefer email? simon@vantageai.cc