Prerequisites
- Target agents turned on under Manage → Agents with compatible model providers connected (Agents).
- Representative tasks for your workload.
- Runs in your run balance (model inference is billed separately by your providers).
Steps
1
Choose tasks
Select tasks spanning simple to complex scenarios—more than 6 tasks so Comparisons has at least 6 differing tasks to separate agents cleanly.
2
Hold the treatment constant
Pick Raw Agent to benchmark agents natively, or one shared treatment to test how well each agent uses your product’s tools.
3
Select agents, models, and effort levels
In New experiment, select each agent, model, and effort level (add an agent multiple times to compare different models or effort levels on it). Default sends no effort level, letting the agent use its native default. Set Trials to
2 or more and launch.4
Weigh pass rate against cost and speed
Open Matrix on Evals to compare pass rates, then switch Matrix metric to Cost, Tokens, Duration, Steps, or Tool calls to inspect efficiency.
5
Confirm differences on Comparisons
Expand Comparisons below the Matrix grid to verify which gaps hold up across tasks (check All N comparisons under Agents with when testing multiple treatments). For a Likely difference, select Run again to be sure · N runs before committing to a higher-cost model.
6
Inspect agent-specific frictions
On Frictions, filter Where to fix to Agent behavior and check More often with / Rate by agent to see why one agent stumbled where another passed.
Interpret the results
From the CLI and the plugin
Example CLI check for agent-specific frictions:
Other ways to run it
- Start fast: run the Latest model shootout template on Home across all connected agents.
- Test reasoning effort: run one model across multiple effort levels to check whether extra reasoning justifies higher cost.
- Benchmark custom agents: include your ACP custom agent (private beta) alongside built-in agents.
Next steps
Measure a tool's impact
See which tools boost each agent’s pass rate.
Track results over time
Relaunch on a schedule when new models ship.
Agents
Supported built-in agents and provider rules.
Frictions
Diagnose agent-specific failures.

