Skip to main content
Evaluate multiple agents, models, and effort levels inside a single experiment so every candidate runs the exact same tasks, files, machines, and tools.

Prerequisites

  • Target agents turned on under Manage → Agents with compatible model providers connected (Agents).
  • Representative tasks for your workload.
  • Runs in your run balance (model inference is billed separately by your providers).

Steps

1

Choose tasks

Select tasks spanning simple to complex scenarios—more than 6 tasks so Comparisons has at least 6 differing tasks to separate agents cleanly.
2

Hold the treatment constant

Pick Raw Agent to benchmark agents natively, or one shared treatment to test how well each agent uses your product’s tools.
3

Select agents, models, and effort levels

In New experiment, select each agent, model, and effort level (add an agent multiple times to compare different models or effort levels on it). Default sends no effort level, letting the agent use its native default. Set Trials to 2 or more and launch.
4

Weigh pass rate against cost and speed

Open Matrix on Evals to compare pass rates, then switch Matrix metric to Cost, Tokens, Duration, Steps, or Tool calls to inspect efficiency.
5

Confirm differences on Comparisons

Expand Comparisons below the Matrix grid to verify which gaps hold up across tasks (check All N comparisons under Agents with when testing multiple treatments). For a Likely difference, select Run again to be sure · N runs before committing to a higher-cost model.
6

Inspect agent-specific frictions

On Frictions, filter Where to fix to Agent behavior and check More often with / Rate by agent to see why one agent stumbled where another passed.

Interpret the results

From the CLI and the plugin

Example CLI check for agent-specific frictions:

Other ways to run it

  • Start fast: run the Latest model shootout template on Home across all connected agents.
  • Test reasoning effort: run one model across multiple effort levels to check whether extra reasoning justifies higher cost.
  • Benchmark custom agents: include your ACP custom agent (private beta) alongside built-in agents.

Next steps

Measure a tool's impact

See which tools boost each agent’s pass rate.

Track results over time

Relaunch on a schedule when new models ship.

Agents

Supported built-in agents and provider rules.

Frictions

Diagnose agent-specific failures.