Skip to main content
Run the same tasks with and without your tool, then compare pass rates on Matrix and Comparisons to measure your tool’s lift.

Prerequisites

Steps

1

Write tasks that test real product work

Under Manage → Tasks, create realistic tasks with rubrics, files, and machines. Avoid naming your tool in task instructions so you measure whether agents discover it naturally. Include more than 6 tasks—Comparisons needs at least 6 differing tasks (ties do not count).
2

Add your tool

Add the tool under Manage → Tools, and enter any required keys or OAuth sign-ins in the dashboard.
3

Create a treatment for your tool

Under Manage → Treatments (Toolsets), create a treatment containing your tool. Every project already includes Raw Agent (no added tools) as the standard baseline.
4

Launch the experiment

Open New experiment, select your tasks, both treatments, and your target agents (Clear any extra pre-selected agents so you only spend runs on the ones you want to measure). Set Baseline to Raw Agent, set Trials to 2 or more, lower Max Concurrency to 1 if tasks modify shared remote state, and launch.
5

Compare pass rates and tool usage on Matrix

On Matrix with Evals selected, compare your treatment’s cells against Raw Agent for the same task and agent (for example, 1 of 4 trials passed vs. 3 of 4 is a lift of +50 percentage points). Switch Matrix metric to Treatment use to confirm agents actually called your tool (a lower bound that excludes script calls and stays blank for SDKs).
6

Verify whether the difference holds up

Expand Comparisons below the Matrix grid. A Clear difference is settled enough to act on; for a Likely difference, select Run again to be sure · N runs to launch a focused follow-up. Read the Brief inside any difference block to see frictions observed with the worse side, and expand All N comparisons for per-agent results (Comparisons).
7

Fix remaining product frictions

On Frictions, filter Where to fix to Product, fix the highest-impact problems in your tool, API, or docs (Find and fix frictions), then select Launch again.

Interpret the results

From the CLI and the plugin

Example prompt for your AI assistant:

Other ways to run it

  • Start from a template: try Skill lift or Stripe interface evals under Home → Templates (Templates).
  • Compare versions or interfaces: put v1 vs. v2—or MCP vs. CLI vs. SDK—into separate treatments, or combine tools (such as CLI + skill) in one treatment.
  • Test across multiple agents: include several agents to see which harnesses benefit most (Compare agents and models).

Next steps

Find and fix frictions

Fix top product blockers and verify the improvement.

Track results over time

Rerun on a schedule after every release.

Treatments (Toolsets)

Organize tools and disable individual MCP tools.

Read results

Explore Matrix metrics and run comparisons.