Prerequisites
- Connected model providers, Oqoqo AI feature models, and at least one enabled agent under Manage → Agents.
- Sufficient runs in your run balance (
tasks × treatments × agents × trials). - The skill, MCP server, CLI, or SDK you want to evaluate.
Steps
1
Write tasks that test real product work
Under Manage → Tasks, create realistic tasks with rubrics, files, and machines. Avoid naming your tool in task instructions so you measure whether agents discover it naturally. Include more than 6 tasks—Comparisons needs at least 6 differing tasks (ties do not count).
2
Add your tool
Add the tool under Manage → Tools, and enter any required keys or OAuth sign-ins in the dashboard.
3
Create a treatment for your tool
Under Manage → Treatments (Toolsets), create a treatment containing your tool. Every project already includes Raw Agent (no added tools) as the standard baseline.
4
Launch the experiment
Open New experiment, select your tasks, both treatments, and your target agents (Clear any extra pre-selected agents so you only spend runs on the ones you want to measure). Set Baseline to Raw Agent, set Trials to
2 or more, lower Max Concurrency to 1 if tasks modify shared remote state, and launch.5
Compare pass rates and tool usage on Matrix
On Matrix with Evals selected, compare your treatment’s cells against Raw Agent for the same task and agent (for example, 1 of 4 trials passed vs. 3 of 4 is a lift of +50 percentage points). Switch Matrix metric to Treatment use to confirm agents actually called your tool (a lower bound that excludes script calls and stays blank for SDKs).
6
Verify whether the difference holds up
Expand Comparisons below the Matrix grid. A Clear difference is settled enough to act on; for a Likely difference, select Run again to be sure · N runs to launch a focused follow-up. Read the Brief inside any difference block to see frictions observed with the worse side, and expand All N comparisons for per-agent results (Comparisons).
7
Fix remaining product frictions
On Frictions, filter Where to fix to Product, fix the highest-impact problems in your tool, API, or docs (Find and fix frictions), then select Launch again.
Interpret the results
From the CLI and the plugin
Example prompt for your AI assistant:
Other ways to run it
- Start from a template: try Skill lift or Stripe interface evals under Home → Templates (Templates).
- Compare versions or interfaces: put v1 vs. v2—or MCP vs. CLI vs. SDK—into separate treatments, or combine tools (such as CLI + skill) in one treatment.
- Test across multiple agents: include several agents to see which harnesses benefit most (Compare agents and models).
Next steps
Find and fix frictions
Fix top product blockers and verify the improvement.
Track results over time
Rerun on a schedule after every release.
Treatments (Toolsets)
Organize tools and disable individual MCP tools.
Read results
Explore Matrix metrics and run comparisons.

