> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oqoqo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Measure a tool's impact

> Find out whether your skill, MCP server, CLI, or SDK helps agents, whether the gain holds up, and what still blocks them.

Run the same tasks with and without your tool, then compare pass rates on **Matrix** and **Comparisons** to measure your tool's [lift](https://docs.oqoqo.ai/experiments/overview.md#baseline-and-lift).

```mermaid theme={null}
flowchart LR
  T["1. Write > 6 tasks"] --> A["2. Add tool &<br/>treatment"] --> L["3. Launch vs<br/>Raw Agent"] --> C["4. Check Matrix &<br/>Comparisons"] --> F["5. Fix frictions &<br/>relaunch"]
```

## Prerequisites

* Connected [model providers](https://docs.oqoqo.ai/setup/model-providers.md), [Oqoqo AI feature models](https://docs.oqoqo.ai/setup/oqoqo-ai.md), and at least one enabled [agent](https://docs.oqoqo.ai/setup/agents.md) under **Manage → Agents**.
* Sufficient runs in your [run balance](https://docs.oqoqo.ai/setup/billing.md) (`tasks × treatments × agents × trials`).
* The skill, MCP server, CLI, or SDK you want to evaluate.

## Steps

**1. Write tasks that test real product work**

Under **Manage → Tasks**, create realistic [tasks](https://docs.oqoqo.ai/experiments/tasks.md) with rubrics, [files, and machines](https://docs.oqoqo.ai/experiments/machines-and-files.md). Avoid naming your tool in task instructions so you measure whether agents discover it naturally. Include more than 6 tasks—[Comparisons](https://docs.oqoqo.ai/evaluation/comparisons.md) needs at least 6 differing tasks (ties do not count).

**2. Add your tool**

Add the [tool](https://docs.oqoqo.ai/experiments/tools.md) under **Manage → Tools**, and enter any required keys or OAuth sign-ins in the dashboard.

**3. Create a treatment for your tool**

Under **Manage → Treatments (Toolsets)**, create a [treatment](https://docs.oqoqo.ai/experiments/treatments.md) containing your tool. Every project already includes **Raw Agent** (no added tools) as the standard baseline.

**4. Launch the experiment**

Open **New experiment**, select your tasks, both treatments, and your target agents (**Clear** any extra pre-selected agents so you only spend runs on the ones you want to measure). Set **Baseline** to **Raw Agent**, set **Trials** to `2` or more, lower **Max Concurrency** to `1` if tasks modify shared remote state, and [launch](https://docs.oqoqo.ai/experiments/launch.md).

**5. Compare pass rates and tool usage on Matrix**

On **Matrix** with **Evals** selected, compare your treatment's cells against **Raw Agent** for the same task and agent (for example, 1 of 4 trials passed vs. 3 of 4 is a lift of **+50 percentage points**). Switch **Matrix metric** to **Treatment use** to confirm agents actually called your tool (a lower bound that excludes script calls and stays blank for SDKs).

**6. Verify whether the difference holds up**

Expand **Comparisons** below the **Matrix** grid. A **Clear difference** is settled enough to act on; for a **Likely difference**, select **Run again to be sure · N runs** to launch a focused follow-up. Read the **Brief** inside any difference block to see frictions observed with the worse side, and expand **All N comparisons** for per-agent results ([Comparisons](https://docs.oqoqo.ai/evaluation/comparisons.md#answers)).

**7. Fix remaining product frictions**

On **Frictions**, filter **Where to fix** to **Product**, fix the highest-impact problems in your tool, API, or docs ([Find and fix frictions](https://docs.oqoqo.ai/guides/fix-frictions.md)), then select **Launch again**.

## Interpret the results

| Pattern | Meaning and next action |
| - | - |
| Positive **Clear difference** and high **Treatment use** | Your tool helps agents complete tasks reliably. |
| Lift near zero and low **Treatment use** | Agents failed to discover or trigger the tool. Improve skill descriptions, MCP tool names, or CLI `--help` text. |
| Lift near zero and high **Treatment use** | Agents called the tool, but it did not solve their problems. Read product frictions and failed [traces](https://docs.oqoqo.ai/evaluation/traces.md). |
| Negative **Clear difference** | The tool hurt pass rates—check frictions for broken schemas, confusing errors, or misleading instructions. |
| **No clear difference** | Current runs cannot separate the two treatments (not proof of zero effect). Add tasks and trials and relaunch. |
| **Machine** or **Task quality** frictions | Fix setup or rubric issues first before trusting pass rates. |

## From the CLI and the plugin

| Step | CLI | Plugin (MCP) |
| - | - | - |
| Browse tasks, tools, and treatments | [`oqoqo catalog list`](https://docs.oqoqo.ai/cli/catalog#oqoqo-catalog-list) | [`inspect_catalog`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#inspect_catalog) |
| Create a tool or treatment | [`oqoqo resources apply`](https://docs.oqoqo.ai/cli/catalog#oqoqo-resources-apply) | [`author_resource`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#author_resource) |
| Preflight and launch | [`experiments preflight`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-preflight), [`launch`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-launch) | [`preflight_experiment`](https://docs.oqoqo.ai/mcp-server/experiments#preflight_experiment), [`launch_experiment`](https://docs.oqoqo.ai/mcp-server/experiments#launch_experiment) |
| Read the Brief | [`oqoqo experiments brief`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-brief) | [`get_experiment_brief`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_experiment_brief) |
| Compare treatments and check significance | [`results analyze compare_treatments`](https://docs.oqoqo.ai/cli/runs#oqoqo-results-analyze), [`evals compare`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-compare) | [`analyze_results`](https://docs.oqoqo.ai/mcp-server/runs-and-results#analyze_results), [`get_evals_comparisons`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_evals_comparisons) |
| List tool experiments and frictions | [`experiments list --tool-id`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-list), [`frictions list --tool-id`](https://docs.oqoqo.ai/cli/frictions#oqoqo-frictions-list) | [`list_experiments`](https://docs.oqoqo.ai/mcp-server/experiments#list_experiments), [`list_frictions`](https://docs.oqoqo.ai/mcp-server/frictions#list_frictions) |

Example prompt for your AI assistant:

```text theme={null}
Launch my checkout tasks with the Raw Agent and Orders MCP treatments on Claude Code, 3 trials each. When it finishes, read the Brief, tell me which differences from Raw Agent hold up, and list the product frictions.
```

## Other ways to run it

* **Start from a template:** try **Skill lift** or **Stripe interface evals** under **Home → Templates** ([Templates](https://docs.oqoqo.ai/experiments/templates.md)).
* **Compare versions or interfaces:** put v1 vs. v2—or MCP vs. CLI vs. SDK—into separate treatments, or combine tools (such as CLI + skill) in one treatment.
* **Test across multiple agents:** include several agents to see which harnesses benefit most ([Compare agents and models](https://docs.oqoqo.ai/guides/compare-agents-and-models.md)).

## Next steps

> [card] [Find and fix frictions](https://docs.oqoqo.ai/guides/fix-frictions.md)
>
> Fix top product blockers and verify the improvement.

> [card] [Track results over time](https://docs.oqoqo.ai/guides/track-agents-over-time.md)
>
> Rerun on a schedule after every release.

> [card] [Treatments (Toolsets)](https://docs.oqoqo.ai/experiments/treatments.md)
>
> Organize tools and disable individual MCP tools.

> [card] [Read results](https://docs.oqoqo.ai/evaluation/results.md)
>
> Explore Matrix metrics and run comparisons.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.