> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oqoqo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Comparisons

> See who did better, how sure the result is, and what to do next.

Because agents vary from run to run, a raw pass-rate gap on the **Matrix** can be misleading. Open an experiment's **Matrix** and expand **Comparisons** below the grid, or select the link in the line above the grid, to see which differences in [Evals](https://docs.oqoqo.ai/evaluation/evals.md) pass rates hold up across tasks and what to do next.

> [!NOTE]
>
> A difference requires at least **6 tasks** where one side outperformed the other. With fewer differing tasks, no difference can appear regardless of how many trials you run.

## Answers

**Comparisons** opens with a one-sentence headline answering your experiment's main question, followed by one block per standout difference (up to 3 initially, with **Show all N differences** expanding up to 5 total Brief findings or any remaining blocks).

![Stripe CLI did better than Raw Agent on all 7 tasks, then its block: 93% and 0%, Clear difference, then the Brief and two frictions.](https://mintcdn.com/oqoqo/ASKPbBxA4zN-Nu4d/images/screenshots/comparisons-light.png?fit=max\&auto=format\&n=ASKPbBxA4zN-Nu4d\&q=85\&s=0db668dd214dbf689c65c2cf5f02a5ba)

The headline covers your **main comparisons**: each treatment versus the [baseline](https://docs.oqoqo.ai/experiments/overview.md#baseline-and-lift) across all agents (or between pairs of agents when testing a single treatment):

* *"Stripe CLI did better than Raw Agent on all 7 tasks."*
* *"Stripe MCP likely did better than Raw Agent on 6 tasks, same on 4."* (Likely differences are never stated as settled.)
* *"No treatment did clearly better than All official CLIs across all agents."*

Below the headline, a note counts any likely breakdown differences (for example, *"7 other comparisons show a likely difference. See All 40 comparisons."*). Each difference block shows both sides' pass rates, task-by-task counts (*"Better on 14 tasks, worse on 2, same on 26."*), the verdict and next step, and per-agent breakdowns (**See each agent**).

Comparisons evaluate the latest Evals attempt by default, or any earlier attempt selected in the attempt menu. **Comparisons** is a collapsed section below the [Matrix](https://docs.oqoqo.ai/evaluation/results.md#matrix) grid. The line above the grid shows the headline, and its link expands the section and scrolls to it.

### Brief

When [Insights](https://docs.oqoqo.ai/evaluation/insights.md) has reviewed the runs, up to 5 difference blocks include a **Brief** (the fourth and fifth appear after selecting **Show all N differences**). Each Brief pairs a one-sentence explanation from your **Experiment insights model** with up to 3 frictions observed on the tasks where that side did worse, showing counts for both sides (for example, *"Raw Agent 24 of 28 failed runs"*, *"Stripe CLI 0 of 26 passed runs"*). Select any friction to open its suggested fix.

Frictions listed in a Brief were observed alongside the side that did worse—that does not prove they caused the difference.

* **After a fix:** on a **Clear difference** that includes a product friction, **After a fix, run the N tasks that differed again · N runs** launches a focused follow-up experiment on just those differing tasks to verify your fix.
* **[What the runs did differently](https://docs.oqoqo.ai/evaluation/agent-mode.md#what-the-runs-did-differently):** with Insights in Agent mode (private beta), each Brief ends with paired examples comparing one failed run on the worse side against one passed run on the better side, highlighting the step where their paths diverged.

| Brief state | What to do |
| - | - |
| No Brief shown | [Run Insights](https://docs.oqoqo.ai/evaluation/insights.md#turn-on-insights) if Insights were off, or wait if Evals/Insights are still running. |
| **Brief** with *"Being written."* | Wait; the sentence loads automatically once complete. |
| *"The Brief could not be written."* | Listed frictions still appear; verify your **Experiment insights model** under **Oqoqo AI defaults** if prompted. The Brief updates automatically the next time Evals or Insights finish. |

> [accordion] How Oqoqo builds the Brief
>
> * **Selection:** Oqoqo counts runs with completed Insights reviews (excluding Oqoqo error runs) and lists a friction when it appears in at least 3 runs of the worse side across at least 2 tasks at a higher rate than the better side.
> * **Accuracy:** Oqoqo computes all counts deterministically; the **Experiment insights model** writes only that one-sentence explanation (without custom instructions, rejecting any response that invents numbers or claims causality).
> * **Updates:** the Brief regenerates whenever Evals or Insights finish (and once more in Insights Agent mode once all run pairs are analyzed).

## Result labels

```mermaid theme={null}
flowchart TD
  S["Compare tasks"] --> T{"At least 6 tasks differ?"}
  T -- "No" --> N["No clear difference"]
  T -- "Yes" --> P{"Passes family correction?"}
  P -- "Yes (adjusted p < 0.05)" --> C["Clear difference"]
  P -- "No, but unadjusted p < 0.05" --> L{"Can extra trials settle it?"}
  P -- "No (p >= 0.05)" --> N
  L -- "Yes (<= 10 trials)" --> R["Likely difference · Run again to be sure"]
  L -- "No" --> M["Likely difference · Needs more tasks to be sure"]
```

| Result | Meaning | Recommended action |
| - | - | - |
| **Clear difference** | One side outperformed the other on enough tasks that the difference holds up. | Act on the result. |
| **Likely difference · Run again to be sure · N runs** | One side is ahead, and extra trials should settle whether it holds up. | Select **Run again to be sure · N runs**. |
| **Likely difference · Needs more tasks to be sure** | One side is ahead, but repeating existing tasks cannot settle it. | Add more tasks and launch a new experiment. |
| **No clear difference** | Neither side is reliably ahead here (this does not prove they perform identically). | Add tasks if the gap matters. |
| **No clear difference** (too few tasks) | Fewer than 6 tasks had a difference between sides. | Add tasks and launch again. |

## All comparisons

**All N comparisons** appears below the difference blocks and lists every comparison across four groups:

| Column | What it shows |
| - | - |
| **Compared** | The two treatments or agents being compared |
| **Runs passed** | Each side's pass rate and passing run fraction |
| **Task by task** | Task win/tie counts (for example, *"Stripe CLI better on 6, same on 1"*) |
| **Result** | The verdict label and next step |

![All 6 comparisons: Stripe CLI and Raw Agent across all agents is a Clear difference. With each agent on its own, it's a Likely difference with Run again to be sure · 42 runs. The two agents show no clear difference with either treatment.](https://mintcdn.com/oqoqo/ASKPbBxA4zN-Nu4d/images/screenshots/comparisons-all-light.png?fit=max\&auto=format\&n=ASKPbBxA4zN-Nu4d\&q=85\&s=6b99a35a9a9b9f6a66979233b490ee2e)

Rows are grouped into **Treatments across all agents** (or **Treatments** for a single agent), **Treatments with** each agent, **Agents with** each treatment (or **Agents** for a single treatment; omitted above 6 agents), and **Does a treatment help one agent more?**.

## Run a comparison again to be sure

For a **Likely difference**, Oqoqo can plan a targeted follow-up experiment using the fewest extra trials (**3 to 10** per combination) needed to settle the gap. This button appears on the latest Evals attempt of finished experiments whenever at least 6 tasks can still swing and 10 or fewer trials per combination suffice; otherwise it displays **Needs more tasks to be sure**.

> [tab] Dashboard
>
> Select **Run again to be sure · N runs** on the block or table row. Review the pre-filled **New experiment** run count and select **Launch**.
>
> If the comparison can no longer be run again, for example because Evals ran again or a treatment was deleted, the **Matrix** opens with **Comparisons** expanded and a note that explains why.

> [tab] CLI
>
> Find likely comparisons, clone the targeted plan, run preflight to populate current revisions, and launch:
>
> ```bash theme={null}
> oqoqo evals compare exp_8f2c41d0 --label likely
> oqoqo experiments clone-plan exp_8f2c41d0 \
>   --comparison cmp_4a0e2c9d --output json \
>   | jq '.data.plan' > run-again.json
> oqoqo experiments preflight --plan-file ./run-again.json --output json \
>   | jq '.data.plan' > ready-plan.json
> oqoqo experiments launch --plan-file ./ready-plan.json
> ```
>
> See [`oqoqo evals compare`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-compare) and [`oqoqo experiments clone-plan`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-clone-plan).

> [tab] Plugin (MCP)
>
> Ask your AI assistant to check and plan the follow-up:
>
> ```text theme={null}
> Did MCP + Skills do better than Raw Agent in my last experiment? If the result is likely, plan the runs that check it again and tell me the run count.
> ```
>
> It calls [`get_evals_comparisons`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_evals_comparisons), [`get_experiment_plan`](https://docs.oqoqo.ai/mcp-server/experiments#get_experiment_plan) (`comparison_id`), [`preflight_experiment`](https://docs.oqoqo.ai/mcp-server/experiments#preflight_experiment), and [`launch_experiment`](https://docs.oqoqo.ai/mcp-server/experiments#launch_experiment). Keep tool approval enabled so it asks before spending runs.

The follow-up experiment isolates only the two sides and agents from that comparison, so a settled main or breakdown comparison becomes a **Clear difference** if the gap holds (interaction comparisons can also settle cleanly when they are the sole test, or remain breakdowns when tested within a multi-comparison launch).

> [accordion] What the follow-up experiment includes
>
> * **Tasks:** tasks from the original comparison that can still differ (minimum 6; tasks with at least 4 total runs that already passed or failed 100% on both sides are excluded).
> * **Sides:** the comparison's treatments and agents (all agents for cross-agent comparisons).
> * **Trials & settings:** the minimum trials per combination (3 to 10), current treatment tools, original task versions, and the same Evals model and mode, named `"Run again: …"`.

## Empty states

| Message | What to do |
| - | - |
| **Nothing to compare** | Launch with at least two treatments or two agents. |
| **Too many agents to compare** | Single-treatment comparisons support at most 6 agents; launch with fewer. |
| **Evals off** | [Run Evals](https://docs.oqoqo.ai/evaluation/evals.md#run-evals-again) on the finished runs. |
| **No evaluated runs yet** / **No evaluated runs to compare** | Wait for Evals to finish, or rerun Evals if both sides lack verdicts. |
| **Too large to compare** | Experiment exceeds 10,000 Matrix cells; read pass rates directly on **Matrix**. |

[Public experiment links](https://docs.oqoqo.ai/evaluation/share.md) do not include **Comparisons**.

## How results are decided

Expand **How results are decided** below **All N comparisons** to inspect the exact rules and constraints for your experiment.

![How results are decided, open: five lines on how two sides are compared and what each result means, then what applies to this experiment, such as the main comparisons and breakdowns, and that with 7 tasks a result for one agent alone can be a likely difference at most.](https://mintcdn.com/oqoqo/ASKPbBxA4zN-Nu4d/images/screenshots/comparisons-decided-light.png?fit=max\&auto=format\&n=ASKPbBxA4zN-Nu4d\&q=85\&s=7ea4c51536ea43ea969881682959c7ba)

Oqoqo compares outcomes **task by task**, never combining runs across tasks. For each comparison, it counts how many tasks favored each side (ignoring ties) and runs an exact two-sided sign test. Five unanimous tasks give `p = 0.063`, whereas six give `p = 0.031`—which is why at least 6 differing tasks are required. Both sides compare the same number of scored runs per task and weight every agent equally, so **Runs passed** can be lower than total runs on **Matrix**.

### Main comparisons and breakdowns

Testing many comparisons increases false positives, so Oqoqo applies the Holm correction separately within two groups:

| Group | What it includes |
| - | - |
| **Main comparisons** | The primary comparisons answering the experiment's question ([Answers](#answers)). |
| **Breakdowns** | Per-agent treatment comparisons, agent-vs-agent comparisons alongside varying treatments, and treatment-by-agent interactions. Corrected more strictly because there are more of them. |

* **Clear difference:** adjusted `p < 0.05` after correction within its group.
* **Likely difference:** unadjusted `p < 0.05`, but `>= 0.05` after correction.
* **No clear difference:** unadjusted `p >= 0.05`.

**Worked example:** An experiment with 2 treatments, 2 agents, and 7 tasks has 1 main comparison and 5 breakdowns. If **Stripe CLI** beats **Raw Agent** on all 7 tasks for both agents (`p = 0.016`), the main comparison (corrected across 1 test) is a **Clear difference** (`0.016 < 0.05`), while each individual agent's breakdown (corrected across 5 tests) is a **Likely difference** (`5 × 0.016 = 0.078 >= 0.05`).

## From the CLI and the plugin

| Task | CLI | Plugin (MCP) |
| - | - | - |
| Read comparisons and verdicts | [`oqoqo evals compare`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-compare) | [`get_evals_comparisons`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_evals_comparisons) |
| Read the Brief | [`oqoqo experiments brief`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-brief) | [`get_experiment_brief`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_experiment_brief) |
| Plan a follow-up experiment | [`oqoqo experiments clone-plan`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-clone-plan) | [`get_experiment_plan`](https://docs.oqoqo.ai/mcp-server/experiments#get_experiment_plan) |

## Next steps

> [card] [Find and fix frictions](https://docs.oqoqo.ai/guides/fix-frictions.md)
>
> Fix the frictions flagged in the Brief and retest.

> [card] [Measure a tool's impact](https://docs.oqoqo.ai/guides/measure-tool-lift.md)
>
> Verify that your tool's lift holds up.

> [card] [Compare agents and models](https://docs.oqoqo.ai/guides/compare-agents-and-models.md)
>
> Compare agents and cost on settled differences.

> [card] [Frictions](https://docs.oqoqo.ai/evaluation/frictions.md)
>
> Explore every friction and its evidence.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.