Skip to main content
Because agents vary from run to run, a raw pass-rate gap on the Matrix can be misleading. Open an experiment’s Matrix and expand Comparisons below the grid, or select the link in the line above the grid, to see which differences in Evals pass rates hold up across tasks and what to do next.
A difference requires at least 6 tasks where one side outperformed the other. With fewer differing tasks, no difference can appear regardless of how many trials you run.

Answers

Comparisons opens with a one-sentence headline answering your experiment’s main question, followed by one block per standout difference (up to 3 initially, with Show all N differences expanding up to 5 total Brief findings or any remaining blocks).
Stripe CLI did better than Raw Agent on all 7 tasks, then its block: 93% and 0%, Clear difference, then the Brief and two frictions.Stripe CLI did better than Raw Agent on all 7 tasks, then its block: 93% and 0%, Clear difference, then the Brief and two frictions.
The headline covers your main comparisons: each treatment versus the baseline across all agents (or between pairs of agents when testing a single treatment):
  • “Stripe CLI did better than Raw Agent on all 7 tasks.”
  • “Stripe MCP likely did better than Raw Agent on 6 tasks, same on 4.” (Likely differences are never stated as settled.)
  • “No treatment did clearly better than All official CLIs across all agents.”
Below the headline, a note counts any likely breakdown differences (for example, “7 other comparisons show a likely difference. See All 40 comparisons.”). Each difference block shows both sides’ pass rates, task-by-task counts (“Better on 14 tasks, worse on 2, same on 26.”), the verdict and next step, and per-agent breakdowns (See each agent). Comparisons evaluate the latest Evals attempt by default, or any earlier attempt selected in the attempt menu. Comparisons is a collapsed section below the Matrix grid. The line above the grid shows the headline, and its link expands the section and scrolls to it.

Brief

When Insights has reviewed the runs, up to 5 difference blocks include a Brief (the fourth and fifth appear after selecting Show all N differences). Each Brief pairs a one-sentence explanation from your Experiment insights model with up to 3 frictions observed on the tasks where that side did worse, showing counts for both sides (for example, “Raw Agent 24 of 28 failed runs”, “Stripe CLI 0 of 26 passed runs”). Select any friction to open its suggested fix. Frictions listed in a Brief were observed alongside the side that did worse—that does not prove they caused the difference.
  • After a fix: on a Clear difference that includes a product friction, After a fix, run the N tasks that differed again · N runs launches a focused follow-up experiment on just those differing tasks to verify your fix.
  • What the runs did differently: with Insights in Agent mode (private beta), each Brief ends with paired examples comparing one failed run on the worse side against one passed run on the better side, highlighting the step where their paths diverged.
  • Selection: Oqoqo counts runs with completed Insights reviews (excluding Oqoqo error runs) and lists a friction when it appears in at least 3 runs of the worse side across at least 2 tasks at a higher rate than the better side.
  • Accuracy: Oqoqo computes all counts deterministically; the Experiment insights model writes only that one-sentence explanation (without custom instructions, rejecting any response that invents numbers or claims causality).
  • Updates: the Brief regenerates whenever Evals or Insights finish (and once more in Insights Agent mode once all run pairs are analyzed).

Result labels

All comparisons

All N comparisons appears below the difference blocks and lists every comparison across four groups:
All 6 comparisons: Stripe CLI and Raw Agent across all agents is a Clear difference. With each agent on its own, it's a Likely difference with Run again to be sure · 42 runs. The two agents show no clear difference with either treatment.All 6 comparisons: Stripe CLI and Raw Agent across all agents is a Clear difference. With each agent on its own, it's a Likely difference with Run again to be sure · 42 runs. The two agents show no clear difference with either treatment.
Rows are grouped into Treatments across all agents (or Treatments for a single agent), Treatments with each agent, Agents with each treatment (or Agents for a single treatment; omitted above 6 agents), and Does a treatment help one agent more?.

Run a comparison again to be sure

For a Likely difference, Oqoqo can plan a targeted follow-up experiment using the fewest extra trials (3 to 10 per combination) needed to settle the gap. This button appears on the latest Evals attempt of finished experiments whenever at least 6 tasks can still swing and 10 or fewer trials per combination suffice; otherwise it displays Needs more tasks to be sure.
Select Run again to be sure · N runs on the block or table row. Review the pre-filled New experiment run count and select Launch.If the comparison can no longer be run again, for example because Evals ran again or a treatment was deleted, the Matrix opens with Comparisons expanded and a note that explains why.
The follow-up experiment isolates only the two sides and agents from that comparison, so a settled main or breakdown comparison becomes a Clear difference if the gap holds (interaction comparisons can also settle cleanly when they are the sole test, or remain breakdowns when tested within a multi-comparison launch).
  • Tasks: tasks from the original comparison that can still differ (minimum 6; tasks with at least 4 total runs that already passed or failed 100% on both sides are excluded).
  • Sides: the comparison’s treatments and agents (all agents for cross-agent comparisons).
  • Trials & settings: the minimum trials per combination (3 to 10), current treatment tools, original task versions, and the same Evals model and mode, named "Run again: …".

Empty states

Public experiment links do not include Comparisons.

How results are decided

Expand How results are decided below All N comparisons to inspect the exact rules and constraints for your experiment.
How results are decided, open: five lines on how two sides are compared and what each result means, then what applies to this experiment, such as the main comparisons and breakdowns, and that with 7 tasks a result for one agent alone can be a likely difference at most.How results are decided, open: five lines on how two sides are compared and what each result means, then what applies to this experiment, such as the main comparisons and breakdowns, and that with 7 tasks a result for one agent alone can be a likely difference at most.
Oqoqo compares outcomes task by task, never combining runs across tasks. For each comparison, it counts how many tasks favored each side (ignoring ties) and runs an exact two-sided sign test. Five unanimous tasks give p = 0.063, whereas six give p = 0.031—which is why at least 6 differing tasks are required. Both sides compare the same number of scored runs per task and weight every agent equally, so Runs passed can be lower than total runs on Matrix.

Main comparisons and breakdowns

Testing many comparisons increases false positives, so Oqoqo applies the Holm correction separately within two groups:
  • Clear difference: adjusted p < 0.05 after correction within its group.
  • Likely difference: unadjusted p < 0.05, but >= 0.05 after correction.
  • No clear difference: unadjusted p >= 0.05.
Worked example: An experiment with 2 treatments, 2 agents, and 7 tasks has 1 main comparison and 5 breakdowns. If Stripe CLI beats Raw Agent on all 7 tasks for both agents (p = 0.016), the main comparison (corrected across 1 test) is a Clear difference (0.016 < 0.05), while each individual agent’s breakdown (corrected across 5 tests) is a Likely difference (5 × 0.016 = 0.078 >= 0.05).

From the CLI and the plugin

Next steps

Find and fix frictions

Fix the frictions flagged in the Brief and retest.

Measure a tool's impact

Verify that your tool’s lift holds up.

Compare agents and models

Compare agents and cost on settled differences.

Frictions

Explore every friction and its evidence.