> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oqoqo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Compare agents and models

> Run the same tasks across agents, models, and effort levels, and read pass rate next to cost, tokens, and duration to pick the best fit.

Evaluate multiple agents, models, and effort levels inside a single experiment so every candidate runs the exact same tasks, files, machines, and tools.

```mermaid theme={null}
flowchart LR
  T["Pick > 6 tasks &<br/>1 treatment"] --> A["Select agents,<br/>models & effort"] --> M["Compare Evals vs<br/>Cost & Duration"] --> C["Verify Clear difference<br/>on Comparisons"]
```

## Prerequisites

* Target agents turned on under **Manage → Agents** with compatible [model providers](https://docs.oqoqo.ai/setup/model-providers.md) connected ([Agents](https://docs.oqoqo.ai/setup/agents.md)).
* Representative [tasks](https://docs.oqoqo.ai/experiments/tasks.md) for your workload.
* Runs in your [run balance](https://docs.oqoqo.ai/setup/billing.md) (model inference is billed separately by your providers).

## Steps

**1. Choose tasks**

Select tasks spanning simple to complex scenarios—more than 6 tasks so [Comparisons](https://docs.oqoqo.ai/evaluation/comparisons.md) has at least 6 differing tasks to separate agents cleanly.

**2. Hold the treatment constant**

Pick **Raw Agent** to benchmark agents natively, or one shared [treatment](https://docs.oqoqo.ai/experiments/treatments.md) to test how well each agent uses your product's tools.

**3. Select agents, models, and effort levels**

In **New experiment**, select each agent, model, and effort level (add an agent multiple times to compare different models or effort levels on it). **Default** sends no effort level, letting the agent use its native default. Set **Trials** to `2` or more and [launch](https://docs.oqoqo.ai/experiments/launch.md).

**4. Weigh pass rate against cost and speed**

Open **Matrix** on **Evals** to compare pass rates, then switch **Matrix metric** to **Cost**, **Tokens**, **Duration**, **Steps**, or **Tool calls** to inspect efficiency.

**5. Confirm differences on Comparisons**

Expand **Comparisons** below the **Matrix** grid to verify which gaps hold up across tasks (check **All N comparisons** under **Agents with** when testing multiple treatments). For a **Likely difference**, select **Run again to be sure · N runs** before committing to a higher-cost model.

**6. Inspect agent-specific frictions**

On [**Frictions**](https://docs.oqoqo.ai/evaluation/frictions.md), filter **Where to fix** to **Agent behavior** and check **More often with** / **Rate by agent** to see why one agent stumbled where another passed.

## Interpret the results

| Observation | Action |
| - | - |
| Higher pass rate at higher cost | Upgrade only for a **Clear difference**; run **Likely difference** comparisons again first. |
| Similar pass rates, lower cost or duration | Pick the faster or cheaper agent/model. |
| One agent fails while others pass | Inspect its **Agent behavior** frictions and [**Where it happens most**](https://docs.oqoqo.ai/evaluation/frictions.md#where-it-happens-most). |
| All agents fail the same task | Check **Machine** and **Task quality** frictions before judging the agents. |

## From the CLI and the plugin

| Step | CLI | Plugin (MCP) |
| - | - | - |
| List agents and models | [`oqoqo catalog list --kind agent`](https://docs.oqoqo.ai/cli/catalog#oqoqo-catalog-list) | [`inspect_catalog`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#inspect_catalog) |
| Launch experiment | [`oqoqo experiments launch`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-launch) | [`launch_experiment`](https://docs.oqoqo.ai/mcp-server/experiments#launch_experiment) |
| Read Brief and agent comparisons | [`experiments brief`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-brief), [`evals compare --kind agent`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-compare) | [`get_experiment_brief`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_experiment_brief), [`get_evals_comparisons`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_evals_comparisons) |
| Summarize metrics by agent | [`oqoqo results analyze summarize_scope`](https://docs.oqoqo.ai/cli/runs#oqoqo-results-analyze) | [`analyze_results`](https://docs.oqoqo.ai/mcp-server/runs-and-results#analyze_results) |
| Inspect agent frictions | [`frictions list --fix-in agent_behavior`](https://docs.oqoqo.ai/cli/frictions#oqoqo-frictions-list), [`frictions get`](https://docs.oqoqo.ai/cli/frictions#oqoqo-frictions-get) | [`list_frictions`](https://docs.oqoqo.ai/mcp-server/frictions#list_frictions), [`get_friction`](https://docs.oqoqo.ai/mcp-server/frictions#get_friction) |

Example CLI check for agent-specific frictions:

```bash theme={null}
oqoqo frictions list --experiment-id exp_8f2c41d0 --fix-in agent_behavior
oqoqo frictions get frc_7d20e6b4 --experiment-id exp_8f2c41d0
```

## Other ways to run it

* **Start fast:** run the **Latest model shootout** [template](https://docs.oqoqo.ai/experiments/templates.md) on **Home** across all connected agents.
* **Test reasoning effort:** run one model across multiple effort levels to check whether extra reasoning justifies higher cost.
* **Benchmark custom agents:** include your ACP [custom agent](https://docs.oqoqo.ai/setup/custom-agents.md) (private beta) alongside built-in agents.

## Next steps

> [card] [Measure a tool's impact](https://docs.oqoqo.ai/guides/measure-tool-lift.md)
>
> See which tools boost each agent's pass rate.

> [card] [Track results over time](https://docs.oqoqo.ai/guides/track-agents-over-time.md)
>
> Relaunch on a schedule when new models ship.

> [card] [Agents](https://docs.oqoqo.ai/setup/agents.md)
>
> Supported built-in agents and provider rules.

> [card] [Frictions](https://docs.oqoqo.ai/evaluation/frictions.md)
>
> Diagnose agent-specific failures.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.