> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oqoqo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> See each run's pass or fail verdicts, choose a scoring mode, check what Evals read, and score runs again after a rubric change.

**Evals** grade each finished run against the task's rubric. Your **Evals model** marks every criterion **Pass** or **Fail** with an explanation, and those verdicts determine pass rates on **Matrix** and **Comparisons**.

While Evals judge the outcome, [Insights](https://docs.oqoqo.ai/evaluation/insights.md) analyze how the agent worked to find [frictions](https://docs.oqoqo.ai/evaluation/frictions.md). Both run independently and can be rerun separately.

## Turn on Evals

**1. Connect a model provider**

Add a [model provider](https://docs.oqoqo.ai/setup/model-providers.md) under **Organization settings → Model providers** or **Profile → Integrations**.

**2. Assign an Evals model**

Set your **Evals model** under [**Organization settings → Oqoqo AI defaults**](https://docs.oqoqo.ai/setup/oqoqo-ai.md).

**3. Enable Evals on an experiment**

Keep **Evals** toggled on when you [launch](https://docs.oqoqo.ai/experiments/launch.md), or select **Run evals** on a finished experiment.

## Read the verdicts

Open any run and select **Evals** to see its overall verdict, passed criterion count, and the **Evals model** that graded it. Expand any criterion to read the model's reasoning.

![Run Evals tab with a pass or fail verdict for each rubric criterion and the Evals model that scored it.](https://mintcdn.com/oqoqo/YO1ysyUx4SPwBwoz/images/screenshots/run-evals-light.png?fit=max\&auto=format\&n=YO1ysyUx4SPwBwoz\&q=85\&s=0b3b804df59d60855c5f914b3fb31f96)

Because an AI model grades the rubric, write [criteria](https://docs.oqoqo.ai/experiments/tasks.md#write-the-rubric) so each one is verifiable from a single run's trace and workspace changes.

## Choose a mode

By default, Evals run in **Standard** mode (one low-cost pass per run). If your organization has enabled [**Agent mode**](https://docs.oqoqo.ai/evaluation/agent-mode.md) (private beta), you can switch to **Agent** at launch or when rerunning so Oqoqo's evals agent inspects full runs and verifies claims before grading. See [Agent mode](https://docs.oqoqo.ai/evaluation/agent-mode.md#standard-and-agent-modes) for cost, speed, and model requirements.

## What Evals read

Your **Evals model** inspects the task rubric, recorded trajectory steps, and changed workspace files. When a run exceeds the model's context window in Standard mode, a coverage note on the **Evals** tab (and in **Compare**) reports what was condensed or omitted—such as summarized step ranges, shortened long steps, truncated changed files, or traces exceeding the 32 MB storage limit.

No coverage note means the model read the entire run (older runs created before coverage tracking show no note regardless). If a run still cannot fit after condensing, Evals fail for that run without creating a friction—select an **Evals model** with a larger context window and rerun Evals.

## Which models ran

Each Evals attempt records the exact provider, model, and effort it used (such as **OpenAI · GPT-5.4 · high**), so later changes under **Oqoqo AI defaults** never alter historical attributions. You can see the model on each run's **Evals** tab, the attempt selector on **Matrix**, the experiment info popover, public experiment links, CLI and plugin reads, and the `evals` dataset in `oqoqo data query`. Older attempts predating model tracking display **Not recorded**.

## Pending verdicts

Evals grade each run as soon as it finishes; **Rerun evals** clears old verdicts and displays **Evals pending** until new scores arrive. In the CLI and plugin, [`oqoqo evals get`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-get) reports per-run status (`queued`, `running`, `completed`, `skipped`, `failed`) and [`oqoqo results analyze`](https://docs.oqoqo.ai/cli/runs#oqoqo-results-analyze) tracks `pendingRuns` separately from inconclusive runs.

## Run Evals again

Rerunning Evals regrades existing recorded work without re-executing agents—ideal after editing a task's rubric or switching to a stronger Evals model. (If you changed the task prompt, files, or machine, launch a new experiment instead.) When Evals finish, Oqoqo automatically updates the experiment's [Brief](https://docs.oqoqo.ai/evaluation/comparisons.md#brief) if Insights ran.

> [tab] Dashboard
>
> Open the experiment and select **Rerun evals** (or **Run evals**), optionally picking a different **Evals model** or mode for that attempt.

> [tab] CLI
>
> ```bash theme={null}
> oqoqo evals preview exp_8f2c41d0
> oqoqo evals trigger exp_8f2c41d0
> ```
>
> `preview` checks eligible runs, rubric diffs, and provider readiness ([`oqoqo evals trigger`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-trigger)).

> [tab] Plugin (MCP)
>
> Ask your AI assistant to rescore the experiment; it calls **Start Evals for an experiment** ([`trigger_evals`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#trigger_evals)) after your approval.

## From the CLI and the plugin

| Task | CLI | Plugin (MCP) |
| - | - | - |
| Preview Evals eligibility and rubric changes | [`oqoqo evals preview`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-preview) | [`preview_evals`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#preview_evals) |
| Run or rerun Evals | [`oqoqo evals trigger`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-trigger) | [`trigger_evals`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#trigger_evals) |
| List Evals attempts | [`oqoqo evals list`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-list) | [`list_evals`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#list_evals) |
| Read per-run verdicts | [`oqoqo evals get`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-get) | [`get_evals_results`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#get_evals_results) |
| Compare results across Evals models or modes | [`oqoqo data query`](https://docs.oqoqo.ai/cli/data#oqoqo-data-query) | [`query_data`](https://docs.oqoqo.ai/mcp-server/tags-and-data#query_data) |

## Next steps

> [card] [Insights](https://docs.oqoqo.ai/evaluation/insights.md)
>
> Analyze trajectories to surface frictions.

> [card] [Agent mode](https://docs.oqoqo.ai/evaluation/agent-mode.md)
>
> Let Oqoqo's evals agent verify claims across full runs.

> [card] [Read results](https://docs.oqoqo.ai/evaluation/results.md)
>
> Read pass rates and lift on Matrix.

> [card] [oqoqo evals and insights](https://docs.oqoqo.ai/cli/evals-and-insights.md)
>
> Full CLI reference for Evals commands.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.