Skip to main content
Evals grade each finished run against the task’s rubric. Your Evals model marks every criterion Pass or Fail with an explanation, and those verdicts determine pass rates on Matrix and Comparisons. While Evals judge the outcome, Insights analyze how the agent worked to find frictions. Both run independently and can be rerun separately.

Turn on Evals

1

Connect a model provider

Add a model provider under Organization settings → Model providers or Profile → Integrations.
2

Assign an Evals model

3

Enable Evals on an experiment

Keep Evals toggled on when you launch, or select Run evals on a finished experiment.

Read the verdicts

Open any run and select Evals to see its overall verdict, passed criterion count, and the Evals model that graded it. Expand any criterion to read the model’s reasoning.
Run Evals tab with a pass or fail verdict for each rubric criterion and the Evals model that scored it.Run Evals tab with a pass or fail verdict for each rubric criterion and the Evals model that scored it.
Because an AI model grades the rubric, write criteria so each one is verifiable from a single run’s trace and workspace changes.

Choose a mode

By default, Evals run in Standard mode (one low-cost pass per run). If your organization has enabled Agent mode (private beta), you can switch to Agent at launch or when rerunning so Oqoqo’s evals agent inspects full runs and verifies claims before grading. See Agent mode for cost, speed, and model requirements.

What Evals read

Your Evals model inspects the task rubric, recorded trajectory steps, and changed workspace files. When a run exceeds the model’s context window in Standard mode, a coverage note on the Evals tab (and in Compare) reports what was condensed or omitted—such as summarized step ranges, shortened long steps, truncated changed files, or traces exceeding the 32 MB storage limit. No coverage note means the model read the entire run (older runs created before coverage tracking show no note regardless). If a run still cannot fit after condensing, Evals fail for that run without creating a friction—select an Evals model with a larger context window and rerun Evals.

Which models ran

Each Evals attempt records the exact provider, model, and effort it used (such as OpenAI · GPT-5.4 · high), so later changes under Oqoqo AI defaults never alter historical attributions. You can see the model on each run’s Evals tab, the attempt selector on Matrix, the experiment info popover, public experiment links, CLI and plugin reads, and the evals dataset in oqoqo data query. Older attempts predating model tracking display Not recorded.

Pending verdicts

Evals grade each run as soon as it finishes; Rerun evals clears old verdicts and displays Evals pending until new scores arrive. In the CLI and plugin, oqoqo evals get reports per-run status (queued, running, completed, skipped, failed) and oqoqo results analyze tracks pendingRuns separately from inconclusive runs.

Run Evals again

Rerunning Evals regrades existing recorded work without re-executing agents—ideal after editing a task’s rubric or switching to a stronger Evals model. (If you changed the task prompt, files, or machine, launch a new experiment instead.) When Evals finish, Oqoqo automatically updates the experiment’s Brief if Insights ran.
Open the experiment and select Rerun evals (or Run evals), optionally picking a different Evals model or mode for that attempt.

From the CLI and the plugin

Next steps

Insights

Analyze trajectories to surface frictions.

Agent mode

Let Oqoqo’s evals agent verify claims across full runs.

Read results

Read pass rates and lift on Matrix.

oqoqo evals and insights

Full CLI reference for Evals commands.