Skip to main content
Agent mode for Evals and Insights is in private beta. Email support@oqoqo.ai to enable it for your organization.
In Agent mode, Oqoqo runs an autonomous review agent on your Evals or Insights model—reading the entire run and calling your model multiple times to verify facts instead of relying on a single condensed pass.
  • Evals Agent mode: Oqoqo’s evals agent inspects the full trace and changed files, verifies claims before scoring, and attaches clickable step and file citations.
  • Insights Agent mode: Oqoqo’s insights agent verifies each friction against tool outputs and adds What the runs did differently to Comparisons.
Oqoqo enables Agent mode for Evals and Insights independently, so specify Evals, Insights, or both when requesting access.

Standard and Agent modes

Both modes use the same models and produce the same verdict and friction formats: Agent mode never consumes extra experiment runs from your Oqoqo balance; your model provider bills the extra model tokens directly (Runs and billing).

Choose models for Agent mode

Agent mode requires models that support tool calling and have a known context window (for Evals model, or both Insights model and Experiment insights model). Claude Pro / Max models from Anthropic (Claude Subscription) cannot run Agent mode; select a model from an API-key provider under Oqoqo AI defaults or use Standard mode.

Turn on Agent mode

Select Agent under Mode at launch or when rerunning Evals or Insights.
  1. In New experiment (or Run evals / Rerun evals / Run insights / Rerun Insights), ensure Evals or Insights is enabled.
  2. Set Mode to Agent.
The Configuration panel previews how many runs the agent will review and on which model before you launch.
New experiment Configuration panel. Evals and Insights are on, each with Mode set to Agent and the line Up to 56 agent reviews on Claude Opus 5.5.New experiment Configuration panel. Evals and Insights are on, each with Mode set to Agent and the line Up to 56 agent reviews on Claude Opus 5.5.

Check what Agent mode uses

Before launching or rerunning, Mode estimates the review count and model (such as “Up to 56 agent reviews on Claude Opus 5.5”); for Insights, friction clustering and pair comparisons add calls on top of per-run reviews.
Rerun Insights dialog with Mode set to Agent and the line Up to 56 agent reviews on Claude Opus 5.5, above the Insights and Experiment insights models.Rerun Insights dialog with Mode set to Agent and the line Up to 56 agent reviews on Claude Opus 5.5, above the Insights and Experiment insights models.
Once running or finished, the attempt menu on Matrix (when multiple attempts exist) reports cumulative input and cached tokens (agentUsage in CLI and plugin attempt lists; agentModeEstimates in preflight and mode.estimate in previews).

Read Evals in Agent mode

Runs graded in Agent mode show an Agent mode badge next to Evals model. Each criterion lists the evals agent’s explanation with direct links to cited run steps, diff lines, and changed files (selecting a file citation jumps directly to Output).
Run Evals tab in Agent mode. A failed criterion shows the evals agent's explanation, a cited step of the run, and two cited changed files.Run Evals tab in Agent mode. A failed criterion shows the evals agent's explanation, a cited step of the run, and two cited changed files.

Read Insights in Agent mode

Runs analyzed in Agent mode show Agent mode next to Insights model above Traces. Like Standard mode, the insights agent inspects only the trajectory (never changed files, rubrics, or Evals verdicts).

What the runs did differently

When Insights run in Agent mode, each difference block with a Brief ends with What the runs did differently. For every task where the worse side lost, the insights agent pairs one failed run from the worse side with one passed run from the better side and pinpoints the exact step and quoted lines where their paths diverged. Once all pairs are analyzed, the Brief updates once more to summarize the cross-task pattern, highlighting one representative task (Like N other tasks) and any exceptions (Went differently).
What the runs did differently, 1 of 7 tasks: Create a customer with metadata, Like 6 other tasks, with a line and a quoted step for each run, then Show all 7 tasks.What the runs did differently, 1 of 7 tasks: Create a customer with metadata, Like 6 other tasks, with a line and a quoted step for each run, then Show all 7 tasks.
Each row compares one specific pair of runs (when trials disagreed, a subtitle notes “Raw Agent failed 2 of 3 runs”). Select ⋯ → Open both runs to inspect both trajectories side by side.

When Agent mode fails

If an Agent mode run stops early, the Evals tab (for Evals) or CLI and plugin reads (oqoqo insights get, oqoqo insights list, get_insights_results, list_insights) report the exact cause—such as rate limits or exceeding context—while preserving any runs that succeeded. See Agent mode did not return a result to rerun only failed runs.

Next steps

Comparisons

See which differences hold up and inspect paired runs.

Frictions

Prioritize verified frictions by unrecovered runs.

Oqoqo AI defaults

Choose tool-capable models for Agent mode.

oqoqo evals and insights

Trigger and inspect Agent mode attempts from the CLI.