- Badges are explained under Approvals.
project_idis optional on most tools; without it, the server picks a project for that call only (Projects).expected_organization_idis an optional string on every write tool, omitted from parameter tables; passorganization.idfromget_identity(Organization check).limitvalues above a tool’s maximum are capped, not refused.
Tools
preview_evals
Preview Evals for an experiment. Shows which runs Evals can score, rubrics that need attention or changed, model readiness, and the modes available. Read-only · CLI:oqoqo evals preview
Result fields
ready, modelsReady, userProviderReady, setupUrl, eligibleRunCount, runsWithoutRubricsCount, runsWithInvalidRubricsCount, runsNotJudgeableCount, rubricChanges, failedRunIds, and mode: experimentMode (the mode the experiment launched with), latestMode, agentAvailable, and, only while your organization can use Agent mode, estimate (runs and model). Their meanings are under oqoqo evals preview.
trigger_evals
Start Evals for an experiment. Starts an Evals attempt, or reruns Evals on the same recorded work after a rubric change. Changes data · You approve each call · CLI:oqoqo evals trigger
Standard scores each run in one pass with your Evals model and condenses very long runs to fit its context window. It’s the fastest and lowest-cost mode. Agent has Oqoqo’s evals agent read the full run and the changed files and check the agent’s claims before it scores. It makes many calls to your Evals model, so it costs more and takes a few minutes per run. Both modes score against the same rubric (how the modes differ).
A request for Agent mode, stated or inherited, is refused with
agent_mode_unavailable when your organization can’t use it: “Agent mode isn’t available for your organization. Use Standard mode instead.” Without mode, the refusal has details.inheritedMode set to agent and says “This experiment launched Evals in Agent mode, which your organization can’t use now. Rerun with mode: “standard”.” When the Evals model can’t run Agent mode, the refusal is agent_mode_model_unsupported, with a setupUrl to Oqoqo AI defaults. Agent mode is in private beta.
Result fields
evalsId (the new eval_ attempt), status, mode (the mode the attempt uses), pollAfterMs (60 seconds in Agent mode, 10 seconds in Standard mode, or null when the attempt has already finished), and replayed (true when a repeated trigger_attempt_id returned the earlier attempt).
list_evals
List Evals history. Returns the experiment’s Evals attempts, newest first, without reports, each with itsmodels and mode. A failed attempt has error (code and message). An Agent mode attempt also has agentUsage once its agent has reviewed runs: the inputTokens, cacheReadTokens, and outputTokens that the evals agent has used so far.
Read-only · CLI: oqoqo evals list
get_evals_results
Get Evals results. Returns a summary for each run of an attempt, or the exact rubric and verdict JSON for one trial. Read-only · CLI:oqoqo evals get
Result fields
get_evals_comparisons
Compare treatments and agents. Compares an experiment’s treatments and agents task by task, from the Matrix’s Evals verdicts. It returns the sentence that answers the experiment’s question and a block for each difference first, as at the top of Comparisons, then every comparison with its result. Read-only · CLI:oqoqo evals compare
kind is treatment (a treatment against the baseline, for all agents together and for each agent, or each pair of treatments without a baseline), agent (two agents with the same treatment), or interaction (Does a treatment help one agent more?). label is clear (Clear difference), likely (Likely difference), or unclear (No clear difference). unclear means that the runs can’t tell the two sides apart, not that the sides are the same.
The most decisive comparisons come first, the main comparisons before the breakdowns within a label. kind and label only filter the list: labels don’t change, and summary, differences, restLine, and comparisonCount stay the same. A read returns up to 200 comparisons that fit in the response. A cursor fails with invalid_cursor once the comparisons change. More than 10,000 Matrix cells fails with experiment_too_large.
Result fields
Each comparison has
comparisonId (cmp_), kind, label, family (primary for a main comparison, or breakdown), title, scope, headline, sides, verdict, nextStep, sentence, groupLabel, taskSplit, treatmentKey, referenceTreatmentKey, agentKey, referenceAgentKey, arm and reference (or first and second for an interaction), tasks (higher, lower, same), pValue (an exact two-sided sign test), adjustedPValue (the Holm correction within its family), runsLeftOut, confirmation, and confirmationUnavailable (earlier_evals, in_progress, or cannot_settle). confirmation holds the runs that check a likely difference again. cannot_settle means that running these tasks again can’t settle it, and the experiment needs more tasks. To plan them, pass its comparisonId to get_experiment_plan as comparison_id, with a new launch_attempt_id.
Say the result in the words of summary and each verdict, which every surface shows, with verdict and nextStep on one line. nextStep is run_again (with text and runs) or more_tasks (with text), or null. Each field is explained under oqoqo evals compare.
get_experiment_brief
Get an experiment’s Brief. Reads an experiment’s Brief: its clear and likely differences, worded as inget_evals_comparisons, with one written sentence and the frictions seen with the side that did worse. Oqoqo counts every number from the experiment’s Evals verdicts and frictions. A model writes only each sentence and, in Agent mode, each pair’s line.
Read-only · CLI: oqoqo experiments brief
state says how complete the Brief is. ready, failed, not_written, and no_insights are final for the Brief. While state is writing, the differences and counts are already in the read: call again after waitMs milliseconds until state changes. In Agent mode, also call again after waitMs while any finding’s pendingTaskCount is above 0. The tool takes no wait_ms.
Result fields
Each finding has the fields of its block in
differences (comparisonId, label, name, sides, taskLine, verdict, nextStep, byAgent, headline), and kind, title, scope, arm and reference, tasks, pValue, adjustedPValue, worseSide, worseTaskCount, notReviewed, sentence, confirmation, confirmationUnavailable, frictions, and afterFix. Each friction has frictionId (frc_), title, summary (what goes wrong, in one line), fixIn, tasks, dashboardUrl, failedRuns (how many of the worse side’s failed runs had it, out of runs), passedRuns (the same for the other side’s passed runs), and, for arm and reference, runsWithFriction out of runsReviewed, which is how the friction was picked. The counts use only the worseTaskCount tasks and the runs that Insights reviewed. afterFix is set for a clear difference with a friction to fix in the product: its text, taskCount, and runs.
When the experiment’s latest Insights ran in Agent mode, each finding also has these fields:
Each pair has
taskVersionId (tskver_), taskName, dashboardUrl, and state: pending while the agent reads it, completed when its lines are written, or failed when the agent gave no answer that passed Oqoqo’s checks, which shows only the two runs. Its worse and better runs each have these fields:
Show the pairs in that order, as the dashboard does: the example with “Like N other tasks” (from
similarTaskCount), each task that went differently with “Went differently”, then the rest. A pair says where two runs went different ways, not why the difference happened. What the agent used to read the pairs counts in the Insights attempt’s agentUsage (list_insights).
When you report a friction, give both sides’ counts (“Raw Agent 24 of 28 failed runs · Stripe CLI 0 of 26 passed runs”) and say it was seen with the side that did worse. Don’t call it the cause. To run a likely difference again, pass its comparisonId to get_experiment_plan as comparison_id. To run a clear difference again after a fix, add after_fix set to true. Each field is explained under oqoqo experiments brief.
preview_insights
Preview Insights for an experiment. Shows whether the Insights models and your provider access are ready, with a dashboard link for any secure setup, and the modes this experiment can use. Read-only · CLI:oqoqo insights preview
Result fields
ready, modelsReady, userProviderReady, setupUrl, failedRunIds, and mode: experimentMode (the mode the experiment launched with), latestMode, agentAvailable, and, only while your organization can use Agent mode, estimate (runs, model, and experimentModel when the Experiment insights model is a different model). Their meanings are under oqoqo insights preview.
trigger_insights
Start Insights for an experiment. Starts an Insights attempt, or reruns Insights on the same recorded work. Changes data · You approve each call · CLI:oqoqo insights trigger
Standard reviews each run in one pass with your Insights model. It’s the fastest and lowest-cost mode. Agent has Oqoqo’s insights agent read the whole run and check each friction against what the tools returned. It costs more and takes a few minutes per run. It also does the experiment’s grouping of frictions, which Standard mode does in one pass. For each task of each difference in the Brief, it says what a failed run and a passed run did differently (
get_experiment_brief). Both modes report frictions in the same way. Without mode, the attempt uses the experiment’s launch mode, not the latest attempt’s mode as the dashboard does.
A request for Agent mode, stated or inherited, is refused with agent_mode_unavailable when your organization can’t use it: “Agent mode isn’t available for your organization. Use Standard mode instead.” Without mode, the refusal has details.inheritedMode set to agent and says “This experiment launched Insights in Agent mode, which your organization can’t use now. Rerun with mode: “standard”.” When the Insights model or the Experiment insights model can’t run Agent mode, the refusal is agent_mode_model_unsupported, with a setupUrl to Oqoqo AI defaults. Agent mode is in private beta.
Result fields
insightsId (the new insights_ attempt), status, mode (the mode the attempt uses), pollAfterMs (60 seconds in Agent mode, 10 seconds in Standard mode, or null when the attempt has already finished), and replayed (true when a repeated trigger_attempt_id returned the earlier attempt).
list_insights
List Insights history. Returns the experiment’s Insights attempts, newest first, without reports, each with itsmodels and mode. A failed attempt has error (code and message); when the insights agent couldn’t group the frictions, error.message starts with “Insights could not group this experiment’s frictions.” An Agent mode attempt also has agentUsage once its agent has reviewed runs: the inputTokens, cacheReadTokens, and outputTokens that the insights agent has used so far, including grouping the frictions and the pairs of runs in the Brief.
Read-only · CLI: oqoqo insights list
get_insights_results
Get Insights results. Returns a summary for each run of an attempt, the attempt’s public report, or the exact Insights JSON for one trial. Read-only · CLI:oqoqo insights get
Result fields
attempt (with its insightsId, models, mode, error, and, in Agent mode, agentUsage), and for each row its status, its error (a failed row’s message starts with “Insights could not analyze this run.” and gives the cause), which follows the same rules as in get_evals_results, and evidenceCoverage (null for older runs). hasMore and nextCursor page the summary view.
Related
Frictions tools
Inspect the frictions Insights found.
oqoqo evals and insights
The matching CLI commands and output formats.

