Tasks and rubrics

A task is the work you want an agent to do, and everything that must stay fixed for that work to be fair. On a task you set:

• Instructions — what the agent should do
• Rubric — how Oqoqo decides pass or fail
• Files — folders mounted before the run starts
• Machine — the environment the run executes in

Create and edit tasks under Manage → Tasks. Files and the machine attach to the task. Skills, MCP servers, CLIs, and SDKs attach to treatments. Agents are chosen at launch. See Core concepts.

​

Write the instructions

Brief a capable engineer who is new to your product. Be specific about the goal, the output you expect, and any limits. Vague tasks are hard to score. Prefer something like: “Refactor the auth module so require() calls are deterministic, without changing the public API.”

​

Add a rubric

A rubric is a set of requirements. Each one is a clear pass or fail criterion.

• “Returns the five prices in page order” checks the output.
• “Calls the batch endpoint once instead of one request per page” checks a step.

Write each requirement so a judge can mark it true or false for a single run.

​

Draft a rubric with Oqoqo

If Rubric generation is assigned under Organization settings → Oqoqo AI features, you can ask Oqoqo to draft criteria. Review and edit before you save. Generation is a draft, not a final answer. See Oqoqo AI features.

​

Attach files and a machine

1. Create folders under Manage → Files, then attach them on the task Files tab.
2. Create environments under Manage → Machines, then pin one on the task Machine tab, or leave the Standard Dev Machine.

Every run of this task starts from those files on that machine, for every treatment and agent you compare. See Machines and files.

​

How runs are scored

Evals score each finished run with an automated judge against every requirement. You see pass or fail per requirement, not only one overall result. Scoring is judge-based. It is not a separate deterministic checker. See Evals and Insights.

​

Versions

Tasks and rubrics are versioned. Past results stay tied to the exact task that produced them. After runs finish, Re-trigger Evals scores the same recorded work under an updated rubric. Launch a new experiment when the task itself changed: new instructions, files, or machine.

​

What does not belong on a task

Skills, MCP servers, CLIs, and SDKs attach to a treatment. Agents are chosen when you launch. See Supported agents.