Skip to main content
A task defines the work you ask an agent to perform, along with the files and machine that stay identical across every treatment and agent you compare. Its rubric defines the pass or fail criteria for scoring each run.

What a task holds

A task’s Experiments, Runs, and Frictions tabs track all experiments, runs, and frictions for that task version. Configure starting files and environments under Machines and files. Tools attach to treatments, and you choose agents when you launch. Use tags to organize tasks as your catalog grows.

Create a task

  1. Open Manage → Tasks and select New task.
  2. Enter a Task name and Instructions.
  3. Add criteria under Rubric.
  4. Attach starting folders on Files and pick an environment on Machine (or keep the project default).
  5. Select Create task.
You can also select New task directly from the task list in New experiment.

Write the instructions

Brief the agent as you would a capable engineer unfamiliar with your codebase: state the goal, expected output, and constraints clearly (for example, “Refactor the auth module so require() calls are deterministic without changing the public API”). Reference attached folders by their relative path, such as helpdesk-web/, rather than local machine paths like /home/user/helpdesk-web.
Task page with the task name, the Instructions tab, and the instructions text.Task page with the task name, the Instructions tab, and the instructions text.

Write the rubric

A rubric is a list of criteria—one concrete check per row—added with Add criterion. Write each criterion so a reviewer can verify it from a single run:
  • Outcome check: “Returns the five prices in page order.”
  • Behavior check: “Calls the batch endpoint once instead of one request per page.”
After a run finishes, Evals uses your Evals model to read the recorded work (trajectory and changed files) and mark each criterion. Criteria use Yes / no by default and pass when every criterion passes. Turn on Advanced mode to customize per-criterion scoring (boolean, likert, or numeric, plus negate) and aggregation rules (all_pass, required_pass, any_pass, weighted_mean, or threshold).
Task Rubric tab with a list of criteria and the Add criterion control.Task Rubric tab with a list of criteria and the Add criterion control.

Draft a rubric with Ask Oqo or your AI assistant

Both Ask Oqo and your AI assistant can draft criteria from a description of what “done” means. Review the generated criteria before launching.
Open Ask Oqo and ask it to draft the rubric:

Task versions

Saving any edit to a task or rubric creates a new version (tskver_). Past versions stay immutable, so finished runs always point to the exact version they ran.

Rerun Evals or launch a new experiment

Import tasks

Import multiple tasks at once from a ZIP archive.
  1. Open Manage → Tasks.
  2. Under More task actions, select Download template to see the ZIP structure.
  3. Under More task actions, select Import tasks and upload your ZIP file.

From the CLI and the plugin

Next steps

Machines and files

Configure starting folders and runtime environments for your tasks.

Tools

Add the skills, MCP servers, CLIs, and SDKs you want to test.

Launch an experiment

Run your tasks across treatments and agents.

Evals

See how Evals grade runs against your rubric.