What a task holds
A task’s Experiments, Runs, and Frictions tabs track all experiments, runs, and frictions for that task version. Configure starting files and environments under Machines and files. Tools attach to treatments, and you choose agents when you launch. Use tags to organize tasks as your catalog grows.
Create a task
- Dashboard
- CLI
- Plugin (MCP)
- Open Manage → Tasks and select New task.
- Enter a Task name and Instructions.
- Add criteria under Rubric.
- Attach starting folders on Files and pick an environment on Machine (or keep the project default).
- Select Create task.
Write the instructions
Brief the agent as you would a capable engineer unfamiliar with your codebase: state the goal, expected output, and constraints clearly (for example, “Refactor the auth module sorequire() calls are deterministic without changing the public API”).
Reference attached folders by their relative path, such as helpdesk-web/, rather than local machine paths like /home/user/helpdesk-web.


Write the rubric
A rubric is a list of criteria—one concrete check per row—added with Add criterion. Write each criterion so a reviewer can verify it from a single run:- Outcome check: “Returns the five prices in page order.”
- Behavior check: “Calls the batch endpoint once instead of one request per page.”
boolean, likert, or numeric, plus negate) and aggregation rules (all_pass, required_pass, any_pass, weighted_mean, or threshold).


Draft a rubric with Ask Oqo or your AI assistant
Both Ask Oqo and your AI assistant can draft criteria from a description of what “done” means. Review the generated criteria before launching.- Dashboard
- CLI
- Plugin (MCP)
Open Ask Oqo and ask it to draft the rubric:
Task versions
Saving any edit to a task or rubric creates a new version (tskver_). Past versions stay immutable, so finished runs always point to the exact version they ran.
Rerun Evals or launch a new experiment
Import tasks
Import multiple tasks at once from a ZIP archive.- Dashboard
- CLI
- Plugin (MCP)
- Open Manage → Tasks.
- Under More task actions, select Download template to see the ZIP structure.
- Under More task actions, select Import tasks and upload your ZIP file.
From the CLI and the plugin
Next steps
Machines and files
Configure starting folders and runtime environments for your tasks.
Tools
Add the skills, MCP servers, CLIs, and SDKs you want to test.
Launch an experiment
Run your tasks across treatments and agents.
Evals
See how Evals grade runs against your rubric.

