> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oqoqo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tasks and rubrics

> Write tasks for agents, define the rubric that decides pass or fail, import tasks in bulk, and know when to rerun evals.

A **task** defines the work you ask an agent to perform, along with the files and machine that stay identical across every treatment and agent you compare. Its **rubric** defines the pass or fail criteria for scoring each run.

## What a task holds

| Tab | What it contains |
| - | - |
| **Instructions** | Task name and the prompt given to the agent |
| **Rubric** | Scoring criteria that determine whether a run passes |
| **Files** | Starting folders mounted in the agent's working directory |
| **Machine** | Runtime environment the task executes in |

A task's **Experiments**, **Runs**, and **Frictions** tabs track all experiments, runs, and frictions for that task version. Configure starting files and environments under [Machines and files](https://docs.oqoqo.ai/experiments/machines-and-files.md). Tools attach to [treatments](https://docs.oqoqo.ai/experiments/treatments.md), and you choose agents when you launch. Use [tags](https://docs.oqoqo.ai/experiments/tags.md) to organize tasks as your catalog grows.

## Create a task

> [tab] Dashboard
>
> 1. Open **Manage → Tasks** and select **New task**.
> 2. Enter a **Task name** and **Instructions**.
> 3. Add criteria under **Rubric**.
> 4. Attach starting folders on **Files** and pick an environment on **Machine** (or keep the project default).
> 5. Select **Create task**.
>
> You can also select **New task** directly from the task list in **New experiment**.

> [tab] CLI
>
> Define the task in a JSON operation file and apply it:
>
> ```bash theme={null}
> oqoqo schema resources apply
> oqoqo resources apply --operation-file task.json
> ```
>
> The schema documents all fields, including `instructionsMarkdown`, `rubric`, and `assetAttachments` with each file's `mountPath`. See [`oqoqo resources apply`](https://docs.oqoqo.ai/cli/catalog#oqoqo-resources-apply) for a complete `task.json` example.

> [tab] Plugin (MCP)
>
> Ask your AI assistant to create the task. It calls **Create or update a resource** ([`author_resource`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#author_resource)).
>
> ```text theme={null}
> Create an Oqoqo task named "Fix the checkout total" that asks the agent to fix the rounding bug in helpdesk-web/, with a rubric of three yes/no criteria.
> ```

## Write the instructions

Brief the agent as you would a capable engineer unfamiliar with your codebase: state the goal, expected output, and constraints clearly (for example, *"Refactor the auth module so `require()` calls are deterministic without changing the public API"*).

Reference attached folders by their [relative path](https://docs.oqoqo.ai/experiments/machines-and-files.md#mount-paths), such as `helpdesk-web/`, rather than local machine paths like `/home/user/helpdesk-web`.

![Task page with the task name, the Instructions tab, and the instructions text.](https://mintcdn.com/oqoqo/_r_byoercOECoPlh/images/screenshots/task-detail-light.png?fit=max\&auto=format\&n=_r_byoercOECoPlh\&q=85\&s=2dfd286edd33d820ecef3d00394b7956)

## Write the rubric

A rubric is a list of **criteria**—one concrete check per row—added with **Add criterion**. Write each criterion so a reviewer can verify it from a single run:

* **Outcome check:** *"Returns the five prices in page order."*
* **Behavior check:** *"Calls the batch endpoint once instead of one request per page."*

After a run finishes, **Evals** uses your Evals model to read the [recorded work](https://docs.oqoqo.ai/evaluation/evals.md#what-evals-read) (trajectory and changed files) and mark each criterion. Criteria use **Yes / no** by default and pass when every criterion passes. Turn on **Advanced mode** to customize per-criterion scoring (`boolean`, `likert`, or `numeric`, plus `negate`) and aggregation rules (`all_pass`, `required_pass`, `any_pass`, `weighted_mean`, or `threshold`).

![Task Rubric tab with a list of criteria and the Add criterion control.](https://mintcdn.com/oqoqo/YO1ysyUx4SPwBwoz/images/screenshots/task-rubric-light.png?fit=max\&auto=format\&n=YO1ysyUx4SPwBwoz\&q=85\&s=aaaeb19e388d4f831000b8b8df2b8130)

## Draft a rubric with Ask Oqo or your AI assistant

Both **Ask Oqo** and your AI assistant can draft criteria from a description of what "done" means. Review the generated criteria before launching.

> [tab] Dashboard
>
> Open [Ask Oqo](https://docs.oqoqo.ai/operate/ask-oqo.md) and ask it to draft the rubric:
>
> ```text theme={null}
> Draft a rubric for the task "Fix the checkout total". Done means the total is correct for discounts and the existing tests still pass.
> ```

> [tab] CLI
>
> The CLI applies JSON operations directly rather than drafting criteria. Edit the `rubric` field in your JSON operation and run [`oqoqo resources apply`](https://docs.oqoqo.ai/cli/catalog#oqoqo-resources-apply).

> [tab] Plugin (MCP)
>
> Ask your AI assistant to draft and save the rubric. It reads the task with **Get a resource and its versions** ([`inspect_resource`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#inspect_resource)) and updates it with **Create or update a resource** ([`author_resource`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#author_resource)).

## Task versions

Saving any edit to a task or rubric creates a new version (`tskver_`). Past versions stay immutable, so finished runs always point to the exact version they ran.

| Launch path | Task version used |
| - | - |
| **New experiment** | Current version (or any older version you select) |
| **Launch again** and schedules | Latest version |
| `oqoqo experiments rerun --versions experiment` | Original experiment's task versions |

## Rerun Evals or launch a new experiment

| What changed | What to do |
| - | - |
| Only the rubric | Select **Rerun evals** on the finished experiment (**Run evals** if Evals have not run yet) to rescore the existing recorded work. |
| Instructions, files, or machine | Launch a new experiment. Rescoring old recorded work cannot reflect changes the agent never saw. |

## Import tasks

Import multiple tasks at once from a ZIP archive.

> [tab] Dashboard
>
> 1. Open **Manage → Tasks**.
> 2. Under **More task actions**, select **Download template** to see the ZIP structure.
> 3. Under **More task actions**, select **Import tasks** and upload your ZIP file.

> [tab] CLI
>
> ```bash theme={null}
> oqoqo uploads create ./tasks.zip --kind task
> ```
>
> [`oqoqo uploads create`](https://docs.oqoqo.ai/cli/catalog#oqoqo-uploads-create) uploads the archive and imports its tasks.

> [tab] Plugin (MCP)
>
> Your AI assistant uploads the archive with **Start an archive upload** ([`create_upload`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#create_upload)) and **Complete an archive upload** ([`complete_upload`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#complete_upload)).

## From the CLI and the plugin

| Task | CLI | Plugin (MCP) |
| - | - | - |
| List tasks | [`oqoqo catalog list --kind task`](https://docs.oqoqo.ai/cli/catalog#oqoqo-catalog-list) | [`inspect_catalog`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#inspect_catalog) |
| Inspect a task and its versions | [`oqoqo resources inspect task`](https://docs.oqoqo.ai/cli/catalog#oqoqo-resources-inspect) | [`inspect_resource`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#inspect_resource) |
| Create or update a task | [`oqoqo resources apply`](https://docs.oqoqo.ai/cli/catalog#oqoqo-resources-apply) | [`author_resource`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#author_resource) |
| Batch-author up to 20 resources | [`oqoqo resources bulk-apply`](https://docs.oqoqo.ai/cli/catalog#oqoqo-resources-bulk-apply) | [`bulk_author_resources`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#bulk_author_resources) |
| Import tasks from a ZIP | [`oqoqo uploads create --kind task`](https://docs.oqoqo.ai/cli/catalog#oqoqo-uploads-create) | [`create_upload`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#create_upload), [`complete_upload`](https://docs.oqoqo.ai/mcp-server/catalog-and-uploads#complete_upload) |
| Rerun evals after rubric edits | [`oqoqo evals trigger`](https://docs.oqoqo.ai/cli/evals-and-insights#oqoqo-evals-trigger) | [`trigger_evals`](https://docs.oqoqo.ai/mcp-server/evals-and-insights#trigger_evals) |

## Next steps

> [card] [Machines and files](https://docs.oqoqo.ai/experiments/machines-and-files.md)
>
> Configure starting folders and runtime environments for your tasks.

> [card] [Tools](https://docs.oqoqo.ai/experiments/tools.md)
>
> Add the skills, MCP servers, CLIs, and SDKs you want to test.

> [card] [Launch an experiment](https://docs.oqoqo.ai/experiments/launch.md)
>
> Run your tasks across treatments and agents.

> [card] [Evals](https://docs.oqoqo.ai/evaluation/evals.md)
>
> See how Evals grade runs against your rubric.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.