> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oqoqo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Launch an experiment

> Choose the tasks, treatments, agents, and settings for an experiment, then launch it, launch it again, or cancel it.

Pick the tasks, treatments, and agents to compare, set the trials, and launch. A launch spends runs from your organization's run balance. If you're new to experiments, read [Experiments](https://docs.oqoqo.ai/experiments/overview.md) first.

## Prerequisites

* At least one [task](https://docs.oqoqo.ai/experiments/tasks.md). A task needs a rubric for its runs to get an Evals result.
* At least one [treatment](https://docs.oqoqo.ai/experiments/treatments.md). Every project has **Raw Agent**.
* At least one [agent](https://docs.oqoqo.ai/setup/agents.md) with a model, under **Manage → Agents**.
* Runs in your organization's [run balance](https://docs.oqoqo.ai/setup/billing.md).

## Start a new experiment

Select **New experiment** in the sidebar and fill in **Experiment name**. You can also start from a [template](https://docs.oqoqo.ai/experiments/templates.md) on **Home**, which fills in its tasks, treatments, and agents. Tasks already carry their files and machine, and treatments carry their tools, so in the **Configuration** panel you only choose what to combine.

![New experiment page with the selected tasks and the Configuration panel: Agents, Treatments (Toolsets), Trials, Max Concurrency, Evals, and Insights.](https://mintcdn.com/oqoqo/YO1ysyUx4SPwBwoz/images/screenshots/launch-config-light.png?fit=max\&auto=format\&n=YO1ysyUx4SPwBwoz\&q=85\&s=d54482136e6a4f36493351c50db3b6b7)

## Choose tasks

Select tasks in the task list. Search by name, or use the [**Tags**](https://docs.oqoqo.ai/experiments/tags.md) filter to select all the tasks of a release or an area at once. To create a task without leaving the page, select **New task**.

Each task runs its current version. To run an older version, choose it in the task's version menu.

To compare treatments or agents, select at least 6 tasks, and more if you can. [Comparisons](https://docs.oqoqo.ai/evaluation/comparisons.md) needs at least 6 tasks where one side did better, and tied tasks don't count, however many trials you run.

## Choose agents

A new experiment selects every built-in agent that has a recommended model. To pick your own, select **Clear** beside the agent search, then select the agents you want. You need at least one agent to launch. The number beside **Clear** is the number of agents, so an agent with two models counts once.

For each agent, choose a model and an effort level. To compare models, add the same agent again with another model.

A new experiment doesn't select a [custom agent](https://docs.oqoqo.ai/setup/custom-agents.md) for you unless you open **New experiment** from that agent's page. A selected custom agent starts on **Agent default**, and one that is **Not tested** or **Needs attention** shows a notice before you launch.

## Choose treatments and a baseline

Select the treatments to compare under **Treatments (Toolsets)**. A new experiment starts with **Raw Agent** selected.

When you select two or more treatments, a **Baseline** control appears. Choose the treatment you want to compare the others with. It can be any treatment you selected, and **Raw Agent** starts as the baseline when it's selected.

Oqoqo saves the baseline with the experiment, and you read each treatment's [lift](https://docs.oqoqo.ai/experiments/overview.md#baseline-and-lift) by comparing it with the baseline in [Pass rate and lift](https://docs.oqoqo.ai/evaluation/results.md#pass-rate-and-lift).

## Set trials

Set **Trials** from 1 to 20. A new experiment starts at 1. Use two or more when you compare treatments or agents, because agents don't give the same result every time.

The footer shows how many runs the experiment uses: tasks × treatments × agents × trials (see [What an experiment combines](https://docs.oqoqo.ai/experiments/overview.md#what-an-experiment-combines)). If your run balance doesn't have enough, the footer links to where you can [add runs](https://docs.oqoqo.ai/setup/billing.md).

## Set Max Concurrency

**Max Concurrency** limits how many runs are active at once. It starts at the number of runs, and it can't be higher.

Lower it when runs change shared resources through connected MCP servers, CLIs, or other tools. A lower limit helps prevent race conditions, unexpected results, and rate-limit spikes at your model provider. Set it to 1 to run one at a time.

Max Concurrency doesn't apply to Evals or Insights. Oqoqo also applies organization and global limits, so the effective limit can be lower than the value you set. For help choosing a limit or for more capacity, email [support@oqoqo.ai](mailto:support@oqoqo.ai).

## Turn on Evals and Insights

| Control | What it does |
| - | - |
| [**Evals**](https://docs.oqoqo.ai/evaluation/evals.md) | Scores each run against its task rubric. When it's off, runs aren't scored. |
| [**Insights**](https://docs.oqoqo.ai/evaluation/insights.md) | Reads each run's saved trace to find frictions, and finds patterns across the experiment. |

Each control needs its models set under [Oqoqo AI features](https://docs.oqoqo.ai/setup/oqoqo-ai.md). Leave one off now and you can select **Run evals** or **Run insights** on the experiment later.

When **Evals** is on and your organization has Agent mode, choose a **Mode**:

| Mode | How it scores | Cost and time |
| - | - | - |
| **Standard** | Scores each run in one pass with your Evals model. | Fastest and lowest cost |
| **Agent** | Oqoqo's evals agent reviews each run in depth, reading the full run and the changed files. | Costs more and takes a few minutes per run |

> [!NOTE]
>
> Evals Agent mode is in private beta. Email [support@oqoqo.ai](mailto:support@oqoqo.ai) to enable it for your organization. See [Agent mode](https://docs.oqoqo.ai/evaluation/agent-mode.md).

## Override models for this experiment

Open **Advanced** to choose a different **Evals model**, **Insights model**, or **Experiment insights model** for this experiment only. **Default** uses your organization's setting.

If your organization has [custom instructions](https://docs.oqoqo.ai/setup/oqoqo-ai.md#add-custom-instructions), turn on **Customize** under a feature to replace them for this experiment. Left off, the experiment uses the organization's instructions, or Oqoqo's built-in guidance when there are none.

In a launch plan from the CLI or the plugin, the role overrides use these names. Omit a role to use the organization's model.

| Role | Name in the plan | Needs |
| - | - | - |
| **Evals model** | `judge` | Evals on |
| **Insights model** | `sweep` | Insights on |
| **Experiment insights model** | `consolidation` | Insights on |

## Launch

> [tab] Dashboard
>
> Check the run count in the footer and select **Launch**. If something blocks the launch, such as a tool that needs values or an agent without a provider, the panel names it and links to the fix.

> [tab] CLI
>
> Check the plan, launch it, and wait for it to finish:
>
> ```bash theme={null}
> oqoqo experiments preflight --plan-file plan.json
> oqoqo experiments launch --plan-file plan.json
> oqoqo experiments watch exp_8f2c41d0
> ```
>
> The CLI launches as soon as you run `launch`, so run `preflight` first. A preflight changes nothing and lists each value a person must add, with its setup link.
>
> The plan names the tasks, treatments, agents, `trialCount`, `maxParallelAgentSessions`, `evals`, `insights`, and `evalsMode`, and its `referenceTreatmentClientId` names the baseline. To write a plan, start from the [example plan](https://docs.oqoqo.ai/cli/experiments.md#example). Every flag is in [`oqoqo experiments launch`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-launch).

> [tab] Plugin (MCP)
>
> Ask your AI assistant to launch it. The assistant checks the plan with **Check an experiment plan before launch** ([`preflight_experiment`](https://docs.oqoqo.ai/mcp-server/tools#preflight_experiment)), launches it with **Launch an experiment** ([`launch_experiment`](https://docs.oqoqo.ai/mcp-server/tools#launch_experiment)), and follows it with **Get experiment status** ([`get_experiment_status`](https://docs.oqoqo.ai/mcp-server/tools#get_experiment_status)).
>
> A launch spends runs, so turn on approval for Oqoqo tools in your AI assistant's tool approval settings, and approve each launch. Ask Oqo always asks before a launch.
>
> ```text theme={null}
> Launch an Oqoqo experiment with the checkout tasks, Raw Agent and our Stripe MCP treatment, Claude Code and Codex, and 2 trials.
> ```

When you launch, Oqoqo saves each task's version, files, and machine, and each treatment's tools, so later edits don't change the runs. Results fill in on the experiment page as runs finish: **Matrix** shows each task against each agent and treatment for a metric such as Evals or cost, **Frictions** shows where agents got stuck, and **Runs** lists every run. Learn how to interpret them in [Read results](https://docs.oqoqo.ai/evaluation/results.md), or get a plain-language summary from [Ask Oqo](https://docs.oqoqo.ai/operate/ask-oqo.md).

## Launch again

**Launch again** starts a new experiment with the same name, tasks, treatments, agents, and settings, and uses as many runs as the original planned. It's on the experiment and in its row menu in **All experiments** once the experiment is **Completed**, **Needs review**, **Error**, or **Cancelled**.

* **Latest versions:** each task runs its latest version, and each treatment uses its current saved state. If tasks changed since the experiment ran, the dialog says how many.
* **Blockers:** the dialog lists anything that blocks the launch, such as a disconnected provider, each with a fix link that opens in a new tab. When you come back, the dialog checks again, and **Launch** turns on as soon as nothing blocks it.
* **Deleted tasks or treatments:** **Launch again** won't launch, so it never runs a smaller experiment or compares against a different baseline. Use **Edit first** to choose others.

**Edit first** opens **New experiment** with the same choices so you can change them. To launch again on a repeat, select [**Schedule**](https://docs.oqoqo.ai/experiments/schedules.md).

> [tab] Dashboard
>
> Select **Launch again**, and then select **Launch**.

> [tab] CLI
>
> ```bash theme={null}
> oqoqo experiments rerun exp_8f2c41d0
> ```
>
> Pass `--versions experiment` to run the experiment's own task versions instead of the latest. To change the plan first, save it with [`oqoqo experiments clone-plan`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-clone-plan), edit it, and launch it. See [`oqoqo experiments rerun`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-rerun).

> [tab] Plugin (MCP)
>
> Your AI assistant calls **Launch an experiment again** ([`rerun_experiment`](https://docs.oqoqo.ai/mcp-server/tools#rerun_experiment)). To change the plan first, it reads it with **Get an experiment's launch plan** ([`get_experiment_plan`](https://docs.oqoqo.ai/mcp-server/tools#get_experiment_plan)), edits it, and launches it.

The new experiment is a child of the one you launched it from, its parent experiment. In **All experiments**, its row shows **Parent:** and the parent's name, and its **⋯** menu has **Open parent experiment**:

![All experiments with two rows named Run again: Stripe CLI and Raw Agent, each showing Parent: Stripe CLI and Raw Agent, 7 create tasks. The first row's menu is open, with Open parent experiment at the top.](https://mintcdn.com/oqoqo/k4Yj05MbUPbdkSQD/images/screenshots/experiments-list-parent-light.png?fit=max\&auto=format\&n=k4Yj05MbUPbdkSQD\&q=85\&s=b681c5e9d59c35f1eee780794cd3c8ed)

## Cancel an experiment

To stop an experiment that is **Running**, select **Cancel experiment** on the experiment or in its row menu. It stops unfinished runs, and completed runs stay available. To stop one run, select **Cancel run** on that run.

> [tab] Dashboard
>
> Select **Cancel experiment**, then confirm.

> [tab] CLI
>
> ```bash theme={null}
> oqoqo experiments cancel exp_8f2c41d0
> ```
>
> Add `--run-id` to cancel one run. The CLI asks you to confirm. See [`oqoqo experiments cancel`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-cancel).

> [tab] Plugin (MCP)
>
> Your AI assistant calls **Cancel an experiment or run** ([`cancel_experiment`](https://docs.oqoqo.ai/mcp-server/tools#cancel_experiment)). It asks you to approve the call.

## From the CLI and the plugin

| Task | CLI | Plugin (MCP) |
| - | - | - |
| Check a plan before launch | [`oqoqo experiments preflight`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-preflight) | [`preflight_experiment`](https://docs.oqoqo.ai/mcp-server/tools#preflight_experiment) |
| Launch a plan | [`oqoqo experiments launch`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-launch) | [`launch_experiment`](https://docs.oqoqo.ai/mcp-server/tools#launch_experiment) |
| Follow progress | [`oqoqo experiments watch`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-watch), [`oqoqo experiments status`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-status) | [`get_experiment_status`](https://docs.oqoqo.ai/mcp-server/tools#get_experiment_status), [`get_launch_status`](https://docs.oqoqo.ai/mcp-server/tools#get_launch_status) |
| Launch again | [`oqoqo experiments rerun`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-rerun) | [`rerun_experiment`](https://docs.oqoqo.ai/mcp-server/tools#rerun_experiment) |
| Read a plan to edit | [`oqoqo experiments clone-plan`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-clone-plan) | [`get_experiment_plan`](https://docs.oqoqo.ai/mcp-server/tools#get_experiment_plan) |
| Cancel | [`oqoqo experiments cancel`](https://docs.oqoqo.ai/cli/experiments#oqoqo-experiments-cancel) | [`cancel_experiment`](https://docs.oqoqo.ai/mcp-server/tools#cancel_experiment) |

## Next steps

> [card] [Read results](https://docs.oqoqo.ai/evaluation/results.md)
>
> Compare Evals results and cost in Matrix.

> [card] [Schedules](https://docs.oqoqo.ai/experiments/schedules.md)
>
> Launch the experiment again on a repeat.

> [card] [Frictions](https://docs.oqoqo.ai/evaluation/frictions.md)
>
> See where agents got stuck.

> [card] [oqoqo experiments](https://docs.oqoqo.ai/cli/experiments.md)
>
> Every flag for launch, watch, and cancel.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.