> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oqoqo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom agents

> Add an Agent Client Protocol (ACP) agent to your organization, test it, and run it in experiments next to the built-in agents.

> [!NOTE]
>
> Custom agents are in private beta. Email [support@oqoqo.ai](mailto:support@oqoqo.ai) to enable them for your organization.

A **custom agent** lets you evaluate your own agent—or any agent supporting the [Agent Client Protocol](https://agentclientprotocol.com/) (ACP)—alongside Oqoqo's built-in agents. Use custom agents to test pre-release agents on real tasks, evaluate third-party ACP agents, or compare an agent's ACP build with its built-in counterpart.

Connecting a custom agent is optional step 4 of the [setup checklist](https://docs.oqoqo.ai/setup/organization.md#setup-checklist). Oqoqo installs the custom agent inside each run's machine so it shares the task's files, services, and treatment tools. Like built-in agents, you choose a custom agent at launch rather than attaching it to a treatment.

> [!NOTE]
>
> An agent that runs tools on a remote host (such as Ask Oqo) cannot access the run's files, services, or treatment tools. Give it tasks that match what it can reach ([Run Ask Oqo as a custom agent](https://docs.oqoqo.ai/guides/ask-oqo-custom-agent.md)).

## Visibility across projects

Custom agents belong to your organization and appear under **Manage → Agents** in every project with a **Custom** tag. A newly added custom agent starts enabled in the project where you created it and disabled in other projects until you turn it on. All projects share the same encrypted environment variable values.

## Connect a custom agent

> [tab] Dashboard
>
> 1. Open **Manage → Agents** and select **Connect custom agent**.
> 2. Enter a unique **Name** (120 characters or fewer) and an optional description.
> 3. Under **Install**, choose **ACP Registry**, **npm package**, **Python package**, **Download**, or **Import JSON**.
> 4. Under **Environment variables**, select **Add variable** for each credential or setting the agent needs.
> 5. Select **Connect agent**, then run [**Test agent**](#test-a-custom-agent) on the agent page.

> [tab] CLI
>
> Create a custom agent from the ACP Registry or a manual package, declaring environment variable names only:
>
> ```bash theme={null}
> oqoqo project ai apply --operation '{"action":"create_custom_agent","name":"Gemini CLI","source":{"kind":"registry","registryId":"gemini"},"env":["GEMINI_API_KEY"]}' --output json
> ```
>
> ```bash theme={null}
> oqoqo project ai apply --operation '{"action":"create_custom_agent","name":"My agent","source":{"kind":"manual","install":{"kind":"npm","package":"my-agent@1.4.2"},"args":"acp --verbose"},"env":["MY_AGENT_API_KEY"]}' --output json
> ```
>
> The output returns the `acp-…` key and a dashboard link for each secret value ([`oqoqo project ai apply`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-apply)).

> [tab] Plugin (MCP)
>
> Ask your AI assistant to connect the custom agent. It calls **Create custom agents and change agents, models, and Oqoqo AI settings** ([`manage_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#manage_project_ai)) and returns dashboard links for any secret values.
>
> ```text theme={null}
> Add the Gemini CLI from the ACP Registry as a custom agent in Oqoqo.
> ```

The **Connect custom agent** form collects the agent name, install source, environment variables, and **Advanced** options such as a custom skills directory.

![Connect custom agent form with the name, the Install choices, the ACP Registry search list, and Environment variables.](https://mintcdn.com/oqoqo/YO1ysyUx4SPwBwoz/images/screenshots/custom-agent-new-light.png?fit=max\&auto=format\&n=YO1ysyUx4SPwBwoz\&q=85\&s=6c1447de6c27face4b5686e9637583e7)

### Install sources

Oqoqo installs the agent in the run's machine and communicates with it over **ACP on standard input and output (stdio)**. Your **Arguments** (or `agent.json` entrypoint) must start the agent in ACP server mode rather than an interactive terminal UI.

| Source | What to provide | Notes |
| - | - | - |
| **ACP Registry** | Search by name, ID, or description | Pins the current registry version; **Update to** appears when a newer version ships. Paused registry entries are hidden. |
| **npm package** | Exact package version (`my-agent@1.4.2`) and **Arguments** | Requires an exact version so experiments stay comparable. |
| **Python package** | Exact package version (`my-agent==1.4.2`) and **Arguments** | Installed with `uv`; requires an exact version. |
| **Download** | HTTPS **Download URL** (Linux x64), **SHA-256** (64 hex chars), **Command**, and **Arguments** | Extracts `.tar.gz`, `.tgz`, `.tar.bz2`, and `.zip` archives and runs **Command** relative to the extracted archive; runs any single downloaded file directly by that **Command** name. |
| **Import JSON** | Paste or upload an ACP Registry `agent.json` | Requires `id`, `name`, `version`, and a Linux x64 distribution (`npx`, `uvx`, or `linux-x86_64` binary with SHA-256). |

* For recognized **ACP Registry** entries (such as **goose**), Oqoqo pre-populates common variable names with empty values, and shows an **Also built in** tag when Oqoqo already includes a built-in version.
* Separate **Arguments** with spaces and quote any argument that contains spaces.
* In the CLI and plugin, npm package, Python package, and Download all use `source.kind: "manual"`.

## Set environment variables

ACP agents cannot complete browser sign-ins during runs, so supply API keys, tokens, and default model configuration (for example `GOOSE_MODEL` when **Test agent** runs with **Agent default**) through environment variables (up to 64).

* Variable names must start with a letter and use uppercase letters, digits, and underscores. Reserved runtime names such as `PATH`, `HOME`, and `OQO_*` are blocked.
* You cannot rename a variable after saving it, so remove and re-add it instead.
* Values stay secret: you enter them only in the dashboard, and Oqoqo scrubs them from saved run artifacts.

The ACP Registry does not declare required variables, so most agents still need their provider API key even when the initial list is empty.

## Test a custom agent

**Test agent** verifies that the custom agent installs, starts, and responds to a prompt. Each test uses **1 run** (**Agent check** task, **Agent default** model, **Raw Agent** treatment, with Evals and Insights off). Test runs do not appear in **All experiments** or on **Home**, though the CLI and plugin can list them. Before testing in another project, confirm the agent is enabled there and **Agent default** is turned on.

```mermaid theme={null}
flowchart LR
  V["Needs values"] -- "Set secrets" --> N["Not tested"]
  N -- "Test agent" --> T["Testing"]
  T -- "Answers" --> R["Ready"]
  T -- "Fails" --> A["Needs attention"]
  A -- "Fix & retest" --> T
```

> [tab] Dashboard
>
> Open the custom agent's page and select **Test agent** under **Status**. Select **Open test run** to watch it live.

> [tab] CLI
>
> ```bash theme={null}
> oqoqo project ai test-agent acp-0123456789abcdef
> oqoqo experiments watch <launch-id>
> ```
>
> Pass `--attempt-id` to retry without spending a second run ([`oqoqo project ai test-agent`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-test-agent)).

> [tab] Plugin (MCP)
>
> Ask your AI assistant to test the custom agent. It calls **Test a custom agent** ([`test_custom_agent`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#test_custom_agent)) and polls **Get launch status** ([`get_launch_status`](https://docs.oqoqo.ai/mcp-server/experiments#get_launch_status)). Because testing uses one run, approve the call in your host.

### Statuses

When multiple states apply, the highest status in this list takes precedence:

| Status | Meaning |
| - | - |
| **Needs values** | One or more environment variables have no value yet |
| **Testing** | A test run with current settings is in progress |
| **Needs attention** | The latest run with current settings failed before answering |
| **Not tested** | No run with current settings has finished yet |
| **Ready** | The latest run with current settings answered |

Any finished experiment run with current settings updates the status (unless cancelled or the agent refused MCP servers). Saving any settings or secret change resets the status to **Not tested**.

### Fix a failing test

When an agent shows **Needs attention**, **Status** highlights the failed step, the agent's error message, and a link to fix it.

| Failure | Fix |
| - | - |
| Agent refused MCP servers | Launch only with treatments that have no MCP servers |
| Agent asked to sign in | Add the provider credentials shown in the error under **Environment variables** |
| Agent rejected the model | Disable that model under **Models**, or use **Agent default** |
| SHA-256 mismatch or install error | Verify the Download URL, SHA-256, or exact npm/Python package version |
| Incompatible Node.js version | [Update Node.js in the machine setup](https://docs.oqoqo.ai/reference/troubleshooting.md#a-run-could-not-start-because-of-its-machine) or use another agent |
| Registry install or startup failure | Retry **Test agent**, switch to an npm/Python/Download install, or inspect the agent log |

## Confirm the agent's models

Oqoqo discovers a custom agent's models from actual runs rather than a **Refresh models** button. **Agent default** lets the agent pick its own default model (or read a variable like `GOOSE_MODEL`). After a run completes, any models the agent reported appear in **Models** so you can toggle them on for launches.

| Model state | Meaning |
| - | - |
| **Current** | Reported by a run with current settings (or **Agent default**) |
| **Not confirmed** | Reported under older settings; test again to confirm |
| **Unavailable** | Rejected by the agent under current settings |

Custom agents do not support effort levels.

## Run a custom agent in an experiment

**New experiment** lists custom agents below built-in agents with a **Custom** badge. Select **New experiment** from a ready custom agent's page to pre-select it alongside built-in agents on **Agent default**.

* Oqoqo blocks the launch if variable values are missing or if a treatment includes MCP servers the agent refuses (`mcpServersRefused`).
* Custom agents cannot [turn off individual MCP tools](https://docs.oqoqo.ai/experiments/treatments.md#turn-off-mcp-tools) (`mcp_tools_off_unsupported`), so leave all server tools enabled when launching a custom agent with MCP servers.
* Running a custom agent that is **Not tested** or **Needs attention** shows a warning notice but allows launching.
* By default, Oqoqo copies treatment skills into `~/.agents/skills`; change **Advanced → Skills folder** on the agent page if the agent reads a different path.

## Read a custom agent's results

Custom agent results appear on **Matrix** (with **Comparisons**), **Frictions**, and **Runs** alongside built-in agents, with a few notes:

* **Agent default** displays the reported model once known (for example, **Agent default: opus**).
* If a run hits a token limit or stops early, a note explains why; Evals still score whatever work finished.
* If the agent fails before answering, the run is marked errored, excluded from pass rates, and does not consume your [run balance](https://docs.oqoqo.ai/setup/billing.md).
* Token and cost columns remain blank unless the custom agent reports them.

### Where a cost comes from

Every cost displays its source on the run page and on hover in **Runs**:

| Label | Meaning |
| - | - |
| **Reported by the agent** | Total reported directly by the agent |
| **Estimated from list prices** | Calculated by Oqoqo from token counts and published prices |
| No label | Cost source was not recorded |

If a Matrix cell mixes runs with different cost sources, it displays **Mixed sources** with separate averages instead of combining them.

## Change or remove a custom agent

Update settings anytime on the agent page and select **Save changes** (past runs keep their launch settings). Wait for any running test or experiment to finish before removing an agent.

> [tab] Dashboard
>
> Open the custom agent's page, open **⋯**, select **Remove**, and click **Remove agent** to confirm.

> [tab] CLI
>
> ```bash theme={null}
> oqoqo project ai delete --operation '{"action":"remove_custom_agent","agentKey":"acp-0123456789abcdef","expectedRevision":4}'
> ```
>
> Pass the current `revision` from `oqoqo project ai inspect --view agents` (add `--yes` with `--output json` or outside a terminal) ([`oqoqo project ai delete`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-delete)).

> [tab] Plugin (MCP)
>
> Ask your AI assistant to remove the custom agent. It calls **Remove custom agents or provider connections** ([`delete_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#delete_project_ai)) after your approval.

Removing a custom agent deletes its stored environment secrets and hides it from new experiments while preserving past run names.

## From the CLI and the plugin

| Task | CLI | Plugin (MCP) |
| - | - | - |
| List custom agents, status, variables, and fixes | [`oqoqo project ai inspect --view agents`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-inspect) | [`inspect_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#inspect_project_ai) |
| Search the ACP Registry | [`oqoqo project ai inspect --view agent_registry`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-inspect) | [`inspect_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#inspect_project_ai) |
| Create or update a custom agent | [`oqoqo project ai apply`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-apply) | [`manage_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#manage_project_ai) |
| Test a custom agent | [`oqoqo project ai test-agent`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-test-agent) | [`test_custom_agent`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#test_custom_agent) |
| List or toggle a custom agent's models | [`oqoqo project ai inspect --view models`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-inspect), [`apply`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-apply) | [`inspect_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#inspect_project_ai), [`manage_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#manage_project_ai) |
| Read stop reason and failures | [`oqoqo runs trajectory`](https://docs.oqoqo.ai/cli/runs#oqoqo-runs-trajectory), [`profile`](https://docs.oqoqo.ai/cli/runs#oqoqo-runs-profile) | [`get_run_trajectory`](https://docs.oqoqo.ai/mcp-server/runs-and-results#get_run_trajectory), [`get_run_profile`](https://docs.oqoqo.ai/mcp-server/runs-and-results#get_run_profile) |
| Remove a custom agent | [`oqoqo project ai delete`](https://docs.oqoqo.ai/cli/agents-and-models#oqoqo-project-ai-delete) | [`delete_project_ai`](https://docs.oqoqo.ai/mcp-server/agents-and-setup#delete_project_ai) |

## Next steps

> [card] [Oqoqo AI defaults](https://docs.oqoqo.ai/setup/oqoqo-ai.md)
>
> Step 5: choose models for Evals, Insights, and Ask Oqo.

> [card] [Run Ask Oqo as a custom agent](https://docs.oqoqo.ai/guides/ask-oqo-custom-agent.md)
>
> Evaluate Ask Oqo using a pinned CLI download.

> [card] [Compare agents and models](https://docs.oqoqo.ai/guides/compare-agents-and-models.md)
>
> Compare your custom agent against built-in agents.

> [card] [Launch an experiment](https://docs.oqoqo.ai/experiments/launch.md)
>
> Select your custom agent in an experiment.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.