Your AI assistant picks a skill from your request. To make sure it uses one, name the skill in your request, for example “Use product-eval-results to compare the treatments in my latest experiment.”
What every skill does
- Stays in one project. The AI assistant calls
get_project_contextfirst and passes the sameproject_idon every call. When your organization has more than one project and you don’t name one, it says which project it will use and continues. Name the project in your request to be sure. - Stays in one organization. Every change sends
expected_organization_id. If someone signs in to another organization while the AI assistant works, the change fails withorganization_scope_changedand changes nothing (Organization check). - Leaves secrets to you. The AI assistant declares each key and OAuth connection by name and gives you the dashboard link where you enter it (Secrets stay in the dashboard).
- Keeps working. When the AI assistant has a question, it says what it will assume, asks, and continues. Reply right away if you want a change.
- Reads the product docs. For anything about how Oqoqo works, the skills read
https://docs.oqoqo.ai/llms.txtand then the pages they need.
product-eval-authoring
Builds a complete eval set in an Oqoqo project from a product’s documentation. Your AI assistant uses it when you point at a product and ask for an eval set, or ask what you should evaluate about your product. What it does1
Sets the scope
It covers the area of the product you name. If you name none, it covers the product broadly, across its documented user jobs.
2
Reads the product's docs
It reads an index (such as
llms.txt) and five or six pages, and it looks for the product’s MCP server, CLI, SDK, and published skills on the product’s own domain.3
Picks task kinds
It writes at least one task of each kind the product supports: integration code, live interaction (only when the product has an account you can sign up for), and knowledge.
4
Posts a plan
Before it creates anything, it posts the scope, each task and its kind, the machine, files, tools, and treatments, and the runs for one trial and for two trials (tasks × treatments × trials, for each agent). Then it continues.
5
Writes the resources
It checks the write schema, reads the catalog, and creates the machine, files, tools, treatments, and tasks, in that order, because each one points at the one before it.
6
Hands it over
It runs preflight and gives you the link for each key or connection you still have to enter.
get_project_context, inspect_catalog, inspect_project_ai, author_resource, bulk_author_resources, create_upload, complete_upload, get_upload, manage_resource, preflight_experiment, get_setup_links. With the CLI, it uses oqoqo resources apply, oqoqo resources bulk-apply, and oqoqo uploads create.
What you get: a finished eval set in your project, and a list of what is left for you:
- Enter the keys and connections at the links it gives you.
- Turn on the agents you want under Manage → Agents.
- Run preflight again.
- Launch with at least two trials.
- It can create MCP servers, CLIs, and SDKs as tools, but it can’t sign in to them or enter their keys.
- Preflight doesn’t check a CLI or SDK variable that isn’t marked required, so the AI assistant lists each one with the page where you enter it.
- It uses
product-eval-tasksto write the tasks,product-eval-setupfor the machine and data, andproduct-eval-resultswhen you ask it to launch.
product-eval-tasks
Turns a product’s documentation into tasks with fair rubrics. Your AI assistant uses it when you point at a product’s docs, README, or skills repository and ask for tasks, or when you ask it to review or fix a rubric. What it does- Starts from the example prompts that the product’s own docs publish, because they are the work the product claims to support.
- Lists the product’s surface areas and gives each one at least one task. The number of tasks comes from the product, not from a target or from your run balance.
- Puts one cheap task first that only reports what the agent can see, so you know the tools reached the run.
- Writes each prompt the way a user would ask, and doesn’t name the tool, the steps, or the file to change.
- Builds each task so that an agent with your tools and an agent without them can get different results.
- Writes two to five criteria for each rubric. Each criterion checks one thing that the judge can see in the trajectory or the workspace, and it doesn’t require one function, endpoint, or command name.
product-eval-authoring saves them to your project.
What you get: tasks ordered from simple to complex, each with its kind, what it needs (files, a machine, an account), and a rubric. It suggests that you run one task on one agent first and read its trajectory.
Example prompts: “Write eval tasks for our CLI from its README.” · “Review the rubric on task_08. Is it fair?”
Notes
- The judge reads only the trajectory and the files left at the end, never the product’s own account. A live interaction task therefore asks the agent for a short written report, and the rubric grades that report.
- Rubric rules are on Tasks.
product-eval-setup
Builds the machine, files, treatments, and test data that an experiment runs on. Your AI assistant uses it when you set up an experiment’s environment, ask what belongs on a machine or in a treatment, or seed a product account with test data. What it does- Keeps the product off the machine. The machine attaches to the task, so the baseline gets it too. Every part of the product goes into a treatment as a skill, MCP server, CLI, or SDK.
- Gives each tool its credential in one of three ways: the agent creates its own account during the run, a person enters a key or connects OAuth in the dashboard, or the tool needs none. It marks each key a task can’t run without as required, so that preflight blocks the launch until the key is set.
- Plans for runs that happen at the same time. It prefers tasks that only read from a shared account, and gives each task that writes its own target.
- Seeds data so that a shortcut gives the wrong answer, then checks the data after it lands.
- Makes the files and the data agree, for example stack traces that point at real lines in the task’s files.
get_project_context, get_setup_links (product-eval-authoring calls author_resource and bulk_author_resources to save them).
What you get: a summary that says how each tool gets its credential, and each key you still have to enter, the page where you enter it, and the task it blocks.
Example prompts: “Set up a machine for these Next.js tasks.” · “Should the Acme CLI be on the machine or in a treatment?”
Notes
- It suggests one cheap run on a task that only reports what the agent can see, before you launch the whole experiment.
- Machines are on Machines and files, and treatments on Treatments (Toolsets).
product-eval-results
Launches an experiment, follows it to the end, and turns its results into conclusions that hold up. Your AI assistant uses it when you ask to launch or rerun an experiment, check on one, compare treatments, explain why runs passed or failed, find out whether agents used a tool, find frictions, or count anything across runs. What it does- Launches once. It says the run count (tasks × treatments × agents × trials) before it launches, suggests at least two trials, runs
preflight_experiment, and launches with an ID it keeps, so that a retry doesn’t spend the runs again. - Follows the experiment. It waits in
get_experiment_statusuntil the experiment finishes, then reads the final runs. It says which Evals and Insights results are still pending. - Reads the Brief first. For “What did the experiment show?”, it says the Brief’s sentence and each result as written, and lists the frictions seen with the side that did worse, with both sides’ counts.
- Compares fairly. Before it calls one treatment better, it checks Comparisons, the runs behind each pass rate, whether the agents used the tool, and each task.
- Runs a likely difference again once, after you say to go ahead, and says the run count first.
- Explains a run with evidence. It starts from the verdict and reads only the steps of the trajectory that it needs. When a criterion asked for the wrong thing, it suggests a rubric fix and Rerun evals on the same recorded work.
- Schedules. It previews a schedule and tells you the next launches and the runs in the next 30 days before it creates one.
What you get: a report in which every number comes from a tool result, with the public IDs of the experiment, runs, and trials it cites. It says what is still pending, what was too small to decide, and what to read next.
Example prompts: “Launch my Stripe evals experiment again with two trials.” · “Did Stripe CLI do better than Raw Agent in exp_8f2c41d0, and did the agents use it?”
Notes
- A launch has no approval step on the server. Turn on tool approval in your host so that you approve each launch.
Install, update, or remove the skills
The skills come with the plugin, and they update with it. Install, update, and remove steps for each host are on Install the Oqoqo Plugin. To use the skills with a direct MCP connection or the CLI, see Add the skills without the plugin. Each skill’s full text is aSKILL.md file in the public Oqoqo-Inc/agent-plugin repository.
Related
Install the Oqoqo Plugin
Install the plugin in your host and sign in.
Plugin (MCP) tools
Index of all 83 tools by area and matching CLI commands.
Tasks and rubrics
Write tasks and rubrics in the dashboard.
Comparisons
Whether a difference holds up, and the Brief.

