Skip to main content
Keep a fixed benchmark running after product updates—or automatically on a schedule—to catch drops in pass rate and newly introduced frictions early.

Prerequisites

Steps

1

Freeze and tag your benchmark tasks

Choose a stable task suite and avoid editing task prompts unnecessarily so trend comparisons stay apples-to-apples. Tag them (for example, regression) for quick filtering.
2

Run your baseline experiment

Launch the initial experiment to establish your reference pass rates and frictions.
3

Launch again after product changes

After shipping updates to your product, tools, or dependencies, open the baseline experiment and select Launch again (uses latest task and tool versions; select Edit first to swap models).
4

Schedule recurring runs for drift

To catch upstream model updates or external API changes automatically, select Schedule, pick Repeat, time, and Time zone, review the Uses run forecast, and select Create schedule. For team-owned schedules, prefer shared organization providers and API-key headers over personal integrations so teammate edits do not pause the schedule for review.
5

Compare Matrix pass rates across runs

Open your schedule under Automate → Schedules to inspect its Experiments and Runs tabs, and compare Matrix pass rates against earlier runs.
6

Sort frictions by First seen

Open the schedule’s Frictions tab (or All frictions) and sort by First seen to isolate newly introduced problems immediately.

Get notified

Clean scheduled experiments send no completion alert. You receive in-app and email notices (controlled by Experiment updates under Profile → Notifications) only when something fails or needs attention:

Interpret the results

From the CLI and the plugin

To gate builds on pass-rate drops in CI (schedules only alert on run or Evals/Insights errors), trigger oqoqo experiments rerun from your pipeline (Run an experiment in CI; pass --versions experiment to keep original task versions).

Next steps

Schedules

Cadence, time zones, pause rules, and run forecasts.

Find and fix frictions

Turn new frictions into verified fixes.

Notifications

Control email and in-app schedule alerts.

Tags

Tag regression tasks and experiments.