Skip to main content
After an experiment runs, the results are organized as runs and trials. Knowing the difference is the key to reading a scorecard correctly.

​
Run and trial

A run is one task’s execution under one treatment. A trial is one repeat of that run. A run groups its trials, and the best trial represents the run in lists and headlines. You open a run to see its trials underneath. You repeat trials because agents are not deterministic. The same task can pass once and fail the next time for reasons that have nothing to do with your change. Running several trials per treatment turns a single lucky or unlucky result into a pass rate you can trust.
A run's evals: a pass or fail for each requirement, with steps, tokens, cost, and duration.

​
Pass rate

Pass rate is the fraction of trials that passed. Oqoqo reports it with a 95% confidence interval, so you see not just the rate but how sure it is. A pass rate of 8 out of 10 with a wide interval is weaker evidence than 80 out of 100 with a narrow one. More trials narrow the interval.

​
Lift

Lift is the change in pass rate from a treatment, measured against the raw agent. Positive lift means the treatment helped. Lift is what tells you whether your skill, MCP server, or other change made agents more successful.

​
Compare

Use Compare to line up agents and treatments side by side. You see pass rate, lift, cost, tokens, and duration together, so you can pick the agent, model, or treatment that does the job well and cheaply.

​
Metrics

Every run records its metrics: steps taken, tokens used, cost, and duration. Metrics tell you how hard an agent worked, not just whether it passed. Two agents can both pass a task while one uses half the tokens. Lower steps and tokens for the same result mean a treatment or interface change made the agent more efficient.