What a rubric is
A rubric is a set of criteria that determine whether a trial passed or failed. Each criterion is a testable assertion about the agent’s output or behavior. Oqoqo evaluates every criterion independently per trial, so you can see exactly which aspects of the task succeeded and which didn’t. You write the rubric in plain language. You don’t need to know how LLM-based evaluation works under the hood — Oqoqo rewrites your criteria to follow best practices before scoring begins.Your rubric is held constant across all trials in an experiment, so results are directly comparable.
Writing a rubric
Write each criterion as a testable assertion — something that is unambiguously either true or false given the agent’s output. Vague or compound criteria tend to produce noisy results. Tips:- One assertion per criterion. Don’t combine two checks into one sentence.
- Focus on observable outputs. What did the agent return, write, or do?
- Avoid subjective language like “good” or “appropriate” — phrase it so a reviewer could verify it mechanically.
Returns requested shape
The output matches the expected data structure (e.g., a JSON array).
Preserves order
Items in the output appear in the same order as the input.
Reports cache state
The agent’s answer includes the current cache status.
No unhandled exceptions
The agent completed the task without throwing an unhandled error.
Completes within cost budget
Total token cost for the trial is below the defined threshold.
Auto-rewrite
When you submit a rubric, Oqoqo rewrites it to follow industry best practices for LLM-based evaluation. This means your criteria are reformulated to reduce ambiguity, improve scoring consistency, and minimize false positives. You don’t need to be an eval expert to get high-quality results. Write what you want the agent to do — Oqoqo handles the translation into a robust eval prompt.You can review the rewritten rubric before running your experiment. If a criterion produces unexpected results, adjusting the original wording and re-running is often enough to fix it.
Validators
Alongside rubrics, you can add validators — programmatic checks for deterministic assertions. Use validators when the check is mechanical and doesn’t need LLM judgment. Good candidates for validators include:- Schema shape — confirm the output matches a JSON schema
- Output format — check that a file was written in the right format
- Exact value match — assert a specific field equals a known value
- Presence of required fields — verify all required keys exist in the response
Viewing results
Rubric results appear in the Evals tab for each trial. Each criterion shows:- ✓ Pass — with supporting context from the trial output
- ✕ Fail — with the reason the criterion was not met
Iterating on rubrics
If a criterion catches too many false positives or misses real failures, update it and rerun. Oqoqo re-evaluates all trials against the updated rubric, so you don’t have to re-execute the experiment.Identify the noisy criterion
Look at trials where the result seems wrong — pass when it should fail, or fail when it should pass.
Update the criterion
Rewrite the assertion to be more precise. Add context, tighten scope, or split a compound criterion into two.