Skip to content
Caplo DocsDocs

Write and run evals

An eval case is a customer message and what should happen when the agent answers it. Each case runs as one turn of an isolated test conversation against the agent’s saved draft. Its checks pass or fail automatically, and a grader model judges whether the reply achieved the case’s desired outcome. By default a workspace publishes an agent only when every case passes. For the ideas behind this, see Evaluations.

Case fields

Section titled “Case fields”
YAML field Console label Required Description
name Case name Yes Up to 80 characters, unique within the agent. The CLI matches cases by name.
input Customer message Yes The one message the customer sends. Up to 8,000 characters.
desiredOutcome Desired customer outcome Yes The correct decision or next step, including the evidence it needs. A grader model judges every run against it. Up to 4,000 characters.
answerContains Answer must contain No A phrase the reply must include.
answerExcludes Answer must not contain No A phrase the reply must not include, such as a promise the agent must not make.
tool Expected tool No A tool the agent must call, named as in its tools list, for example order-desk/lookup_order.
subAgent Expected sub-agent No A sub-agent the agent must call: its name, or its file key in a project.
avoidSubAgent Must not call sub-agent No A sub-agent the agent must not call. Must differ from subAgent.
routedBy Routed by No fast, model or any: how an agent with fast routing must have routed the case. See Routing cases.
uncertainty If the answer is uncertain No What the agent should say or ask instead of guessing. A note for reviewers; runs do not check it.
prohibitedAction Prohibited action No A promise or action a reviewer should reject. A note for reviewers; runs do not check it.

Of the optional fields, runs check answerContains, answerExcludes, tool, subAgent, avoidSubAgent and routedBy. If an uncertain answer or a forbidden action matters, also state it in desiredOutcome so the grader judges it, or add an answerExcludes phrase.

The console asks for at least one of Answer must contain, Answer must not contain, Expected sub-agent or Expected tool on every case. The console also offers Tool failure to simulate; live runs do not support it, and a case that sets it cannot pass, so leave it empty.

An agent can have up to 100 cases.

What every run checks

Section titled “What every run checks”

Besides the case’s own fields, every case is checked for these:

Check Passes when
Turn completed The turn ended with a reply.
Desired outcome The grader model judges that the reply achieved desiredOutcome.
No business action The test did not carry out a write in your systems.
Within limits The turn stayed within its delegation and tool limits.

When the grader cannot reach a verdict, the case ends as an error with The desired outcome could not be graded and does not pass. Run it again.

Add cases

Section titled “Add cases”
  1. Open the agent and select the Test tab. It lists the agent’s cases with the result of the latest run.
  2. Select Add case, fill in the fields, and select Save case. Cases save immediately; they are not part of the draft.
  3. Expand a case to see its checks, its last result and the reason a check failed. Edit and Remove change it.

Routing cases

Section titled “Routing cases”

An agent with fast routing publishes only when the run proves routing works. Add cases that set routedBy and name a direct sub-agent in subAgent:

evals:
  - name: Routes an order question
    input: Where is my order SO-1002?
    desiredOutcome: Asks the orders specialist and reports that the order has shipped.
    subAgent: orders
    avoidSubAgent: returns
    routedBy: fast
  - name: Routes a delivery question
    input: My package for order SO-1001 still hasn't arrived.
    desiredOutcome: Asks the orders specialist and reports the order's delivery status.
    subAgent: orders
    routedBy: fast

routedBy: fast passes when fast routing picked the sub-agent, model when the model did, and any when either did. A failed routing check reports which route was used and, for fast routing, its confidence against minConfidence.

While the workspace requires evals, each direct sub-agent needs a set number of passing routing cases: two by default (see Publication requirements). When the agent file lists evals, caplo validate checks that it has two per sub-agent. The workspace must also have Fast decisions turned on; see Fast decisions.

Run evals

Section titled “Run evals”

Runs start from the publish dialog. Select Publish agent, then Run tests under Agent evals. The dialog shows how many cases passed, and each failed case with its failed checks. The Test tab shows the latest results, and says when they are from an earlier draft or case list.

Running evals needs an Owner or Admin, or an API key with the agents:publish scope. A run takes up to about 10 minutes. Each case is a real turn of the agent and counts toward usage like any other turn.

Publication gate

Section titled “Publication gate”

Publishing runs the gate against the exact saved draft and the current case list:

  • Every case must pass in a run of that draft and those cases. Editing either one needs a new run.
  • Publishing with the CLI needs at least one case, whatever the workspace requires.
  • A release does not pin eval cases. When only the cases change, caplo publish runs them against the released draft and keeps the release.
  • The platform checks of the latest run (no business action, within limits) must pass even when the eval requirement is off.

Publication requirements

Section titled “Publication requirements”

An Owner or Admin sets the workspace’s rules in Settings > General > Agent publication requirements. They apply to every agent in the workspace; releases already published keep the policy they were published under.

Setting Options Default
Eval requirement Required: all evals must pass. Disabled: results do not block publication. Required
Passing routing cases per sub-agent 1 to 10 2

With the requirement disabled, eval results are advisory and fast routing coverage is not enforced, but fast routing still needs Fast decisions turned on.

AI prompt

Write eval cases for a Caplo agent

Adds focused eval cases to an agent file in this repository and runs them with caplo eval until they pass.

Open in ClaudeOpen in ChatGPT
Show prompt
Goal: give the Caplo agent in <AGENT_FILE> (for example agents/support.yaml) a set of eval cases that cover its main jobs, its tools, its sub-agents and the mistakes it must not make, then run them until they pass.

Read these pages first:
- https://docs.caplo.si/agents/evals.md
- https://docs.caplo.si/agents/sub-agents-and-routing.md (if the agent has sub-agents)
- https://docs.caplo.si/reference/agent-yaml.md

Steps:
1. Read <AGENT_FILE>: its instructions, `tools`, `knowledgeBases`, `subAgents` and `routing`, and any existing `evals`. Read the tool packages it uses under the project's tools directory to learn what each tool returns.
2. Draft 5 to 15 cases under `evals:` as a YAML list (the CLI form, no `id` fields). Each case has a unique `name` (up to 80 characters), one customer message in `input`, and a `desiredOutcome` that states the correct decision or next step and the evidence it needs. A grader model judges every run against `desiredOutcome`, so make it specific and checkable.
3. Add automatic checks where they fit: `answerContains` or `answerExcludes` for a phrase that must or must not appear, `tool` (named as in the agent's `tools` list) for a tool the agent must call, `subAgent` and `avoidSubAgent` (file keys) for routing. Do not rely on `uncertainty` or `prohibitedAction`; runs do not check them, so put those expectations in `desiredOutcome` or `answerExcludes`.
4. Cover: each main task, each tool, a question the agent must decline or cannot answer, a request it must not fulfil (for example promising a refund), and a request for a person if `actions` includes `transfer`.
5. If `routing.mode` is `fast`, add at least two cases per sub-agent with `routedBy: fast` and `subAgent` naming it.
6. Validate offline: `caplo validate` in a project, or `caplo agent validate -f <AGENT_FILE>` for a single file. Fix every problem.
7. Show me the cases. After I approve them, run `caplo eval agent/<AGENT_KEY>`. This pushes the agent's draft and replaces its eval cases in the workspace, so ask before running it.
8. For each failing case, decide with me whether the agent or the case is wrong. Fix the instructions or the case, then run `caplo eval agent/<AGENT_KEY>` again.

Verify:
- `caplo eval agent/<AGENT_KEY>` exits with status 0 and prints ok for every case.

Rules:
- Never paste secrets, API keys or customer data into cases or this chat. Use made-up order numbers and names.
- Do not run `caplo publish` or `caplo agent publish`; ask me before anything that publishes.
- Do not weaken a case only to make it pass; tell me when a case exposes a real problem in the agent.

Next steps

Section titled “Next steps”