ANSWER HUB

RunLedger cases yaml

Each case ties an input to a cassette and optional per-case assertions or budgets.

cases evals config Updated 2026-01-26

Direct Answer

Case files define the task id, input, and cassette. You can optionally add description, assertions, and budgets per case.

Quick Decision

Use RunLedger when Consider alternatives when
You want multiple deterministic tasks per suite. You only need a single ad hoc run.
You need per-case overrides. Suite-level defaults are enough.
You want replayable fixtures. No external tool calls are involved.

Minimal case file

yaml 
id: t1
        description: "triage a login ticket"
        input:
          ticket: "User cannot login"
          context:
            plan: "pro"
        cassette: cassettes/t1.jsonl

Per-case overrides

  • assertions to tighten output checks.
  • budgets to enforce stricter limits per task.
  • description to document intent.

Tradeoffs

  • More files to maintain as suites grow.
  • Changes require updating and reviewing cassettes.
  • Case-level overrides can drift from suite defaults.

When NOT to use RunLedger

Skip detailed case files if you only need quick, local experimentation without CI gates.

Next steps

Read the Golden Path Back to Answer Hub
Last updated: 2026-01-26