Sample report · recorded replay

A schema regression, caught before merge.

This is a real report from RunLedger's demo agent with an intentionally stricter output schema. Explore the failed case and its recorded tool trace below. No account or API key is needed.

Failed check: required field missing

The schema now requires next_steps, but the agent still returns only category and reply. RunLedger exits with code 1. When configured as a required CI check, this failure blocks the merge.

Cases

Case Status Wall (ms) Tool Calls Tool Errors

Case detail

Reproduce this failure

This sample uses the repository's demo_repo suite and regression-schema fixture. Recorded tool responses are replayed locally; no model API is called. Timings describe this sample run, not a performance benchmark.

git clone https://github.com/runledger/Runledger.git
cd Runledger
python -m pip install -e .
cd demo_repo
python -c "from shutil import copyfile; copyfile('regressions/regression-schema/evals/demo/schema.json', 'evals/demo/schema.json')"
python -m runledger run ./evals/demo --mode replay

The final command should exit with code 1 and report the missing next_steps field. Open report.html in the artifacts directory printed by the CLI. These commands intentionally change the demo schema in your new clone.

Try your first replay View the demo source Back to RunLedger