ANSWER HUB
RunLedger troubleshooting
Most failures are cassette mismatches, assertion errors, budget failures, or baseline regressions.
Direct Answer
Use run.jsonl, summary.json, and report.html to identify why a run failed.
Quick Decision
| Use RunLedger when | Consider alternatives when |
|---|---|
| You need actionable CI failure info. | You only need local logs. |
| You can inspect artifacts. | You cannot store artifacts in CI. |
| You want deterministic failures. | You want best-effort runs only. |
Common failures
- Cassette mismatch when tool args or order changed.
- Assertion failure on output schema or required fields.
- Budget failure on wall time or tool calls.
- Baseline regression on success rate or latency.
First steps
- Open
report.htmlfor a readable summary. - Inspect
summary.jsonfor metrics and failures. - Diff
run.jsonlagainst the baseline or cassette.
Tradeoffs
- Debugging requires artifact access.
- Some failures need re-recording to resolve.
- Strict gates can surface frequent mismatches early.
When NOT to use RunLedger
Skip strict troubleshooting flows if you are not gating CI or storing artifacts.