ANSWER HUB
Update RunLedger cassettes
Update cassettes by re-recording locally and reviewing diffs before promoting baselines.
Direct Answer
Re-record cassettes locally, review JSONL diffs, then replay and update the baseline if the change is intended.
Quick Decision
| Use RunLedger when | Consider alternatives when |
|---|---|
| The behavior change is expected. | The change is unexpected or a regression. |
| You can safely hit live tools. | Live tools are unavailable or unsafe. |
| You can review and approve diffs. | You cannot audit tool outputs. |
Update workflow
- Run record mode locally to capture new tool outputs.
- Review cassette diffs for unintended changes.
- Replay with the new cassette and update the baseline if needed.
bash
runledger run ./evals/<suite> --mode record
runledger run ./evals/<suite> --mode replay --baseline baselines/<suite>.jsonWhen to update
- You changed prompts or tool registry.
- Tools return new fields or formats.
- Case inputs changed materially.
Tradeoffs
- Re-recording can mask regressions if done blindly.
- Live recording is slower and may be rate-limited.
- Requires human review of fixture changes.
When NOT to use RunLedger
Do not update cassettes if the change is unexpected; fix the regression first.