ANSWER HUB
RunLedger cassette mismatch
Cassette mismatch means the agent called a tool differently than the recorded cassette.
Direct Answer
Cassette mismatch means the agent made a tool call that does not match the recorded cassette. Fix it by restoring expected behavior or re-recording after confirming the change is intended.
Quick Decision
| Use RunLedger when | Consider alternatives when |
|---|---|
| The change is expected and you want to update fixtures. | The change is unexpected and may be a regression. |
| You can re-record safely against live tools. | You cannot hit live tools right now. |
| Tool args legitimately changed. | Tool args changed due to a bug. |
Common causes
- Prompt or tool registry changes alter tool arguments.
- New tool calls were added or ordering changed.
- Case input changed without re-recording cassettes.
Safe fix workflow
- Run record mode locally to capture updated tool outputs.
- Review cassette diffs and confirm they are expected.
- Replay with the new cassette and update the baseline if needed.
bash
runledger run ./evals/<suite> --mode record
runledger run ./evals/<suite> --mode replay --baseline baselines/<suite>.jsonTradeoffs
- Re-recording can hide real regressions if done blindly.
- Live recording is slower than replay.
- Frequent tool changes require cassette maintenance.
When NOT to use RunLedger
Do not re-record if the mismatch is unexpected. Fix the agent or tool behavior first, then record intentionally.