# Production incident imports and replay Turn a captured agent failure into a local, reviewable regression fixture. Import an export from your application, redact it on your machine, inspect explicit business rules, and rerun your agent against the recorded tool responses in order. RunLedger does not upload your incident to runledger.io. The website example is **fictional production-format data**, generated by the same importer and replay engine. It demonstrates a refund committing twice after a timeout. No customer incident is published in the download. ## Install and try the complete workflow Install or upgrade the CLI from PyPI in your Python environment: ```sh python -m pip install --upgrade runledger runledger incident --help ``` Production incident commands require RunLedger **0.3.0 or later** and Python 3.9+. Use a virtual environment if your system Python is managed by your operating system. Existing pipx users can run `pipx upgrade runledger`; new pipx users can run `pipx install runledger`. To try the fictional example, download https://runledger.io/templates/support-agent/runledger-scenario-source.zip and extract it into a separate folder. The example files are separate from the pip-installed CLI. From the extracted folder, with your Python environment active: ```sh python examples/incidents/replay_example.py ``` You do not need to install the source download after installing the CLI with pip. Developers who want an editable source install can use `python -m pip install -e .` in that folder instead. The Python example runner works on Windows, macOS, and Linux without shell JSON quoting. Use fresh output folders on repeated runs. Open `incident-example-imported/report.html` to review the redaction and captured effects. Open `incident-example-replayed/report.html` to see the agent reproduce the failure: **replay MATCHED; business rules FAIL**. The included adapter makes no live provider or model calls. Replace its policy with your application's JSONL adapter before testing your own agent. ## Import your own export ```sh runledger incident import production-export.json --out imported runledger incident inspect imported/incident.json --out reviewed ``` The canonical JSON export has this versioned shape. The minimal example below records a tool-free task. The included refund example shows calls, responses, audit effects, and a captured failure in full. ```json { "schema_version": 1, "input": {"request": "Check my order", "order_id": "order-1042"}, "effects_complete": false, "events": [ {"type": "final_output", "output": {"status": "pending"}} ], "assertions": [ {"id": "status-present", "type": "required_output", "fields": ["status"]} ] } ``` Optional root fields are `incident_id`, `title`, `metadata`, and `source`. They are discarded, and a new opaque incident ID is generated. Export arbitrary supported payloads from your logging system; there is no vendor-specific cloud connector or automatic collection agent in this version. Unknown fields and unsupported formats are rejected instead of silently approximated. ### Map your trace to events Keep events in their recorded order. Tool calls must be serial and paired with one response; concurrent calls and incomplete traces are not supported yet. | Event | Required fields | Meaning | | --- | --- | --- | | `tool_call` | `call_id`, `name`, `args` (object) | Agent request; call ID unique in the trace | | `tool_result` | `call_id`, `ok` (boolean) | Matching response; `result` and `error` optional, default null | | `business_effect` | `call_id`, `effect` (object with `type`) | Independently captured committed action | | `final_output` | `output` (object) | Terminal structured agent result | | `task_error` | `message` (string) | Terminal recorded agent error | Event `timestamp` fields are accepted and discarded. Call IDs are normalized to `c1`, `c2`, and so on. A terminal event must finish the trace after all responses. An effect can arrive after its response, but must reference an earlier call and appear before the terminal event. Capture an ambiguous timeout as a failed tool response, not as an absent response. Do not manufacture missing results or state. For a refund, an effect might be: ```json {"type":"business_effect","call_id":"refund-attempt-1","effect":{"type":"refund","order_id":"order-1042","refund_id":"refund-1","amount_cents":4900,"currency":"USD"}} ``` Only set `effects_complete: true` when your export includes the complete relevant audit ledger. A successful response does not prove an effect committed; a timeout does not prove it failed. RunLedger never synthesizes effects from tool responses. Without a complete ledger, effect-count checks are **inconclusive** and fail the gate. Completeness is your exporter declaration, not independently verified by RunLedger. ### Import existing RunLedger JSONL logs ```sh runledger incident import run.jsonl --format runledger --case refund-case --rules rules.json --out imported-log ``` `rules.json` contains an assertion array like the one above. Choose `--case` when the log contains multiple case IDs. Task start/input, tool events, terminal events, and explicit audit effects are selected; logs and prior assertion/budget results are discarded. Existing ordinary RunLedger logs do not contain a provider audit ledger. Add explicit `business_effect` events in your exporter if needed, and use `--effects-complete` only when you can establish that the ledger is complete. ## Redaction happens locally The importer replaces detected credentials, personal details, opaque identifiers, and freeform text. Repeated values use consistent aliases within an import, so order IDs and idempotency keys can still be correlated. Every import uses a new random secret; the secret and original-to-alias lookup are never exported. Repeated imports intentionally produce different aliases. Payload fields such as amounts, currencies, statuses, and business-rule limits are retained where possible. Protocol structure and known machine error codes are preserved. Unknown error text and freeform requests are removed. Numeric sensitive values become string aliases, which may change type-sensitive behavior. An unrecognized field or encoded/derived copy can contain private data: automated redaction is not a guarantee of anonymity. **Review `incident.json`, `trace.jsonl`, and `report.html` before sharing.** Add application-specific JSON Pointer paths to remove fields the defaults do not know: ```sh runledger incident import production-export.json --redact /input/company --redact /events/*/args/internal_reference --out imported-private ``` Pointers address the normalized incident's `input`, `events`, and `assertions`; `*` matches one level. A missing path or a path that would remove protocol structure is rejected. Use only paths present in your export. The report records value-free counts and warnings, not raw secrets, original field names, or mappings. Raw source files remain where you supplied them; no raw copy is put in artifacts. Imported bundles are data only and never choose or execute an agent command. ## Declare rules explicitly | Rule | Fields | Check | | --- | --- | --- | | `effect_count` | `effect_type`, `max`, optional `min` and `group_by` | Number of captured effects, optionally per top-level effect field | | `output_equals` | `path`, `value` | Exact typed value at an output JSON Pointer | | `required_output` | `fields` | Required top-level output keys are present | | `max_tool_errors` | `max` | Recorded tool error count is within the limit | Each rule has a unique `id`. At least one explicit rule is required. No code, regular expressions, templates, or executable assertions are loaded from imports. For example, this rule detects the duplicate refund in the public example: ```json {"id":"one-refund-per-order","type":"effect_count","effect_type":"refund","group_by":"order_id","min":1,"max":1} ``` ## Replay your agent and keep the regression This command uses POSIX shell quoting. On Windows, use the supplied Python runner or call `runledger.incidents.cli.main` with a Python argument list as it does. ```sh runledger incident replay imported/incident.json --agent-command '["python","my_agent.py"]' --out replayed runledger incident promote imported/incident.json --to cases/refund-incident.json runledger incident replay cases/refund-incident.json --agent-command '["python","my_agent.py"]' --out ci-result --gate ``` The adapter receives `task_start` with the sanitized input, emits `tool_call` messages, reads `tool_result`, and finishes with `final_output` or `task_error`. The included `examples/incidents/replay_agent.py` shows this protocol. Agent call IDs may differ; ordered tool names and arguments must match the recording exactly. Repeated identical requests consume separate recorded responses, so a timeout followed by success is preserved. An extra, missing, reordered, or changed call is a divergence. A changed final output is also a divergence. RunLedger stops and does not execute a live fallback tool. Unmatched candidate payloads, logs, and stderr are withheld from exported replay artifacts so a new candidate cannot introduce unreviewed values into them. | Command outcome | Exit code | | --- | --- | | Valid import or inspect, including a recorded rule violation | 0 | | Replay matches all recorded calls and terminal output, even a failed incident | 0 | | `inspect --gate` or `replay --gate` with failed/inconclusive business rules | 1 | | Replay divergence or agent execution error | 1 | | Invalid input, output path, or command | 2 | Every output folder must be new or empty. Artifacts include the sanitized bundle, redaction audit, evaluation JSON, escaped HTML report, trace JSONL, and JUnit XML. Replay also adds `replay.json`. JUnit represents the combined replay/business gate; it reports the known failure even when a successful reproduction exits zero. Promotion saves the validated bundle without overwriting an existing fixture. The SHA-256 checksum detects accidental edits; it is not a signature or proof of where the evidence came from. ## What this establishes This workflow can reproduce a recorded path with your explicit agent adapter and show its captured rule violation. It freezes responses at the JSONL tool boundary. It does not recreate an external provider's state or calculate the result of a new agent branch. Use the stateful support scenario pack, a suitable test environment, or new recordings to validate a fix that takes a different path. Your supplied agent command runs locally with your account's permissions and is not sandboxed. Model calls or network activity outside the JSONL protocol are not intercepted. Capture or stub those dependencies in your adapter when needed. Redaction changes text, identifiers, and sometimes types; this can affect model judgment, derived keys, and exact replay. A complete original failure is not guaranteed to survive redaction unchanged. Review divergence rather than treating it as proof of a fix. Limits: one incident per canonical file, 4 MiB per input, at most 500 events, 20 levels of nesting, and 30 assertions. The replay runner also bounds execution time and message/tool counts. This version supports serial structured traces, not arbitrary binary logs, vendor-native telemetry archives, or automatic crash-state reconstruction.