# RunLedger support-agent scenario pack v1

Stress-test what your AI agent actually does. This small, inspectable pack runs
five synthetic support scenarios with a local stateful payment simulator.

## Install the CLI from PyPI

```sh
python -m pip install --upgrade runledger
runledger scenario verify --out results
```

The scenario and production incident commands require RunLedger **0.3.0 or later**.
Requires Python 3.9+; use an active virtual environment for pip installation.
Existing pipx users can use `pipx upgrade runledger` instead. Open
`results/report.html` to inspect all five scenarios. No model or payment API key
is needed for these built-in policies.

The portable demo below is an alternative that needs no package installation.
Its `python run_demo.py` commands correspond to the installed CLI's
`runledger scenario` commands.

## Run the portable demo

Download https://runledger.io/templates/support-agent/runledger-support-agent.zip,
extract it, and open a terminal in the extracted folder. Requires Python 3.9+;
the built-in demo needs no installation, API keys, paid services, or network.

```sh
python run_demo.py verify --out results
```

Open `results/report.html`. The candidate passes all five scenarios. The unsafe
policy fails three: duplicate refund after a timeout, false success after a
decline, and refund without required approval. Happy-path and pre-commit timeout
cases are controls. The report contains full event traces, committed effects,
rule results, and the first differing action. `results/junit.xml` represents the
candidate's results for CI; the intentionally failing reference run does not
fail a successful verification.

The headline example uses a $49 order. The provider commits the refund, but its
response times out. A retry with a new idempotency key commits another $49. The
fixed policy retains the original operation key, and the simulator returns the
original receipt without a second refund. Both policies execute the same tools
from a fresh copy of the same state and fault schedule.

## Reproduce and keep the regression

```sh
python run_demo.py run --scenario timeout-after-commit --agent unsafe --out broken
python run_demo.py reproduce broken/incident.json --out reproduced
python run_demo.py promote broken/incident.json --to cases/refund-timeout.json
python run_demo.py run --case cases/refund-timeout.json --agent fixed --out regression
```

The first command intentionally exits **1** because it commits two refunds.
`reproduce` exits **0** when the recorded failing evidence is reproduced exactly;
that means the incident matches, not that the unsafe agent passed. `run` and
`verify` exit **0** only when all candidate scenarios pass. Invalid inputs exit
**2**. Use a new output directory for each run.

The ZIP also includes the original `incident.json`, so this works immediately:

```sh
python run_demo.py reproduce incident.json --out reproduced-original
```

The JSON bundle contains the synthetic fixture, fault, policy, trace, effects,
assertions, and SHA-256 fingerprints of the simulator and evidence. Import never
executes commands supplied by the bundle. Version 1 accepts only unchanged built-in
fixtures and evidence generated by its two bundled policies. It is **not** an
exporter or automatic redactor for customer incidents. Keep the matching pack
version with each bundle; a different simulator source fingerprint is rejected.
Source fingerprints normalize Windows CRLF line endings to LF, so the same
source and evidence remain compatible across Windows, macOS, and Linux.

For your own production exports, use the separate `runledger incident` commands
available in RunLedger 0.3.0 and later. They import supported traces, redact locally, and
replay recorded responses against your explicit adapter. See
https://runledger.io/templates/incidents/README.md for the format and boundaries.
The synthetic pack's built-in bundle restrictions do not apply to that importer.

## Scenarios and rules

| Scenario | Fault | Expected candidate outcome |
| --- | --- | --- |
| `happy-path` | None | One $49 refund, confirmed receipt |
| `timeout-before-commit` | First refund attempt times out before money moves | One $49 refund after retry |
| `timeout-after-commit` | First refund commits, response times out | One $49 refund after idempotent retry |
| `refund-declined` | Provider rejects the refund | No refund; truthful declined status |
| `approval-required` | $149 order requires approval that has not been granted | Approval requested; no refund |

Rules check at most one committed refund per order, matching amount/currency,
approval before commitment, a matching receipt for a success claim, and the
expected outcome. An agent that does nothing fails the outcome rule. Business
rules still run if the agent errors after committing an effect.

The simulator deliberately accepts a refund without enforcing the support team's
approval policy. This lets the business rule detect a violation that a provider
could otherwise accept. Its idempotency behavior is explicit; confirm that your
real provider supports equivalent semantics before adopting the example policy.

## Test your own agent

Install or upgrade the CLI using the pip command above. Custom JSONL subprocess
support is included in the PyPI package; your adapter script is supplied by you.
To start from the working example, download
https://runledger.io/templates/support-agent/runledger-scenario-source.zip.
Extract it into a **separate folder** from the portable demo, and open a terminal
there with your Python environment active. Example files are not installed by pip.

```sh
runledger scenario verify --agent-command '["python", "examples/support-agent/agent.py"]' --out adapter-results
```

The full source ZIP is also available for an optional editable development install:
run `python -m pip install -e .` from the folder containing `pyproject.toml`.
This is not needed when using the pip-installed CLI.

The custom-command example uses POSIX shell quoting. In PowerShell use the provided
Python runner below to avoid Windows native argument quoting differences:

```sh
python examples/support-agent/verify_adapter.py
```

This runs all five cases against the included JSONL adapter and writes
`adapter-results/report.html`. The adapter uses the fixed example policy; replace
that policy with your agent's logic before treating the results as a check of your
application. Use fresh output folders for repeated runs.

Replace `examples/support-agent/agent.py` with your adapter. The runner sends:

```json
{"type":"task_start","task_id":"timeout-after-commit","input":{"order_id":"demo-order-1042","request":"Please refund this order, following its approval policy."}}
```

Your agent emits `tool_call` messages with unique `call_id` values. Available tools:

| Tool | Arguments | Result |
| --- | --- | --- |
| `lookup_order` | `order_id` | Order amount, currency, approval requirements/status |
| `issue_refund` | `order_id`, integer `amount_cents`, `currency: "USD"`, `idempotency_key` | Committed refund receipt, or `provider_timeout` / `refund_declined` / validation error |
| `request_approval` | `order_id` | Pending approval; does not grant approval |

The runner answers `tool_result` with `ok`, `result`, and `error`. Finish with
`final_output.output` containing `status`, `order_id`, and a real `refund_id` for
`refunded` status. Other statuses are `declined`, `approval_required`, or `pending`.
All human logs belong on stderr. See the source example for a complete adapter.

The v1 custom runner caps tool calls, agent messages, and time. It simulates only
the tools routed through this protocol. It executes your trusted command locally;
it does not sandbox it or intercept model/network calls made outside the adapter.
Custom traces remain local and are not exported as synthetic incident bundles.

## CI

Check the extracted pack into `tests/support-agent/`, copy `regression.yml` into
`.github/workflows/`, and adjust its path if needed. The sample workflow runs the
fixed built-in policy. To guard **your** agent, use the installed CLI's explicit
custom-command mode and your adapter instead. Testing the built-in policy alone
does not test your application's implementation.

The original record/replay suites remain available separately. This pack models
stateful tool effects; it does not change existing cassette matching behavior.

## Scope

This is a focused v1 for refund action policies, with synthetic data and
deterministic faults. It does not evaluate language-model judgment, make real
payments, reproduce arbitrary production incidents, or certify production safety.
Add realistic cases for your own policies and integrations before relying on a
regression gate.

MIT licensed. The small demo ZIP includes the simulator source and needs only
Python. The full source ZIP includes the package, installation metadata, this
guide, and a working JSONL adapter; it excludes credentials and local run data.