# JSON Report Mode

## Goal

Add a `--report <path>` CLI option that replaces normal stdout output with a JSON file containing structured evaluation data. Designed for benchmarking models, settings, agent skills, and AGENT.md effectiveness.

Incompatible with `--repl` (error out if both are specified).

## Design decision: what "outcome" means

`outcome` is purely operational, not a correctness judgment:
- `"success"` — the model produced a final text answer before max_turns
- `"exhausted"` — max_turns hit (may still contain a partial answer)
- `"error"` — the loop terminated due to an unrecoverable error (e.g. triple context overflow)

True task correctness requires an external evaluator and is out of scope. The report provides the raw data (answer text, tool trace, stats) that an evaluator can consume.

## Report schema

```json
{
  "version": 1,
  "timestamp": "2026-02-25T14:30:00Z",
  "task": "the original user question",
  "model": "model-id-string",
  "provider": "lmstudio|huggingface",
  "settings": {
    "temperature": 0.55,
    "top_p": 1.0,
    "seed": null,
    "max_turns": 50,
    "max_output_tokens": 32768,
    "context_length": null,
    "yolo": false,
    "allowed_commands": ["ls", "git"],
    "skills_discovered": ["skill-a", "skill-b"],
    "instructions_loaded": ["CLAUDE.md", "AGENT.md"]
  },
  "result": {
    "outcome": "success|exhausted|error",
    "answer": "final model text or null",
    "error_message": "only present when outcome is error",
    "exit_code": 0
  },
  "stats": {
    "turns": 12,
    "tool_calls_total": 34,
    "tool_calls_succeeded": 30,
    "tool_calls_failed": 4,
    "tool_calls_by_name": {
      "read_file": { "succeeded": 10, "failed": 1 },
      "edit_file": { "succeeded": 8, "failed": 2 },
      "think": { "succeeded": 5, "failed": 0 },
      "grep": { "succeeded": 7, "failed": 1 }
    },
    "compactions": 1,
    "turn_drops": 0,
    "guardrail_interventions": 0,
    "truncated_responses": 0,
    "llm_calls": 14,
    "total_llm_time_s": 45.2,
    "total_tool_time_s": 3.8
  },
  "timeline": [
    {
      "turn": 1,
      "type": "llm_call",
      "duration_s": 3.2,
      "prompt_tokens_est": 1200,
      "finish_reason": "tool_calls",  // or "stop", "length", "context_overflow" (failed attempt)
      "is_retry": false
    },
    {
      "turn": 1,
      "type": "tool_call",
      "name": "read_file",
      "arguments": { "path": "src/main.py" },
      "succeeded": true,
      "duration_s": 0.01,
      "result_length": 4200
    },
    {
      "turn": 1,
      "type": "tool_call",
      "name": "edit_file",
      "arguments": { "path": "src/main.py", "old_string": "...", "new_string": "..." },
      "succeeded": false,
      "error": "error: old_string not found in file",
      "duration_s": 0.003,
      "result_length": 35
    },
    {
      "turn": 3,
      "type": "compaction",
      "strategy": "compact_messages",
      "tokens_before": 95000,
      "tokens_after": 62000
    },
    {
      "turn": 3,
      "type": "llm_call",
      "duration_s": 2.8,
      "prompt_tokens_est": 62000,
      "finish_reason": "tool_calls",
      "is_retry": true,
      "retry_reason": "compact_messages"
    },
    {
      "turn": 5,
      "type": "guardrail",
      "tool": "edit_file",
      "level": "nudge"
    },
    {
      "turn": 7,
      "type": "truncated_response"
    }
  ]
}
```

## Implementation plan

### 0. Bugfix: missing `seed` in overflow retry calls (prerequisite)

`call_llm()` signature is `(base_url, model_id, messages, max_output_tokens, temperature, top_p, seed, tools, verbose, ...)`. The two retry calls at lines 907-916 and 929-938 skip `seed`, passing `tools` where `seed` should be. This means retries after compaction silently pass `tools` as the seed and the actual tools list as `verbose`.

Fix: add `seed` to both retry call sites, matching the primary call at line 884.

Test: add a test in `tests/test_compact.py` (or new `tests/test_overflow_retry.py`) that mocks `call_llm` to raise `ContextOverflowError` on first call, succeed on second, and asserts `seed` was passed correctly in the retry.

### 1. Add `ReportCollector` class in new file `swival/report.py`

A mutable accumulator that the agent loop populates as events happen.

```python
class ReportCollector:
    def __init__(self):
        self.events: list[dict] = []
        self.tool_stats: dict[str, dict] = {}  # name -> {succeeded, failed}
        self.compactions = 0
        self.turn_drops = 0
        self.guardrail_interventions = 0
        self.truncated_responses = 0
        self.llm_calls = 0
        self.total_llm_time = 0.0
        self.total_tool_time = 0.0

    def record_llm_call(self, turn, duration, token_est, finish_reason, *, is_retry=False, retry_reason=None): ...
    def record_tool_call(self, turn, name, arguments, succeeded, duration, result_length, error=None): ...
    def record_compaction(self, turn, strategy, tokens_before, tokens_after): ...
    def record_guardrail(self, turn, tool, level): ...
    def record_truncated_response(self, turn): ...

    def build_report(self, *, task, model, provider, settings, outcome, answer, exit_code, turns, error_message=None) -> dict:
        """Assemble the final JSON dict."""
        ...

    def write(self, path: str):
        """Write report JSON to disk."""
        ...
```

The collector is optional — `None` when `--report` is not given. Each call site does `if report: report.record_xxx(...)` so there's zero overhead when not collecting.

### 2. Enrich `handle_tool_call()` return value

Currently returns only `{"role": "tool", "tool_call_id": ..., "content": result}`. The parsed args and elapsed time are local variables (lines 255, 271-288) — the caller in `run_agent_loop()` can't see them.

**Decision: use tuple return.** Change `handle_tool_call` to return `(tool_msg_dict, tool_meta_dict)` where `tool_meta_dict = {"name": str, "arguments": dict|None, "elapsed": float, "succeeded": bool}`. The single call site at line 974 destructures it.

**Invalid-JSON early return path** (line 254-263): when `json.loads` fails, the function returns early before `dispatch()` runs. The returned metadata must still have stable keys so reporting code doesn't need to branch:

```python
# JSON parse failure path
return (
    {"role": "tool", "tool_call_id": tool_call.id, "content": error_msg},
    {"name": name, "arguments": None, "elapsed": 0.0, "succeeded": False},
)
```

`arguments: None` signals to the report that args weren't parseable. `elapsed: 0.0` because no dispatch happened.

### 3. CLI changes in `agent.py`

In `build_parser()`:
- Add `--report <filepath>` argument (type=str, default=None, metavar="FILE")

In `main()`:
- Validate: `parser.error()` if both `--report` and `--repl` are given
- If `--report` is set, create a `ReportCollector` instance; otherwise `None`
- Pass `report` into `run_agent_loop()` as a kwarg
- Capture which instruction files were loaded (see step 4)

### 4. Make `load_instructions()` return filenames alongside content

Currently returns only the combined text string. Change to return `(text, filenames_loaded)` where `filenames_loaded` is a `list[str]` like `["CLAUDE.md", "AGENT.md"]` or `["CLAUDE.md"]` or `[]`.

This is a small change — the function already iterates over the filenames. Just collect the ones that were actually found. The single call site in `main()` destructures the new return value.

### 5. Instrument `run_agent_loop()`

Add `report: ReportCollector | None = None` parameter.

**LLM call recording — per attempt, not per turn.** The current code has up to 3 `call_llm()` invocations per turn (primary + 2 overflow retries). Each must be individually timed and recorded. Restructure the timing from the current "one `t0` wrapping everything" to per-call timing:

Every `call_llm()` invocation — whether it succeeds or raises — gets its own `t0`/`elapsed` and a `record_llm_call`. Failed attempts use `finish_reason="context_overflow"`.

```
Primary call:
    t0 = time.monotonic()
    try:
        msg, finish_reason = call_llm(...)
    except ContextOverflowError:
        elapsed = time.monotonic() - t0
        if report: report.record_llm_call(turn, elapsed, token_est, "context_overflow")
        ... compaction + retry 1 below ...
    else:
        elapsed = time.monotonic() - t0
        if report: report.record_llm_call(turn, elapsed, token_est, finish_reason)

    tokens_before = estimate_tokens(...)
    compact_messages(...)
    tokens_after = estimate_tokens(...)
    if report: report.record_compaction(turn, "compact_messages", tokens_before, tokens_after)

    Retry 1:
        t0 = time.monotonic()
        try:
            msg, finish_reason = call_llm(...)  # with seed fix from step 0
        except ContextOverflowError:
            elapsed = time.monotonic() - t0
            if report: report.record_llm_call(turn, elapsed, token_est_after, "context_overflow", is_retry=True, retry_reason="compact_messages")
            ... drop_middle_turns + retry 2 below ...
        else:
            elapsed = time.monotonic() - t0
            if report: report.record_llm_call(turn, elapsed, token_est_after, finish_reason, is_retry=True, retry_reason="compact_messages")

        tokens_before = estimate_tokens(...)
        drop_middle_turns(...)
        tokens_after = estimate_tokens(...)
        if report: report.record_compaction(turn, "drop_middle_turns", tokens_before, tokens_after)

        Retry 2:
            t0 = time.monotonic()
            try:
                msg, finish_reason = call_llm(...)  # with seed fix from step 0
            except ContextOverflowError:
                elapsed = time.monotonic() - t0
                if report: report.record_llm_call(turn, elapsed, token_est_after, "context_overflow", is_retry=True, retry_reason="drop_middle_turns")
                raise AgentError("context window exceeded even after compaction")
            else:
                elapsed = time.monotonic() - t0
                if report: report.record_llm_call(turn, elapsed, token_est_after, finish_reason, is_retry=True, retry_reason="drop_middle_turns")
```

This ensures `llm_calls` count and `total_llm_time_s` include all attempts, and the timeline shows the actual time spent on each failed call before it overflowed.

The existing `elapsed = time.monotonic() - t0` at line 943 (which currently wraps compaction work) is removed. The `fmt.llm_timing()` call moves to after each individual `call_llm()` return so it also reports accurate per-call timing.

**Tool call recording** — after `handle_tool_call()` returns the `(tool_msg, meta)` tuple:

```python
tool_msg, meta = handle_tool_call(...)
messages.append(tool_msg)
if report:
    report.record_tool_call(
        turn, meta["name"], meta["arguments"],
        meta["succeeded"], meta["elapsed"],
        len(tool_msg["content"]),
        error=tool_msg["content"] if not meta["succeeded"] else None,
    )
```

**Other recording points** (same locations as before, just with corrected detail):

| What | Where | Recording call |
|------|-------|----------------|
| Guardrail intervention | Lines 998-1014, where nudge/stop is built | `report.record_guardrail(turn, tool_name, "nudge" or "stop")` |
| Truncated response | `finish_reason == "length"` branch, line 953 | `report.record_truncated_response(turn)` |

### 6. Centralize error exits for report writing

Several `sys.exit(1)` calls in `main()` and one inside `run_agent_loop()` (line 941, triple overflow) would skip report writing. Fix each path:

**Inside `run_agent_loop()`** (line 940-942, triple context overflow):
- Instead of `sys.exit(1)`, raise `AgentError(message)`.
- `main()` catches it and writes the report with accumulated partial stats.

**Inside helper functions called from the loop** — these `sys.exit(1)` calls also bypass report writing and must be converted to `raise AgentError(...)`:

| Location | Function | Current behavior |
|----------|----------|-----------------|
| `agent.py:432-433` | `call_llm()` | Non-overflow `BadRequestError` → `sys.exit(1)` |
| `agent.py:435-436` | `call_llm()` | Unexpected `Exception` → `sys.exit(1)` |

These are mid-loop failures — the report collector already has partial data. Convert both to `raise AgentError(f"LLM call failed: {e}")`. The loop doesn't catch `AgentError`, so it propagates to `main()` where the report is written.

**Inside helper functions called during setup** — these happen before the loop starts:

| Location | Function | Current behavior |
|----------|----------|-----------------|
| `agent.py:314-315` | `discover_model()` | Can't connect to LM Studio → `sys.exit(1)` |
| `agent.py:317-318` | `discover_model()` | Invalid JSON from LM Studio → `sys.exit(1)` |
| `agent.py:366-367` | `configure_context()` | Model reload failed → `sys.exit(1)` |

Convert all three to `raise AgentError(...)`. `main()` catches these around the pre-loop setup and writes a minimal report (zero stats, empty timeline, `outcome: "error"`).

**Inside `main()` itself** — pre-loop `sys.exit(1)` calls:
- Lines 622-626: no model found in LM Studio
- Lines 664-669: bad `--allow-dir` path
- Line 690: allowed command not found on PATH
- Line 698: allowed command resolves inside base directory

Convert all four to `raise AgentError(...)`.

Keep `parser.error()` calls as-is — those are usage errors (wrong flags, missing required args), not runtime failures. Argparse exits with code 2 before we even know `--report` was valid, so no report is expected.

**`main()` structure:**

```python
def main():
    args = build_parser().parse_args()
    ...
    report = ReportCollector() if args.report else None

    try:
        # all setup + loop
        ...
    except AgentError as e:
        fmt.error(str(e))
        if report:
            report.build_report(..., outcome="error", error_message=str(e), ...)
            report.write(args.report)
        sys.exit(1)
```

**Do NOT catch bare `SystemExit` broadly.** `SystemExit(2)` is used by both argparse (usage errors) and max-turn exhaustion, so a broad catch would misclassify outcomes. Only catch `AgentError`.

### 7. Change output behavior in `main()`

When `--report` is active, after `run_agent_loop()` returns (or raises `AgentError`):
- Build the report via `report.build_report(...)` with metadata from CLI args
- Write to the specified path via `report.write(path)`
- Do NOT print the answer to stdout (the report replaces it)
- Still use the same exit codes (0 / 1 / 2)
- Stderr confirmation: guard with `if verbose:` before calling `fmt.info(f"Report written to {path}")`. `fmt.info()` itself always prints (it doesn't check a quiet flag), so the caller must gate it. This matches the existing pattern throughout the codebase where `verbose` guards all `fmt.*` diagnostic calls.

### 8. Update existing test fixtures

Three categories of existing tests are affected by signature changes:

**a) Namespace objects missing `report` attr.**
Tests that monkeypatch `parse_args` with hand-built namespace objects (e.g. `test_logging.py` lines 32-61, `test_guardrails.py` lines 26-54) will break because they lack the new `report` attribute. Fix: add `report=None` to every hand-built namespace. Grep for `argparse.Namespace(` and `SimpleNamespace(` across `tests/`.

**b) `run_agent_loop()` callers missing `report` kwarg.**
Tests that call `run_agent_loop()` directly need the new `report=None` kwarg. Grep for `run_agent_loop(` across `tests/`.

**c) `handle_tool_call()` callers expecting dict return.**
`test_think.py` lines 655 and 671 call `handle_tool_call()` directly and assert on `result_msg["role"]`. With the tuple return, these must destructure: `result_msg, meta = handle_tool_call(...)`. Both call sites need updating.

**d) `load_instructions()` callers expecting string return.**
`test_instructions.py` (starting line 18) calls `load_instructions()` and asserts on the returned string. With the `(text, filenames)` tuple return, every `result = load_instructions(...)` becomes `result, loaded = load_instructions(...)`. Add assertions on `loaded` where useful (e.g. verify `["CLAUDE.md"]` when only CLAUDE.md exists).

### 9. Tests in `tests/test_report.py`

- **Unit tests for `ReportCollector`**: record events, verify `build_report()` output matches schema, check stats aggregation
- **`handle_tool_call` tuple return**: verify old behavior still works, verify metadata fields are populated
- **Integration test**: mock LLM, run single-shot with `--report /tmp/report.json`, verify file contents, schema structure, stats, timeline
- **Validation test**: `--report` + `--repl` produces argparse error
- **Edge cases**: zero tool calls, all failures, compaction events, triple overflow producing error outcome
- **Overflow retry with seed**: mock `call_llm` to overflow then succeed, verify `seed` passed correctly (step 0 bugfix)
- **Quiet mode**: `--report --quiet` still writes report, no stderr output
- **Invalid-JSON tool call**: verify meta has `arguments: None`, `elapsed: 0.0`, `succeeded: False`

## File change summary

| File | Change |
|------|--------|
| `swival/report.py` | New — `ReportCollector` class + `AgentError` exception |
| `swival/agent.py` | `--report` arg, validation, `load_instructions()` returns `(text, filenames)`, `handle_tool_call()` returns `(msg, meta)` tuple with stable keys on all paths, per-attempt LLM timing, report instrumentation in loop, `AgentError` instead of `sys.exit(1)` in loop and pre-loop failures, error-catching in `main()`, fix missing `seed` in retry calls |
| `tests/test_report.py` | New — tests for report feature + seed bugfix |
| `tests/test_think.py` | Update `handle_tool_call()` callers to destructure tuple (lines 655, 671) |
| `tests/test_instructions.py` | Update `load_instructions()` callers to destructure tuple, add `loaded` filenames assertions |
| `tests/test_logging.py` | Add `report=None` to monkeypatched namespaces |
| `tests/test_guardrails.py` | Add `report=None` to monkeypatched namespaces |
| `tests/test_*.py` (others) | Add `report=None` to any remaining namespace objects and `run_agent_loop()` calls |
