# Plan: Retry LLM Calls on Transient Network Errors

## Problem

When the network is unstable, `call_llm` raises `AgentError("LLM call failed: ...")` for transient errors like `[Errno 54] Connection reset by peer`. This propagates up through `run_agent_loop` → `main` → `sys.exit(1)`, killing the session immediately with no retry.

## Files to Change

| File | What |
|------|------|
| `swival/agent.py` | `_completion_with_retry()` helper, `_is_transient()`, refactor `call_llm()`, `--retries` CLI flag, `_build_self_review_cmd()` mirroring, validation |
| `swival/config.py` | `CONFIG_KEYS`, `_ARGPARSE_DEFAULTS`, config template, `args_to_session_kwargs()` |
| `swival/session.py` | `Session.__init__()` — add `retries` param; `_setup()` — inject into `self._llm_kwargs` |
| `swival/reviewer.py` | `run_as_reviewer()` (line ~85) — pass `max_retries` to `call_llm()` |
| `swival/report.py` | `record_llm_call()` — new `provider_retries` field |
| `docs.md/reports.md` | Document `provider_retries` in timeline schema |
| `tests/test_retry.py` | New: retry-specific tests |
| `tests/test_command_provider.py`, `test_provider.py`, `test_empty_assistant.py`, `test_cache.py`, `test_secrets_integration.py` | Update all direct `call_llm()` 3-value unpacking sites to 4-value |

## Current Flow

```
litellm.completion()
  → ContextWindowExceededError  → ContextOverflowError (handled by compaction)
  → BadRequestError             → AgentError (some special-cased)
  → Exception                   → AgentError (fatal, exits)
```

The generic `except Exception` catch-all at line 2274 turns every non-context, non-bad-request error into a fatal `AgentError`. Network errors (connection reset, timeout, DNS failure) land here.

## Design

### Core: `_completion_with_retry()` helper

Extract the `litellm.completion()` call into a shared helper that both the normal path and the empty-assistant sanitization retry use. This avoids the current problem where the sanitization retry at agent.py:2263 would bypass the retry policy.

Returns `(response, provider_retries)` where `provider_retries` is the number of retries performed (0 = first attempt succeeded).

```python
def _completion_with_retry(completion_kwargs, *, max_retries, verbose):
    """Call litellm.completion() with retry on transient errors.

    Returns (response, provider_retries).
    Raises ContextOverflowError, litellm.BadRequestError, or the original
    exception for non-transient errors. Never wraps in AgentError — caller does that.
    """
    for attempt in range(max_retries):
        try:
            return litellm.completion(**completion_kwargs), attempt
        except litellm.ContextWindowExceededError:
            raise ContextOverflowError("context window exceeded (typed)")
        except litellm.BadRequestError:
            raise  # caller handles special cases
        except Exception as e:
            if _CONTEXT_OVERFLOW_RE.search(str(e)):
                raise ContextOverflowError(f"context window exceeded (inferred): {e}")
            if not _is_transient(e) or attempt == max_retries - 1:
                raise
            delay = min(2 * (2 ** attempt), 30)
            delay *= 0.75 + 0.5 * random.random()
            if verbose:
                fmt.warning(
                    f"Network error: {e} — retrying in {delay:.0f}s "
                    f"(attempt {attempt + 2}/{max_retries})"
                )
            time.sleep(delay)
```

Then `call_llm()` becomes:

```python
try:
    response, retries = _completion_with_retry(
        completion_kwargs, max_retries=max_retries, verbose=verbose
    )
except ContextOverflowError:
    raise  # must propagate for compaction in run_agent_loop
except litellm.BadRequestError as e:
    msg_text = str(e)
    if _CONTEXT_OVERFLOW_RE.search(msg_text):
        raise ContextOverflowError(f"context window exceeded (inferred): {e}")
    if _EMPTY_ASSISTANT_RE.search(msg_text) and _sanitize_assistant_messages(messages):
        if verbose:
            fmt.warning("Fixed empty assistant message in history, retrying...")
        try:
            response, retries = _completion_with_retry(
                completion_kwargs, max_retries=max_retries, verbose=verbose
            )
        except ContextOverflowError:
            raise  # must propagate for compaction
        except Exception as e2:
            raise AgentError(f"LLM call failed after message sanitization: {e2}")
    else:
        raise AgentError(f"LLM call failed: {e}")
except Exception as e:
    raise AgentError(f"LLM call failed: {e}")

# ... existing choice/decrypt logic ...
return _decrypt_msg(choice.message), choice.finish_reason, [], retries
```

### Command Provider Consistency

The `provider == "command"` early-return paths (agent.py:2106–2116) must also return 4-tuples:

```python
if provider == "command":
    if command_tool_kwargs is not None:
        return (*_call_command_with_tools(...), 0)
    msg, stop = _call_command(model_id, messages, verbose, max_output_tokens)
    return msg, stop, [], 0
```

All callers already use `_llm_result[0]`, `_llm_result[1]`, `_llm_result[2] if len(_llm_result) > 2 else []`. The 4th element is accessed explicitly only where we need it (report recording). Existing unpacking patterns like `msg, reason, _` in tests need updating to `msg, reason, _, _`.

### Transient Error Detection

Classify by type first, then by status code on broader `APIError`, then by message pattern as a fallback:

```python
def _is_transient(exc):
    import litellm

    # Explicit non-transient types — never retry
    if isinstance(exc, (litellm.BadRequestError, litellm.AuthenticationError,
                        litellm.NotFoundError, litellm.ContextWindowExceededError)):
        return False

    # Known transient types
    if isinstance(exc, (litellm.InternalServerError, litellm.APIConnectionError,
                        litellm.Timeout, litellm.RateLimitError)):
        return True

    # Broader APIError with server-side status code (500-599)
    if isinstance(exc, litellm.APIError):
        status = getattr(exc, "status_code", None)
        if status is not None and 500 <= status < 600:
            return True

    # Fallback: string pattern match for OS-level / socket errors
    return bool(_TRANSIENT_PATTERNS.search(str(exc)))

_TRANSIENT_PATTERNS = re.compile(
    r"Connection reset by peer|Connection refused|timed out"
    r"|RemoteDisconnected|Temporary failure in name resolution"
    r"|SSLError|EOF occurred|BrokenPipeError",
    re.IGNORECASE,
)
```

### Retry Parameters

| Parameter | Value | Notes |
|-----------|-------|-------|
| Max attempts | 5 | Total, including the first try |
| Initial delay | 2s | |
| Backoff factor | 2× | Delays: 2s, 4s, 8s, 16s |
| Max delay cap | 30s | Won't exceed this per wait |
| Jitter | ±25% | Avoid thundering herd |

Total worst-case wait before giving up: ~30s (2+4+8+16).

**`Retry-After` is deferred.** The litellm `RateLimitError` shape needs to be verified against our pinned version before we parse headers from it. For v1, the exponential backoff is sufficient for 429s. Can be added later as an optimization.

### Config Plumbing

**`swival/config.py`:**
- Add `"retries": int` to `CONFIG_KEYS` (line ~28)
- Add `"retries": 5` to `_ARGPARSE_DEFAULTS` (line ~94)
- Add `"retries"` passthrough in `args_to_session_kwargs()` (line ~741) — no rename needed, name matches
- Add commented line in config template (line ~908): `# retries = 5                     # max provider retries on transient network errors (1 = no retry)`

**`swival/agent.py` CLI:**
- Add `--retries` to `behavior_group` (near `--max-turns`, line ~2554):
  ```python
  behavior_group.add_argument(
      "--retries", type=int, default=_UNSET,
      help="Max provider retries on transient network errors (default: 5, 1 = no retry).",
  )
  ```
- Add validation (near line ~2972): `if args.retries < 1: parser.error("--retries must be >= 1")`
- Mirror in `_build_self_review_cmd()` (line ~2298): `if args.retries != 5: parts.extend(["--retries", str(args.retries)])`

**`swival/session.py`:**
- Add `retries: int = 5` parameter to `Session.__init__()` (line ~45)
- Store as `self.retries = retries`
- In `_setup()` (line ~200), inject into `self._llm_kwargs` after the existing `extra_body`/`reasoning_effort` lines — `_build_loop_kwargs()` just passes `self._llm_kwargs` through, so no changes needed there

### Threading `max_retries` to `call_llm()`

There are three execution paths that call `call_llm()`:

**1. CLI agent loop** (`_run_main` → `run_agent_loop`):
`run_agent_loop` receives `llm_kwargs` which already gets extended with `cache` and `secret_shield` at lines 4119–4122. Add `max_retries` the same way:

```python
# In run_agent_loop, at ~line 4122:
llm_kwargs = {**llm_kwargs, "max_retries": max_retries}
```

Where does `max_retries` come from? Add it as a `run_agent_loop()` parameter. In `_run_main`, pass `args.retries`. In `Session._build_loop_kwargs()`, pass `self.retries`.

Wait — simpler: inject it into `llm_kwargs` before passing to `run_agent_loop`, same as `extra_body` and `reasoning_effort`. Both paths already do this:

- CLI path (`_run_main`, ~line 3690): `llm_kwargs["max_retries"] = args.retries`
- Session path (`_build_loop_kwargs` isn't where `llm_kwargs` is built — it's built in `_setup()` at ~line 200). Add: `self._llm_kwargs["max_retries"] = self.retries` after the existing `extra_body`/`reasoning_effort` lines.

Then `call_llm()` extracts it: `max_retries = kwargs.pop("max_retries", 5)` or just accept it as a keyword argument.

The `_call_llm_for_secondary` wrapper at line 4126 also passes through `**kwargs`, so compaction summaries and proactive checkpoints get retry automatically.

**2. Reviewer mode** (`run_reviewer_mode` in `reviewer.py`):
Currently calls `call_llm()` directly at reviewer.py:165. Pass `max_retries=args.retries`:

```python
_llm_result = call_llm(
    ...,
    max_retries=getattr(args, "retries", 5),
)
```

Using `getattr` with default because reviewer mode can be invoked by external scripts that may not have the flag.

**3. Direct `call_llm` in tests:**
Tests call `call_llm()` directly. They get `max_retries=5` by default, which is fine. Tests that specifically test retry behavior are in the new `test_retry.py`.

### Reporting

The outer agent loop records one `record_llm_call()` event per `call_llm()` invocation. `is_retry` means "retry after compaction." These meanings are preserved.

`_completion_with_retry()` returns the retry count as its second element. `call_llm()` returns it as a 4th tuple element. The agent loop threads it to `record_llm_call()`.

**`swival/report.py`:** Add optional `provider_retries` field:

```python
def record_llm_call(self, turn, duration, token_est, finish_reason, *,
                    is_retry=False, retry_reason=None, provider_retries=0):
    ...
    if provider_retries:
        event["provider_retries"] = provider_retries
```

**Agent loop call sites** (lines ~4284, ~4391, ~4403, ~4417, ~4490, ~4538):
Extract `provider_retries = _llm_result[3] if len(_llm_result) > 3 else 0` and pass to `record_llm_call()`.

**`docs.md/reports.md`:**

Update `stats` section (~line 89) — current text says `llm_calls` is "total model API calls" which will be misleading after this change. New wording:

> `llm_calls` is total logical LLM calls (each `call_llm()` invocation counts as one, regardless of internal provider retries). Includes retries after compaction.

Update the `llm_call` timeline event description (~line 101):

> For `llm_call`, fields include `duration_s`, `prompt_tokens_est`, `finish_reason`, `is_retry`, and optionally `provider_retries` (number of transient-error retries within this call; omitted when 0). Retry calls include `retry_reason`...

### User Feedback

On each retry, emit `fmt.warning()` to stderr:

```
Network error: [Errno 54] Connection reset by peer — retrying in 4s (attempt 3/5)
```

No spinner during sleep — the outer loop already wraps `call_llm()` in `fmt.llm_spinner()`, and nesting a second spinner would cause rendering issues. The warning line is sufficient feedback.

## Implementation Order

1. Add `_is_transient()` and `_TRANSIENT_PATTERNS` to `agent.py`
2. Extract `_completion_with_retry()` helper, returning `(response, retries)`
3. Refactor `call_llm()` to use the helper for both normal and sanitization paths
4. Update command provider early-return to 4-tuple `(msg, stop, [], 0)`
5. Add `max_retries` keyword to `call_llm()` signature
6. Wire `--retries` through CLI flag, validation, config schema/defaults/template, `args_to_session_kwargs()`
7. Inject `max_retries` into `llm_kwargs` in both CLI path (`_run_main`) and Session path (`_setup`)
8. Pass `max_retries` in reviewer mode (`reviewer.py:165`)
9. Mirror `--retries` in `_build_self_review_cmd()`
10. Update `report.py:record_llm_call()` to accept `provider_retries`
11. Thread `provider_retries` from agent loop `_llm_result[3]` into all `record_llm_call()` sites
12. Update `docs.md/reports.md` timeline schema
13. Update all direct `call_llm()` unpacking sites in tests (`test_command_provider.py`, `test_provider.py`, `test_empty_assistant.py`, `test_cache.py`, `test_secrets_integration.py`)
14. New `tests/test_retry.py`

## Testing

1. **Transient succeeds on retry**: Mock `litellm.completion` to raise `APIConnectionError` twice then succeed. Assert 3 calls total, correct result returned, `provider_retries=2`.
2. **Non-transient fails immediately**: Mock with `BadRequestError`. Assert single call, immediate `AgentError`.
3. **Exhausted retries**: Mock with transient error 5 times. Assert `AgentError` after all attempts.
4. **Generic `APIError(status_code=500)` is transient**: Assert `_is_transient()` returns True.
5. **`APIError(status_code=400)` is NOT transient**: Assert `_is_transient()` returns False.
6. **Sanitization retry also retries transient errors**: Mock to raise `BadRequestError` (empty assistant) first, then `APIConnectionError` on sanitized retry, then succeed. Assert recovery.
7. **`provider_retries` in report**: Run agent loop with mocked transient+success, assert report event has `provider_retries=1`.
8. **`--retries 1` disables retry**: Mock with transient error. Assert single call, immediate failure.
9. **`--retries 0` rejected**: Assert `parser.error` on validation.
10. **Command provider returns 4-tuple**: Assert `len(call_llm(..., provider="command")) == 4` and 4th element is 0.
11. **Reviewer mode passes retries**: Mock `call_llm` in reviewer, assert `max_retries` kwarg is forwarded.
12. **ContextOverflowError propagates**: Mock with `ContextWindowExceededError`. Assert `ContextOverflowError` is raised (not wrapped in `AgentError`).
13. **ContextOverflowError propagates through sanitization retry**: Mock to raise `BadRequestError` (empty assistant) first, then `ContextWindowExceededError` on sanitized retry. Assert `ContextOverflowError` propagates.

## Rollout

Ship behind `--retries` defaulting to 5. Users who want fail-fast behavior can set `--retries 1` or `retries = 1` in config.

**Post-implementation:** Run the full test suite, not just `tests/test_retry.py`. The 4-tuple return change touches many call sites and existing tests; any missed unpacking will surface as test failures.
