# Plan: Local LLM Response Cache

## Goal

Add optional local caching of LLM request/response pairs in a SQLite database. When an identical request is seen again, return the cached response without contacting the LLM.

## Design Decisions

- **Storage**: SQLite via `sqlite3` (stdlib, zero dependencies). Single file at `.swival/cache.db`.
- **Cache key**: SHA-256 hash of the canonicalized request. This avoids storing the full request as a lookup key while still being deterministic.
- **Canonicalization**: `json.dumps(..., sort_keys=True)` on the fields that affect the response. The key includes:
  - `model` — the final model string after provider mapping
  - `provider` and `api_base` — two different backends can map to overlapping model strings. Without these, a lmstudio response could be replayed for a generic backend or vice versa.
  - `messages` — the full conversation history. The dynamic timestamp line (`Current date and time: ...`) in system messages is stripped before hashing so the cache key remains stable across runs.
  - `tools`, `tool_choice`
  - `temperature`, `top_p`, `seed`
  - `max_tokens` — the runtime clamps output budget per turn via `clamp_output_tokens()`, and a `finish_reason == "length"` response changes control flow. Two requests with different output caps are not equivalent.
  - `extra_body`, `reasoning_effort` (when present)
- **Activation**: `--cache` CLI flag / `cache = true` in config TOML / `cache=True` in `Session()`. Off by default.
- **Cache location override**: `--cache-dir PATH` for sharing a cache across projects.
- **Record and replay are transparent**: When cache is enabled, every `call_llm()` checks the cache first. On miss, the LLM is called and the response is stored. On hit, the cached response is returned. No separate record/replay modes.
- **Timestamp stripping**: The system prompt always uses the live date/time, but `_cache_key()` strips the dynamic `Current date and time: ...` line from system messages before hashing. This means the same conversation produces the same cache key across runs regardless of when it's executed, while non-timestamp differences in the system prompt still produce different keys.

## Schema

```sql
CREATE TABLE IF NOT EXISTS cache (
    key      TEXT PRIMARY KEY,  -- SHA-256 hex digest
    request  TEXT NOT NULL,     -- full JSON completion_kwargs (for debugging/inspection)
    response TEXT NOT NULL,     -- JSON-serialized (message_dict, finish_reason)
    model    TEXT NOT NULL,     -- for human filtering
    created  TEXT NOT NULL      -- ISO-8601 timestamp
);

CREATE TABLE IF NOT EXISTS meta (
    key   TEXT PRIMARY KEY,
    value TEXT NOT NULL
);
```

Single file, no migrations needed. The `request` column is stored for transparency/debugging but is not used for lookup (the hash is). The `meta` table stores arbitrary key-value pairs.

## New Module: `swival/cache.py`

```
class LLMCache:
    __init__(self, db_path: Path)
    open(self) -> None              # create tables if needed
    close(self) -> None
    get(self, completion_kwargs: dict) -> tuple[message_dict, finish_reason] | None
    put(self, completion_kwargs: dict, message_dict: dict, finish_reason: str) -> None
    _cache_key(completion_kwargs: dict) -> str           # @staticmethod, canonical hash

    # Meta
    get_meta(self, key: str) -> str | None
    set_meta(self, key: str, value: str) -> None

    # Utility
    stats(self) -> dict            # entry count, db size
    clear(self) -> None            # drop all entries
```

- `get()` returns `(message_dict, finish_reason)` on hit; the caller reconstructs a litellm `Message` via `_reconstruct_message()`.
- `put()` takes an already-serialized message dict (via `model_dump(exclude_none=True)`).

## Message Serialization

Cached messages are stored as plain dicts and reconstructed using `litellm.types.utils.Message` with proper nested types (`ChatCompletionMessageToolCall`, `Function`). This gives Pydantic `model_dump()` support and correct attribute access for tool calls.

## Integration: `call_llm()` in `agent.py`

The cache wraps the existing `litellm.completion()` call:

```python
def call_llm(..., cache: LLMCache | None = None):
    # ... build completion_kwargs ...

    if cache is not None:
        cache_kwargs = {**completion_kwargs, "_provider": provider, "_api_base": api_base_for_key}
        hit = cache.get(cache_kwargs)
        if hit is not None:
            return _reconstruct_message(hit[0]), hit[1]

    response = litellm.completion(**completion_kwargs)
    choice = _pick_best_choice(response.choices)

    if cache is not None:
        cache.put(cache_kwargs, choice.message.model_dump(exclude_none=True), choice.finish_reason)

    return choice.message, choice.finish_reason
```

## Secondary LLM Call Paths

The main loop call gets cache via `**llm_kwargs`. Secondary call sites pass `call_llm` as a function reference and don't go through `**llm_kwargs`. These are handled by `_call_llm_for_secondary`, a wrapper created at the top of `run_agent_loop()`:

```python
_call_llm_for_secondary = call_llm
if cache is not None:
    def _call_llm_for_secondary(*args, **kwargs):
        kwargs.setdefault("cache", cache)
        return call_llm(*args, **kwargs)
```

This wrapper is passed to:
- Compaction summaries (`_llm_summary_kwargs["call_llm_fn"]`)
- Proactive checkpoints (`compaction_state.maybe_checkpoint()`)
- Continue-file enrichment (`write_continue_file(call_llm_fn=...)`)

## Cache Lifecycle

- **CLI `main()`**: Opens `LLMCache` in `_run_main()` before `build_system_prompt()`. Stashes on `args._llm_cache`. Closed in `finally` block of `main()` (covers both success and error paths, including early `return` and `sys.exit`).
- **`Session`**: Opens `LLMCache` in `_setup()`. Closed in `__exit__()`. Shared across `run()` and `ask()` calls.
- **`run_agent_loop()`**: Receives the already-opened cache handle; does not own the lifecycle.

## Config & CLI

### Config keys (`config.py`)
- `cache` (bool) in `CONFIG_KEYS` and `_ARGPARSE_DEFAULTS`
- `cache_dir` (str) in `CONFIG_KEYS`, `_ARGPARSE_DEFAULTS`, and `_resolve_paths()`
- Both pass through `config_to_session_kwargs()` unchanged

### CLI arguments (`agent.py`)
```
--cache              Enable LLM response caching (.swival/cache.db)
--cache-dir PATH     Custom cache database directory
```

### Session API (`session.py`)
```python
Session(cache=True, cache_dir="/path/to/cache")
```

## File Changes Summary

| File | Change |
|------|--------|
| `swival/cache.py` | **New** — `LLMCache` class, `open_cache()`, `_reconstruct_message()`, `_strip_timestamps()` |
| `swival/agent.py` | Add `--cache`, `--cache-dir` CLI args; `cache` param in `call_llm()`; `_call_llm_for_secondary` wrapper in `run_agent_loop()`; cache lifecycle in `_run_main()`/`main()` |
| `swival/config.py` | Add `cache`, `cache_dir` to `CONFIG_KEYS`, `_ARGPARSE_DEFAULTS`, `generate_config()`; add `cache_dir` to `_resolve_paths()` |
| `swival/session.py` | Accept `cache`, `cache_dir` params; open/close `LLMCache` in `_setup()`/`__exit__()` |
| `tests/test_cache.py` | **New** — unit + integration tests |

## Non-Goals (for now)

- Cache invalidation / TTL — not needed for general use.
- Partial matching / fuzzy cache — exact match only.
- Caching tool execution results — tools run locally, no need.
- Cache warming CLI command — can be added later; for now, just run once online to populate.
