# Plan: Token-based inline limits for tool output

## Goal

Replace byte-based `MAX_INLINE_OUTPUT` (10 KB) and `MCP_INLINE_LIMIT` (20 KB) with token-based equivalents so that the thresholds reflect actual context window cost rather than raw byte size.

## Why

Bytes are a poor proxy for context usage. A 10 KB block of dense code uses far fewer tokens than 10 KB of whitespace-padded log output. Token-based limits give more consistent context management — the same limit works well for both terse binary-ish output and verbose prose.

## Current state

| Constant | Value | Used in |
|---|---|---|
| `MAX_INLINE_OUTPUT` | 10 KB (bytes) | `_capture_process` — run_command output; `_save_large_output` fallback truncation |
| `MCP_INLINE_LIMIT` | 20 KB (bytes) | `_guard_mcp_output` — MCP tool results; MCP error truncation in `dispatch()` |
| `MAX_FILE_OUTPUT` | 1 MB (bytes) | Stream-capture cap in `_capture_process` — stays byte-based (disk budget, not context) |
| `MCP_FILE_LIMIT` | 10 MB (bytes) | Hard cap before file write in `_guard_mcp_output` — stays byte-based (disk budget) |

Token counting lives in `agent.py`: `_encoder = tiktoken.get_encoding("cl100k_base")` and `estimate_tokens()`. A separate rough estimator (`len(text) // 4`) exists in `snapshot.py:32`.

## Plan

### 1. New `swival/tokens.py` module

Extract the tiktoken encoder out of `agent.py` into a shared module:

```python
import tiktoken

_encoder = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(_encoder.encode(text))

def truncate_to_tokens(text: str, max_tokens: int) -> str:
    tokens = _encoder.encode(text)
    if len(tokens) <= max_tokens:
        return text
    return _encoder.decode(tokens[:max_tokens])
```

- `agent.py` imports `_encoder` (or `count_tokens`) from here instead of creating its own.
- `snapshot.py:_estimate_tokens` is replaced by `count_tokens` from this module, eliminating the duplicate `len(text) // 4` heuristic.

### 2. Replace the constants

```python
# tools.py
MAX_INLINE_TOKENS = 2_500   # ~10 KB of typical text
MCP_INLINE_TOKENS = 5_000   # ~20 KB of typical text
```

Exact values TBD — pick token counts that roughly match the old byte budgets for average content, then tune.

### 3. Token-aware `_safe_truncate_tokens`

```python
def _safe_truncate_tokens(text: str, max_tokens: int, suffix: str) -> str:
    """Truncate text to fit within max_tokens *including* the suffix.

    Precondition: suffix must fit within max_tokens. Raises ValueError
    if it doesn't — callers own their suffix strings, so this is a
    programming error, not a runtime condition.
    """
    suffix_tokens = count_tokens(suffix)
    if suffix_tokens > max_tokens:
        raise ValueError(
            f"suffix ({suffix_tokens} tokens) exceeds budget ({max_tokens})"
        )
    body_budget = max_tokens - suffix_tokens
    truncated_body = truncate_to_tokens(text, body_budget)
    if truncated_body == text:
        return text  # no truncation needed
    return truncated_body + suffix
```

The suffix is reserved inside the budget so the total never exceeds `max_tokens`. The helper enforces that the suffix itself fits within the budget — since suffixes are static strings controlled by callers, violations are programming errors and raise `ValueError`. This matters for the MCP error path (`dispatch()` line ~2054) where the result goes directly into context.

Edge-case tests to add:
- Exact fit: body + suffix together equal `max_tokens` — returned unchanged.
- Over-limit body with short suffix — body truncated, suffix appended, total equals `max_tokens`.
- Suffix longer than budget — `ValueError` raised.

Keep the old byte-based `_safe_truncate` — it's still needed for the disk-budget caps (`MAX_FILE_OUTPUT`, `MCP_FILE_LIMIT`).

### 4. Add `inline_tokens` parameter to `_save_large_output`

Currently `_save_large_output` hard-codes `MAX_INLINE_OUTPUT` in its fallback truncation paths. Since it's shared by both `run_command` and MCP flows (which have different budgets), add an explicit parameter:

```python
def _save_large_output(
    output: str,
    base_dir: str,
    *,
    tool_name: str | None = None,
    was_truncated: bool = False,
    inline_tokens: int = MAX_INLINE_TOKENS,  # caller sets its own policy
) -> str:
```

Fallback truncation uses `_safe_truncate_tokens(output, inline_tokens, ...)` instead of the byte-based version.

Callers pass their budget explicitly:
- `_capture_process` passes `inline_tokens=MAX_INLINE_TOKENS`
- `_guard_mcp_output` passes `inline_tokens=MCP_INLINE_TOKENS`

### 5. Update call sites

| Call site | Change |
|---|---|
| `_capture_process` (line ~1842) | `count_tokens(result) > MAX_INLINE_TOKENS` instead of byte-length check |
| `_guard_mcp_output` (line ~1767) | `count_tokens(result) > MCP_INLINE_TOKENS` |
| `dispatch()` MCP error branch (line ~2054) | `_safe_truncate_tokens(result, MCP_INLINE_TOKENS, ...)` |
| `_save_large_output` fallback paths (lines ~1717, ~1731) | Use `_safe_truncate_tokens` with the `inline_tokens` parameter |
| Size reporting in `_save_large_output` (line ~1753) | Report both tokens and bytes: `"({n_tokens} tokens, {size_kb:.1f}KB)"` |

### 6. Performance fast path

Use UTF-8 byte length as a conservative lower bound for token count (every token is at least 1 byte of UTF-8):

```python
if len(text.encode("utf-8")) <= max_tokens:
    return text  # guaranteed under the token limit, skip encoding
```

This is a valid proof: `len(encoder.encode(text)) <= len(text.encode("utf-8"))` always holds because every token maps to at least 1 byte. For short outputs (under the limit in bytes) this skips the tiktoken call entirely.

### 7. Absorb `snapshot.py:_estimate_tokens`

Replace the `len(text) // 4` heuristic in `snapshot.py:32` with `from swival.tokens import count_tokens`. This eliminates the second token-counting story in the codebase.

### 8. Update tests

Grep for `MAX_INLINE_OUTPUT`, `MCP_INLINE_LIMIT`, `_safe_truncate`, and `_guard_mcp_output` in `tests/` and update:
- Assertions that check byte lengths need to check token counts.
- Monkeypatched constants need new names/values.
- Any test that crafts output of a specific byte size to trigger the threshold needs adjustment.

## Things that stay byte-based

- `MAX_FILE_OUTPUT` (1 MB) — disk/memory budget for captured subprocess output, not a context concern.
- `MCP_FILE_LIMIT` (10 MB) — disk budget for saved MCP output.
- `_capture_process` stream reading (the `remaining = MAX_FILE_OUTPUT - output_total` loop) — operates on raw bytes from the pipe.
- The byte-based `_safe_truncate` helper — still used for disk-budget truncation before file write.

## Risks

- **Performance**: `tiktoken.encode()` is fast but not free. For large outputs (hundreds of KB), encoding just to check the length adds latency. Mitigated by the UTF-8 byte-length fast path described above.
- **Truncation fidelity**: Token-boundary truncation can split mid-word. Acceptable since the output is being saved to a file anyway; the inline message is just a pointer.
