# Vision Model Support for Swival

## Goal

Allow Swival to answer questions about image files (PNG, JPEG, GIF, WebP, and best-effort BMP) by sending them to vision-capable LLMs. A user should be able to say "describe the content of screenshot.png" and get a useful answer.

## Current State

- `read_file` detects binary files (null-byte check) and returns an error — images are unreadable
- All messages use `content` as a plain string throughout the codebase
- `_msg_content()` returns `str`, and callers use `.startswith()`, `len()`, slicing freely
- `handle_tool_call` (agent.py:1333) calls `result.startswith("error:")` on dispatch output
- The agent loop (agent.py:3704, 3715, 3730-3731) slices, measures, and `.startswith()`-checks `tool_msg["content"]`
- `call_llm` wraps `litellm.BadRequestError` as `AgentError` (agent.py:1587-1607), which propagates up and **aborts the run** — there is no recovery path for unsupported-model errors
- Token estimation in `estimate_tokens` (agent.py:226) passes `_msg_content()` to tiktoken
- Snapshot compaction in `snapshot.py:212` also calls `_msg_content()` for token estimates
- Cache (cache.py) serializes the full `messages` list into `_cache_key()` via `json.dumps` — base64 image payloads would bloat both cache keys and stored request JSON (`put()` persists `completion_kwargs` at cache.py:89)
- MCP client already discards image content blocks with `[image: mime, N bytes]` placeholders (mcp_client.py:503-506)
- The OpenAI-compatible Chat Completions schema (used by LM Studio, LiteLLM, and all supported providers) only allows **text** content in `role: "tool"` messages — not `image_url` parts
- `SYNTHETIC_USER_PREFIXES` (agent.py:70-78) is a tuple of string prefixes; `continue_here._is_synthetic()` checks `content.startswith(prefix)` to skip synthetic messages when finding the user's real task
- `is_pinned()` (agent.py:499) preserves all user turns during `drop_middle_turns` — including synthetic ones

## Design

### Core insight: inject the image into a user message, not a tool message

The Chat Completions schema (the OpenAI-compatible format that LM Studio and LiteLLM already speak) restricts `tool` messages to text-only content. Images can only go in `user` messages. So the flow is:

1. Agent calls `view_image` tool with a file path
2. Tool returns a **plain text** result and stashes the image payload in a per-turn `image_stash` list passed through from the agent loop
3. `handle_tool_call` appends the normal text tool message (no schema violations)
4. **After all tool calls in the turn are processed**, the agent loop checks the `image_stash`. If populated and the model supports vision, it injects a synthetic `user` message containing the image data-URL(s) and the question text. If the model lacks vision support, it injects a text-only fallback
5. The next `call_llm` invocation sees the image in a user message, which is valid for all providers

This keeps tool messages as plain strings everywhere — no content-array changes needed in `_msg.py`, no breakage in error checking, reporting, guardrails, or cache key computation.

### Synthetic image messages are marked and treated as synthetic everywhere

The injected user message always starts with a recognizable prefix: `"[image]"`. This prefix is added to `SYNTHETIC_USER_PREFIXES` so that:

- `continue_here._is_synthetic()` skips it when searching for the user's real task
- `is_pinned()` is updated to recognize synthetic image turns and **not** pin them during `drop_middle_turns`, since the image data is ephemeral context, not a real user request

### Multi-provider considerations

Swival supports LM Studio, OpenRouter, HuggingFace, ChatGPT, and generic OpenAI-compatible servers. Vision support varies:

- **LM Studio:** Users load arbitrary models (Qwen 2.5 VL, LLaVA, etc.). `litellm.supports_vision()` won't recognize these model names — the three-way check returns `None`, so we try optimistically. LM Studio's `/v1/chat/completions` endpoint passes through `image_url` content parts when the loaded model supports vision.
- **OpenRouter:** Routes to many backends. Vision support depends on the specific model. Litellm usually knows these models.
- **HuggingFace Inference API:** Vision models work via the standard messages format.
- **Generic:** Any OpenAI-compatible server — same optimistic approach as LM Studio.
- **ChatGPT:** Supports vision on GPT-4o and later. Litellm knows these.

The design handles all of these uniformly: the multimodal user message uses the OpenAI-compatible content-array format — the same transport LM Studio and LiteLLM already speak. The recovery path catches rejections regardless of provider.

### Why not put the image in the tool message?

The Chat Completions schema only allows `type: "text"` content parts in tool messages. While LiteLLM might silently pass through a list-valued content field to some providers, it would break on others. The user-message injection approach works universally across all providers Swival supports.

### Alternative considered: extending `read_file`

Tempting, but wrong. `read_file` returns numbered lines meant for code editing. An image has no lines. Mixing text-file semantics with image semantics in one tool would confuse the model and complicate the code. A dedicated tool is cleaner.

## Implementation Plan

### 1. Add the `view_image` tool definition

**File:** `swival/tools.py`

Add to `TOOLS` list:

```python
{
    "type": "function",
    "function": {
        "name": "view_image",
        "description": "Load an image file so you can see and analyze it. "
                       "Supports PNG, JPEG, GIF, and WebP. "
                       "BMP is accepted but may not work with all providers. "
                       "After calling this tool, the image will be visible "
                       "in your next response.",
        "parameters": {
            "type": "object",
            "properties": {
                "image_path": {
                    "type": "string",
                    "description": "Path to the image file",
                },
                "question": {
                    "type": "string",
                    "description": "Optional question about the image. "
                                   "If omitted, describe what you see.",
                },
            },
            "required": ["image_path"],
        },
    },
}
```

PNG, JPEG, GIF, and WebP are widely supported: LM Studio documents JPEG/PNG/WebP, OpenRouter documents PNG/JPEG/WebP/GIF, and hosted APIs support all four. BMP is accepted as best-effort — some local backends may handle it, but no provider explicitly documents BMP support. Since the data is base64-encoded with a MIME type, the worst case for an unsupported format is an API error, which the recovery path handles.

### 2. Implement `_view_image()` in tools.py

```python
import base64

IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp"}
IMAGE_MIME = {
    ".png": "image/png",
    ".jpg": "image/jpeg",
    ".jpeg": "image/jpeg",
    ".gif": "image/gif",
    ".webp": "image/webp",
    ".bmp": "image/bmp",
}
MAX_IMAGE_BYTES = 20 * 1024 * 1024  # 20 MB

def _view_image(image_path, question=None, *, base_dir, allowed_dirs,
                image_stash, ...):
    resolved = _resolve_and_check(image_path, base_dir, allowed_dirs, ...)
    if isinstance(resolved, str):  # error string
        return resolved

    ext = resolved.suffix.lower()
    if ext not in IMAGE_EXTENSIONS:
        return f"error: unsupported image format ({ext}). Supported: PNG, JPEG, GIF, WebP (and BMP on some providers)."

    size = resolved.stat().st_size
    if size > MAX_IMAGE_BYTES:
        return f"error: image too large ({size:,} bytes, max {MAX_IMAGE_BYTES:,})"
    if size == 0:
        return "error: image file is empty"

    data = resolved.read_bytes()
    b64 = base64.b64encode(data).decode("ascii")
    mime = IMAGE_MIME[ext]
    data_url = f"data:{mime};base64,{b64}"

    # Stash the data URL and question on the per-turn list that the agent
    # loop will consume after all tool calls are processed.
    image_stash.append({
        "data_url": data_url,
        "question": question or "",
        "path": str(image_path),
    })

    size_kb = len(data) / 1024
    return (
        f"Image loaded: {resolved.name} ({size_kb:.0f} KB). "
        "The image has been attached and will be visible on your next turn. "
        "Proceed to analyze it."
    )
```

The `image_stash` list is created per-turn in `run_agent_loop` and threaded through `handle_tool_call` → `dispatch` → `_view_image`. No module-level mutable state.

Wire into `dispatch()` alongside other tool handlers.

### 3. Inject images into the message stream after tool processing

**File:** `swival/agent.py`, after the tool-call loop (around line 3800)

After all tool calls in a turn are processed and appended:

```python
_IMAGE_SYNTHETIC_PREFIX = "[image]"

if image_stash:
    vision_support = _model_supports_vision(model_str)  # True / False / None

    if vision_support is False:
        # Known no-vision — don't attempt, inject text-only fallback
        messages.append({
            "role": "user",
            "content": (
                _IMAGE_SYNTHETIC_PREFIX + " The current model does not support "
                "vision/image analysis. The image could not be displayed. "
                "Please inform the user and suggest they use a vision-capable model."
            ),
        })
    else:
        # True or None (unknown) — inject multimodal message
        parts = []
        questions = [img["question"] for img in image_stash if img["question"]]
        text = _IMAGE_SYNTHETIC_PREFIX + " " + (
            " ".join(questions) if questions
            else "Describe and analyze the attached image(s)."
        )
        parts.append({"type": "text", "text": text})
        for img in image_stash:
            parts.append({
                "type": "image_url",
                "image_url": {"url": img["data_url"]},
            })
        messages.append({"role": "user", "content": parts})
        # Tag for recovery — see step 5
        _vision_pending = True
    image_stash.clear()
```

The text always starts with `"[image]"`, which is added to `SYNTHETIC_USER_PREFIXES`:

```python
SYNTHETIC_USER_PREFIXES: tuple[str, ...] = (
    "Your response was empty.",
    "Your response was cut off.",
    "IMPORTANT:",
    "STOP:",
    "Tip:",
    "Reminder:",
    "[REVIEWER FEEDBACK",
    "[image]",  # synthetic image injection from view_image tool
)
```

This ensures `continue_here._is_synthetic()` correctly skips these messages.

### 4. Extract `_resolve_model_str()` helper

**File:** `swival/agent.py`

The provider → model_str mapping is currently inline inside `call_llm` (agent.py:1485-1521). Extract it into a pure function so the image injection code can call it without duplicating the logic:

```python
def _resolve_model_str(provider: str, model_id: str) -> str:
    """Map (provider, model_id) to the litellm model string."""
    if provider == "lmstudio":
        return f"openai/{model_id}"
    elif provider == "huggingface":
        return f"huggingface/{model_id.removeprefix('huggingface/')}"
    elif provider == "openrouter":
        bare = model_id[len("openrouter/"):] if model_id.startswith("openrouter/openrouter/") else model_id
        return f"openrouter/{bare}"
    elif provider == "generic":
        return f"openai/{model_id}"
    elif provider == "chatgpt":
        return f"chatgpt/{model_id.removeprefix('chatgpt/').removeprefix('chatgpt/')}"
    else:
        return model_id
```

`call_llm` calls this internally. The image injection code in `run_agent_loop` also calls it once to get `model_str` for the `_model_supports_vision()` check.

### 5. Vision capability check and recovery

**File:** `swival/agent.py`

Two layers: a pre-check to avoid the call when we *know* it will fail, and a narrow recovery path when the call fails on the first attempt after image injection.

**Pre-check:**

```python
def _model_supports_vision(model_str: str) -> bool | None:
    """Check if the resolved model supports vision via litellm.

    Returns True, False, or None (unknown / not in registry).
    """
    try:
        import litellm
        return litellm.supports_vision(model=model_str)
    except Exception:
        return None
```

**Injection logic** is in step 3 — it uses the three-way result to either skip (False), inject optimistically (True/None), or set `_vision_pending = True` for the recovery path.

**Recovery path — narrow, index-free, one-shot:**

The problem with tracking a message index is that compaction rewrites `messages[:]` in place (agent.py:3486), invalidating any stored index. Instead, use a boolean flag `_vision_pending` that is:

- Set to `True` immediately after injecting a multimodal image user message (step 3)
- Cleared to `False` after a successful `call_llm` return
- Cleared to `False` if compaction runs (compaction already strips image content — see step 7)

On failure, instead of `pop()`-ing by index, scan backward for the image message by content type:

```python
# In run_agent_loop, around the call_llm site:
try:
    msg, finish_reason = call_llm(*_llm_args, **llm_kwargs)
    _vision_pending = False  # success — clear the flag
except AgentError as e:
    if _vision_pending and _is_vision_rejection(e):
        _vision_pending = False
        # Find and replace the image message (scan from end)
        for i in range(len(messages) - 1, -1, -1):
            if (isinstance(messages[i], dict)
                    and isinstance(messages[i].get("content"), list)
                    and any(
                        p.get("type") == "image_url"
                        for p in messages[i]["content"]
                        if isinstance(p, dict)
                    )):
                messages[i] = {
                    "role": "user",
                    "content": (
                        _IMAGE_SYNTHETIC_PREFIX + " The image could not be sent "
                        "to the model — it does not support image analysis. "
                        "Please inform the user and suggest a vision-capable model."
                    ),
                }
                break
        continue  # retry the LLM call with text-only
    raise  # Not vision-related, propagate normally
```

**Narrowing the catch with `_is_vision_rejection()`:**

The guard is not just "an image was in flight" — it also checks the error content to avoid swallowing unrelated failures (auth, network, rate limits):

```python
_VISION_REJECTION_PATTERNS = (
    "image",
    "vision",
    "multimodal",
    "content type",
    "does not support",
    "image_url",           # field name echoed in error by OpenAI-compat servers
    "invalid content",     # generic OpenAI-compat servers
)

def _is_vision_rejection(error: AgentError) -> bool:
    """Heuristic: does this error look like a vision/multimodal rejection?

    Tuned for multiple providers:
    - Hosted APIs (OpenAI, etc.): "This model does not support image input"
    - LM Studio / llama.cpp: "image_url is not supported" or similar
    - OpenRouter: wraps upstream errors, usually contains "image" or "vision"
    - vLLM: "multimodal" or "content type" in validation errors

    Intentionally heuristic — false negatives are safe (error propagates),
    false positives are unlikely (these keywords don't appear in auth/network errors).
    """
    msg = str(error).lower()
    return any(pattern in msg for pattern in _VISION_REJECTION_PATTERNS)
```

**Provider-specific notes on the heuristic:**

- **LM Studio:** When a non-vision model (e.g. a text-only Qwen) receives `image_url` content, LM Studio's `/v1/chat/completions` endpoint returns a 400 with a message mentioning "image_url" or "image". The heuristic catches this. Vision-enabled models (Qwen2-VL, Qwen3-VL, LLaVA, etc.) accept the same content-array format natively.
- **Local servers (llama.cpp, vLLM, MLX):** Error messages vary but generally reference the content type or "multimodal". If a server returns a truly generic 400 with no vision-related keywords, the heuristic will false-negative — the error propagates and the run aborts with a clear message, which is the safe direction.
- **OpenRouter / HuggingFace:** These proxy upstream errors, which usually contain "image" or "vision" from the underlying model provider.

**Why this is safe:**

- `_vision_pending` is only True for the single LLM call immediately after image injection — compaction clears it, success clears it
- `_is_vision_rejection()` further narrows the catch to vision-related error messages
- The recovery replaces the image message in-place (no index math) by scanning for `image_url` content
- It fires at most once per image injection — on retry, `_vision_pending` is False
- Auth, network, provider, and rate-limit errors propagate normally because they won't match the rejection patterns

### 6. Update `_msg_content()` defensively

**File:** `swival/_msg.py`

Even though we only put arrays in `user` messages (not `tool` messages), `_msg_content()` is called on all messages including user messages during compaction, token estimation, and continue-here serialization. Make it handle both:

```python
def _msg_content(msg) -> str:
    c = msg.get("content", "") if isinstance(msg, dict) else getattr(msg, "content", "")
    if isinstance(c, list):
        # Extract text parts from multimodal content arrays
        return " ".join(
            part.get("text", "")
            for part in c
            if isinstance(part, dict) and part.get("type") == "text"
        )
    return c or ""
```

This is a defensive change that prevents crashes if a list-valued content reaches any of the ~20 call sites. It does **not** preserve image data — that's intentional for compaction and token estimation.

`_set_msg_content()` needs no changes — it already writes a plain string, which naturally replaces any list.

### 7. Make synthetic image turns droppable in compaction

**File:** `swival/agent.py`

Currently `is_pinned()` (agent.py:499) pins all user turns, which would prevent synthetic image messages from ever being dropped. Update it:

```python
def is_pinned(turn: list) -> bool:
    """User turns are always preserved — except synthetic image injections."""
    for msg in turn:
        if _msg_role(msg) == "user":
            content = _msg_content(msg)
            if content.startswith(_IMAGE_SYNTHETIC_PREFIX):
                return False  # synthetic image turn, droppable
            return True
    return False
```

Additionally, in `compact_messages` / `drop_middle_turns` / `aggressive_drop_turns`, flatten any surviving image user messages to text-only early — before the scoring or LLM summarization step:

```python
# Early pass: strip base64 from image user messages that survived to this point
for msg in messages:
    if _msg_role(msg) == "user" and isinstance(
        msg.get("content") if isinstance(msg, dict) else getattr(msg, "content", None),
        list,
    ):
        text = _msg_content(msg)  # extracts text parts
        _set_msg_content(msg, text + " [image data removed during compaction]")
```

This ensures base64 blobs never survive compaction, even if the turn isn't dropped entirely.

### 8. Image-aware token estimation

**File:** `swival/agent.py`, in `estimate_tokens` (line 226)

When a message has list-valued content, estimate the image tokens separately. Use a worst-case bound rather than a flat average — the standard formula (used by hosted APIs) is 85 base + 170 per 512x512 tile, and high-detail images can use up to ~16 tiles. Local servers may count differently, but overestimating is safe for context-window budgeting:

```python
# Image token budget: worst-case high-detail (85 base + 170 * 16 tiles = 2805)
# This overestimates for small images but prevents context overflow surprises.
_IMAGE_TOKEN_ESTIMATE = 2805

def estimate_tokens(messages: list, tools: list | None = None) -> int:
    total = 0
    for m in messages:
        content_raw = _msg_get(m, "content", "")
        if isinstance(content_raw, list):
            for part in content_raw:
                if isinstance(part, dict):
                    if part.get("type") == "text":
                        total += len(_encoder.encode(part.get("text", "")))
                    elif part.get("type") == "image_url":
                        total += _IMAGE_TOKEN_ESTIMATE
        else:
            total += len(_encoder.encode(_msg_content(m)))
        # ... rest unchanged (tool_calls handling)
```

Also update `snapshot.py:212` to use `_msg_get` and handle list content with the same estimate, or call a shared helper.

### 9. Skip caching for vision requests entirely

**File:** `swival/cache.py`

Rather than trying to store base64 blobs in the cache DB (which would balloon `cache.db` — `put()` persists the full `completion_kwargs` JSON at cache.py:89), skip caching when messages contain image data:

```python
def _has_image_content(messages: list) -> bool:
    """Check if any message contains image_url parts."""
    for m in messages:
        if isinstance(m, dict) and isinstance(m.get("content"), list):
            for part in m["content"]:
                if isinstance(part, dict) and part.get("type") == "image_url":
                    return True
    return False
```

In `call_llm`, before the cache lookup:

```python
# Skip cache for vision requests — base64 payloads would bloat the DB
if cache is not None and _has_image_content(messages):
    cache = None  # disable for this call
```

This is simpler than partial hashing and avoids the DB growth problem entirely. Vision requests are unlikely to benefit from caching anyway — each image is unique.

### 10. Update `read_file` to suggest `view_image`

**File:** `swival/tools.py`

Change the binary-file error in `_read_file` to detect image extensions and return a helpful hint:

```python
ext = resolved.suffix.lower()
if b"\x00" in chunk:
    if ext in IMAGE_EXTENSIONS:
        return f"error: {file_path} is an image file. Use view_image to analyze it."
    return f"error: binary file detected: {file_path}"
```

## Failure Strategy

### Model doesn't support vision

Three-layer defense:

1. **Pre-check (known no-vision):** `_model_supports_vision()` returns `False` → text-only fallback injected, image never sent. No API call wasted.
2. **Optimistic attempt (unknown support):** `_model_supports_vision()` returns `None` (model not in litellm's registry, e.g. local LM Studio) → image is injected and the API call proceeds.
3. **Narrow recovery (API rejects image):** If `call_llm` raises `AgentError` while `_vision_pending` is True **and** `_is_vision_rejection(e)` matches the error text, the agent loop replaces the image message with a text-only explanation and retries. Non-vision errors (auth, network, rate limits) propagate normally because they don't match the rejection patterns.

The `_vision_pending` flag is scoped tightly: set after image injection, cleared on success, cleared on compaction. Combined with the error-content check, this ensures the recovery only fires for genuine vision rejections on the immediate next LLM call.

### Image file doesn't exist / wrong format / too large

These are handled by the tool returning `"error: ..."` strings — the existing guardrail system kicks in if the model retries with the same bad arguments.

## What NOT to do

- **Don't put image data in tool messages.** The Chat Completions schema forbids it.
- **Don't add `--vision` config flags.** Use `litellm.supports_vision()` at runtime.
- **Don't support TIFF.** Rare in practice, large files, and no mainstream vision API accepts it natively.
- **Don't resize or compress images.** Let the provider handle it.
- **Don't add image generation.** Different feature entirely.
- **Don't change `_msg_content` return type.** Keep it returning `str` everywhere.
- **Don't cache vision requests.** Base64 payloads would balloon the SQLite DB.

## Testing

1. **`_view_image` unit tests:** path validation, extension filtering, size cap, base64 correctness, stash population, error messages for unsupported formats
2. **Image injection tests:** mock `handle_tool_call` to populate the stash, verify the synthetic user message has correct multimodal structure, verify text starts with `"[image]"` prefix
3. **`_msg_content` with list content:** verify text extraction, verify empty list returns `""`
4. **Compaction with image messages:** verify base64 is stripped, text summary remains, verify image turns are **not** pinned by `is_pinned()`
5. **Token estimation with images:** verify list-content messages get worst-case per-image estimate
6. **Cache skip:** verify `_has_image_content()` detection, verify cache is disabled for vision calls
7. **`_model_supports_vision` three-way:** mock returning False → text-only fallback; returning True → multimodal injection; raising → returns None
8. **`_is_vision_rejection` matching:** verify matches "image", "vision", "multimodal", "does not support"; verify does NOT match "authentication", "rate limit", "timeout", "connection"
9. **Vision recovery fires narrowly:** inject image, mock `call_llm` raising `AgentError("model does not support image input")` → verify image message replaced in-place, loop continues, `_vision_pending` cleared
10. **Recovery skips non-vision errors:** `_vision_pending` True but error is "authentication failed" → verify `AgentError` propagates (not swallowed)
11. **Recovery skips when no image in-flight:** `_vision_pending` False, `AgentError` raised → verify propagates normally
12. **Recovery after compaction:** inject image, trigger compaction (which clears `_vision_pending`) → verify subsequent `AgentError` propagates, not caught by recovery
13. **`read_file` hint:** call `_read_file` on a PNG, verify error mentions `view_image`
14. **continue_here synthetic detection:** verify `_is_synthetic("[image] ...")` returns True
15. **`_resolve_model_str` helper:** verify all providers produce correct model strings, verify `call_llm` still works after refactor

Test images: 1x1 PNGs generated in fixtures with `struct.pack` — no real photos needed.

## Files Changed

| File | Change |
|------|--------|
| `swival/tools.py` | `view_image` tool def, `_view_image()` impl, `dispatch` routing, `IMAGE_EXTENSIONS`/`IMAGE_MIME` constants, `image_stash` param, `read_file` image hint |
| `swival/agent.py` | `_resolve_model_str()` extracted from `call_llm`, `_model_supports_vision()` (three-way), `_is_vision_rejection()` heuristic, `_IMAGE_SYNTHETIC_PREFIX`, image stash plumbing through `handle_tool_call`, post-tool-loop injection with three-way check (~50 lines), `_vision_pending` flag + narrow `AgentError` recovery around `call_llm`, `estimate_tokens` list-content handling, `is_pinned()` excludes image turns, compaction base64 stripping + `_vision_pending` clearing, `SYNTHETIC_USER_PREFIXES` updated, cache skip for vision |
| `swival/_msg.py` | `_msg_content()` handles list content defensively |
| `swival/cache.py` | `_has_image_content()` helper (used by agent.py) |
| `swival/snapshot.py` | Token estimation handles list-content messages |
| `tests/test_vision.py` | New test file covering all above |

## Rough size estimate

~300 lines of production code, ~450 lines of tests.
