# Gen Module — Developer Reference

> **Audience:** contributors and Claude Code maintainers.
> The engine layer (`modules/gen/engine/`) is installed as the `sunwaee` PyPI package and is also imported by `api/` — changes here affect both consumers.

---

## File Tree

```
modules/gen/
├── __init__.py                  # lazy re-exports (stream_run, Session, get_engine, …)
├── agent.py                     # ReAct loop — run() + stream_run() + _execute()
├── session.py                   # JSONL session persistence
├── tools.py                     # aggregates Tool lists from other modules
│
├── commands/
│   ├── __init__.py              # Typer app ("gen") with sub-apps wired in
│   ├── code.py                  # `sunwaee gen` — entry point, starts the REPL
│   ├── models.py                # `gen models list`
│   ├── sessions.py              # `gen sessions list`
│   └── set_model.py             # `gen set-model` interactive picker
│
├── engine/
│   ├── __init__.py              # public surface: get_engine, Message, Response, Tool, …
│   ├── base.py                  # BaseEngine ABC (chat + stream)
│   ├── factory.py               # get_engine() — maps provider string → engine instance
│   ├── model.py                 # Model dataclass + tiered cost computation
│   ├── types.py                 # Message, Response, ToolCall, Usage, Cost, Performance, …
│   │
│   ├── models/
│   │   ├── __init__.py          # get_model(), list_models() registry helpers
│   │   ├── anthropic.py         # 4 models  (Sonnet 4.6, Opus 4.6/4.5, Haiku 4.5)
│   │   ├── openai.py            # 27+ models (GPT 5.4 Pro/Mini/Nano … GPT 4.1-nano)
│   │   ├── google.py            # 11+ models (Gemini 3.1 Flash Lite … Gemini 2.0 Flash)
│   │   ├── deepseek.py          # 2 models   (deepseek-reasoner, deepseek-chat)
│   │   ├── xai.py               # 7 models   (Grok 4.20 Beta … Grok 3-mini)
│   │   └── moonshot.py          # 1 model    (Kimi K2.5)
│   │
│   └── providers/
│       ├── __init__.py          # empty
│       ├── anthropic.py         # AnthropicEngine — streaming + non-streaming
│       ├── openai.py            # OpenAIEngine — shared by OpenAI, DeepSeek, xAI, Moonshot
│       └── google.py            # GoogleEngine  — streaming + non-streaming
│
└── repl/
    ├── __init__.py              # exports run()
    ├── loop.py                  # main REPL event loop (stream_run, vim commands, display)
    ├── input.py                 # InputSession + ReplCompleter (prompt_toolkit)
    └── display.py               # Rich formatting helpers (banner, tool calls, cost, …)

tests/gen/
├── __init__.py
├── test_agent.py                # unit — run() with mock engine; tool dispatch, iteration cap
├── test_stream_agent.py         # unit — stream_run() with mock engine; streaming tool loop
├── test_session.py              # unit — JSONL round-trips, create/load/append/delete
├── test_tools.py                # unit — TOOLS list contains all expected module names
│
└── engine/
    ├── __init__.py
    ├── test_types.py            # unit — resolve_tokens(), make_non_stream_performance(), serialization
    ├── test_factory.py          # unit — get_engine() dispatch, env var fallback, error paths
    ├── test_model.py            # unit — compute_cost() tiered pricing, get_model(), list_models()
    │
    ├── models/
    │   └── __init__.py          # (placeholder, model catalog tests live in test_model.py)
    │
    ├── providers/
    │   ├── __init__.py
    │   ├── test_anthropic.py    # unit — AnthropicEngine payload building + response parsing (mocked HTTP)
    │   └── test_openai.py       # unit — OpenAIEngine payload building + response parsing (mocked HTTP)
    │
    └── live/
        ├── __init__.py
        ├── test_providers.py    # live — real API calls, all providers × all scenarios × chat+stream
        └── run/                 # output dir — JSON snapshots written after each live run
            └── .gitkeep
```

---

## Architecture

```
User types message
        │
        ▼
repl/loop.py  run()                        ← REPL event loop, vim commands
        │  builds [system_prompt] + history + [user_msg]
        ▼
agent.py  stream_run()                     ← ReAct orchestrator (max 10 iterations)
        │
        ├─ engine.stream(messages, tools)  ← provider-specific SSE client
        │     ├─ AnthropicEngine.stream()
        │     ├─ OpenAIEngine.stream()
        │     └─ GoogleEngine.stream()
        │
        │  yields Response chunks:
        │    reasoning_content / reasoning_signature
        │    content
        │    tool_calls
        │    (final chunk: stop_reason, usage, cost, performance)
        │
        ├─ if stop_reason == TOOL_USE:
        │     asyncio.gather(*_execute(tc) for tc in tool_calls)
        │     append tool_result Messages
        │     yield synthetic Response (signals UI)
        │     → next iteration
        │
        └─ else: return
                │
                ▼
        session.py  append_messages()      ← JSONL persistence
```

---

## Key Types (`engine/types.py`)

| Type          | Purpose                                                                                     |
| ------------- | ------------------------------------------------------------------------------------------- |
| `Role`        | `SYSTEM / USER / ASSISTANT / TOOL`                                                          |
| `StopReason`  | `END_TURN / TOOL_USE / MAX_TOKENS`                                                          |
| `Message`     | One conversation turn — role, content, reasoning_content, reasoning_signature, tool_calls   |
| `ToolCall`    | A single tool invocation — id, name, arguments, results, error, duration, thought_signature |
| `Tool`        | A callable with description + JSON schema (from `@tool` decorator)                          |
| `Usage`       | Token counts — input, output, total, cache_read, cache_write                                |
| `Cost`        | Pricing breakdown — input, output, cache_read, cache_write, total                           |
| `Performance` | Latency metrics — latency, reasoning_duration, content_duration, throughput                 |
| `Response`    | Full provider response — all of the above plus provider, model, streaming flag, error       |

---

## Generation Flows

### Streaming (normal REPL path)

1. `repl/loop.py` calls `agent.stream_run(all_msgs, TOOLS, engine)`.
2. `stream_run` calls `engine.stream(messages, tools)` which opens an SSE connection and yields `Response` chunks:
   - `reasoning_content` — extended thinking text (styled italic in REPL)
   - `reasoning_signature` — opaque bytes to echo back next turn
   - `content` — plain answer text
   - `tool_calls` — accumulated once per tool batch
   - final chunk — `stop_reason`, `usage`, `cost`, `performance`
3. On `stop_reason == TOOL_USE`: tools run concurrently (`asyncio.gather`), results appended as `Role.TOOL` messages, loop continues.
4. On `stop_reason == END_TURN` or `MAX_TOKENS`: stream ends.
5. `repl/loop.py` appends user + assistant messages to the session file.

### Non-Streaming

`agent.run()` wraps `engine.chat()` synchronously. Used only for programmatic callers, not the REPL.

### Tool Execution (`agent._execute`)

```
_execute(tc, tools_by_name)
    ├─ unknown tool  → error JSON {"ok": false, "error": "Unknown tool …"}
    ├─ no callable   → error JSON {"ok": false, "error": "Tool … has no callable"}
    ├─ call fn (async via await, sync via run_in_executor)
    ├─ catch ALL exceptions → error JSON + tc.error + tc.duration
    ├─ try JSON-decode result
    │    success: wrap dict/list → tc.results
    │    failure: store raw string → tc.results
    └─ return raw string
```

---

## Provider Dispatch (`engine/factory.py`)

```python
def get_engine(
    provider: str,
    model: str,
    api_key: str | None = None,         # falls back to {PROVIDER_UPPERCASE}_API_KEY env var
    max_tokens: int = 8192,
    thinking_budget: int | None = None, # Anthropic/Google extended thinking
    reasoning_effort: str | None = None # OpenAI: "low" | "medium" | "high"
) -> BaseEngine
```

```python
_OPENAI_COMPATIBLE = {
    "openai":    "https://api.openai.com/v1",
    "deepseek":  "https://api.deepseek.com/v1",
    "moonshot":  "https://api.moonshot.ai/v1",
    "xai":       "https://api.x.ai/v1",
}
```

DeepSeek, Moonshot, and xAI are **all routed through `OpenAIEngine`** with their respective base URLs. Only Anthropic and Google have dedicated engine classes.

### Connection pooling

`factory.py` maintains a module-level dict of persistent `httpx.AsyncClient` instances:

```python
_HTTP_CLIENTS: dict[tuple[int, str], httpx.AsyncClient] = {}
```

The key is `(id(asyncio.get_running_loop()), base_url)`. `_get_client(base_url)` looks up or creates the client for the current event loop + base URL pair, then passes it into every engine constructor via the `client=` parameter.

**Why loop-scoped, not just URL-scoped:** `httpx.AsyncClient` is tied to the event loop it was created in. Keying only by URL meant that in tests (where each test function gets its own event loop) the second test would receive a client bound to a closed loop and raise `RuntimeError: Event loop is closed`. In production the server has one long-lived loop, so the key is effectively constant and connection reuse is global for the process lifetime.

**Why this matters:** without pooling, every `engine.chat()` / `engine.stream()` call opened a fresh TCP connection + TLS handshake (~150–300 ms overhead). With a persistent client, the connection is reused across the `autoselect_engine` call, the main generation stream, and the `name_chat` background call — all of which hit the same provider base URL.

---

## Pricing (`engine/model.py`)

Cost tiers kick in based on **input token count**:

| Threshold     | Tier suffix |
| ------------- | ----------- |
| default       | (base rate) |
| > 128k tokens | `_128k`     |
| > 200k tokens | `_200k`     |
| > 272k tokens | `_272k`     |

The 272k tier exists **only for OpenAI** (comment in source: "hi there we're openai and we love to fk dev experience up").

`compute_cost(usage, model)` bills each component independently:

```
regular_input = input_tokens − cache_read_tokens − cache_write_tokens
cost = regular_input × input_rate
     + cache_read_tokens  × cache_read_rate
     + cache_write_tokens × cache_write_rate
     + output_tokens      × output_rate
```

---

## Prompt Caching

| Provider  | Mechanism                                                                 |
| --------- | ------------------------------------------------------------------------- |
| Anthropic | `cache_control: {"type": "ephemeral"}` on system message and root payload |
| OpenAI    | Implicit; tracked via `prompt_tokens_details.cached_tokens`               |
| Google    | Implicit; tracked via `cachedContentTokenCount` in usage metadata         |

Cache token costs are billed separately in `compute_cost()`.

---

## Streaming Event Walkthrough per Provider

### Anthropic SSE

| Event                                           | Action                                               |
| ----------------------------------------------- | ---------------------------------------------------- |
| `message_start`                                 | Extract input tokens                                 |
| `content_block_start` (type=`tool_use`)         | Begin tool buffer                                    |
| `content_block_delta` (type=`thinking_delta`)   | Yield `reasoning_content`                            |
| `content_block_delta` (type=`signature_delta`)  | Yield `reasoning_signature`                          |
| `content_block_delta` (type=`text_delta`)       | Yield `content`                                      |
| `content_block_delta` (type=`input_json_delta`) | Accumulate tool JSON                                 |
| `content_block_stop`                            | Finalize tool call (parse JSON args)                 |
| `message_delta`                                 | Extract final `stop_reason` + output tokens          |
| Stream close                                    | Yield final Response with usage + cost + performance |

### OpenAI SSE

| Event                                 | Action                                                           |
| ------------------------------------- | ---------------------------------------------------------------- |
| First chunk (with `reasoning_effort`) | Yield synthetic "reasoning in progress" Response immediately     |
| `delta.reasoning_content`             | Yield `reasoning_content`                                        |
| `delta.content`                       | Yield `content`; record `t_reasoning_end` on first content chunk |
| `delta.tool_calls[i]`                 | Accumulate in buffer by index                                    |
| `finish_reason` set                   | Determine stop reason                                            |
| Final `[DONE]`                        | Yield final Response                                             |

### Google SSE

| Part type      | Action                                                                                         |
| -------------- | ---------------------------------------------------------------------------------------------- |
| `thought`      | Yield `reasoning_content`, then yield `reasoning_signature` separately                         |
| `text`         | Yield `content`                                                                                |
| `functionCall` | Yield `tool_calls` immediately (complete, no buffering); include `thoughtSignature` if present |
| `finishReason` | Determine stop reason; if has_tool → override to `TOOL_USE`                                    |

---

## Exceptions and Provider-Specific Quirks

Every item below documents a load-bearing non-obvious behavior. Do not remove without understanding the reason.

---

### 1. Token count normalization (`types.py` — `resolve_tokens()`)

**Problem:** xAI and Google (with thinking enabled) exclude reasoning tokens from `output_tokens` but include them in `total_tokens`. This makes `input + output ≠ total`.

**Fix:** `resolve_tokens()` treats `total_tokens` as the source of truth. If `input + output ≠ total`, it back-calculates the correct output figure from `total − input`. This runs on every Response before cost computation.

---

### 2. OpenAI reasoning is silent — tokens not streamed (`providers/openai.py`)

**Problem:** When `reasoning_effort` is set, OpenAI reasoning models do **not** emit reasoning tokens over SSE. The connection just hangs until the answer starts, giving no visual feedback.

**Fix:** When `reasoning_effort` is set and the stream begins, `OpenAIEngine.stream()` immediately yields a synthetic `Response` with `reasoning_content = "Reasoning in progress - tokens not available for this model"` and records `t_reasoning_start = t_start` so the timing span appears correctly in the final performance block.

---

### 3. Google `thoughtSignature` lives on `functionCall` parts (`providers/google.py`)

**Problem:** When Gemini thinking is enabled and a tool is called, the `thoughtSignature` (the opaque blob Gemini requires echoed back) is **embedded on the `functionCall` part** — not in a separate top-level thought block. Unlike Anthropic, where the signature lives alongside the reasoning block in the assistant turn.

**Fix:**

- When parsing a `functionCall` part, capture its `thoughtSignature` into `ToolCall.thought_signature`.
- When building the next request (`_build_contents`), emit `thoughtSignature` on the `functionCall` part of each echoed tool call.
- The response-level `thoughtSignature` (top-level reasoning block) is only applied to `parts[0]` **if no per-ToolCall signature exists**.
- Importantly: Google requires the signature on **every assistant turn**, not just the last one (unlike Anthropic).

---

### 4. Google has no tool call IDs (`providers/google.py`)

**Problem:** Google Gemini does not assign IDs to tool calls. Our `ToolCall` type requires an ID for correlation.

**Fix:** Use the **function name** as the tool call ID throughout the Google provider. `_build_contents()` builds a `name → function_name` mapping for tool result echo so the round-trip still matches up.

---

### 5. Reasoning echoed only on the last assistant turn (all providers)

**Problem:** Reasoning content includes provider-specific opaque signatures (Anthropic's `signature`, Google's `thoughtSignature`). Sending these to the wrong provider — or sending old signatures in earlier history turns — breaks the API.

**Fix:** All three providers strip `reasoning_content` and `reasoning_signature` from all assistant messages **except the very last one** before building the request payload. This lets sessions switch providers mid-conversation without corrupting earlier turns.

---

### 6. System-only input promoted to user message (`providers/anthropic.py`, `providers/google.py`)

**Problem:** Both Anthropic and Google require at least one non-system message. If only a system prompt exists, the API rejects the request.

**Fix:** `_split_system()` (Anthropic) and `_build_contents()` (Google) detect this case and promote the system content to a `user` role message.

---

### 7. Anthropic thinking budget constraints (`providers/anthropic.py`)

**Rule:** Anthropic requires `1024 ≤ thinking_budget < max_tokens`.

**Default:** `thinking_budget = max(1024, max_tokens − 1024)` — leaves at least 1024 tokens for the answer.

**Override:** A custom `thinking_budget` bypasses this default but must still satisfy Anthropic's constraints or the API will reject it.

---

### 8. OpenAI uses `max_completion_tokens`, not `max_tokens` (`providers/openai.py`)

OpenAI's API parameter is `max_completion_tokens`. All other providers (including OpenAI-compatible ones routing through `OpenAIEngine`) accept `max_tokens`. The OpenAI provider translates this at payload-build time.

---

### 9. Tool call JSON buffered across streaming chunks (all providers)

Streaming tool arguments arrive as incremental JSON deltas. All providers buffer these:

- **Anthropic:** Buffer keyed by `content_block` index; merged via `input_json_delta`.
- **OpenAI:** Buffer keyed by `tool_calls[i]` index; `arguments` string concatenated per delta. If no ID present in a chunk, generates `f"tc_{i}"` as fallback.
- **Google:** Emits complete `functionCall` objects in a single chunk — no buffering needed.

---

### 10. Google streaming uses `streamGenerateContent?alt=sse` (`providers/google.py`)

Google's REST streaming endpoint requires the `alt=sse` query parameter. Without it, the response is returned as a JSON array rather than an SSE stream. On 4xx/5xx responses, the provider reads the full body before raising so the error message is useful.

---

### 11. Max iterations guard (`agent.py`)

`stream_run` / `run` allow at most **10 ReAct iterations** by default. If the tool loop hits this limit, it logs a warning and returns the last assistant message or a generic "Max iterations reached" string. This prevents runaway tool loops.

---

### 12. All tool calls executed concurrently (`agent.py`)

Within a single iteration, all tool calls from one assistant turn run via `asyncio.gather()`. Sync tool functions are dispatched to a thread pool (`run_in_executor`) so they don't block the event loop. There is no per-tool timeout (only the HTTP client's 120 s timeout applies).

---

### 13. `update_session_meta` rewrites the whole file (`session.py`)

Updating session metadata (e.g. title or `updated_at`) requires changing line 1 of the JSONL file. There is no in-place line rewrite, so the implementation reads all lines, replaces line 0, and writes the file back. For large sessions this is O(n) in message count.

---

### 14. Gitignore cache keyed by cwd (`repl/input.py`)

The `@file` completer caches parsed `.gitignore` rules in a module-level dict keyed by current working directory. The cache is never invalidated during a session — changes to `.gitignore` mid-session are not reflected in completions.

---

## Testing

```bash
pytest tests/gen/                          # all gen unit tests
pytest tests/gen/ -m "not live"            # explicit unit-only (same as above)
pytest tests/gen/ -m live                  # live tests (real API calls, keys required)
pytest tests/gen/engine/live/              # live tests only
pytest tests/gen/test_stream_agent.py      # single file
pytest tests/gen/ -k "anthropic"           # filter by name
```

### Unit Tests (no API keys needed, safe for CI)

All unit tests mock HTTP clients — no real provider calls are made.

| File                                 | What it covers                                                                                        |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| `test_agent.py`                      | `run()` — tool dispatch, error handling, iteration cap, sync/async tools                              |
| `test_stream_agent.py`               | `stream_run()` — streaming tool loop, concurrent execution, synthetic chunks                          |
| `test_session.py`                    | JSONL round-trips — create, load, append, update meta, delete; `_msg_to_dict` / `_dict_to_msg`        |
| `test_tools.py`                      | `TOOLS` list contains all expected module names                                                       |
| `engine/test_types.py`               | `resolve_tokens()`, `make_non_stream_performance()`, dataclass serialization                          |
| `engine/test_factory.py`             | `get_engine()` dispatch for all providers, env var fallback, missing key error                        |
| `engine/test_model.py`               | `compute_cost()` tiered pricing (base / 128k / 200k / 272k), `get_model()`, `list_models()`           |
| `engine/providers/test_anthropic.py` | `AnthropicEngine` — payload building, response parsing, streaming events (mocked HTTP)                |
| `engine/providers/test_openai.py`    | `OpenAIEngine` — payload building, response parsing, streaming events, DeepSeek variant (mocked HTTP) |

### Live Tests (`-m live`)

Live tests in `tests/gen/engine/live/test_providers.py` call real provider APIs and assert that every `Response` field is correctly populated — including `cost`, `usage`, and `performance`.

**Scenarios tested per provider × both `chat` and `stream`:**

| Scenario           | Description                                                                                          |
| ------------------ | ---------------------------------------------------------------------------------------------------- |
| `ONLY_SYSTEM`      | System message only — edge case; lenient assertions                                                  |
| `ONLY_USER`        | Single user message                                                                                  |
| `SYSTEM_AND_USER`  | System prompt + user message; asserts system prompt is respected                                     |
| `TOOL_CALL`        | Model must issue a tool call                                                                         |
| `TOOL_CALL_RESULT` | Full multi-turn: runs TOOL_CALL first to capture real IDs/signatures, then replays with tool results |

`TOOL_CALL_RESULT` is run as a separate test because Google has no tool call IDs (function name used as ID) and Anthropic/Google require the reasoning signature echoed verbatim — capturing a live call first avoids format mismatches.

**Providers covered:** Anthropic (`claude-haiku-4-5`), OpenAI (`gpt-5-nano`), Google (`gemini-3.1-flash-lite-preview`), xAI (`grok-4-1-fast`), DeepSeek (`deepseek-chat`), Moonshot (`kimi-k2.5`).

A test is **skipped** (not failed) when its API key is absent.

**API keys read from environment:**

```bash
ANTHROPIC_API_KEY
OPENAI_API_KEY
GOOGLE_API_KEY
XAI_API_KEY
DEEPSEEK_API_KEY
MOONSHOT_API_KEY
```

**Output:** Results saved to `tests/gen/engine/live/run/{provider}_{model}_{scenario}_{mode}.json` after each run for inspection.

There are also **provider cross-turn switch** snapshots in `run/` (e.g. `anthropic_…_to_openai_…_turn_switch.json`) validating that mid-session provider switches don't corrupt the reasoning signature handling.

---

## Adding a New Provider

1. Create `engine/models/<provider>.py` — list of `Model` instances.
2. Add the model list to `engine/models/__init__.py` registry.
3. If the provider is OpenAI-compatible: add an entry to `_OPENAI_COMPATIBLE` in `factory.py` with the base URL. `_get_client()` picks it up automatically.
4. Otherwise: create `engine/providers/<provider>.py` extending `BaseEngine`, implement `chat()` and `stream()`. Accept `client: httpx.AsyncClient | None = None` in `__init__` and store it as `self._client = client or httpx.AsyncClient()` — this allows the factory to inject the shared pooled client while keeping the class usable standalone.
5. Wire the new engine class into `factory.py`: call `_get_client(YourEngine.BASE_URL)` and pass the result as `client=` to the constructor.
6. Add live test coverage in `tests/gen/engine/live/test_providers.py` — add the `(provider, model)` tuple to `ENGINES`.

---

## User Guide

### Commands

```bash
sunwaee gen                                # new session with default model
sunwaee gen --model claude-opus-4-6        # override model for this session
sunwaee gen --session <id>                 # resume a previous session
sunwaee gen models list                    # all models + capabilities + pricing
sunwaee gen sessions list                  # saved sessions for current workspace
sunwaee gen set-model                      # interactive default model picker (Tab completion)
```

### REPL Commands

| Command         | Description                      |
| --------------- | -------------------------------- |
| `:help`         | List available commands          |
| `:session`      | Show session metadata            |
| `:cost`         | Accumulated token usage and cost |
| `:model`        | Show current model               |
| `:model <name>` | Switch model mid-session         |
| `:clear`        | Clear conversation history       |
| `:quit` / `:q`  | Exit                             |

### Providers & API Keys

| Provider  | Env var             |
| --------- | ------------------- |
| Anthropic | `ANTHROPIC_API_KEY` |
| OpenAI    | `OPENAI_API_KEY`    |
| Google    | `GOOGLE_API_KEY`    |
| DeepSeek  | `DEEPSEEK_API_KEY`  |
| xAI       | `XAI_API_KEY`       |
| Moonshot  | `MOONSHOT_API_KEY`  |
