# Plan: Multi-Model Support for Cleave

**Goal**: Make cleave useful with models beyond Claude Code, including local inference.

---

## Current State

Cleave TUI is **tightly coupled** to Claude Agent SDK:
- Direct imports from `claude_agent_sdk`
- Hardcoded model names (`claude-sonnet-4-20250514`, etc.)
- Authentication tied to `~/.claude/.credentials.json`
- Event parsing assumes Claude SDK types (`AssistantMessage`, `ResultMessage`)
- Skill invocation uses `/cleave` which is Claude Code specific

**Coupling level: 9/10 - significant refactoring required**

---

## Research Summary

### Local Inference Engines

| Engine | Best For | API |
|--------|----------|-----|
| **Ollama** | Getting started, local dev | OpenAI-compatible, tool calling |
| **llama.cpp** | Resource-constrained, single-user | OpenAI-compatible via llama-server |
| **vLLM** | High throughput, multi-user | OpenAI-compatible |
| **llamafile** | Zero-install distribution | OpenAI-compatible |

### Recommended Models for Agentic Coding

| Model | Size | Tool Calling F1 | Notes |
|-------|------|-----------------|-------|
| **Qwen3:14B** | ~8GB VRAM | 0.971 | Best accuracy, close to GPT-4 (0.974) |
| **Qwen2.5-Coder 7B** | ~5GB VRAM | 0.88+ | Best speed/accuracy trade-off |
| **DeepSeek Coder V2** | Variable | Good | Strong but Qwen edging ahead |

### Key Challenges

1. **Tool calling reliability** - Local models struggle with correct tool invocation
2. **Context length vs VRAM** - 32K context on 8B model = ~5GB VRAM
3. **Latency** - Qwen3:14B takes ~142s vs GPT-4's ~5s per test case
4. **Hallucinations/loops** - Local models prone to infinite agent loops

---

## Architecture Plan

### Phase 1: Abstract Backend Interface

Create a provider-agnostic interface that different backends can implement.

```
src/cleave/tui/
├── backends/
│   ├── __init__.py          # Backend registry
│   ├── base.py              # Abstract Backend class
│   ├── claude.py            # Claude SDK backend
│   ├── openai_compat.py     # OpenAI-compatible backend (Ollama, vLLM, etc.)
│   └── config.py            # Backend configuration
```

**Backend Interface:**
```python
class Backend(ABC):
    """Abstract backend for LLM inference."""

    @abstractmethod
    async def connect(self, config: BackendConfig) -> None: ...

    @abstractmethod
    async def query(self, prompt: str) -> AsyncIterator[Message]: ...

    @abstractmethod
    async def disconnect(self) -> None: ...

    @classmethod
    @abstractmethod
    def check_auth(cls) -> tuple[bool, str]: ...

    @classmethod
    @abstractmethod
    def available_models(cls) -> list[str]: ...
```

### Phase 2: OpenAI-Compatible Backend

Implement a single backend that works with any OpenAI-compatible API:

```python
class OpenAICompatibleBackend(Backend):
    """Backend for Ollama, vLLM, llama.cpp, llamafile, etc."""

    def __init__(self, base_url: str, model: str, api_key: str | None = None):
        self.base_url = base_url  # e.g., "http://localhost:11434/v1"
        self.model = model        # e.g., "qwen2.5-coder:7b"
        self.api_key = api_key
```

**Tool calling via OpenAI format:**
```python
{
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "read_file",
                "description": "Read a file from disk",
                "parameters": {...}
            }
        }
    ]
}
```

### Phase 3: Configuration System

Extend `~/.cleave/settings.yaml` to include backend configuration:

```yaml
# ~/.cleave/settings.yaml
cloven_report_naming: descriptor

# Backend configuration
backend: ollama  # or: claude, openai, vllm, llamacpp

backends:
  claude:
    # Uses Claude Agent SDK (existing behavior)
    default_model: claude-sonnet-4-20250514

  ollama:
    base_url: http://localhost:11434/v1
    default_model: qwen2.5-coder:7b
    models:
      - qwen2.5-coder:7b
      - qwen3:14b
      - deepseek-coder-v2:16b

  openai:
    base_url: https://api.openai.com/v1
    api_key_env: OPENAI_API_KEY
    default_model: gpt-4o

  vllm:
    base_url: http://localhost:8000/v1
    default_model: Qwen/Qwen2.5-Coder-7B-Instruct
```

**CLI commands:**
```bash
cleave config set backend ollama
cleave config set backends.ollama.base_url http://localhost:11434/v1
cleave config show
```

### Phase 4: Skill Invocation Refactoring

Current `/cleave` skill assumes Claude Code. Options:

**Option A: Direct Library Calls (Recommended)**

Instead of invoking `/cleave` through SDK, call cleave library functions directly:

```python
# Instead of:
prompt = f"/cleave parallel=true\n{directive}"
async for msg in sdk.query(prompt):
    yield msg

# Do:
from cleave import assess_directive, init_workspace, reunify_workspace

assessment = assess_directive(directive, threshold)
if assessment.decision == "cleave":
    workspace = init_workspace(directive=directive, children=children, ...)
    # Execute children with backend
    for child in children:
        await execute_child(backend, child)
    result = reunify_workspace(workspace)
```

**Option B: Tool-Based Decomposition**

Define cleave operations as tools the backend can call:

```python
tools = [
    {"name": "assess_complexity", "fn": assess_directive},
    {"name": "create_workspace", "fn": init_workspace},
    {"name": "execute_subtask", "fn": ...},
    {"name": "reunify_results", "fn": reunify_workspace},
]
```

### Phase 5: UI Updates

**Control Panel Changes:**

```python
# Current
model: Literal["sonnet", "opus", "haiku"] = "opus"

# New
class ModelSelection:
    backend: str  # "claude", "ollama", "openai", etc.
    model: str    # Model name (dynamic based on backend)
```

**New UI elements:**
- Backend selector dropdown
- Model selector (populated from backend.available_models())
- Connection status indicator
- Local inference progress/stats

---

## Implementation Order

### Milestone 1: Backend Abstraction (Week 1)

1. Create `backends/base.py` with abstract `Backend` class
2. Create `backends/claude.py` wrapping existing SDK code
3. Refactor `SDKSession` to use backend interface
4. Tests pass with Claude backend (no regression)

### Milestone 2: OpenAI-Compatible Backend (Week 2)

1. Create `backends/openai_compat.py`
2. Implement connection to Ollama (`http://localhost:11434/v1`)
3. Implement streaming response handling
4. Implement basic tool calling
5. Test with Qwen2.5-Coder via Ollama

### Milestone 3: Configuration (Week 2-3)

1. Extend settings system for backend configuration
2. Add CLI commands: `cleave config set backend ...`
3. Add backend auto-detection (is Ollama running? is Claude auth present?)
4. Model discovery from backends

### Milestone 4: Direct Execution (Week 3)

1. Replace `/cleave` skill invocation with direct library calls
2. Implement child task execution loop
3. Test full decomposition workflow with local models
4. Handle tool calling for file operations

### Milestone 5: UI Polish (Week 4)

1. Update ControlPanel for backend/model selection
2. Add connection status indicator
3. Add local inference metrics (tokens/sec, VRAM usage)
4. Documentation and examples

---

## Technical Decisions

### Tool Calling Strategy

**Recommendation: Use Instructor + Structured Outputs**

Local models are unreliable at tool calling. Mitigate with:

1. **Instructor library** - Validates tool calls against Pydantic schemas
2. **Constrained decoding** - Guidance library forces valid JSON
3. **Retry logic** - Re-prompt on validation failure

```python
import instructor
from pydantic import BaseModel

class ToolCall(BaseModel):
    name: str
    arguments: dict

client = instructor.patch(openai_client)
response = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    response_model=ToolCall,
    messages=[...]
)
```

### Handling Cleave-Specific Operations

For operations that need file system access (create workspace, read task files, etc.):

**Option 1: Tool-Based** (More agentic, less reliable with local models)
```python
tools = [
    Tool(name="write_file", fn=lambda path, content: Path(path).write_text(content)),
    Tool(name="read_file", fn=lambda path: Path(path).read_text()),
]
```

**Option 2: Direct Control** (Recommended for reliability)
```python
# Don't let the model decide - orchestrate directly
assessment = assess_directive(directive)
if assessment.decision == "cleave":
    children = suggest_decomposition(directive)  # Model generates this
    init_workspace(directive, children)          # We execute this
```

### Context Management

Local models have limited context. Strategy:

1. **Summarize history** - Don't pass full conversation, summarize key points
2. **Lazy loading** - Only load file contents when needed
3. **RAG for codebase** - Use retrieval instead of full context

---

## Dependencies

### New Dependencies

```toml
[project.optional-dependencies]
local = [
    "openai>=1.0.0",      # OpenAI-compatible client
    "instructor>=1.0.0",  # Structured outputs
    "httpx>=0.27.0",      # Async HTTP client
]

tui = [
    "textual>=0.47.0",
    # claude-agent-sdk becomes optional
]

claude = [
    "claude-agent-sdk>=0.1.0",
]
```

### Installation Options

```bash
# Full Claude Code integration (current)
pip install styrene-cleave[tui,claude]

# Local-only with TUI
pip install styrene-cleave[tui,local]

# Headless local
pip install styrene-cleave[local]
```

---

## Risk Mitigation

### Risk: Local Model Quality

**Problem**: Local models may produce worse decompositions than Claude.

**Mitigation**:
- Recommend Qwen3:14B or Qwen2.5-Coder for best results
- Add quality warnings in UI when using smaller models
- Provide model comparison benchmarks in docs

### Risk: Tool Calling Failures

**Problem**: Local models frequently fail at tool calling.

**Mitigation**:
- Use Instructor for schema validation
- Implement retry with exponential backoff
- Fallback to direct orchestration (not agentic)
- Clear error messages when tool calling fails

### Risk: Latency

**Problem**: Local inference is slow, poor UX.

**Mitigation**:
- Show progress indicators
- Recommend GPU for 7B+ models
- Support async/background execution
- Batch operations where possible

---

## Success Criteria

1. **Cleave TUI works with Ollama + Qwen2.5-Coder:7b** - Full workflow
2. **No regression with Claude backend** - Existing users unaffected
3. **Configuration is straightforward** - `cleave config set backend ollama`
4. **Graceful degradation** - Clear errors when backend unavailable
5. **Documentation** - Setup guides for each backend

---

## Future Considerations

### MCP Integration

If Model Context Protocol (MCP) becomes standard:
- Cleave tools could be exposed as MCP tools
- Any MCP-compatible client could use cleave

### Hybrid Execution

For complex tasks:
- Use local model for assessment and decomposition
- Delegate execution to more capable model (or human)
- Reunify locally

### Custom Model Fine-Tuning

Long-term: Fine-tune a model specifically for cleave-style decomposition.
- Training data from accumulated `.cloven.md` reports
- Specialized for task splitting and complexity assessment
