# autocrucible v0.4.0 Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Implement six improvements (P5→P2→P3→P4→P0→P1) to add structured logging, cost tracking, plateau detection, eval quality guard, Docker sandbox, and Ollama agent backend.

**Architecture:** Each task builds on the previous. P5 (JSONL) changes the data layer, P2 adds cost tracking on top, P3/P4 are independent enhancements, P0 adds execution sandboxing, P1 refactors the agent abstraction. All changes maintain backward compatibility via TSV fallback parsing.

**Tech Stack:** Python 3.10+, dataclasses, json/jsonl, Docker CLI, Ollama REST API, pytest, click CLI

---

### Task 1: JSONL Results — Data Layer (`results.py`)

**Files:**
- Modify: `src/crucible/results.py`
- Test: `tests/test_results.py`

**Step 1: Write failing tests for UsageInfo and new ExperimentRecord**

Add to `tests/test_results.py`:

```python
import json
from crucible.results import ResultsLog, ExperimentRecord, UsageInfo, results_filename


def test_usage_info_defaults():
    u = UsageInfo()
    assert u.input_tokens is None
    assert u.output_tokens is None
    assert u.estimated_cost_usd is None


def test_usage_info_with_values():
    u = UsageInfo(input_tokens=100, output_tokens=50, estimated_cost_usd=0.01)
    assert u.input_tokens == 100


def test_new_record_has_all_fields():
    r = ExperimentRecord(
        commit="abc1234", metric_value=1.0, status="keep", description="test",
        iteration=1, timestamp="2026-03-15T00:00:00Z",
        delta=None, delta_percent=None,
        files_changed=["train.py"], diff_stats={"insertions": 5, "deletions": 2},
        duration_seconds=12.3, usage=None,
    )
    assert r.iteration == 1
    assert r.files_changed == ["train.py"]


def test_jsonl_log_and_read_all(tmp_path):
    log = ResultsLog(tmp_path / "results.jsonl")
    log.init()
    log.log(ExperimentRecord(
        commit="abc1234", metric_value=0.95, status="keep", description="first",
        iteration=1, timestamp="2026-03-15T00:00:00Z",
        delta=None, delta_percent=None,
        files_changed=["train.py"], diff_stats={"insertions": 3, "deletions": 1},
        duration_seconds=10.0, usage=None,
    ))
    records = log.read_all()
    assert len(records) == 1
    assert records[0].commit == "abc1234"
    assert records[0].iteration == 1
    assert records[0].files_changed == ["train.py"]


def test_jsonl_forward_compat(tmp_path):
    """Missing fields in JSONL should default to None/empty."""
    path = tmp_path / "results.jsonl"
    # Write a record with only the original 4 fields (simulating old format)
    path.write_text(json.dumps({
        "commit": "abc1234", "metric_value": 0.5,
        "status": "keep", "description": "old record",
    }) + "\n")
    log = ResultsLog(path)
    records = log.read_all()
    assert len(records) == 1
    assert records[0].iteration is None
    assert records[0].files_changed is None
    assert records[0].usage is None


def test_tsv_fallback_read_from_string():
    """read_from_string should still parse TSV content."""
    tsv = "commit\tmetric_value\tstatus\tdescription\nabc1234\t0.5\tkeep\tfirst\n"
    records = ResultsLog.read_from_string(tsv)
    assert len(records) == 1
    assert records[0].commit == "abc1234"
    assert records[0].iteration is None  # TSV records lack new fields


def test_jsonl_best_and_is_improvement(tmp_path):
    log = ResultsLog(tmp_path / "results.jsonl")
    log.init()
    for i, val in enumerate([1.0, 0.9, 1.1]):
        status = "keep" if val <= 1.0 else "discard"
        log.log(ExperimentRecord(
            commit=f"aaa{i:04d}", metric_value=val, status=status,
            description=f"exp {i}", iteration=i+1,
            timestamp="2026-03-15T00:00:00Z",
            delta=None, delta_percent=None,
            files_changed=[], diff_stats={},
            duration_seconds=5.0, usage=None,
        ))
    best = log.best("minimize")
    assert best.metric_value == 0.9
    assert log.is_improvement(0.85, "minimize") is True
    assert log.is_improvement(0.95, "minimize") is False


def test_jsonl_seed_baseline(tmp_path):
    log = ResultsLog(tmp_path / "results.jsonl")
    log.init()
    log.seed_baseline(600.0, "abc1234", "run1")
    records = log.read_all()
    assert len(records) == 1
    assert records[0].status == "baseline"
    assert records[0].metric_value == 600.0


def test_jsonl_summary_excludes_baseline(tmp_path):
    log = ResultsLog(tmp_path / "results.jsonl")
    log.init()
    log.seed_baseline(600.0, "abc1234", "run1")
    log.log(ExperimentRecord(
        commit="def5678", metric_value=700.0, status="keep",
        description="improvement", iteration=1,
        timestamp="2026-03-15T00:00:00Z",
        delta=100.0, delta_percent=16.7,
        files_changed=["train.py"], diff_stats={"insertions": 5, "deletions": 0},
        duration_seconds=8.0, usage=None,
    ))
    s = log.summary()
    assert s["total"] == 1
    assert s["kept"] == 1
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_results.py -v -x`
Expected: FAIL — `UsageInfo` not importable, `ExperimentRecord` missing new fields

**Step 3: Implement new results.py**

Rewrite `src/crucible/results.py`:

```python
"""Results module for JSONL experiment logging (with TSV fallback)."""

from __future__ import annotations

import json
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Optional


@dataclass
class UsageInfo:
    input_tokens: int | None = None
    output_tokens: int | None = None
    estimated_cost_usd: float | None = None


@dataclass
class ExperimentRecord:
    commit: str
    metric_value: float
    status: str
    description: str
    # New fields (all optional for backward compat)
    iteration: int | None = None
    timestamp: str | None = None
    delta: float | None = None
    delta_percent: float | None = None
    files_changed: list[str] | None = None
    diff_stats: dict | None = None
    duration_seconds: float | None = None
    usage: UsageInfo | None = None


def results_filename(tag: str) -> str:
    """Return the results JSONL filename for a given experiment tag."""
    return f"results-{tag}.jsonl"


def _parse_jsonl(content: str) -> list[ExperimentRecord]:
    """Parse JSONL content into experiment records."""
    records: list[ExperimentRecord] = []
    for line in content.splitlines():
        line = line.strip()
        if not line:
            continue
        try:
            data = json.loads(line)
        except json.JSONDecodeError:
            continue
        usage_data = data.get("usage")
        usage = UsageInfo(**usage_data) if isinstance(usage_data, dict) else None
        records.append(ExperimentRecord(
            commit=data["commit"],
            metric_value=float(data["metric_value"]),
            status=data["status"],
            description=data.get("description", ""),
            iteration=data.get("iteration"),
            timestamp=data.get("timestamp"),
            delta=data.get("delta"),
            delta_percent=data.get("delta_percent"),
            files_changed=data.get("files_changed"),
            diff_stats=data.get("diff_stats"),
            duration_seconds=data.get("duration_seconds"),
            usage=usage,
        ))
    return records


def _parse_tsv(content: str) -> list[ExperimentRecord]:
    """Parse legacy TSV content into experiment records (skips header)."""
    records: list[ExperimentRecord] = []
    for line in content.splitlines()[1:]:
        if not line.strip():
            continue
        parts = line.split("\t", maxsplit=3)
        if len(parts) < 4:
            continue
        try:
            metric = float(parts[1])
        except ValueError:
            continue
        records.append(ExperimentRecord(
            commit=parts[0],
            metric_value=metric,
            status=parts[2],
            description=parts[3],
        ))
    return records


def _detect_format(content: str) -> str:
    """Detect whether content is JSONL or TSV."""
    first_line = content.strip().split("\n", 1)[0].strip()
    if first_line.startswith("{"):
        return "jsonl"
    return "tsv"


class ResultsLog:
    """Append-only JSONL log of experiment results."""

    def __init__(self, path: Path | str) -> None:
        self.path = Path(path)

    def init(self) -> None:
        """Create an empty JSONL file."""
        self.path.write_text("")

    def seed_baseline(self, value: float, commit: str, source_tag: str) -> None:
        """Write a baseline record from a previous run's best result."""
        self.log(ExperimentRecord(
            commit=commit,
            metric_value=value,
            status="baseline",
            description=f"Forked from {source_tag} best",
        ))

    def log(self, record: ExperimentRecord) -> None:
        """Append one experiment record to the log."""
        data = {}
        for k, v in asdict(record).items():
            if v is not None:
                data[k] = v
        with self.path.open("a") as f:
            f.write(json.dumps(data) + "\n")

    def read_all(self) -> list[ExperimentRecord]:
        """Read every record from the log."""
        if not self.path.exists():
            return []
        content = self.path.read_text()
        if not content.strip():
            return []
        fmt = _detect_format(content)
        if fmt == "jsonl":
            return _parse_jsonl(content)
        return _parse_tsv(content)

    def read_last(self, n: int) -> list[ExperimentRecord]:
        """Return the last *n* records."""
        return self.read_all()[-n:]

    def best(self, direction: str) -> Optional[ExperimentRecord]:
        """Return the best record among those with status 'keep' or 'baseline'."""
        candidates = [r for r in self.read_all() if r.status in ("keep", "baseline")]
        if not candidates:
            return None
        if direction == "minimize":
            return min(candidates, key=lambda r: r.metric_value)
        return max(candidates, key=lambda r: r.metric_value)

    def is_improvement(self, value: float, direction: str) -> bool:
        """Check whether *value* improves on the current best."""
        current_best = self.best(direction)
        if current_best is None:
            return True
        if direction == "minimize":
            return value < current_best.metric_value
        return value > current_best.metric_value

    @staticmethod
    def read_from_string(content: str) -> list[ExperimentRecord]:
        """Parse records from string content (auto-detects JSONL or TSV)."""
        if not content.strip():
            return []
        fmt = _detect_format(content)
        if fmt == "jsonl":
            return _parse_jsonl(content)
        return _parse_tsv(content)

    def summary(self) -> dict[str, int]:
        """Return counts by status category (excludes baseline)."""
        records = [r for r in self.read_all() if r.status != "baseline"]
        return {
            "total": len(records),
            "kept": sum(1 for r in records if r.status == "keep"),
            "discarded": sum(1 for r in records if r.status == "discard"),
            "crashed": sum(1 for r in records if r.status == "crash"),
        }
```

**Step 4: Run tests to verify they pass**

Run: `uv run pytest tests/test_results.py -v`
Expected: ALL PASS

**Step 5: Update old tests for new API**

The old tests call `log.log(commit=..., metric_value=..., status=..., description=...)` with keyword args — this must be updated to pass an `ExperimentRecord`. Update all old tests in `test_results.py` to use the new `log.log(ExperimentRecord(...))` API. Remove the old TSV-specific tests (like `test_init_creates_file_with_header` which checks for TSV header).

Run: `uv run pytest tests/test_results.py -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/crucible/results.py tests/test_results.py
git commit -m "feat: migrate results from TSV to JSONL with backward compat"
```

---

### Task 2: JSONL Results — Orchestrator Integration

**Files:**
- Modify: `src/crucible/orchestrator.py`
- Modify: `src/crucible/agents/base.py` (add `usage` and `duration_seconds` to AgentResult)
- Test: `tests/test_orchestrator.py`

**Step 1: Write failing test for enriched logging**

Add to `tests/test_orchestrator.py`:

```python
def test_iteration_logs_enriched_record(tmp_path):
    """run_one_iteration should log ExperimentRecord with iteration, timestamp, etc."""
    setup_repo(tmp_path)
    cfg = make_config()
    mock_agent = MagicMock()
    orch = Orchestrator(cfg, tmp_path, tag="test", agent=mock_agent)
    orch.init()

    with patch.object(orch.runner, "execute") as mock_exec, \
         patch.object(orch.runner, "parse_metric") as mock_parse:
        mock_exec.return_value = MagicMock(exit_code=0, timed_out=False, stderr_tail="")
        mock_parse.return_value = 0.95

        def modify_file(*args, **kwargs):
            (tmp_path / "train.py").write_text("x = 2")
            return AgentResult(modified_files=[Path("train.py")], description="optimize x")
        mock_agent.generate_edit.side_effect = modify_file

        result = orch.run_one_iteration()

    assert result == "keep"
    records = orch.results.read_all()
    assert len(records) == 1
    r = records[0]
    assert r.iteration == 1
    assert r.timestamp is not None
    assert r.files_changed == ["train.py"]
    assert r.diff_stats is not None
    assert r.duration_seconds is not None
```

**Step 2: Run test to verify it fails**

Run: `uv run pytest tests/test_orchestrator.py::test_iteration_logs_enriched_record -v`
Expected: FAIL — `AgentResult` lacks `usage`/`duration_seconds`, orchestrator doesn't pass enriched record

**Step 3: Add usage/duration_seconds to AgentResult**

Modify `src/crucible/agents/base.py`:

```python
from __future__ import annotations
from abc import ABC, abstractmethod
from dataclasses import dataclass
from pathlib import Path
from crucible.results import UsageInfo


@dataclass
class AgentResult:
    modified_files: list[Path]
    description: str
    usage: UsageInfo | None = None
    duration_seconds: float | None = None


class AgentInterface(ABC):
    @abstractmethod
    def generate_edit(self, prompt: str, workspace: Path) -> AgentResult:
        ...
```

**Step 4: Update orchestrator to log ExperimentRecord with enriched fields**

Modify `src/crucible/orchestrator.py`:
- Add `import time` and `from datetime import datetime, timezone`
- Add `self._iteration = 0` in `__init__`
- In `resume()`, set `self._iteration = len(existing)`
- In `run_one_iteration()`:
  - Increment `self._iteration` at start
  - Wrap `agent.generate_edit()` with `time.monotonic()` timing
  - After git commit, call `_get_diff_stats()` to get insertions/deletions
  - Compute `delta` and `delta_percent` from best record
  - Build `ExperimentRecord` with all fields for every `log()` call (keep/discard/crash)
- Add `_get_diff_stats(self, commit: str) -> dict` helper using `git diff --numstat HEAD~1 HEAD`
- Update `results_filename` import and gitignore pattern from `results-*.tsv` to `results-*.jsonl`

**Step 5: Update all orchestrator log() calls**

Every `self.results.log(commit=..., ...)` must become `self.results.log(ExperimentRecord(...))`. There are 3 call sites: crash (line 142), keep (line 161), discard (line 173).

**Step 6: Run all orchestrator tests**

Run: `uv run pytest tests/test_orchestrator.py -v`
Expected: ALL PASS (may need to fix old tests that check record fields)

**Step 7: Fix downstream tests**

Run: `uv run pytest -v`
Expected: Some tests in `test_context.py`, `test_cli.py`, `test_integration.py` may fail because they create `ResultsLog` with `.tsv` extension or use old `log()` API. Fix all callers:
- `test_context.py`: uses `log.log("aaa0001", 1.0, "keep", ...)` — update to `log.log(ExperimentRecord(...))`
- `test_cli.py`: uses `results_filename(tag)` which now returns `.jsonl` — tests should adapt
- `test_integration.py`: uses `FakeAgent` with old API — update
- `test_postmortem.py`: may reference TSV format — update
- `cli.py`: update `_scan_previous_runs` to scan `results-*.jsonl` (with `.tsv` fallback), update `.gitignore` patterns

**Step 8: Run full test suite**

Run: `uv run pytest -v`
Expected: ALL PASS

**Step 9: Commit**

```bash
git add -A
git commit -m "feat: enrich orchestrator logging with iteration, timing, diff stats"
```

---

### Task 3: JSONL Results — CLI Updates

**Files:**
- Modify: `src/crucible/cli.py`
- Test: `tests/test_cli.py`

**Step 1: Update `history` command for JSONL**

Modify `src/crucible/cli.py`:
- `history` command: add `--format` option (`table` default, `jsonl` for raw output)
- When `--format jsonl`: read JSONL file and output raw lines
- When `--format table` (default): render human-readable table as before
- Update `--json` flag to use the enriched record fields (iteration, timestamp, delta, etc.)
- Update `_scan_previous_runs` to glob both `results-*.jsonl` and `results-*.tsv`
- Update `.gitignore` generation in `new` command scaffold to include `results-*.jsonl`
- Update `status` command: no changes needed (already uses `ResultsLog` API)

**Step 2: Run CLI tests**

Run: `uv run pytest tests/test_cli.py -v`
Expected: ALL PASS

**Step 3: Commit**

```bash
git add src/crucible/cli.py tests/test_cli.py
git commit -m "feat: update CLI for JSONL results format"
```

---

### Task 4: Cost Tracking — BudgetGuard

**Files:**
- Create: `src/crucible/budget.py`
- Modify: `src/crucible/config.py`
- Test: `tests/test_budget.py` (new)

**Step 1: Write failing tests for BudgetGuard**

Create `tests/test_budget.py`:

```python
import pytest
from crucible.budget import BudgetGuard, BudgetConfig
from crucible.results import UsageInfo


def test_budget_guard_no_config():
    guard = BudgetGuard(None)
    assert guard.check(None) == "ok"


def test_budget_guard_accumulate():
    cfg = BudgetConfig(max_cost_usd=10.0)
    guard = BudgetGuard(cfg)
    guard.accumulate(UsageInfo(estimated_cost_usd=3.0))
    assert guard.total_cost == 3.0
    guard.accumulate(UsageInfo(estimated_cost_usd=2.0))
    assert guard.total_cost == 5.0


def test_budget_guard_accumulate_none():
    cfg = BudgetConfig(max_cost_usd=10.0)
    guard = BudgetGuard(cfg)
    guard.accumulate(None)
    assert guard.total_cost == 0.0
    guard.accumulate(UsageInfo())  # all None fields
    assert guard.total_cost == 0.0


def test_budget_guard_exceeded():
    cfg = BudgetConfig(max_cost_usd=5.0)
    guard = BudgetGuard(cfg)
    guard.accumulate(UsageInfo(estimated_cost_usd=5.01))
    assert guard.check(None) == "exceeded"


def test_budget_guard_warning():
    cfg = BudgetConfig(max_cost_usd=10.0, warn_at_percent=80)
    guard = BudgetGuard(cfg)
    guard.accumulate(UsageInfo(estimated_cost_usd=8.5))
    assert guard.check(None) == "warning"


def test_budget_guard_per_iter_exceeded():
    cfg = BudgetConfig(max_cost_per_iter_usd=0.50)
    guard = BudgetGuard(cfg)
    usage = UsageInfo(estimated_cost_usd=0.60)
    assert guard.check(usage) == "exceeded"


def test_budget_guard_ok():
    cfg = BudgetConfig(max_cost_usd=10.0, max_cost_per_iter_usd=1.0)
    guard = BudgetGuard(cfg)
    guard.accumulate(UsageInfo(estimated_cost_usd=0.50))
    assert guard.check(UsageInfo(estimated_cost_usd=0.50)) == "ok"


def test_budget_guard_percent_used():
    cfg = BudgetConfig(max_cost_usd=10.0)
    guard = BudgetGuard(cfg)
    guard.accumulate(UsageInfo(estimated_cost_usd=3.0))
    assert guard.percent_used == 30.0
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_budget.py -v`
Expected: FAIL — `budget` module doesn't exist

**Step 3: Implement budget.py**

Create `src/crucible/budget.py`:

```python
"""Budget tracking and control for experiment runs."""

from __future__ import annotations

from dataclasses import dataclass
from crucible.results import UsageInfo


@dataclass
class BudgetConfig:
    max_cost_usd: float | None = None
    max_cost_per_iter_usd: float | None = None
    warn_at_percent: int = 80


class BudgetGuard:
    """Tracks cumulative cost and enforces budget limits."""

    def __init__(self, config: BudgetConfig | None) -> None:
        self.config = config
        self.total_cost: float = 0.0
        self.iteration_count: int = 0

    @property
    def percent_used(self) -> float:
        if not self.config or not self.config.max_cost_usd:
            return 0.0
        return (self.total_cost / self.config.max_cost_usd) * 100

    def accumulate(self, usage: UsageInfo | None) -> None:
        """Add cost from one iteration."""
        if usage and usage.estimated_cost_usd:
            self.total_cost += usage.estimated_cost_usd
        self.iteration_count += 1

    def check(self, usage: UsageInfo | None) -> str:
        """Check budget limits. Returns 'ok', 'warning', or 'exceeded'."""
        if not self.config:
            return "ok"

        # Per-iteration check
        if (self.config.max_cost_per_iter_usd
                and usage and usage.estimated_cost_usd
                and usage.estimated_cost_usd > self.config.max_cost_per_iter_usd):
            return "exceeded"

        # Total budget check
        if self.config.max_cost_usd and self.total_cost > self.config.max_cost_usd:
            return "exceeded"

        # Warning threshold
        if (self.config.max_cost_usd and self.config.warn_at_percent
                and self.percent_used >= self.config.warn_at_percent):
            return "warning"

        return "ok"
```

**Step 4: Run tests**

Run: `uv run pytest tests/test_budget.py -v`
Expected: ALL PASS

**Step 5: Add BudgetConfig to config.py**

Modify `src/crucible/config.py`:
- Add `from crucible.budget import BudgetConfig` (or inline the dataclass)
- Add `budget: BudgetConfig | None = None` to `ConstraintsConfig`
- In `load_config()`, parse `constraints.budget` section

**Step 6: Run config tests**

Run: `uv run pytest tests/test_config.py -v`
Expected: ALL PASS (budget is optional, existing configs unaffected)

**Step 7: Commit**

```bash
git add src/crucible/budget.py tests/test_budget.py src/crucible/config.py
git commit -m "feat: add BudgetGuard for cost tracking and budget control"
```

---

### Task 5: Cost Tracking — Orchestrator Integration

**Files:**
- Modify: `src/crucible/orchestrator.py`
- Modify: `src/crucible/cli.py` (status command)
- Test: `tests/test_orchestrator.py`

**Step 1: Write failing test**

Add to `tests/test_orchestrator.py`:

```python
from crucible.budget import BudgetConfig


def test_budget_exceeded_stops_loop(tmp_path):
    setup_repo(tmp_path)
    cfg = make_config()
    cfg.constraints.budget = BudgetConfig(max_cost_usd=0.01)
    mock_agent = MagicMock()
    orch = Orchestrator(cfg, tmp_path, tag="test", agent=mock_agent)
    orch.init()

    def modify_file(*args, **kwargs):
        (tmp_path / "train.py").write_text("x = 2")
        return AgentResult(
            modified_files=[Path("train.py")], description="optimize",
            usage=UsageInfo(estimated_cost_usd=0.02),
        )
    mock_agent.generate_edit.side_effect = modify_file

    with patch.object(orch.runner, "execute") as mock_exec, \
         patch.object(orch.runner, "parse_metric") as mock_parse:
        mock_exec.return_value = MagicMock(exit_code=0, timed_out=False, stderr_tail="")
        mock_parse.return_value = 0.95
        result = orch.run_one_iteration()

    assert result == "budget_exceeded"
```

**Step 2: Run test to verify it fails**

Run: `uv run pytest tests/test_orchestrator.py::test_budget_exceeded_stops_loop -v`
Expected: FAIL

**Step 3: Integrate BudgetGuard into orchestrator**

Modify `src/crucible/orchestrator.py`:
- Import `BudgetGuard`
- In `__init__`, create `self.budget = BudgetGuard(config.constraints.budget)`
- After `agent.generate_edit()` returns, call `self.budget.accumulate(agent_result.usage)`
- Check `self.budget.check(agent_result.usage)` — if `"exceeded"`, return `"budget_exceeded"`
- In `run_loop()`, add `"budget_exceeded"` to stop conditions

**Step 4: Update `crucible status` in CLI**

Add cost display to `status` command: read all records, sum `usage.estimated_cost_usd` from each, display total. If budget config exists, show as `$X / $Y (Z%)`.

**Step 5: Run all tests**

Run: `uv run pytest -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/crucible/orchestrator.py src/crucible/cli.py tests/test_orchestrator.py
git commit -m "feat: integrate budget guard into orchestrator and status command"
```

---

### Task 6: Plateau Detection

**Files:**
- Modify: `src/crucible/context.py`
- Modify: `src/crucible/config.py`
- Test: `tests/test_context.py`

**Step 1: Write failing tests**

Add to `tests/test_context.py`:

```python
def test_plateau_hint_no_records():
    from crucible.context import _plateau_hint
    assert _plateau_hint([], threshold=8) is None


def test_plateau_hint_below_threshold():
    from crucible.context import _plateau_hint
    records = [_make_record("keep")] + [_make_record("discard")] * 5
    assert _plateau_hint(records, threshold=8) is None


def test_plateau_hint_triggers():
    from crucible.context import _plateau_hint
    records = [_make_record("keep")] + [_make_record("discard")] * 8
    hint = _plateau_hint(records, threshold=8)
    assert hint is not None
    assert "8 consecutive" in hint
    assert "fundamentally different" in hint


def test_plateau_hint_all_failures():
    from crucible.context import _plateau_hint
    records = [_make_record("crash")] * 10
    hint = _plateau_hint(records, threshold=8)
    assert hint is not None
    assert "10 consecutive" in hint


def test_plateau_in_assembled_output(tmp_path):
    cfg = make_config(tmp_path)
    cfg.constraints.plateau_threshold = 3  # low threshold for test
    (tmp_path / "program.md").write_text("Instructions.")
    tsv = tmp_path / "results.jsonl"
    log = ResultsLog(tsv)
    log.init()
    for i in range(5):
        log.log(ExperimentRecord(
            commit=f"aaa{i:04d}", metric_value=1.0, status="discard",
            description=f"failed attempt {i}",
            iteration=i+1, timestamp="2026-03-15T00:00:00Z",
            delta=None, delta_percent=None,
            files_changed=[], diff_stats={}, duration_seconds=5.0, usage=None,
        ))
    ctx = ContextAssembler(cfg, tmp_path, branch_name="crucible/test")
    prompt = ctx.assemble(log)
    assert "5 consecutive" in prompt
    assert "fundamentally different" in prompt
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_context.py -v -k plateau`
Expected: FAIL — `_plateau_hint` doesn't exist, `plateau_threshold` not in config

**Step 3: Add `plateau_threshold` to config**

Modify `src/crucible/config.py`:
- Add `plateau_threshold: int = 8` to `ConstraintsConfig`
- In `load_config()`, parse it from `constraints.plateau_threshold`

**Step 4: Implement `_plateau_hint` in context.py**

Add to `src/crucible/context.py`:

```python
def _plateau_hint(records: list[ExperimentRecord], threshold: int) -> str | None:
    """Generate a strong prompt when metric hasn't improved for N iterations."""
    if not records:
        return None

    streak = 0
    for r in reversed(records):
        if r.status == "keep":
            break
        streak += 1

    if streak < threshold:
        return None

    recent_failures = [r.description for r in records[-streak:] if r.description]

    return (
        f"⚠️ The metric has NOT improved for {streak} consecutive iterations.\n"
        f"Recent attempts: {'; '.join(recent_failures[-5:])}\n"
        "You MUST try a fundamentally different approach:\n"
        "- Change the core algorithm entirely\n"
        "- Restructure the data representation\n"
        "- Challenge an assumption from earlier iterations\n"
        "- Do NOT make small tweaks to previous failed approaches"
    )
```

**Step 5: Integrate into `_section_history` or `assemble`**

In `ContextAssembler.assemble()`, after building the history section, call `_plateau_hint(real_records, self.config.constraints.plateau_threshold)`. If non-None, append to the history section.

**Step 6: Run tests**

Run: `uv run pytest tests/test_context.py -v`
Expected: ALL PASS

**Step 7: Commit**

```bash
git add src/crucible/context.py src/crucible/config.py tests/test_context.py
git commit -m "feat: add plateau detection with configurable threshold"
```

---

### Task 7: Eval Quality — Stability Check

**Files:**
- Modify: `src/crucible/validator.py`
- Modify: `src/crucible/cli.py`
- Test: `tests/test_validator.py`

**Step 1: Write failing tests**

Add to `tests/test_validator.py`:

```python
from unittest.mock import MagicMock, patch
from crucible.validator import check_stability, StabilityResult


def test_check_stability_stable(tmp_path):
    """Identical metrics across runs → stable."""
    config = MagicMock()
    config.commands.run = "echo ok"
    config.commands.eval = "echo 'metric: 1.0'"
    config.metric.name = "metric"
    config.constraints.timeout_seconds = 30

    with patch("crucible.validator.ExperimentRunner") as MockRunner:
        runner = MockRunner.return_value
        runner.execute.return_value = MagicMock(exit_code=0, timed_out=False)
        runner.parse_metric.return_value = 1.0
        result = check_stability(tmp_path, config, runs=5)

    assert result.stable is True
    assert result.cv == 0.0


def test_check_stability_unstable(tmp_path):
    """Widely varying metrics → unstable."""
    config = MagicMock()
    config.commands.run = "echo ok"
    config.commands.eval = "echo 'metric: 1.0'"
    config.metric.name = "metric"
    config.constraints.timeout_seconds = 30

    values = iter([100.0, 200.0, 150.0, 300.0, 50.0])
    with patch("crucible.validator.ExperimentRunner") as MockRunner:
        runner = MockRunner.return_value
        runner.execute.return_value = MagicMock(exit_code=0, timed_out=False)
        runner.parse_metric.side_effect = lambda *a: next(values)
        result = check_stability(tmp_path, config, runs=5)

    assert result.stable is False
    assert result.cv > 5.0
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_validator.py -v -k stability`
Expected: FAIL — `check_stability`, `StabilityResult` don't exist

**Step 3: Implement in validator.py**

Add to `src/crucible/validator.py`:

```python
import statistics

@dataclass
class StabilityResult:
    stable: bool
    cv: float = 0.0
    values: list[float] | None = None
    mean: float = 0.0
    stdev: float = 0.0
    reason: str = ""


def check_stability(workspace: Path, config, runs: int = 5) -> StabilityResult:
    """Run experiment N times and compute coefficient of variation."""
    runner = ExperimentRunner(workspace=workspace)
    timeout = min(config.constraints.timeout_seconds, 120)
    values: list[float] = []
    for _ in range(runs):
        run_result = runner.execute(config.commands.run, timeout)
        if run_result.exit_code != 0 or run_result.timed_out:
            continue
        metric = runner.parse_metric(config.commands.eval, config.metric.name)
        if metric is not None:
            values.append(metric)

    if len(values) < 2:
        return StabilityResult(stable=False, reason="too few successful runs")

    mean = statistics.mean(values)
    stdev = statistics.stdev(values)
    cv = (stdev / abs(mean) * 100) if mean != 0 else float("inf")

    return StabilityResult(
        stable=cv < 5.0,
        cv=cv, values=values, mean=mean, stdev=stdev,
    )
```

**Step 4: Add CLI flag**

Modify `src/crucible/cli.py` `validate` command:
- Add `--stability` flag and `--runs` option (default 5)
- When `--stability`, call `check_stability()` and display result

**Step 5: Run tests**

Run: `uv run pytest tests/test_validator.py -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/crucible/validator.py src/crucible/cli.py tests/test_validator.py
git commit -m "feat: add metric stability check to crucible validate"
```

---

### Task 8: Eval Quality — Multi-Run Median

**Files:**
- Modify: `src/crucible/config.py`
- Modify: `src/crucible/runner.py`
- Modify: `src/crucible/orchestrator.py`
- Test: `tests/test_runner.py`

**Step 1: Write failing test**

Add to `tests/test_runner.py`:

```python
from unittest.mock import MagicMock, patch


def test_execute_with_repeat_median(tmp_path):
    from crucible.runner import ExperimentRunner
    runner = ExperimentRunner(tmp_path)

    with patch.object(runner, "execute") as mock_exec, \
         patch.object(runner, "parse_metric") as mock_parse:
        mock_exec.return_value = MagicMock(exit_code=0, timed_out=False, stderr_tail="")
        mock_parse.side_effect = [1.0, 3.0, 2.0]
        result, metric = runner.execute_with_repeat(
            run_cmd="echo ok", eval_cmd="echo 'metric: 1'",
            metric_name="metric", repeat=3,
            aggregation="median", timeout=30,
        )
    assert metric == 2.0  # median of [1.0, 3.0, 2.0]


def test_execute_with_repeat_single_failure(tmp_path):
    from crucible.runner import ExperimentRunner
    runner = ExperimentRunner(tmp_path)

    with patch.object(runner, "execute") as mock_exec:
        mock_exec.return_value = MagicMock(exit_code=1, timed_out=False, stderr_tail="error")
        result, metric = runner.execute_with_repeat(
            run_cmd="echo ok", eval_cmd="echo 'metric: 1'",
            metric_name="metric", repeat=3,
            aggregation="median", timeout=30,
        )
    assert metric is None


def test_execute_with_repeat_one(tmp_path):
    """repeat=1 should behave like normal execute."""
    from crucible.runner import ExperimentRunner
    runner = ExperimentRunner(tmp_path)

    with patch.object(runner, "execute") as mock_exec, \
         patch.object(runner, "parse_metric") as mock_parse:
        mock_exec.return_value = MagicMock(exit_code=0, timed_out=False, stderr_tail="")
        mock_parse.return_value = 42.0
        result, metric = runner.execute_with_repeat(
            run_cmd="echo ok", eval_cmd="echo 'metric: 42'",
            metric_name="metric", repeat=1,
            aggregation="median", timeout=30,
        )
    assert metric == 42.0
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_runner.py -v -k repeat`
Expected: FAIL

**Step 3: Add EvaluationConfig to config.py**

Modify `src/crucible/config.py`:

```python
@dataclass
class EvaluationConfig:
    repeat: int = 1
    aggregation: str = "median"  # median | mean
```

Add `evaluation: EvaluationConfig = field(default_factory=EvaluationConfig)` to `Config`.
Parse from `raw.get("evaluation", {})` in `load_config()`.

**Step 4: Implement execute_with_repeat in runner.py**

Add to `src/crucible/runner.py`:

```python
import statistics

def execute_with_repeat(
    self, run_cmd: str, eval_cmd: str, metric_name: str,
    repeat: int, aggregation: str, timeout: int,
) -> tuple[RunResult, float | None]:
    """Execute experiment repeat times, return aggregated metric."""
    values: list[float] = []
    last_result = None
    for _ in range(repeat):
        result = self.execute(run_cmd, timeout)
        last_result = result
        if result.exit_code != 0 or result.timed_out:
            return result, None
        metric = self.parse_metric(eval_cmd, metric_name)
        if metric is None:
            return result, None
        values.append(metric)

    if not values:
        return last_result, None

    if aggregation == "mean":
        return last_result, statistics.mean(values)
    return last_result, statistics.median(values)
```

**Step 5: Update orchestrator to use execute_with_repeat**

In `orchestrator.py` `run_one_iteration()`, replace the `execute()` + `parse_metric()` block with `execute_with_repeat()` when `config.evaluation.repeat > 1`. Otherwise use existing single-run path for backward compat.

**Step 6: Run tests**

Run: `uv run pytest tests/test_runner.py tests/test_orchestrator.py -v`
Expected: ALL PASS

**Step 7: Commit**

```bash
git add src/crucible/runner.py src/crucible/config.py src/crucible/orchestrator.py tests/test_runner.py
git commit -m "feat: add multi-run median evaluation support"
```

---

### Task 9: Eval Quality — Correctness Gate Template

**Files:**
- Modify: `src/crucible/cli.py` (the `new` command scaffold)
- No tests needed (template change only)

**Step 1: Update evaluate.py template in `new` command**

In `src/crucible/cli.py`, find the scaffold `evaluate.py` content (around line 237) and replace with:

```python
(dest_path / "evaluate.py").write_text('''\
"""Evaluation harness (readonly). Measures the metric for the current solution."""

from solution import solve


def evaluate():
    result = solve()

    # === Correctness Gate ===
    # TODO: Add correctness checks here. If correctness fails,
    # print a worst-case metric value and exit early.
    # Example:
    #   if not verify_correctness(result):
    #       print("metric: 999999")  # worst value for minimize
    #       return

    # === Method Verification ===
    # TODO: Verify the solution uses the required approach.
    # Example:
    #   if not uses_required_algorithm(result):
    #       print("metric: 999999")
    #       return

    # === Performance Measurement ===
    metric = abs(result - 0)
    print(f"metric: {metric}")


if __name__ == "__main__":
    evaluate()
''')
```

**Step 2: Verify existing tests still pass**

Run: `uv run pytest tests/test_cli.py -v`
Expected: ALL PASS

**Step 3: Commit**

```bash
git add src/crucible/cli.py
git commit -m "feat: add correctness gate template to evaluate.py scaffold"
```

---

### Task 10: Docker Sandbox — SandboxRunner

**Files:**
- Create: `src/crucible/sandbox.py`
- Modify: `src/crucible/config.py`
- Test: `tests/test_sandbox.py` (new)

**Step 1: Write failing tests**

Create `tests/test_sandbox.py`:

```python
import pytest
from pathlib import Path
from unittest.mock import patch, MagicMock
from crucible.sandbox import SandboxRunner, SandboxConfig
from crucible.runner import RunResult


def test_sandbox_none_delegates_to_native(tmp_path):
    """backend=none should delegate to ExperimentRunner."""
    runner = SandboxRunner(config=None, workspace=tmp_path)
    with patch("crucible.sandbox.ExperimentRunner") as MockRunner:
        mock = MockRunner.return_value
        mock.execute.return_value = RunResult(exit_code=0, timed_out=False)
        result = runner.execute("echo ok", timeout=30)
    assert result.exit_code == 0


def test_sandbox_docker_builds_correct_command(tmp_path):
    """Docker backend should build correct docker run command."""
    cfg = SandboxConfig(backend="docker", base_image="python:3.12-slim", network=False)
    runner = SandboxRunner(config=cfg, workspace=tmp_path, readonly_files=["eval.py"])

    with patch("subprocess.Popen") as mock_popen:
        mock_proc = MagicMock()
        mock_proc.communicate.return_value = ("", "")
        mock_proc.returncode = 0
        mock_popen.return_value = mock_proc
        runner.execute("python eval.py", timeout=30)

    call_args = mock_popen.call_args
    cmd = call_args[0][0] if call_args[0] else call_args[1].get("args", "")
    # Verify key docker flags
    assert "docker" in str(cmd)
    assert "--network=none" in str(cmd) or "--network" in str(cmd)


def test_sandbox_docker_check_available():
    from crucible.sandbox import check_docker_available
    # Just verify it returns a bool without crashing
    result = check_docker_available()
    assert isinstance(result, bool)


def test_sandbox_ensure_image_with_requirements(tmp_path):
    """Should generate a Dockerfile when requirements.txt exists."""
    (tmp_path / "requirements.txt").write_text("numpy\nscipy\n")
    cfg = SandboxConfig(backend="docker", base_image="python:3.12-slim")
    runner = SandboxRunner(config=cfg, workspace=tmp_path)

    with patch("subprocess.run") as mock_run:
        mock_run.return_value = MagicMock(returncode=0)
        tag = runner._ensure_image()

    assert "crucible-" in tag
    # Verify docker build was called
    assert mock_run.called
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_sandbox.py -v`
Expected: FAIL — module doesn't exist

**Step 3: Implement sandbox.py**

Create `src/crucible/sandbox.py` with `SandboxConfig` dataclass, `SandboxRunner` class, `check_docker_available()`, and `_ensure_image()`. The Docker backend wraps commands with `docker run --rm` using `subprocess.Popen`, applying `--network=none`, `--memory`, `--cpus` limits, `-v workspace:/workspace` mount, and `:ro` overrides for readonly files.

**Step 4: Add SandboxConfig to config.py**

Add `SandboxConfig` dataclass, add `sandbox: SandboxConfig | None = None` to `Config`, parse from YAML.

**Step 5: Run tests**

Run: `uv run pytest tests/test_sandbox.py -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/crucible/sandbox.py src/crucible/config.py tests/test_sandbox.py
git commit -m "feat: add Docker sandbox runner with auto image build"
```

---

### Task 11: Docker Sandbox — Orchestrator Integration

**Files:**
- Modify: `src/crucible/orchestrator.py`
- Modify: `src/crucible/validator.py`
- Test: `tests/test_orchestrator.py`

**Step 1: Write failing test**

Add to `tests/test_orchestrator.py`:

```python
def test_sandbox_runner_used_when_configured(tmp_path):
    """When sandbox config exists, orchestrator should use SandboxRunner."""
    from crucible.sandbox import SandboxConfig
    setup_repo(tmp_path)
    cfg = make_config()
    cfg.sandbox = SandboxConfig(backend="docker", base_image="python:3.12-slim")
    mock_agent = MagicMock()
    orch = Orchestrator(cfg, tmp_path, tag="test", agent=mock_agent)
    orch.init()
    # Verify the runner is a SandboxRunner
    from crucible.sandbox import SandboxRunner
    assert isinstance(orch.runner, SandboxRunner)
```

**Step 2: Implement**

In `Orchestrator.__init__()`, check `config.sandbox`. If sandbox config exists and backend != "none", create `SandboxRunner` instead of `ExperimentRunner`. Both share the same `execute()` interface (duck typing or shared base class).

In `validator.py` `validate_project()`, add a Docker availability check when sandbox is configured.

**Step 3: Run tests**

Run: `uv run pytest tests/test_orchestrator.py tests/test_validator.py -v`
Expected: ALL PASS

**Step 4: Commit**

```bash
git add src/crucible/orchestrator.py src/crucible/validator.py tests/test_orchestrator.py
git commit -m "feat: integrate sandbox runner into orchestrator"
```

---

### Task 12: Agent Abstraction — Refactor Base + ClaudeCode

**Files:**
- Modify: `src/crucible/agents/base.py`
- Modify: `src/crucible/agents/claude_code.py`
- Create: `src/crucible/agents/__init__.py`
- Test: `tests/test_agents.py`

**Step 1: Write failing tests**

Add to `tests/test_agents.py`:

```python
def test_agent_result_has_usage_and_duration():
    from crucible.agents.base import AgentResult
    from crucible.results import UsageInfo
    r = AgentResult(
        modified_files=[], description="test",
        usage=UsageInfo(input_tokens=100), duration_seconds=5.0,
    )
    assert r.usage.input_tokens == 100
    assert r.duration_seconds == 5.0


def test_claude_code_agent_reports_duration():
    """ClaudeCodeAgent should report duration_seconds in AgentResult."""
    # This test verifies the timing wrapper exists
    from crucible.agents.claude_code import ClaudeCodeAgent
    agent = ClaudeCodeAgent()
    assert hasattr(agent, 'generate_edit')


def test_capabilities_default():
    from crucible.agents.base import AgentInterface
    # Default capabilities should include all tools
    class DummyAgent(AgentInterface):
        def generate_edit(self, prompt, workspace):
            pass
    agent = DummyAgent()
    assert "read" in agent.capabilities()
    assert "edit" in agent.capabilities()


def test_create_agent_claude_code():
    from crucible.agents import create_agent
    from crucible.config import AgentConfig
    cfg = AgentConfig(type="claude-code")
    agent = create_agent(cfg)
    from crucible.agents.claude_code import ClaudeCodeAgent
    assert isinstance(agent, ClaudeCodeAgent)


def test_create_agent_unknown_raises():
    from crucible.agents import create_agent
    from crucible.config import AgentConfig
    import pytest
    with pytest.raises(ValueError, match="Unknown agent type"):
        create_agent(AgentConfig(type="nonexistent"))
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_agents.py -v -k "capabilities or create_agent"`
Expected: FAIL — `capabilities()` method and `create_agent` don't exist

**Step 3: Add `capabilities()` to AgentInterface**

Already done in Task 2 when we added `usage`/`duration_seconds`. Now add:

```python
class AgentInterface(ABC):
    @abstractmethod
    def generate_edit(self, prompt: str, workspace: Path) -> AgentResult:
        ...

    def capabilities(self) -> set[str]:
        return {"read", "edit", "write", "glob", "grep"}
```

**Step 4: Add timing to ClaudeCodeAgent**

In `claude_code.py` `_run_query()`, wrap the query loop with `time.monotonic()` timing. Set `duration_seconds` on the returned `AgentResult`.

**Step 5: Create agent factory**

Create `src/crucible/agents/__init__.py`:

```python
"""Agent factory for creating agent instances from config."""

from crucible.agents.base import AgentInterface, AgentResult
from crucible.config import AgentConfig


def create_agent(config: AgentConfig, **kwargs) -> AgentInterface:
    """Create an agent instance based on config.type."""
    if config.type == "claude-code":
        from crucible.agents.claude_code import ClaudeCodeAgent
        return ClaudeCodeAgent(**kwargs)
    elif config.type == "ollama":
        from crucible.agents.ollama import OllamaAgent
        return OllamaAgent(
            model=config.model or "qwen2.5-coder:32b",
            base_url=config.base_url or "http://localhost:11434",
            **kwargs,
        )
    else:
        raise ValueError(f"Unknown agent type: {config.type}")
```

**Step 6: Add model/base_url to AgentConfig**

Modify `src/crucible/config.py` — add `model: str | None = None` and `base_url: str | None = None` to `AgentConfig`. Parse from YAML in `_build_agent()`.

**Step 7: Update CLI to use create_agent**

In `src/crucible/cli.py`, replace direct `ClaudeCodeAgent(...)` instantiation in `init` and `run` commands with `create_agent(config.agent, timeout=..., ...)`. Pass extra kwargs based on agent type.

**Step 8: Run tests**

Run: `uv run pytest tests/test_agents.py -v`
Expected: ALL PASS

**Step 9: Commit**

```bash
git add src/crucible/agents/ src/crucible/config.py src/crucible/cli.py tests/test_agents.py
git commit -m "feat: add agent factory and capabilities method"
```

---

### Task 13: Ollama Agent Backend

**Files:**
- Create: `src/crucible/agents/ollama.py`
- Modify: `src/crucible/agents/__init__.py` (already done in Task 12)
- Test: `tests/test_agents.py`

**Step 1: Write failing tests**

Add to `tests/test_agents.py`:

```python
import json
from unittest.mock import patch, MagicMock


def test_ollama_agent_capabilities():
    from crucible.agents.ollama import OllamaAgent
    agent = OllamaAgent(model="qwen2.5-coder:32b")
    caps = agent.capabilities()
    assert caps == {"edit"}
    assert "read" not in caps


def test_ollama_agent_parses_edits(tmp_path):
    """OllamaAgent should parse structured JSON edits from model output."""
    from crucible.agents.ollama import OllamaAgent

    (tmp_path / "train.py").write_text("x = 1\ny = 2\n")

    agent = OllamaAgent(model="test-model")
    response_content = json.dumps({
        "edits": [{"file": "train.py", "search": "x = 1", "replace": "x = 42"}],
        "description": "optimized x value",
    })

    mock_response = MagicMock()
    mock_response.json.return_value = {
        "message": {"content": response_content},
        "prompt_eval_count": 500,
        "eval_count": 200,
        "total_duration": 5_000_000_000,  # 5 seconds in nanoseconds
    }
    mock_response.raise_for_status = MagicMock()

    with patch("requests.post", return_value=mock_response):
        result = agent.generate_edit("optimize x", tmp_path)

    assert len(result.modified_files) == 1
    assert result.modified_files[0] == Path("train.py")
    assert (tmp_path / "train.py").read_text() == "x = 42\ny = 2\n"
    assert result.usage is not None
    assert result.usage.input_tokens == 500
    assert result.usage.output_tokens == 200
    assert result.usage.estimated_cost_usd == 0.0
    assert result.description == "optimized x value"


def test_ollama_agent_handles_api_error(tmp_path):
    """OllamaAgent should return error AgentResult on API failure."""
    from crucible.agents.ollama import OllamaAgent
    import requests

    agent = OllamaAgent(model="test-model")

    with patch("requests.post", side_effect=requests.ConnectionError("refused")):
        result = agent.generate_edit("optimize", tmp_path)

    assert result.modified_files == []
    assert "error" in result.description.lower()


def test_ollama_agent_handles_malformed_json(tmp_path):
    """OllamaAgent should handle non-JSON model output gracefully."""
    from crucible.agents.ollama import OllamaAgent

    agent = OllamaAgent(model="test-model")
    mock_response = MagicMock()
    mock_response.json.return_value = {
        "message": {"content": "I can't do JSON, sorry"},
        "prompt_eval_count": 100,
        "eval_count": 50,
        "total_duration": 1_000_000_000,
    }
    mock_response.raise_for_status = MagicMock()

    with patch("requests.post", return_value=mock_response):
        result = agent.generate_edit("optimize", tmp_path)

    assert result.modified_files == []
```

**Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/test_agents.py -v -k ollama`
Expected: FAIL — `ollama` module doesn't exist

**Step 3: Implement ollama.py**

Create `src/crucible/agents/ollama.py`:

```python
"""Ollama agent — uses Ollama REST API for local LLM inference."""

from __future__ import annotations

import json
import logging
from pathlib import Path

import requests

from crucible.agents.base import AgentInterface, AgentResult
from crucible.results import UsageInfo

logger = logging.getLogger(__name__)

OLLAMA_SYSTEM_PROMPT = (
    "You are a code optimization agent. You receive code files and must "
    "propose edits to improve a metric.\n\n"
    "RESPOND ONLY WITH VALID JSON in this exact format:\n"
    '{"edits": [{"file": "filename.py", "search": "exact text to find", '
    '"replace": "replacement text"}], "description": "one line summary"}\n\n'
    "Rules:\n"
    "- search must be an EXACT substring of the current file content\n"
    "- Make ONE focused change per edit\n"
    "- description must be under 120 characters\n"
)


class OllamaAgent(AgentInterface):
    def __init__(
        self,
        model: str,
        base_url: str = "http://localhost:11434",
        timeout: int = 600,
    ):
        self.model = model
        self.base_url = base_url.rstrip("/")
        self.timeout = timeout

    def capabilities(self) -> set[str]:
        return {"edit"}

    def generate_edit(self, prompt: str, workspace: Path) -> AgentResult:
        try:
            return self._call_ollama(prompt, workspace)
        except Exception as e:
            return AgentResult(modified_files=[], description=f"ollama error: {e}")

    def _read_workspace_files(self, workspace: Path) -> str:
        """Read all non-hidden files in workspace for context."""
        parts = []
        for f in sorted(workspace.rglob("*.py")):
            rel = f.relative_to(workspace)
            if str(rel).startswith(".") or "__pycache__" in str(rel):
                continue
            try:
                content = f.read_text()
                parts.append(f"--- {rel} ---\n{content}")
            except Exception:
                continue
        return "\n\n".join(parts)

    def _call_ollama(self, prompt: str, workspace: Path) -> AgentResult:
        files_context = self._read_workspace_files(workspace)
        full_prompt = f"{prompt}\n\n--- Current Files ---\n{files_context}"

        response = requests.post(
            f"{self.base_url}/api/chat",
            json={
                "model": self.model,
                "messages": [
                    {"role": "system", "content": OLLAMA_SYSTEM_PROMPT},
                    {"role": "user", "content": full_prompt},
                ],
                "stream": False,
                "options": {"num_predict": 8192},
            },
            timeout=self.timeout,
        )
        response.raise_for_status()
        data = response.json()

        content = data["message"]["content"]
        usage = UsageInfo(
            input_tokens=data.get("prompt_eval_count"),
            output_tokens=data.get("eval_count"),
            estimated_cost_usd=0.0,
        )
        duration = data.get("total_duration", 0) / 1e9

        # Parse structured JSON edits
        try:
            parsed = json.loads(content)
            edits = parsed.get("edits", [])
            description = parsed.get("description", "ollama edit")
        except json.JSONDecodeError:
            logger.warning("Ollama returned non-JSON output, no edits applied")
            return AgentResult(
                modified_files=[], description="ollama: non-JSON response",
                usage=usage, duration_seconds=duration,
            )

        # Apply edits
        modified = self._apply_edits(edits, workspace)

        return AgentResult(
            modified_files=modified,
            description=description[:200],
            usage=usage,
            duration_seconds=duration,
        )

    def _apply_edits(self, edits: list[dict], workspace: Path) -> list[Path]:
        """Apply search/replace edits to workspace files."""
        modified: list[Path] = []
        for edit in edits:
            filepath = workspace / edit.get("file", "")
            search = edit.get("search", "")
            replace = edit.get("replace", "")
            if not filepath.exists() or not search:
                continue
            content = filepath.read_text()
            if search in content:
                new_content = content.replace(search, replace, 1)
                filepath.write_text(new_content)
                modified.append(Path(edit["file"]))
                logger.debug(f"Applied edit to {edit['file']}")
            else:
                logger.warning(f"Search string not found in {edit['file']}")
        return modified
```

**Step 4: Add `requests` as optional dependency**

Check if `requests` is already in `pyproject.toml`. If not, add it. Ollama backend is optional, so consider wrapping the import with a helpful error message.

**Step 5: Run tests**

Run: `uv run pytest tests/test_agents.py -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/crucible/agents/ollama.py tests/test_agents.py
git commit -m "feat: add Ollama agent backend with structured JSON edits"
```

---

### Task 14: Agent Abstraction — Context Assembly for Non-Tool Agents

**Files:**
- Modify: `src/crucible/context.py`
- Modify: `src/crucible/orchestrator.py`
- Test: `tests/test_context.py`

**Step 1: Write failing test**

Add to `tests/test_context.py`:

```python
def test_assemble_with_files_includes_file_content(tmp_path):
    cfg = make_config(tmp_path)
    (tmp_path / "program.md").write_text("Instructions.")
    (tmp_path / "train.py").write_text("x = 1\ny = 2\n")
    log = ResultsLog(tmp_path / "results.jsonl")
    log.init()
    ctx = ContextAssembler(cfg, tmp_path, branch_name="crucible/test")
    prompt = ctx.assemble_with_files(log, tmp_path, ["train.py"])
    assert "x = 1" in prompt
    assert "train.py" in prompt
```

**Step 2: Implement `assemble_with_files`**

Add to `ContextAssembler`:

```python
def assemble_with_files(
    self, log: ResultsLog, workspace: Path, editable_files: list[str],
) -> str:
    """Assemble prompt with file contents inlined (for agents without read tools)."""
    base = self.assemble(log)
    parts = [base, "\n\n---\n\n## Editable File Contents\n"]
    for fname in editable_files:
        fpath = workspace / fname
        if fpath.exists():
            content = fpath.read_text()
            parts.append(f"\n### {fname}\n```\n{content}\n```\n")
    return "".join(parts)
```

**Step 3: Update orchestrator**

In `run_one_iteration()`, check `self.agent.capabilities()`:

```python
if "read" not in self.agent.capabilities():
    prompt = self.context.assemble_with_files(
        self.results, self.workspace, self.config.files.editable,
    )
else:
    prompt = self.context.assemble(self.results)
```

**Step 4: Run tests**

Run: `uv run pytest tests/test_context.py tests/test_orchestrator.py -v`
Expected: ALL PASS

**Step 5: Commit**

```bash
git add src/crucible/context.py src/crucible/orchestrator.py tests/test_context.py
git commit -m "feat: add assemble_with_files for agents without read capability"
```

---

### Task 15: Final Integration Test + Cleanup

**Files:**
- Test: `tests/test_integration.py`
- All source files (review)

**Step 1: Run full test suite**

Run: `uv run pytest -v`
Expected: ALL PASS

**Step 2: Fix any remaining failures**

Common issues:
- Import paths changed (e.g., `from crucible.results import ExperimentRecord, UsageInfo`)
- Old TSV file patterns in tests (`.tsv` → `.jsonl`)
- `results_filename()` returns different extension
- Old `log()` API calls with keyword args
- `make_config()` helpers missing new config fields

**Step 3: Run integration test specifically**

Run: `uv run pytest tests/test_integration.py -v`
Expected: ALL PASS

**Step 4: Final commit**

```bash
git add -A
git commit -m "test: fix all tests for v0.4.0 changes"
```

---

## Summary of Commits

| # | Message | Tasks |
|---|---------|-------|
| 1 | `feat: migrate results from TSV to JSONL with backward compat` | Task 1 |
| 2 | `feat: enrich orchestrator logging with iteration, timing, diff stats` | Task 2 |
| 3 | `feat: update CLI for JSONL results format` | Task 3 |
| 4 | `feat: add BudgetGuard for cost tracking and budget control` | Task 4 |
| 5 | `feat: integrate budget guard into orchestrator and status command` | Task 5 |
| 6 | `feat: add plateau detection with configurable threshold` | Task 6 |
| 7 | `feat: add metric stability check to crucible validate` | Task 7 |
| 8 | `feat: add multi-run median evaluation support` | Task 8 |
| 9 | `feat: add correctness gate template to evaluate.py scaffold` | Task 9 |
| 10 | `feat: add Docker sandbox runner with auto image build` | Task 10 |
| 11 | `feat: integrate sandbox runner into orchestrator` | Task 11 |
| 12 | `feat: add agent factory and capabilities method` | Task 12 |
| 13 | `feat: add Ollama agent backend with structured JSON edits` | Task 13 |
| 14 | `feat: add assemble_with_files for agents without read capability` | Task 14 |
| 15 | `test: fix all tests for v0.4.0 changes` | Task 15 |
