# autocrucible v0.4.0 Improvements Design

Date: 2026-03-15
Status: Approved

## Overview

Six improvements to autocrucible v0.3.0, addressing architecture limitations and design gaps. Implementation order prioritizes value delivery over severity ranking.

**Implementation order: P5 → P2 → P3 → P4 → P0 → P1**

## Decisions Summary

| Item | Decision |
|------|----------|
| P0 Sandbox | Docker + none backends, cross-platform (Win/Mac/Linux), auto image build |
| P1 Agent abstraction | Interface + ClaudeCode + Ollama backends |
| P2 Cost tracking | UsageInfo in AgentResult, budget guard, per-backend fill |
| P3 Exploration | Plateau detection only (prompt injection) |
| P4 Eval quality | Stability check, multi-run median, correctness gate template |
| P5 Structured logs | Migrate to JSONL, TSV fallback parser for compat |

---

## P5 — Structured Experiment Logs (JSONL Migration)

### Changes

**`results.py`** — core rewrite

```python
@dataclass
class ExperimentRecord:
    # Existing fields
    commit: str
    metric_value: float
    status: str               # keep | discard | crash | baseline
    description: str
    # New fields
    iteration: int
    timestamp: str            # ISO 8601
    delta: float | None       # relative to previous keep
    delta_percent: float | None
    files_changed: list[str]
    diff_stats: dict          # {"insertions": N, "deletions": N}
    duration_seconds: float   # experiment execution time
    usage: UsageInfo | None   # token usage (filled by P2)
```

```python
@dataclass
class UsageInfo:
    input_tokens: int | None = None
    output_tokens: int | None = None
    estimated_cost_usd: float | None = None
```

**File format**: `results.jsonl` (one JSON object per line), gitignored.

**Key method changes**:
- `log()` → `json.dumps()` append one line
- `read_all()` → line-by-line `json.loads()`, missing fields default to None (forward compat)
- `read_from_string()` → detect format (is first line JSON?), support TSV fallback
- `best()` / `is_improvement()` — logic unchanged, only data source changes

**`orchestrator.py`** — enrich log calls with new fields:
- `iteration`, `timestamp`, `delta`, `delta_percent` computed at log time
- `files_changed`, `diff_stats` from git after commit
- `usage` from AgentResult (None until P2)

**`_get_diff_stats()`** — new helper using `git diff --stat --numstat`

**CLI changes**:
- `crucible history` — default table format (read JSONL, render human-readable table)
- `crucible history --format jsonl` — raw JSONL output

### Out of scope

- No `category` auto-classification (needs LLM or complex diff analysis)
- No `agent_reasoning` extraction (depends on unstable agent output format)
- No `crucible analyze` command (wait for sufficient JSONL data)

---

## P2 — Cost Tracking & Budget Control

### Data flow

```
Agent backend → UsageInfo → AgentResult → orchestrator → ExperimentRecord (JSONL)
                                              ↓
                                         BudgetGuard (check limits)
```

### AgentResult extension

```python
@dataclass
class AgentResult:
    modified_files: list[Path]
    description: str
    usage: UsageInfo | None = None
    duration_seconds: float | None = None  # agent response time, excludes experiment
```

### BudgetGuard — new module `budget.py`

```python
@dataclass
class BudgetConfig:
    max_cost_usd: float | None = None
    max_cost_per_iter_usd: float | None = None
    warn_at_percent: int = 80

class BudgetGuard:
    def __init__(self, config: BudgetConfig | None):
        self.total_cost = 0.0

    def check(self, usage: UsageInfo | None) -> BudgetVerdict:
        """Returns ok / warning / exceeded"""
        ...

    def accumulate(self, usage: UsageInfo | None):
        """Accumulate cost"""
        ...
```

### Config extension

```yaml
constraints:
  budget:
    max_cost_usd: 10.0
    max_cost_per_iter_usd: 0.50
    warn_at_percent: 80
```

### Orchestrator integration

- After agent returns: `budget.accumulate()` then `budget.check()`
- `exceeded` → stop with new status `"budget_exceeded"`
- `warning` → log warning, continue

### CLI

`crucible status` shows: `Cost: $1.47 / $10.00 (14.7%) — 32 iterations, avg $0.046/iter`
When usage is None: `Cost: unknown (agent backend does not report usage)`

### Out of scope

- No context compression strategy (existing `history_limit` suffices)
- No price table maintenance (each backend computes its own `estimated_cost_usd`)

---

## P3 — Plateau Detection (Prompt Injection)

### Changes: `context.py` only

```python
def _plateau_hint(self, records: list[ExperimentRecord]) -> str | None:
    """When N consecutive non-keep iterations, inject plateau prompt"""
    streak = 0
    for r in reversed(records):
        if r.status == "keep":
            break
        streak += 1

    if streak < self.plateau_threshold:
        return None

    recent_failures = [r.description for r in records[-streak:] if r.description]

    return (
        f"⚠️ The metric has NOT improved for {streak} consecutive iterations.\n"
        f"Recent attempts: {'; '.join(recent_failures[-5:])}\n"
        "You MUST try a fundamentally different approach:\n"
        "- Change the core algorithm entirely\n"
        "- Restructure the data representation\n"
        "- Challenge an assumption from earlier iterations\n"
        "- Do NOT make small tweaks to previous failed approaches"
    )
```

### Config

```yaml
constraints:
  plateau_threshold: 8    # optional, default 8
```

### Relationship with existing strategy hints

- `_strategy_hint()` — based on `consecutive_failures` from orchestrator (crash + discard)
- `_plateau_hint()` — based on results records, longer history window, stronger wording
- Both may fire simultaneously — intentional reinforcement

### Out of scope

- No simulated annealing, no checkpoint branching (deferred to future discussion)

---

## P4 — Evaluation Quality Guard

### 4a. Metric stability check — `crucible validate --stability`

**Changes: `validator.py` + `cli.py`**

```python
def check_stability(workspace, config, runs=5) -> StabilityResult:
    """Run N times, compute CV (coefficient of variation)"""
    # Run experiment N times, collect metrics
    # CV = stdev / |mean| * 100
    # Stable if CV < 5%
```

CLI: `crucible validate --stability --runs 5`

Output:
```
[WARN] Metric stability: CV = 12.3% over 5 runs
       Values: [142000, 128000, 155000, 131000, 148000]
       Consider: fix random seeds or increase sample size
```

### 4b. Multi-run median

**Changes: `config.py` + `runner.py` + `orchestrator.py`**

```yaml
evaluation:
  repeat: 3          # default 1
  aggregation: "median"  # median | mean, default median
```

`runner.py` adds `execute_with_repeat()` — runs experiment N times, returns aggregated metric. Any single failure = crash.

### 4c. Correctness gate template

**Changes: `wizard.py`**

Generated `evaluate.py` includes structured three-phase template:
1. Correctness gate (TODO placeholder)
2. Method verification (TODO placeholder)
3. Performance measurement

Template only — user fills in content.

### Out of scope

- No eval anomaly detection (hard to define universal "anomalous" threshold)

---

## P0 — Sandbox Execution Environment

### Architecture

```
orchestrator → SandboxRunner (new) → Docker / NativeRunner (existing)
                    ↓
              config.sandbox.backend decides path
```

### Config

```yaml
sandbox:
  backend: "docker"           # docker | none (default none)
  base_image: "python:3.12-slim"
  network: false
  memory_limit: "2g"          # optional
  cpu_limit: 2                # optional
```

### SandboxRunner — new module `sandbox.py`

- `backend == "docker"` → `docker run --rm` wrapper
- `backend == "none"` → delegate to existing `ExperimentRunner`

### Auto image build

```python
def _ensure_image(self, workspace: Path) -> str:
    """Auto-build image from requirements.txt or pyproject.toml"""
    tag = f"crucible-{workspace.name}:latest"
    # Generate Dockerfile: FROM base_image + pip install deps
    # Cache by dependency file hash — only rebuild on change
    return tag
```

- `sandbox.base_image` as FROM base
- Auto-detect `requirements.txt` / `pyproject.toml`, build with deps
- Hash-based cache — only rebuild when deps change
- No `.venv` mounting (host binaries incompatible with container)

### Readonly mount strategy

- Entire workspace mounted as rw volume
- readonly + hidden files additionally mounted as `:ro` override
- Agent's modifications write back to host filesystem for git detection

### Docker availability check

`crucible validate` checks `docker info` when `sandbox.backend == "docker"`. Fails with helpful message if Docker unavailable.

### Out of scope

- No bubblewrap backend (Linux-only, excluded by cross-platform requirement)
- No resource monitoring (cgroup usage → JSONL)
- No `sandbox-exec` (macOS deprecated)

---

## P1 — Agent Backend Abstraction

### Core interface

```python
# agents/base.py
class AgentInterface(ABC):
    @abstractmethod
    def generate_edit(self, prompt: str, workspace: Path) -> AgentResult:
        ...

    def capabilities(self) -> set[str]:
        """Capabilities this backend supports"""
        return {"read", "edit", "write", "glob", "grep"}
```

**Design decision**: Keep existing `generate_edit(prompt, workspace)` signature. Context assembly stays in `context.py`. Agent receives final prompt string only.

### ClaudeCode backend — refactor existing

- Add `usage` extraction from ResultMessage (if SDK provides it)
- Add `duration_seconds` timing around `_run_query()`
- Hidden file hooks unchanged

### Ollama backend — new `agents/ollama.py`

```python
class OllamaAgent(AgentInterface):
    def generate_edit(self, prompt, workspace) -> AgentResult:
        # 1. Read all editable files into prompt (no filesystem tools)
        # 2. Call Ollama chat API (stream=False)
        # 3. Parse structured JSON output: {"edits": [{"file", "search", "replace"}]}
        # 4. Apply edits to workspace
        # 5. Collect usage from response (prompt_eval_count, eval_count)

    def capabilities(self) -> set[str]:
        return {"edit"}  # edit only, no autonomous file reading
```

Key difference: Ollama has no filesystem tools → all file content pre-loaded into prompt, output format is structured JSON (search/replace).

### Agent factory

```python
# agents/__init__.py
def create_agent(config: AgentConfig, **kwargs) -> AgentInterface:
    match config.type:
        case "claude-code": return ClaudeCodeAgent(**kwargs)
        case "ollama": return OllamaAgent(model=config.model, ...)
        case _: raise ValueError(f"Unknown agent type: {config.type}")
```

### Config

```yaml
agent:
  type: "ollama"              # claude-code | ollama
  model: "qwen2.5-coder:32b"
  base_url: "http://localhost:11434"
  timeout: 600
```

### Orchestrator adjustment

```python
# If agent can't read files, pre-load into prompt
if "read" not in self.agent.capabilities():
    prompt = self.context.assemble_with_files(workspace, editable_files)
else:
    prompt = self.context.assemble()
```

### Out of scope

- No AnthropicAPI / OpenAI backend (interface ready for future additions)
- No streaming abstraction
- No tool use format abstraction (ClaudeCode uses SDK tools, Ollama uses structured output)
- No model switching strategy (cheap explore + expensive refine)

---

## Files Changed Summary

| Module | P5 | P2 | P3 | P4 | P0 | P1 |
|--------|----|----|----|----|----|----|
| `results.py` | **rewrite** | | | | | |
| `orchestrator.py` | modify | modify | | modify | modify | modify |
| `context.py` | | | **modify** | | | modify |
| `config.py` | | modify | modify | modify | modify | modify |
| `cli.py` | modify | modify | | modify | | |
| `budget.py` | | **new** | | | | |
| `sandbox.py` | | | | | **new** | |
| `agents/base.py` | modify | modify | | | | modify |
| `agents/claude_code.py` | | modify | | | | modify |
| `agents/ollama.py` | | | | | | **new** |
| `agents/__init__.py` | | | | | | **new** |
| `runner.py` | | | | modify | | |
| `validator.py` | | | | modify | | |
| `wizard.py` | | | | modify | | |
