# ABREKA User Guide

ABREKA is a CLI tool that automates the research experiment loop: **propose, implement, test, run, evaluate**. It drives a coding agent to iteratively design experiments, write code, validate it, execute training runs, and interpret results — then loop again, building on what it learned.

This guide covers everything from first setup through advanced configuration.

---

## Table of Contents

- [Getting Started](#getting-started)
  - [Prerequisites](#prerequisites)
  - [Installation](#installation)
  - [Initializing a Project](#initializing-a-project)
  - [Setting the Research Goal](#setting-the-research-goal)
  - [Running Your First Experiment](#running-your-first-experiment)
- [The Autonomous Loop](#the-autonomous-loop)
  - [How It Works](#how-it-works)
  - [Duration Runs](#duration-runs)
  - [Single-Step Mode](#single-step-mode)
  - [Error Recovery](#error-recovery)
- [Experiments](#experiments)
  - [What Is an Experiment?](#what-is-an-experiment)
  - [Status Lifecycle](#status-lifecycle)
  - [The Experiment DAG](#the-experiment-dag)
  - [Metrics](#metrics)
  - [Runtime Costs](#runtime-costs)
  - [Findings](#findings)
  - [Artifacts & Sessions](#artifacts--sessions)
- [Experiment Management](#experiment-management)
  - [Creating Experiments](#creating-experiments)
  - [Updating Experiments](#updating-experiments)
  - [Finishing and Failing](#finishing-and-failing)
  - [Remote Run Recovery](#remote-run-recovery)
  - [Deleting Experiments](#deleting-experiments)
  - [Listing and Filtering](#listing-and-filtering)
  - [Viewing Details](#viewing-details)
  - [Searching](#searching)
  - [Comparing Experiments](#comparing-experiments)
  - [Viewing the Tree](#viewing-the-tree)
  - [Chatting With the Project](#chatting-with-the-project)
  - [Chatting With an Experiment](#chatting-with-an-experiment)
- [Backlog](#backlog)
  - [What Is the Backlog?](#what-is-the-backlog)
  - [Adding Ideas](#adding-ideas)
  - [Listing and Filtering](#listing-and-filtering-1)
  - [Viewing Details](#viewing-details-1)
  - [Resolving and Dismissing](#resolving-and-dismissing)
  - [Prioritizing](#prioritizing)
  - [Deleting Items](#deleting-items)
  - [Agent Integration](#agent-integration)
- [Project Wiki](#project-wiki)
  - [What Is the Wiki?](#what-is-the-wiki)
  - [Adding and Editing Pages](#adding-and-editing-pages)
  - [Reading and Searching](#reading-and-searching)
  - [Renaming, Merging, Deleting](#renaming-merging-deleting)
  - [How the Wiki Reaches Agents](#how-the-wiki-reaches-agents)
  - [Wiki Maintenance During Evaluate](#wiki-maintenance-during-evaluate)
  - [EDA During Propose](#eda-during-propose)
- [Branch Summaries](#branch-summaries)
  - [Inspecting a Branch](#inspecting-a-branch)
  - [Overview of All Branches](#overview-of-all-branches)
  - [Health Classifications](#health-classifications)
- [Agent Commands](#agent-commands)
  - [Propose](#propose)
  - [Implement](#implement)
  - [Test](#test)
  - [Evaluate](#evaluate)
- [Configuration](#configuration)
  - [config.toml Reference](#configtoml-reference)
  - [Agent Configuration](#agent-configuration)
  - [Customizing Prompts](#customizing-prompts)
  - [Template Variables](#template-variables)
- [Project Structure](#project-structure)
  - [Directory Layout](#directory-layout)
  - [Shared Library & Copy-on-Write](#shared-library--copy-on-write)
  - [The Index File](#the-index-file)
- [Metrics Protocol](#metrics-protocol)
  - [Stdout Convention](#stdout-convention)
  - [Setting Metrics via CLI](#setting-metrics-via-cli)
  - [Querying by Metrics](#querying-by-metrics)
- [Web UI](#web-ui)
  - [Starting the Server](#starting-the-server)
  - [Dashboard](#dashboard)
  - [Experiment Browser](#experiment-browser)
  - [Session Viewer](#session-viewer)
  - [Experiment Tree](#experiment-tree)
  - [Backlog Browser](#backlog-browser)
  - [Wiki Browser](#wiki-browser)
  - [Config Page](#config-page)
- [Maintenance Utilities](#maintenance-utilities)
  - [Repairing Bloated Session Files](#repairing-bloated-session-files)
- [Troubleshooting](#troubleshooting)

---

## Getting Started

### Prerequisites

- **Python 3.11+**
- **[uv](https://docs.astral.sh/uv/)** package manager
- **[pi](https://github.com/badlogic/pi-mono)** coding agent (required for agent-driven commands)
- An existing `uv`-managed Python project with a `pyproject.toml` and `src/` layout

### Installation

Install ABREKA as a standalone tool:

```bash
uv tool install abreka
```

Or run it directly without installing:

```bash
uvx abreka --help
```

### Bootstrapping a New Project

The fastest way to start is `abreka bootstrap`, which creates a new uv project, initializes abreka, and launches an interactive pi session to help you set up your research goal, metrics, dependencies, and initial shared library code:

```bash
abreka bootstrap my-research-project
```

This will:
1. Create the directory and run `uv init --app --package`
2. Initialize the abreka directory structure
3. Launch pi in interactive mode to guide you through setup

Use `--no-agent` to skip the interactive session and set things up manually:

```bash
abreka bootstrap my-research-project --no-agent
```

### Initializing an Existing Project

If you already have a `uv`-managed Python project, you can add abreka to it directly:

```bash
cd my-research-project/
abreka init .
```

This reads your `pyproject.toml` to discover the project name and `src/` library layout, then creates an `abreka/` directory alongside your existing code:

```
abreka/
  config.toml        # Project settings, metrics, agent config
  goal.md            # Your research objective (edit this!)
  index.json         # Experiment metadata cache (auto-maintained)
  backlog/           # Backlog ideas
  backlog_index.json # Backlog metadata cache (auto-maintained)
  prompts/           # Agent prompt templates (editable)
    propose_exploit.md
    propose_explore.md
    implement.md
    test.md
    repair_run.md
    evaluate.md
  artifacts/
    shared/          # Project-wide curated artifacts reused across experiments and synced to remote runs
  chats/             # Project-wide interactive chat sessions (created on first use)
  experiments/       # Where experiments live
```

ABREKA never modifies your `pyproject.toml`.

### Setting the Research Goal

The most important step after init is writing your research goal. Open `abreka/goal.md` in your editor:

```bash
$EDITOR abreka/goal.md
```

This file is injected into every agent prompt. Be specific about what you're trying to achieve, how success is measured, and any constraints:

```markdown
# Research Goal

Achieve the highest possible validation accuracy on MNIST digit
classification using models that train in under 60 seconds on a
single GPU (or CPU).

## Success Criteria

- val:accuracy > 0.995
- Training time < 60 seconds
- Model must generalize (train:accuracy - val:accuracy < 0.01)

## Constraints

- Dataset: standard MNIST (torchvision)
- Framework: PyTorch
- Shared library code lives in `src/mnist_lab/`
- Each experiment must print METRIC: lines to stdout
```

The clearer and more specific your goal, the better the agent's proposals.

### Running Your First Experiment

The fastest way to get started is single-step mode, which runs one full cycle and stops:

```bash
abreka run --step
```

This executes the complete loop once: the agent proposes an experiment, writes the code, validates it with tests, runs training, and evaluates results. You can inspect the output, review what happened, and run again.

For fully autonomous operation over a longer period:

```bash
abreka run 2h    # Run for 2 hours
abreka run 24h   # Run overnight
```

---

## The Autonomous Loop

### How It Works

The core of ABREKA is a five-stage loop:

```
1. Propose    Agent reviews the goal and all past experiments, then
              proposes the single most promising next experiment.

2. Implement  Agent writes code: the experiment's run.py entry point,
              any experiment-specific modules, and optionally grows
              the shared library.

3. Test       Agent runs the full test suite. If tests fail, the agent
              attempts to fix them. If they still fail, the experiment
              is marked failed and the loop moves on.

4. Run        ABREKA executes the experiment's run.py via uv. Stdout
              is captured. METRIC: lines are parsed for structured data.
              A configurable timeout prevents runaway processes. If run.py
              exits non-zero, ABREKA invokes the repair_run prompt and retries
              the run (default: up to 3 repairs). Timeouts are treated as
              ordinary failures and are not repaired.

5. Evaluate   Agent reads the run output and artifacts, extracts metrics,
              writes narrative findings, and marks the experiment completed
              or failed.
```

After evaluation, the loop starts again with a new proposal that builds on everything learned so far.

### Duration Runs

Specify how long ABREKA should run autonomously:

```bash
abreka run 24h    # 24 hours
abreka run 2h     # 2 hours
abreka run 30m    # 30 minutes
abreka run 90s    # 90 seconds (mostly for testing)
```

ABREKA tracks wall-clock time. When the duration is exceeded, it finishes the current experiment and stops. Individual experiment timeouts (set in `config.toml`) are separate from the overall duration.

### Single-Step Mode

For human-in-the-loop workflows:

```bash
abreka run --step
```

This runs exactly one iteration of the loop, then stops. You can inspect results, adjust your goal, modify prompts, or manually edit experiment metadata before running again.

### Graceful Stop Requests

If a multi-iteration autonomous run is already in progress and you want it to stop cleanly, queue a stop request from another terminal:

```bash
abreka stop --after-current
```

ABREKA does **not** interrupt the active experiment. It lets the current experiment finish all remaining stages, including `evaluate`, then exits before proposing or reserving the next experiment. After the request is honored, the stop flag is cleared automatically.

If no autonomous run is currently active, the next `abreka run ...` command notices the queued request immediately, prints a message, clears it, and exits without starting new work. Queued stop requests are also surfaced in `abreka status` and as a banner on the web UI dashboard.

### Remote Run Mode

ABREKA can offload **only the run step** to a remote machine while keeping propose, implement, test, and evaluate local. This is useful when you want to save GPU/VM costs by starting remote compute only for training.

In remote mode, ABREKA:

1. optionally runs a local startup hook (for example, booting a VM),
2. syncs the project to a remote workspace with `rsync`,
3. optionally runs a remote bootstrap command (for example, `uv sync --frozen`),
4. launches `run.py` remotely over SSH in detached mode,
5. polls the remote log and mirrors it into local `run_output.txt`,
6. syncs the experiment artifacts back, and
7. optionally runs a local teardown hook.

The local project remains the source of truth: experiment status, metrics extraction, and evaluation still happen locally.

Example configuration:

```toml
[run]
mode = "remote"
timeout_minutes = 30
require_tests = true

[run.remote]
host = "gpu-box"
project_dir = "/workspace/my-project"
bootstrap_command = "uv sync --frozen"
poll_seconds = 5
startup_command = "./scripts/start-gpu-vm.sh"
teardown_command = "./scripts/stop-gpu-vm.sh"
```

`project_dir` should point to a dedicated remote workspace for this project. ABREKA syncs the project there with `rsync --delete`, excluding common local-only directories such as `.git/`, `.venv/`, caches, and per-experiment artifact directories. Project-wide shared artifacts under `abreka/artifacts/shared/` are included in this sync so remote runs can read them.

### Explore Mode

By default the propose step uses `propose_exploit.md`, which biases the agent toward extending the best existing experiment lines. You can add **exploration** — encouraging the agent to try genuinely novel approaches:

```bash
abreka run --explore 2h              # Always explore (every iteration)
abreka run --explore-prob 0.1 2h     # Explore 10% of iterations, exploit the rest
abreka run --step --explore          # Single step in explore mode
```

When exploring, the agent uses `propose_explore.md`, which reframes the task: instead of "single most promising experiment," it asks for a fundamentally different approach the research hasn't tried yet. Explore experiments are tagged with `explore` for easy filtering.

You can also set a default explore probability in `config.toml`:

```toml
[experiment]
explore_probability = 0.1   # 10% of iterations explore by default
```

The `--explore` and `--explore-prob` CLI flags override the config value.

### Focus Mode

Sometimes a branch of the experiment tree shows promise but isn't the top performer on the primary metric. Exploit mode would ignore it in favor of the overall best. Use `--focus` to constrain the agent to continue from within a specific subtree:

```bash
abreka run --step --focus 0004           # Exploit within the 0004 subtree
abreka run 2h --focus 0004               # Multi-iteration focus
abreka run --step --focus 0004 --explore # Explore within the subtree
abreka propose --focus 0004              # Standalone propose with focus
```

When focused, the agent:
- **Must** set `--parent` to a completed experiment within the subtree (the root or any descendant).
- Sees the subtree experiments in detail and outside experiments in a collapsed summary.
- Reports both the best overall result and the best within the subtree.

Focus is composable with explore/exploit — it constrains *where* the agent works, not *how* it works.

### Guidance

You can inject ad-hoc instructions into the propose prompt using `--guidance` or `--guidance-file`. This steers the agent without editing prompt templates:

```bash
# Inline guidance (short)
abreka propose --guidance "Focus on attention-based architectures"
abreka run --step --guidance "Avoid CNN approaches, try transformers"

# Multi-line guidance from a file
abreka propose --guidance-file notes/guidance.md
abreka run 2h --guidance-file notes/guidance.md

# Interactive multi-line input (type until EOF on its own line)
abreka propose --guidance
Enter guidance (end with EOF on its own line):
Focus on transformer architectures.
Avoid anything CNN-based.
EOF
```

The guidance text appears as a "Researcher Guidance" section in the prompt, positioned after the experiment history and backlog context but before the task instructions. When no guidance is provided, the section is omitted entirely.

Guidance is composable with `--focus` and `--explore` — it controls *what direction* the agent considers, while focus constrains *where* and explore/exploit constrains *how*.

### Error Recovery

The loop is designed to keep going:

- If any stage fails, the experiment is marked `failed` with a reason, and the loop continues with a new proposal.
- If `run.py` exits non-zero, ABREKA preserves the failed output, asks an agent to repair the existing experiment, and reruns `run.py`. By default it allows **3 repair attempts** (configurable with `run.max_repair_attempts`).
- Run timeouts are not repaired: exceeding `run.timeout_minutes` means the experiment failed for the allotted window. Any metrics printed before timeout are saved as partial metrics, but the experiment is still marked `failed`.
- The agent sees prior failures when proposing, so it learns from mistakes.
- After **5 consecutive experiment failures**, ABREKA stops to prevent wasting resources on a systemic issue.

ABREKA supports two kinds of interrupted-process recovery:

1. **Native pi stage-session recovery** for interrupted `propose`, `implement`, `test`, and `evaluate` stages.
2. **Remote run recovery** for interrupted remote-only `run` stages.

For **remote run mode**, ABREKA also stores recovery metadata in `abreka/experiments/<id>/artifacts/remote_run.json`. If your local process dies or SSH polling is interrupted, you can inspect and recover the run explicitly:

```bash
abreka exp remote list          # Show all experiments with remote-run metadata
abreka exp remote status 0007   # Inspect host, PID, log/status paths
abreka exp remote sync 0007     # Fetch latest remote artifacts
abreka exp remote resume 0007   # Reattach, finalize, then evaluate
abreka exp remote stop 0007     # Stop the remote run and mark failed
abreka run --recover            # Convenience: recover exactly one interrupted local stage or remote run
```

---

## Experiments

### What Is an Experiment?

An experiment is the fundamental unit of work. Every experiment has:

- **ID**: A 4-digit zero-padded identifier (0001, 0002, ...), assigned automatically.
- **Hypothesis**: What the experiment is testing.
- **Method**: How it will test the hypothesis.
- **Status**: Where it is in the lifecycle.
- **Parents**: Optional references to prior experiments it builds on (forming a DAG). Experiments with no parents represent new research directions.
- **Tags**: For categorization and filtering.
- **Metrics**: Structured numeric results, namespaced by split (train, val, test).
- **Runtime Costs**: Automatic wall-clock timing per stage (`propose`, `implement`, `test`, `run`, `run_repair`, `evaluate`), including retries/resumes.
- **Failures**: A saved list of run failures and preserved failed outputs, used by the repair prompt and useful for audit/debugging.
- **Findings**: Narrative analysis written by the evaluate agent.

Each experiment lives in its own directory under `abreka/experiments/<id>/`.

### Status Lifecycle

Experiments progress through a defined lifecycle:

```
proposing --> proposed --> implementing --> testing --> running --> completed
     |             |             |             |           |
     +----------- failed --------+-------------+-----------+
```

- **proposing** — A proposal session has reserved the experiment ID and is still filling in the hypothesis/method.
- **proposed** — Hypothesis and method defined, no code yet.
- **implementing** — Agent is writing code.
- **testing** — Agent is running tests to validate the code.
- **running** — The experiment's `run.py` is executing, or ABREKA is repairing/retrying a non-timeout run failure.
- **completed** — Run finished, metrics and findings recorded.
- **failed** — Something went wrong. The `failure_reason` field explains what.

`completed` and `failed` are terminal states.

### The Experiment DAG

Experiments reference parents, forming a directed acyclic graph. This captures how ideas branch and converge:

```
0001 Vanilla CNN baseline
 |-- 0002 Add dropout (parent: 0001)
 |    \-- 0005 Dropout + CutMix (parent: 0002)
 |-- 0003 Random augmentation (parent: 0001)
 \-- 0004 Combine dropout + augmentation (parents: 0002, 0003)
0006 Try transformer approach          (no parent — new direction)
```

The agent uses this lineage when proposing new experiments. You can view the DAG with `abreka exp tree` or in the web UI.

### Metrics

Metrics are structured numeric results stored in `experiment.toml`. They are namespaced by **split** (train, val, test) and identified by **key** (accuracy, loss, f1, etc.):

```toml
[metrics.train]
accuracy = 0.9985
loss = 0.005

[metrics.val]
accuracy = 0.9934
loss = 0.028
```

The **primary metric** (configured in `config.toml`, default `val:accuracy`) determines default sorting and "best experiment" tracking.

### Runtime Costs

ABREKA automatically records wall-clock runtime per stage in each `experiment.toml` under `runtime_costs`.

Tracked stages:
- `propose`
- `implement`
- `test`
- `run`
- `run_repair` (automatic LLM repairs after non-timeout run failures)
- `evaluate`

Each stage stores attempt count, cumulative seconds, last-attempt seconds, and last status (`ok`, `error`, or `timed_out`). Run retries, repair attempts, and `--resume` executions append to the same stage totals.

Use `abreka exp show <id>` to inspect these runtime costs from the CLI.

### Findings

The evaluate agent writes `findings.md` for each experiment — a narrative analysis of what happened, whether the hypothesis was supported, how results compare to parent experiments, and suggestions for future work.

### Artifacts & Sessions

ABREKA has two artifact scopes:

- **Project-wide shared artifacts** live under `abreka/artifacts/shared/`. Use this area for curated, reusable files such as EDA reports, reference tables, or plots that future experiments may need. In remote mode, this shared subtree is pushed to the remote workspace as part of the normal project sync.
- **Experiment-local artifacts** live under `abreka/experiments/<id>/artifacts/` and contain outputs from a single run.

Each experiment's `artifacts/` directory contains:

- `checkpoints/` — Model checkpoints saved during training.
- `plots/` — Visualizations generated by the experiment.
- `eda/` — Optional notes, plots, and summary stats produced during the propose stage. Distilled findings are later promoted to the project wiki by evaluate.
- `run_output.txt` — Full captured stdout/stderr from the latest run attempt.
- `run_failures/run_attempt_<n>.txt` — Preserved stdout/stderr snapshots for failed run attempts.
- `remote_run.json` — Local metadata for a remote run (host, workspace, PID, state). Present only in remote mode.
- `remote_status.json` — Final remote exit status synced back after a remote run. Present only in remote mode.
- `pi_session_propose.jsonl` — Native pi session for the propose stage.
- `pi_session_implement.jsonl` — Native pi session for the implement stage.
- `pi_session_test.jsonl` — Native pi session for the test stage.
- `pi_session_run_repair_<n>.jsonl` — Native pi session for automatic run repair attempt `<n>`.
- `pi_session_evaluate.jsonl` — Native pi session for the evaluate stage.

These are native pi session files, not RPC event dumps. You can point pi directly at them with `pi --session <path>` to inspect or resume the exact stage conversation.

---

## Experiment Management

All experiment management commands are under `abreka exp`.

### Creating Experiments

```bash
abreka exp new --hypothesis "Add batch normalization" --method "Insert BN after each conv layer" --parent 0001 --tag architecture
```

Options:

| Flag | Description |
|------|-------------|
| `-h, --hypothesis` | Experiment hypothesis (text) |
| `-m, --method` | Experiment method (text) |
| `-p, --parent` | Parent experiment ID (repeatable for multiple parents) |
| `-t, --tag` | Tag for categorization (repeatable) |

The command prints the new experiment ID (e.g., `0003`).

In practice, experiments are usually created by the propose agent rather than manually. The agent may create experiments with parents (continuing a line of research) or without parents (starting a new direction). Manual creation is useful when you want to inject a specific idea into the DAG.

### Updating Experiments

```bash
abreka exp update 0003 --hypothesis "Revised hypothesis" --metric val:accuracy=0.9934 --metric train:loss=0.005
```

Options:

| Flag | Description |
|------|-------------|
| `--hypothesis` | Set hypothesis text |
| `--method` | Set method text |
| `--status` | Set status (proposing, proposed, implementing, testing, running, completed, failed) |
| `--parent` | Set parent IDs (replaces existing; repeatable) |
| `--metric` | Set metric as `[split:]key=value` (repeatable) |
| `--tag` | Set tags (replaces existing; repeatable) |

Metric format examples:
- `val:accuracy=0.9934` — explicit split and key
- `train:loss=0.005` — train split
- `accuracy=0.85` — split defaults to `val`

### Finishing and Failing

Mark an experiment as completed:

```bash
abreka exp finish 0003
```

This sets the status to `completed` and records a `completed_at` timestamp.

Mark an experiment as failed with a reason:

```bash
abreka exp fail 0003 --reason "OOM on batch size 256"
```

The `--reason` (or `-r`) flag is required.

### Recovery

ABREKA can recover exactly one interrupted unit of work at a time. Recoverable work includes:

- a `proposing` experiment with `pi_session_propose.jsonl`
- an `implementing` experiment with `pi_session_implement.jsonl`
- a `testing` experiment with `pi_session_test.jsonl`
- a `running` experiment with `pi_session_evaluate.jsonl` (interrupted evaluate stage)
- a remote run with recoverable `remote_run.json` metadata

Use `abreka run --recover` when you want ABREKA to detect that single interrupted item automatically, resume it, and finish the rest of the pipeline for that experiment. If you also pass a duration or `--step`, ABREKA recovers first and then continues the normal autonomous loop.

#### Remote Run Recovery

When `[run].mode = "remote"`, only the run stage is offloaded. ABREKA records enough metadata locally to inspect or recover an interrupted remote run later.

```bash
abreka exp remote list
abreka exp remote status 0007
abreka exp remote sync 0007
abreka exp remote resume 0007
abreka exp remote resume 0007 --no-evaluate
abreka exp remote stop 0007
abreka exp remote stop 0007 --kill
abreka run --recover
abreka run --recover 24h
```

What each command does:

- `list` — Show every experiment that currently has `artifacts/remote_run.json`.
- `status` — Read-only inspection of the saved remote metadata plus a live SSH check when the host is reachable.
- `sync` — One-shot sync back of the current remote `artifacts/` directory without waiting.
- `resume` — Reattach to an already-launched remote run, continue mirroring logs, sync artifacts back, parse metrics locally, and by default run the evaluate step. Use `--no-evaluate` to stop after finalizing the run result.
- `stop` — Send `SIGTERM` to the remote run, sync artifacts back, and mark the experiment failed locally. Use `--kill` to send `SIGKILL` instead.
- `run --recover` — Convenience wrapper: if exactly one interrupted local stage or remote run is recoverable, ABREKA resumes it automatically. If you also pass a duration or `--step`, ABREKA recovers first and then continues the normal autonomous loop.

These remote commands do **not** relaunch the run from scratch. They operate on the existing remote workspace and PID recorded in `artifacts/remote_run.json`.

#### Stage Sessions

Each agent stage stores its exact native pi session under `abreka/experiments/<id>/artifacts/`.

```bash
abreka exp session 0007 propose              # Print the path
abreka exp session 0007 implement --open     # Open interactively in pi
abreka exp session 0007 evaluate --export /tmp/eval.html
```

Use this when you want the exact agent-work session rather than a separate `abreka chat` / `abreka exp chat` discussion. Those chat commands create new human-driven conversations under `chats/`; `abreka exp session` targets the original stage session that did the work.

### Deleting Experiments

```bash
abreka exp delete 0003         # Prompts for confirmation
abreka exp delete 0003 --yes   # Skip confirmation
```

This removes the experiment directory and its entry from the index. Use with care — this is not reversible.

### Pruning a Subtree

When a flawed experiment has biased everything proposed downstream of it, prune the bad node and its descendants in one shot so the bias cannot leak into future proposals:

```bash
abreka exp prune 0003               # Delete 0003 and all descendants (prompts first)
abreka exp prune 0003 --exclusive   # Keep 0003; delete only its descendants
abreka exp prune 0003 --yes         # Skip confirmation
```

Prune walks the parent → child DAG from the target node, lists every experiment (and every backlog item whose `source_exp` falls inside the subtree) before prompting for confirmation, then deletes them. This is **destructive and not reversible** — the confirmation prompt shows the full list of ids that will be removed; review it before accepting.

Use `--exclusive` when the target itself is correct and you only want to discard the work that branched from it. Default is inclusive so a single command fully excises a bad node.

### Listing and Filtering

List all experiments:

```bash
abreka exp list
```

Filter by status:

```bash
abreka exp list --status completed
abreka exp list --status failed
```

Filter by parent (find children of an experiment):

```bash
abreka exp list --parent 0001
```

Sort by time or metric:

```bash
abreka exp list --sort time                     # Most recent first
abreka exp list --sort metric:val:accuracy      # By metric value
```

Filter by metric value:

```bash
abreka exp list --where "metric:val:accuracy > 0.95"
abreka exp list --where "metric:train:loss < 0.01" --sort metric:val:accuracy
```

Supported operators: `>`, `>=`, `<`, `<=`, `==`, `!=`.

### Viewing Details

```bash
abreka exp show 0007
```

Shows full experiment metadata, metrics, failure reason (if any), findings, and the **Runtime Costs** table (stage attempts, total seconds, last seconds, and last status).

The Web UI experiment detail page also shows the same runtime-cost summary for each experiment.

### Searching

Search across hypotheses, methods, tags, and findings:

```bash
abreka exp search "SAM"              # word-boundary match (won't match "SAME")
abreka exp search "/CNN|RNN/i"       # regex, case-insensitive
abreka exp search "/dropout.*0\\.5/"  # regex, case-sensitive
```

Plain queries use **word-boundary matching** — searching for `SAM` matches the word "SAM" but not "SAME" or "SAMPLE". Matching is case-insensitive by default.

Wrap a query in `/slashes/` to use **regex** (case-sensitive). Add an `i` flag (`/pattern/i`) for case-insensitive regex.

Filter results by status or tag:

```bash
abreka exp search "CNN" --status completed
abreka exp search "attention" --tag arch --tag transformer
```

### Comparing Experiments

Side-by-side diff of two experiments:

```bash
abreka exp diff 0003 0005
```

Displays a table comparing status, hypothesis, method, parents, and all metrics. Metric values are color-coded: green for better, red for worse (based on the `metric_direction` setting).

### Viewing the Tree

Visualize the experiment DAG:

```bash
abreka exp tree
```

Each node shows the experiment ID, status (color-coded), hypothesis snippet, and primary metric value.

### Chatting With the Project

Open an interactive [pi](https://github.com/badlogic/pi-mono) session briefed on the whole project — the research goal, config, experiment history, open backlog items, project wiki, prompt files, and shared library layout:

```bash
abreka chat
```

The chat runs from the project root and stores its session history under `abreka/chats/`.

```bash
abreka chat --continue                  # Resume the most recent project chat
abreka chat --resume                    # Pick from previous project chats
abreka chat "what should we try next?"
```

Use this when you want to think through strategy, compare branches, refine prompts, inspect shared code, or capture new ideas in the backlog or wiki. For structured state changes, prefer the normal CLI (`abreka backlog add`, `abreka wiki add`, `abreka exp update`, etc.). If you want a deep dive on one run, switch to `abreka exp chat <id>`.

### Chatting With an Experiment

Open an interactive [pi](https://github.com/badlogic/pi-mono) session briefed on a specific experiment — its metadata, hypothesis, method, metrics, findings, parent lineage, the project's research goal, and the abreka layout:

```bash
abreka exp chat 0001
```

The chat runs from the project root so pi can read or edit shared library code in `src/<lib>/` and any sibling experiment. Session files are persisted under `abreka/experiments/<id>/chats/` so each experiment has its own history.

```bash
abreka exp chat 0001 --continue            # Resume the most recent chat
abreka exp chat 0001 --resume              # Pick from previous chats
abreka exp chat 0001 "why did val loss spike at epoch 7?"
```

The chat agent is briefed to inspect files directly and defer to the normal CLI (`abreka exp update`, `abreka exp finish`, `abreka exp fail`, `abreka backlog add`) for any state changes.

---

## Backlog

### What Is the Backlog?

The backlog is a centralized place to capture experiment ideas that aren't ready to become full experiments yet. Ideas can come from anywhere — the evaluate agent might note a promising direction, you might spot an opportunity while reviewing results, or you might want to queue up ideas for later prioritization.

Each backlog item has:

- **ID**: A 4-digit zero-padded identifier (like experiments).
- **Title**: A short summary of the idea.
- **Description**: Longer rationale or notes.
- **Priority**: An integer (higher = more important, default 0).
- **Tags**: For categorization.
- **Source Experiment**: The experiment that inspired this idea (optional).
- **Status**: `open` (new idea), `resolved` (turned into an experiment), or `dismissed` (rejected).

Backlog items are stored as TOML files in `abreka/backlog/` with a fast-query index at `abreka/backlog_index.json`.

### Adding Ideas

```bash
abreka backlog add --title "Try attention mechanism" --description "Based on findings from exp 0002" --source-exp 0002 --tag arch --priority 3
```

Options:

| Flag | Description |
|------|-------------|
| `-t, --title` | Short summary (required) |
| `-d, --description` | Longer rationale or notes |
| `--tag` | Tag for categorization (repeatable) |
| `-p, --priority` | Priority integer (higher = more important, default 0) |
| `--source-exp` | Experiment ID that spawned this idea |

### Listing and Filtering

```bash
abreka backlog list                         # All items, sorted by priority
abreka backlog list --status open           # Only open items
abreka backlog list --tag arch              # Filter by tag
abreka backlog list --sort time             # Sort by creation time
```

Sort options: `priority` (default), `time`, `id`.

### Viewing Details

```bash
abreka backlog show 1
```

Displays all fields including description, source experiment, and dismiss reason.

### Resolving and Dismissing

When a backlog idea becomes an experiment:

```bash
abreka backlog resolve 1 --exp 0015    # Link to the experiment it became
abreka backlog resolve 1                # Resolve without linking
```

When an idea is no longer worth pursuing:

```bash
abreka backlog dismiss 1 --reason "Already tried in exp 0005"
```

### Prioritizing

Set or change the priority of an item:

```bash
abreka backlog prioritize 1 10    # Set priority to 10
```

Higher priority items appear first in listings and in the agent's prompt context.

### Deleting Items

```bash
abreka backlog delete 1         # Prompts for confirmation
abreka backlog delete 1 --yes   # Skip confirmation
```

### Agent Integration

The backlog integrates with the autonomous loop in two ways:

1. **Evaluate agent adds ideas**: The evaluate prompt includes an optional step for the agent to log promising directions to the backlog using `abreka backlog add --source-exp <id>`.

2. **Propose agent reads the backlog**: The propose prompt includes a "Pending Backlog Ideas" section showing all open items sorted by priority. The agent can draw inspiration from the backlog, and when it creates an experiment based on a backlog idea, it can resolve it with `abreka backlog resolve <id> --exp <new_exp_id>`.

When using `--focus` mode, only backlog items whose source experiment is within the focused subtree are shown to the propose agent.

---

## Project Wiki

### What Is the Wiki?

The project wiki is the durable cross-experiment knowledge layer. Where the backlog tracks future-experiment ideas and `findings.md` records what happened in one run, the wiki captures **knowledge that should outlive any single experiment**:

- Distilled EDA results ("9 of 10 MNIST classes are uniform; class 5 ≈9.0%").
- Methodology lessons ("never train on validation folds — produced a +0.04 spurious lift on exp 0007").
- Failure modes and their fixes ("batch > 64 OOMs on the 16 GB GPU").
- Domain knowledge collected from literature or earlier work.

Each page is a markdown file with TOML frontmatter, slug-keyed, stored under `abreka/wiki/pages/<slug>.md`. A flat index lives at `abreka/wiki_index.json` for fast listing and rendering into prompts. Pages are intended to be read on every stage and rewritten freely as understanding sharpens — agents may add, append to, edit, rename, and merge pages, but only humans may delete.

Each page records `created_at`, `updated_at`, `created_by_exp`, and `updated_by_exp` so you can audit who wrote what.

### Adding and Editing Pages

Create a new page:

```bash
abreka wiki add --title "Dataset class balance" --summary "9/10 MNIST classes uniform; class 5 ≈9.0%." --tag eda --tag dataset --content "$(cat <<'EOF'
# Dataset class balance

Body of the page in markdown.
EOF
)"
```

Useful flags:

| Flag | Description |
|------|-------------|
| `-t, --title` | Page title (required) |
| `--slug` | Slug (kebab-case). Derived from the title when omitted. |
| `--summary` | One-line summary surfaced in the wiki index inside agent prompts |
| `--tag` | Categorization tag. Repeatable. |
| `-c, --content` | Page body (markdown). Use `--content-file path` or `--content-file -` for stdin. |
| `--exp` | Experiment id authoring this page (recorded as `created_by_exp`) |

Append a dated note under a page's `## History` section:

```bash
abreka wiki append dataset-class-balance \
  --content "exp 0011 confirmed class 5 still ~9.0% on the held-out split." \
  --exp 0011
```

Edit an existing page (replace title / summary / tags / body):

```bash
abreka wiki edit dataset-class-balance \
  --summary "Refined: 9/10 uniform; class 5 ≈9.0%." \
  --content "$(cat consolidated.md)" \
  --exp 0011
```

### Reading and Searching

```bash
abreka wiki list                              # Index view
abreka wiki list --tag eda                    # Filter by tag
abreka wiki show dataset-class-balance        # Render the page
abreka wiki show dataset-class-balance --raw  # Raw on-disk markdown with frontmatter
abreka wiki search "class balance"            # Word-boundary search over slug/title/summary/tags/body
abreka wiki search "/balance.*\d/"            # /regex/ syntax (case-sensitive); add /i for case-insensitive
abreka wiki tags                              # Tag cloud with counts
```

### Renaming, Merging, Deleting

```bash
abreka wiki rename old-slug new-slug          # Move a page's slug
abreka wiki merge duplicate-page canonical    # Fold one page into another, union tags
abreka wiki delete old-page --yes             # Permanent — user-only, agents cannot run this
```

`merge` appends the source page's body under a `## Merged from <slug>` heading on the destination, unions the tags, then deletes the source. Renames update the file and the index together.

### How the Wiki Reaches Agents

When the runner builds a prompt for any stage (`propose`, `implement`, `test`, `evaluate`), it injects a compact `## Project Wiki` section at the top via a `{{wiki_index}}` placeholder. The section lists each page as a one-liner — `slug — summary (tags)` — and reminds the agent to read full pages with `uv run abreka wiki show <slug>` and to search with `uv run abreka wiki search "<query>"`.

Pages are not stage-scoped. Every page is shown to every stage; the agent decides which to read in detail based on the slug, summary, and tags.

### Wiki Maintenance During Evaluate

The default `evaluate.md` prompt includes a **Wiki Maintenance** task block. After writing findings, the evaluate agent is asked to:

1. **Add new pages** for non-obvious findings worth carrying forward.
2. **Append** a dated `## History` entry to existing pages this run confirmed, contradicted, or refined.
3. **Consolidate one** existing page if it has accumulated cruft (rewrite the body to be tighter; rename or merge for clarity).

Agents may use `wiki add`, `append`, `edit`, `rename`, and `merge`. They may **not** use `wiki delete` — that is a user-only operation. The trust boundary is intentional: the agent can restructure freely, but durable knowledge cannot be silently destroyed.

### EDA During Propose

The propose stage may run small amounts of exploratory data analysis when a relevant data question would sharpen the proposal. EDA artifacts (notes, plots, summary stats) are saved under `abreka/experiments/<id>/artifacts/eda/`. The evaluate stage then promotes the durable EDA findings into the wiki — raw EDA stays attached to the experiment, only the distilled lesson lands on the wiki.

---

## Branch Summaries

A **branch summary** is a compact, structured view of a rooted subtree of the experiment DAG. It acts as a shared search-state abstraction: `propose` and `evaluate` use it internally to reason about which branches deserve more effort, and you can inspect it directly via the CLI.

Each summary includes:

- Subtree size, depth, frontier, and active/completed/failed counts
- Root, best-in-subtree, latest, and global-best primary metric values
- Net gain from root and gap to global best
- Recent trend (parent deltas, improvement ratio, new subtree bests)
- A health classification with explanatory reasons
- A recommended next action (`exploit`, `discriminate`, `explore_elsewhere`, `stop_or_pause`, `validate`)
- Linked open backlog items

Branch summaries are computed deterministically in Python, not inferred by the agent, so they are auditable and cheap.

### Inspecting a Branch

```bash
abreka branch summary 0001              # Default markdown view
abreka branch summary 0001 --compact    # One-screen quick glance
abreka branch summary 0001 --verbose    # Full findings + tag/term detail
abreka branch summary 0001 --json       # Structured output for scripts
```

The default view shows sections for membership, performance, frontier, trend, health, uncertainty, and backlog — plus the recommended next action at the bottom.

### Overview of All Branches

```bash
abreka branch summary --all                         # Every real branch, sorted by attention
abreka branch summary --all --sort best             # Sort by best primary metric
abreka branch summary --all --sort size             # Sort by subtree size
abreka branch summary --all --sort health           # Sort by health status
abreka branch summary --all --sort recent           # Most recently touched first
abreka branch summary --all --json                  # Array of summaries as JSON
```

By default, `--all` enumerates parentless roots and **skips singletons** (branches with no descendants) because they are rarely useful as search branches. Pass `--include-singletons` to see them too.

Supported sorts: `attention` (default), `best`, `recent`, `size`, `health`.

### Health Classifications

| Status | Meaning |
|--------|---------|
| `improving` | Recent completed runs beat their parent and/or set a new subtree best. |
| `plateauing` | Enough completed runs, but no new best and recent deltas hover near zero. |
| `mixed` | Some improvements and some regressions, no strong signal. |
| `declining` | Average recent delta is meaningfully negative and no new bests. |
| `noisy` | Recent deltas alternate in sign with large swings. |
| `underexplored` | Two or fewer completed runs — not enough to judge. |
| `stalled` | All experiments failed; nothing running. |

Each classification ships with `health_reasons` explaining why it was assigned, so you can audit and tune the heuristics against your own project's intuition.

---

## Agent Commands

These commands invoke the coding agent (via pi-rpc) to perform individual stages of the loop. They are called automatically by `abreka run`, but can also be run manually.

All agent commands use the `--debug` flag on the root `abreka` command for verbose pi-rpc logging:

```bash
abreka --debug propose
abreka --debug implement 0003
```

### Propose

```bash
abreka propose
abreka propose --resume 0007
abreka propose --guidance "Focus on attention mechanisms"
abreka propose --guidance-file notes/guidance.md
abreka propose --guidance   # interactive multi-line input until EOF
```

The propose stage now **reserves the experiment ID first** and stores a native pi session at `abreka/experiments/<id>/artifacts/pi_session_propose.jsonl`. The agent then fills in that reserved experiment by setting its hypothesis, method, parents/tags, and final status `proposed`. If the stage is interrupted, resume it with `abreka propose --resume <id>`.

The default propose prompts permit small **exploratory data analysis** when a relevant data question would meaningfully sharpen the proposal. Any EDA artifacts (notes, plots, summary stats) are saved under `abreka/experiments/<id>/artifacts/eda/`. Keep EDA scoped to the question at hand — durable findings are promoted to the project wiki later, during evaluate.

Output: the reserved experiment ID.

Uses `propose_model` from config if set, otherwise falls back to the default `model`.

### Implement

```bash
abreka implement <exp_id>
abreka implement <exp_id> --resume
```

The agent writes code for a proposed experiment:

1. Creates `abreka/experiments/<id>/run.py` as the entry point.
2. Writes experiment-specific modules in `abreka/experiments/<id>/code/`.
3. Imports shared code from `src/<lib_name>/`.
4. May add new code to `src/<lib_name>/` to grow the shared library.
5. May modify existing shared code only if all tests still pass.
6. Writes tests for any new shared library code.

The agent is explicitly instructed **not** to run training or benchmarks during implementation — only write code.

The stage stores a native pi session at `abreka/experiments/<id>/artifacts/pi_session_implement.jsonl`. If the implement stage is interrupted, resume it with `--resume`.

Uses `implement_model` from config if set.

### Test

```bash
abreka test <exp_id>
abreka test <exp_id> --resume
```

The agent runs the full test suite (using the `test_command` from config, default `uv run pytest`). If tests fail, the agent attempts to fix them. The native pi session is stored at `abreka/experiments/<id>/artifacts/pi_session_test.jsonl` and can be resumed with `--resume`.

### Evaluate

```bash
abreka evaluate <exp_id>
abreka evaluate <exp_id> --resume
```

The agent reads the experiment's run output and artifacts, then:

1. Extracts metrics and records them via `abreka exp update`.
2. Writes `findings.md` with analysis of results, hypothesis validation, comparisons to parent experiments, and suggestions for next steps.
3. Marks the experiment `completed` or `failed`.
4. **Maintains the project wiki**: adds new pages for non-obvious durable findings, appends dated `## History` notes to existing pages this run confirmed/contradicted/refined, and optionally consolidates one page (rewrite/rename/merge) if it has accumulated cruft. The agent may use `wiki add`, `append`, `edit`, `rename`, and `merge`; it may **not** use `wiki delete`. Durable EDA from the propose stage is promoted into the wiki here.

The native pi session is stored at `abreka/experiments/<id>/artifacts/pi_session_evaluate.jsonl` and can be resumed with `--resume`.

Uses `evaluate_model` from config if set.

---

## Configuration

### config.toml Reference

Located at `abreka/config.toml`. Created by `abreka init` with sensible defaults.

```toml
[project]
name = "my-project"            # Discovered from pyproject.toml
lib = "my_lib"                 # Discovered from src/ layout
test_command = "uv run pytest" # How to run the test suite

[experiment]
primary_metric = "val:accuracy"  # Used for sorting and "best" tracking
metric_direction = "maximize"    # "maximize" or "minimize"
run_template = "./run.py"        # Default entry point template
# explore_probability = 0.1      # Default exploration probability (0.0 = exploit only)

[run]
mode = "local"                  # "local" or "remote"
timeout_minutes = 30             # Kill experiments exceeding this
require_tests = true             # Enforce TDD (won't run without passing tests)
max_repair_attempts = 3          # LLM repair attempts after non-timeout run failures

# Remote-run settings (only used when mode = "remote"):
# [run.remote]
# host = "gpu-box"               # SSH host / alias
# project_dir = "/workspace/my-project"
# bootstrap_command = "uv sync --frozen"
# poll_seconds = 5
# startup_command = "./scripts/start-gpu-vm.sh"
# teardown_command = "./scripts/stop-gpu-vm.sh"

[agent]
provider = "openai-codex"        # Agent provider
model = "gpt-5.4"               # Default model for all agents
thinking_level = "xhigh"        # Thinking depth: low, medium, high, xhigh

# Per-stage model overrides (optional):
# propose_model = "claude-opus-4-20250514"
# implement_model = "gpt-5.4"
# evaluate_model = "claude-sonnet-4-20250514"

# Extra arguments passed to pi-rpc (optional):
# [agent.pi]
# extra_args = ["--some-flag"]
```

### Agent Configuration

The `[agent]` section controls which coding agent is used:

- **provider**: The pi-rpc provider (e.g., `anthropic`, `openai-codex`, `openai`).
- **model**: The default model used for all agent stages.
- **thinking_level**: Controls the depth of agent reasoning. `xhigh` is the default and recommended for complex research tasks.
- **Per-stage overrides**: You can use a more capable model for propose (where creativity matters) and a faster model for implement/test (where execution matters).
- **Native stage sessions**: propose/implement/test/evaluate persist native pi session files (`pi_session_*.jsonl`) that can be resumed directly with pi or with ABREKA's `--resume` stage flags. Automatic run repairs also store `pi_session_run_repair_<n>.jsonl` files for inspection.

### Customizing Prompts

Agent prompts live in `abreka/prompts/` and are fully editable. ABREKA generates defaults during `abreka init`, but you can restructure them freely.

Each stage has its own template:

| File | Used by | Purpose |
|------|---------|---------|
| `propose_exploit.md` | `abreka propose`, `abreka run` | Analyze history, propose next experiment (exploitation) |
| `propose_explore.md` | `abreka run --explore` | Propose a genuinely novel direction (exploration) |
| `implement.md` | `abreka implement` | Write code for a proposed experiment |
| `test.md` | `abreka test` | Validate code, run tests |
| `repair_run.md` | automatic during `abreka run` | Repair an existing experiment after `run.py` exits non-zero |
| `evaluate.md` | `abreka evaluate` | Interpret results, write findings |

Templates use `{{variable}}` placeholders that ABREKA fills at runtime. You can add domain-specific instructions, change the tone, or restructure the prompts entirely.

### Template Variables

| Variable | Available in | Description |
|----------|-------------|-------------|
| `{{goal}}` | All | Contents of `abreka/goal.md` |
| `{{experiment_context}}` | propose_exploit, propose_explore, implement | Structured summary: current best, frontier experiments sorted by metric, and compact history |
| `{{primary_metric}}` | propose_exploit, propose_explore, implement, repair_run, evaluate | Primary metric from config (e.g., `val:accuracy`) |
| `{{metric_direction}}` | propose_exploit, propose_explore, implement, repair_run, evaluate | `maximize` or `minimize` |
| `{{exp_id}}` | implement, test, repair_run, evaluate | Current experiment ID |
| `{{hypothesis}}` | implement, test, repair_run, evaluate | Experiment hypothesis |
| `{{method}}` | implement, test, repair_run, evaluate | Experiment method |
| `{{parents}}` | implement | Comma-separated parent IDs |
| `{{lib_name}}` | implement, test, repair_run | Your package name from `src/` |
| `{{test_command}}` | test, repair_run | Test command from config |
| `{{run_command}}` | repair_run | Exact command the runner will execute after repair |
| `{{failure_reason}}` | repair_run | Latest non-timeout run failure reason |
| `{{run_attempt}}` | repair_run | The failed run attempt number |
| `{{repair_attempt}}` | repair_run | Current repair attempt number |
| `{{max_repair_attempts}}` | repair_run | Configured repair-attempt cap |
| `{{failure_history}}` | repair_run | Compact list of saved run failures for this experiment |
| `{{run_output}}` | repair_run, evaluate | Captured stdout/stderr from the latest run (tail-bounded in repair_run) |
| `{{artifacts_list}}` | evaluate | List of artifact files |
| `{{parent_context}}` | evaluate | For each parent: hypothesis, method, primary-metric value, a truncated findings snippet, and the signed delta vs the current experiment. Renders a "branch root or new direction" line when the experiment has no parent. |
| `{{branch_summary}}` | evaluate | Full branch summary for the subtree this experiment belongs to (root walked via `parents[0]`) — membership, performance, frontier, trend, health classification, and the recommended next action. |
| `{{global_context}}` | evaluate | Global-best comparison: global-best id and metric, branch-best id and metric, gap-to-global, gap-to-branch, and flags noting if this experiment is the new global or branch best. |
| `{{branch_overview}}` | propose_exploit, propose_explore | Compact ranking of every multi-experiment branch (parentless roots), capped to the top few by "attention". The focused subtree is excluded when `--focus` is used (it's rendered separately as `focused_branch_summary`). |
| `{{focused_branch_summary}}` | propose_exploit, propose_explore | Full branch summary for the focused subtree (membership, performance, frontier, trend, health, recommended action). Empty string when `--focus` is not used. |
| `{{focus_constraint}}` | propose_exploit, propose_explore | Focus constraint block (empty when unfocused, hard parent constraint when `--focus` is used) |
| `{{backlog_context}}` | propose_exploit, propose_explore | Open backlog ideas sorted by priority (filtered to subtree when `--focus` is used) |
| `{{guidance}}` | propose_exploit, propose_explore | Researcher guidance section (empty when `--guidance` is not used) |
| `{{reserved_exp_id}}` | propose_exploit, propose_explore | Pre-reserved experiment ID that the propose stage must populate and mark `proposed` |
| `{{wiki_index}}` | All | Project Wiki section — slug/summary/tags from `abreka/wiki_index.json`. Always rendered, with a "wiki is empty" hint when no pages exist. |

---

## Project Structure

### Directory Layout

After `abreka init` and a few experiments:

```
my-research-project/
  pyproject.toml                      # Your project (ABREKA reads, never modifies)
  src/
    my_lib/                           # Your shared library
      __init__.py
      models.py
      train.py
  tests/
    test_models.py
  abreka/
    config.toml                       # Settings
    goal.md                           # Research objective
    index.json                        # Experiment cache (auto-maintained)
    backlog/                          # Backlog ideas (TOML files)
    backlog_index.json                # Backlog cache (auto-maintained)
    wiki/
      pages/                          # Project wiki pages (markdown + TOML frontmatter)
    wiki_index.json                   # Wiki index cache (auto-maintained)
    prompts/
      propose_exploit.md              # Editable agent prompts
      propose_explore.md
      implement.md
      test.md
      repair_run.md
      evaluate.md
    chats/                            # Project-wide interactive chat sessions
    artifacts/
      shared/                         # Curated shared artifacts (synced to remote workspaces)
    experiments/
      0001/
        experiment.toml               # Structured metadata
        findings.md                   # Narrative findings
        run.py                        # Entry point
        code/                         # Experiment-specific modules
        artifacts/
          checkpoints/                # Model checkpoints
          plots/                      # Visualizations
          eda/                        # Optional EDA notes/plots from propose
          run_output.txt              # Captured stdout/stderr from latest run attempt
          run_failures/               # Preserved failed run outputs
          remote_run.json             # Remote-run metadata (remote mode only)
          remote_status.json          # Remote exit status (remote mode only)
          pi_session_propose.jsonl    # Native pi session for propose
          pi_session_implement.jsonl  # Native pi session for implement
          pi_session_test.jsonl       # Native pi session for test
          pi_session_run_repair_1.jsonl # Native pi session for run repair
          pi_session_evaluate.jsonl   # Native pi session for evaluate
        chats/                        # Experiment-specific interactive chat sessions
      0002/
        ...
```

### Shared Library & Copy-on-Write

Your code in `src/<lib_name>/` is shared across all experiments. The agent may modify it subject to these rules:

1. **Adding new code**: Encouraged. The shared library grows over time.
2. **Modifying existing code**: Allowed **only if all existing tests still pass**. The test suite is the arbiter.
3. **If modifications break tests**: The agent copies the code into the experiment's `code/` directory and modifies the copy instead.

This copy-on-write rule keeps old experiments reproducible while allowing the shared library to evolve.

### The Index File

`abreka/index.json` is a lightweight cache of all experiment metadata. It enables fast listing and filtering without reading individual `experiment.toml` files. The index is updated automatically on every `abreka exp` mutation. You should never need to edit it manually.

---

## Metrics Protocol

### Stdout Convention

Experiments print metrics to stdout during training using a simple line protocol:

```
METRIC:train:loss=0.005
METRIC:train:accuracy=0.9985
METRIC:val:loss=0.028
METRIC:val:accuracy=0.9934
```

Format: `METRIC:<split>:<key>=<value>`

If the split is omitted, it defaults to `val`:

```
METRIC:accuracy=0.85    # Equivalent to METRIC:val:accuracy=0.85
```

ABREKA captures all stdout, extracts `METRIC:` lines, and writes them into `experiment.toml`. Non-metric output is preserved in `artifacts/run_output.txt`.

### Setting Metrics via CLI

Metrics can also be set explicitly:

```bash
abreka exp update 0001 --metric val:accuracy=0.9912
abreka exp update 0001 --metric train:loss=0.005 --metric val:accuracy=0.99
```

### Querying by Metrics

```bash
abreka exp list --sort metric:val:accuracy
abreka exp list --where "metric:val:accuracy > 0.95"
abreka exp list --where "metric:train:loss < 0.01" --sort metric:val:accuracy
```

---

## Web UI

ABREKA includes a built-in web interface for browsing experiments, inspecting agent sessions, and viewing the experiment tree.

### Starting the Server

```bash
abreka webui start                  # http://127.0.0.1:8000
abreka webui start -p 3000          # Custom port
abreka webui start --host 0.0.0.0   # Bind to all interfaces
```

### Dashboard

The home page shows:

- Project name and research goal (double-click to edit inline).
- Experiment counts by status (total, completed, in progress, failed).
- A banner when a graceful stop has been queued with `abreka stop --after-current`.
- Status breakdown with clickable badges to filter.
- **Metric timeline chart** — an interactive scatter plot of the primary metric over experiment number. Each completed experiment with a recorded metric is a dot colored by its branch root (topmost ancestor via `parents[0]`). Solid line segments connect consecutive experiments that share a branch root; segments break when the sequence crosses to a different branch. A dotted grey line tracks the best metric so far. Click any dot to open that experiment; hover for ID, metric value, best-so-far, and hypothesis. Raw metric values are plotted with no axis flipping: higher y is always drawn higher, so `metric_direction = minimize` shows improvement as downward movement. Outliers are suppressed by default via a Tukey-fence (1.5·IQR) Y-range — any values that fall outside the fence are drawn as triangle markers at the axis edge (with the true value shown in the tooltip), and a caption reports how many points were clipped. Toggle **Include outliers in scale** to rescale to the full data range.
- Recent experiments table with links to details.

### Experiment Browser

The `/experiments` page lists all experiments in a filterable table with columns for ID, status, hypothesis, primary metric, parents, tags, and creation date. Click any status badge to filter. Click an experiment ID for the full detail view.

The detail page (`/experiments/<id>`) shows:

- All metadata: status, timestamps, parents, tags.
- Hypothesis and method.
- Metrics table with all splits and keys.
- Available session recordings with links to the session viewer.
- Run output (full-width, collapsible) — displayed below the widgets grid for easier reading of training logs.
- Findings rendered from markdown.

The experiment detail page uses a tabbed interface with two tabs:

- **Overview** — The default view with metadata, hypothesis, method, metrics, sessions, run output, and findings.
- **Files** — An integrated file browser with a VS Code-like split layout.

### File Browser

The file browser is available as the **Files** tab on the experiment detail page (`/experiments/<id>`). It provides a VS Code-like split-pane layout with a file tree on the left and a content viewer on the right. The standalone file list is also still accessible at `/experiments/<id>/files`.

- Click any file in the tree to view its contents with syntax highlighting in the right pane.
- Drag the vertical divider between the explorer and preview panes to resize the file tree; your preferred width is remembered in the browser.
- Use the **Expand** button to collapse the file tree sidebar for a full-width file view; click **Show files** to restore it.
- Supports syntax highlighting for Python, TOML, Markdown, JSON, YAML, shell scripts, and many other file types (powered by Pygments).
- Binary files (raster images, SVGs, model checkpoints, archives) are detected and labeled — images are rendered inline with a preview, while other binary files display a size indicator instead of raw content.
- Large files (over 100 KB) are truncated with a warning showing the original size.
- The file tree shows directories first, with nested files indented for easy navigation.

### Session Viewer

The session viewer (`/experiments/<id>/sessions/<stage>`) is the most powerful debugging tool. It parses the raw pi-rpc event stream and renders it as a readable conversation:

- **User prompts**: The full prompt sent to the agent (collapsible).
- **Assistant thinking**: The agent's reasoning (collapsed by default, expandable).
- **Assistant text**: The agent's visible response.
- **Tool calls**: Each tool invocation (bash, read, write, edit, etc.) with a one-line summary. Arguments and full results are collapsible.

Controls at the top let you expand or collapse all sections at once. Error tool calls are highlighted in red.

Available sessions per experiment:
- **propose** — How the experiment was conceived.
- **implement** — How the code was written.
- **test** — How tests were run and fixed.
- **evaluate** — How results were interpreted.

### Experiment Tree

The `/tree` page visualizes the experiment DAG. Each node shows the experiment ID, status badge, hypothesis snippet, and primary metric value. Click any node to navigate to its detail page. Child experiments are indented under their parents.

### Search

The `/search` page provides search with rich faceted filtering. Type a query to search across hypotheses, methods, and tags. Plain queries use word-boundary matching (e.g. `SAM` won't match "SAME"). Wrap in `/slashes/` for regex, or `/pattern/i` for case-insensitive regex. Use the sidebar facets to narrow results:

- **Status** — Check one or more statuses to filter (e.g. completed + running).
- **Tags** — Check tags to show only experiments with those tags.
- **Metrics** — Filter to experiments that have (or lack) recorded metrics.
- **Lineage** — Show only root experiments (no parents).

Facets can be combined freely. Counts next to each facet value reflect the current filtered result set. Active filters appear as removable chips above the results table.

### Backlog Browser

The `/backlog` page lists all backlog items in a filterable table with columns for ID, priority, status, title, tags, source experiment, and creation date. Click a status badge to filter. Click an item ID for the full detail view, which links to source and resolved experiments.

The dashboard also shows a count of open backlog items.

### Wiki Browser

The `/wiki` page lists every project wiki page with slug, title, tags, summary, and last-updated date. A search bar applies the same word-boundary / `/regex/` matching rules as `abreka exp search`, and tag chips filter by tag. A trailing **Add a Page** form creates a new page in place.

`/wiki/<slug>` renders the page body as markdown alongside its provenance (slug, tags, summary, `updated_at`, `updated_by_exp`, `created_by_exp`). Double-click the rendered view to switch to an inline form that edits the title, summary, tags, and body in one save. A separate Danger Zone exposes a Delete button — deletion is permanent and user-only.

The same data is available over a JSON API:

- `GET /api/wiki` — list (sorted by slug)
- `GET /api/wiki/<slug>` — full page (body + metadata)
- `POST /api/wiki` — create (`title` required; `slug` is derived from the title when omitted)
- `PUT /api/wiki/<slug>` — update title / summary / tags / content
- `DELETE /api/wiki/<slug>` — delete

### Config Page

The `/config` page renders the project's settings and provides inline editors for:

- **Goal** (`abreka/goal.md`) — double-click the card to edit and save.
- **Agent prompts** (`abreka/prompts/*.md`) — double-click any prompt card to edit.
- **Project Wiki** — a card showing the page count and a link to the dedicated `/wiki` view. Page editing happens on the wiki view, not on `/config`.

---

## Maintenance Utilities

The `abreka util` command group collects maintenance operations that don't belong to a specific experiment or agent stage.

### Repairing Legacy RPC Session Dumps

This utility only applies to legacy JSON-array RPC transcripts (`artifacts/pi_session*.json`) from older ABREKA versions. Current releases write native pi session files (`pi_session_*.jsonl`) instead, so most projects will never need this command.

Older RPC transcripts can become very large — in some cases multiple gigabytes — due to two kinds of redundant payload:

1. `toolcall_delta` events (cumulative tool-arg snapshots)
2. cumulative snapshots embedded in `message_update.message` and `message_update.assistantMessageEvent.partial`

To shrink *legacy* files, use:

```bash
abreka util repair-session-files          # dry run: report only
abreka util repair-session-files --apply  # actually rewrite (prompts to confirm)
```

Flags:

- `--apply` — perform the rewrite. Without this flag the command is always a dry run and does not touch any file.
- `-y`, `--yes` — skip the confirmation prompt when `--apply` is set.
- `--keep-backup` — move the original to `<file>.bak` instead of replacing it in place.

You can also pass one or more explicit file paths as positional arguments; with no paths the command discovers all `pi_session*.json` files under `abreka/experiments/*/artifacts/`.

**Safety:** the repair streams each file (never loading the whole thing into memory), writes the repaired content to `<file>.repair.tmp`, validates that the result is a valid JSON array with the expected event count, and only then atomically replaces the original. Files that don't contain repairable payloads are left untouched (mtime preserved). Files that fail to parse are reported as errors and skipped — their bytes are never modified.

---

## Troubleshooting

**"No abreka/config.toml found"**
You're not inside an ABREKA-initialized project. Run `abreka init .` first, or navigate to the correct directory.

**"pi command not found"**
The pi coding agent is not installed. Install it from [pi-mono](https://github.com/badlogic/pi-mono).

**Experiment stuck in "proposing", "implementing", or "testing"**
The agent session may have failed. Check the native pi session in `artifacts/pi_session_propose.jsonl`, `artifacts/pi_session_implement.jsonl`, or `artifacts/pi_session_test.jsonl` (or use the web UI session viewer). You can resume the stage with the corresponding `--resume` flag, or manually update the status: `abreka exp update <id> --status failed` or `abreka exp fail <id> --reason "..."`.

**Timeout during run**
Increase `run.timeout_minutes` in `config.toml`. When a run times out, ABREKA sends SIGTERM for graceful shutdown, captures any partial metrics printed so far, and marks the experiment failed. Timeouts do not trigger automatic run repair; they mean the experiment did not finish in the allotted window.

**5 consecutive failures, stopping**
The autonomous loop stops after 5 consecutive failures to prevent resource waste. Check the recent experiments with `abreka exp list --status failed --sort time` and review their failure reasons. Common causes: the goal is too ambitious for the current model, the test suite has a systemic issue, or the shared library is in a broken state.

**Index out of sync**
If `abreka exp list` shows stale data, the index rebuilds automatically when experiments are queried. You can also delete `abreka/index.json` and run any `abreka exp` command to regenerate it.

**Agent not using the right model**
Check `abreka/config.toml`. The `[agent]` section has `model` for the default and optional per-stage overrides (`propose_model`, `implement_model`, `evaluate_model`). View the current config at a glance with `abreka webui start` and navigating to `/config`.
