# ABREKA — Autonomous Research Agent

## Overview

ABREKA is a standalone CLI tool that automates the research experiment loop: **propose → implement → test → run → evaluate**. It is installed separately from the user's project (via `uv tool install abreka` or `uvx`), initialized inside any `uv`-managed Python project, and uses [pi-rpc](https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/rpc.md) as the agent backend for all LLM-driven steps.

ABREKA is inspired by [autoresearch](https://github.com/karpathy/autoresearch) but is agent-driven end-to-end and designed for long autonomous runs with full auditability.

---

## Core Principles

1. **Experiments are the unit of work.** Every action produces or updates an experiment with a hypothesis, method, metrics, and findings.
2. **Fully automated by default, human-in-the-loop optional.** `abreka run 24h` runs autonomously. `abreka run --step` does one iteration and stops. Individual commands (`abreka propose`, `abreka implement`, etc.) can be run manually.
3. **TDD enforced.** Code must pass tests before experiments run. The agent writes tests as part of implementation.
4. **Full auditability.** Every pi session transcript is saved as an artifact. Every experiment has structured metrics AND narrative findings.
5. **Experiments form a DAG, not a list.** Each experiment can reference parent experiments, forming a graph of ideas that branch and converge.

---

## Installation & Initialization

ABREKA is a standalone `uv` project/CLI, independent of the user's project:

```
uv tool install abreka
```

Then, inside any existing `uv` project:

```
cd my-research-project/
abreka init .
```

### What `abreka init .` Does

1. Reads `pyproject.toml` to discover the package name and `src/` layout
2. Creates `abreka/` directory (not hidden — this is visible project infrastructure)
3. Generates `abreka/config.toml` with discovered project info and sensible defaults
4. Creates `abreka/goal.md` with a placeholder for the user to fill in
5. Populates `abreka/prompts/` with default agent prompts (editable by the user)
6. Creates empty `abreka/experiments/` and `abreka/index.json`

---

## Project Structure

ABREKA lives alongside the user's existing `uv` project. It does not own or modify the project's `pyproject.toml`.

```
my_research_project/
  pyproject.toml              # user's uv project — abreka reads but doesn't modify
  src/
    my_lib/                   # user's package (name discovered from pyproject.toml)
      __init__.py
      data.py
      models.py
  run.py                      # default run template (convention, user-provided)
  tests/
    test_data.py
    test_models.py
  abreka/                     # created by `abreka init .`
    config.toml
    goal.md
    index.json
    prompts/
      propose_exploit.md
      propose_explore.md
      implement.md
      test.md
      evaluate.md
    experiments/
      0001/
        experiment.toml       # structured experiment metadata
        findings.md           # narrative findings (written by evaluate agent)
        run.py                # experiment entry point: `uv run abreka/experiments/0001/run.py`
        code/                 # experiment-specific modules
          augment.py
        artifacts/
          checkpoints/
            model_best.pt
          plots/
            loss_curve.png
          pi_session.json     # saved pi-rpc session transcript
      0002/
        ...
```

### Key Conventions

- **`run.py`**: Every experiment has a `run.py` entry point, executed via `uv run abreka/experiments/<id>/run.py`. The user specifies a default run template in config (`./run.py` by convention), or the agent can borrow from a previous experiment's `run.py`.
- **`src/<lib_name>/`**: The user's shared library code is importable by all experiments via the normal `uv` project layout. Since experiments run via `uv run`, they have access to `src/<lib_name>` automatically.
- **`code/`**: Experiment-specific modules live here. The agent writes new code here and imports shared code from `src/<lib_name>/`.
- **`artifacts/`**: All experiment outputs — checkpoints go in `artifacts/checkpoints/`, plots in `artifacts/plots/`, pi session transcripts as `artifacts/pi_session.json`.

---

## Configuration

### `abreka/config.toml`

```toml
[project]
name = "mnist-exploration"        # discovered from pyproject.toml
lib = "mnist_exp"                 # discovered from src/ layout
test_command = "uv run pytest"    # how to run the test suite

[experiment]
primary_metric = "val:accuracy"   # used for default sorting and "best" tracking
metric_direction = "maximize"     # or "minimize"
run_template = "./run.py"         # default template for new experiment run.py files

[run]
timeout_minutes = 30              # kill experiments that exceed this
require_tests = true              # enforce TDD — won't run without passing tests

[agent]
provider = "anthropic"
model = "claude-sonnet-4-20250514"
# Per-agent-type overrides (optional):
# propose_model = "..."
# implement_model = "..."
# evaluate_model = "..."

[agent.pi]
# pi-rpc connection settings (optional)
# extra_args = ["--thinking", "high"]
```

### `abreka/goal.md`

Free-form markdown describing the research objective. Injected into agent prompts. Example:

```markdown
# Research Goal

Achieve the highest possible test accuracy on MNIST using models that train
in under 60 seconds on a single GPU. Explore architectures, regularization,
data augmentation, and optimization strategies.

## Success Criteria
- val:accuracy > 0.995
- Training time < 60 seconds

## Constraints
- Single GPU
- No pre-trained models
```

---

## Experiment Data Model

### `experiment.toml`

```toml
id = "0003"
status = "completed"                # proposed | implementing | testing | running | completed | failed
created_at = "2026-04-08T10:00:00Z"
completed_at = "2026-04-08T10:05:00Z"

hypothesis = "Adding random rotation (±15°) and horizontal flip augmentation will improve generalization, pushing val accuracy above 0.993"
method = "Modified data pipeline from exp 0001 with torchvision.transforms; same 2-layer CNN architecture"
parents = ["0001"]                  # edges in the experiment DAG

tags = ["augmentation", "cnn"]

[metrics.train]
accuracy = 0.9985
loss = 0.005

[metrics.val]
accuracy = 0.9934
loss = 0.028

[metrics.test]
# Populated only when explicitly evaluated
# accuracy = 0.9928
# loss = 0.030

[failure]
reason = ""                         # populated only if status = "failed"
```

### Status Lifecycle

```
proposed → implementing → testing → running → completed
                                          ↘ failed
```

- **proposed**: Hypothesis and method defined, no code yet
- **implementing**: Agent is writing code via pi-rpc
- **testing**: Agent is running tests to validate the code
- **running**: `uv run abreka/experiments/<id>/run.py` is executing
- **completed**: Run finished, metrics and findings recorded
- **failed**: Something went wrong; `failure.reason` explains what

### Metrics

Metrics are namespaced by split: `train`, `val`, `test`.

- **Default split is `val`** when omitted (in stdout parsing, CLI args, and sorting)
- `test` split should be used sparingly — the point is to not overfit to the test set
- The `primary_metric` in config (e.g., `val:accuracy`) determines default sorting and "best experiment" tracking

### Experiment DAG

Experiments reference parents via the `parents` field, forming a directed acyclic graph. This captures the lineage of ideas:

- Experiment 0001: Vanilla CNN baseline (no parents)
- Experiment 0002: Add dropout (parent: 0001)
- Experiment 0003: Add augmentation (parent: 0001)
- Experiment 0004: Combine dropout + augmentation (parents: 0002, 0003)

The DAG is queryable via `abreka exp tree` and `abreka exp list --parent <id>`.

---

## Metrics Protocol

### Stdout Convention

Experiments print metrics to stdout during training. ABREKA parses lines matching:

```
METRIC:<split>:<key>=<value>
METRIC:<key>=<value>              # split defaults to "val"
```

Examples:

```
METRIC:train:loss=0.005
METRIC:train:accuracy=0.9985
METRIC:val:loss=0.028
METRIC:val:accuracy=0.9934
```

ABREKA captures all stdout, extracts `METRIC:` lines, and writes them into `experiment.toml`. Non-metric stdout is preserved in the run log.

### CLI Convention

Metrics can also be set explicitly:

```bash
abreka exp update 0001 --metric val:accuracy=0.9912
abreka exp update 0001 --metric train:loss=0.005
abreka exp update 0001 --metric accuracy=0.85      # defaults to val:accuracy
```

### Querying by Metrics

```bash
abreka exp list --sort metric:val:accuracy
abreka exp list --where "metric:val:accuracy > 0.95"
abreka exp list --where "metric:train:loss < 0.01" --sort metric:val:accuracy
```

---

## Shared Library & Copy-on-Write Rule

The user's package in `src/<lib_name>/` is shared across all experiments. The agent may modify it subject to these rules:

1. **Adding new code**: The agent can add new modules or functions to `src/<lib_name>/`. This is encouraged — it grows the shared library over time.
2. **Modifying existing code**: The agent can modify existing code in `src/<lib_name>/` **if and only if all existing tests still pass**. The test suite is the arbiter.
3. **If modifications break tests**: The agent must copy the code into the experiment's `code/` directory and modify the copy instead (copy-on-write). This keeps old experiments reproducible.
4. **The test suite enforces this automatically**: `abreka test <id>` runs the full test suite. If it fails, the experiment cannot proceed to `running`.

---

## Agent Prompts

### `abreka/prompts/`

Each agent-driven command uses a corresponding prompt template from `abreka/prompts/`. Defaults are generated by `abreka init` but are fully editable by the user.

| File | Used By | Purpose |
|------|---------|---------|
| `propose_exploit.md` | `abreka propose`, `abreka run` | Analyze experiment history, propose the next experiment (exploitation) |
| `propose_explore.md` | `abreka run --explore` | Propose a genuinely novel direction (exploration) |
| `implement.md` | `abreka implement <id>` | Write code for a proposed experiment |
| `test.md` | `abreka test <id>` | Validate experiment code, write/run tests |
| `evaluate.md` | `abreka evaluate <id>` | Interpret results, write findings, extract metrics |

### Template Variables

Prompts support `{{variable}}` placeholders that ABREKA fills at runtime:

| Variable | Description |
|----------|-------------|
| `{{lib_name}}` | The user's package name (from config) |
| `{{exp_id}}` | Current experiment ID |
| `{{hypothesis}}` | Current experiment's hypothesis |
| `{{method}}` | Current experiment's method |
| `{{parents}}` | Parent experiment IDs |
| `{{experiment_context}}` | Summary of prior experiments (for propose/implement) |
| `{{artifacts_list}}` | List of artifact files (for evaluate) |
| `{{run_output}}` | Captured stdout/stderr from the run (for evaluate) |
| `{{goal}}` | Contents of `abreka/goal.md` |
| `{{primary_metric}}` | Primary metric from config |
| `{{metric_direction}}` | "maximize" or "minimize" |

Users can restructure prompts freely, add domain-specific instructions, or change the tone. If a template variable is missing, ABREKA warns but proceeds.

---

## Pi-RPC Integration

### Architecture

```
abreka CLI (uv + Python)
  └── propose / implement / test / evaluate commands
        └── pi-rpc client (spawns `pi --mode rpc` subprocess)
              └── pi agent uses tools: bash, read, write, edit
                    └── agent can call `abreka exp ...` commands as tools
```

ABREKA communicates with pi via the RPC protocol: JSON commands over stdin, JSON events over stdout, using strict LF-delimited JSONL framing.

### Pi-RPC Client

ABREKA spawns pi as a subprocess:

```python
proc = subprocess.Popen(
    ["pi", "--mode", "rpc", "--no-session", "--provider", provider, "--model", model],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    text=True,
    cwd=project_root,
)
```

Commands are sent as JSON lines:

```json
{"type": "prompt", "message": "..."}
```

Events are read from stdout:

```json
{"type": "message_update", "assistantMessageEvent": {"type": "text_delta", "delta": "..."}}
{"type": "agent_end"}
```

### Session Saving

When a pi session completes (for any agent-driven command), the full event stream is serialized and saved to `abreka/experiments/<id>/artifacts/pi_session.json`. This provides:

- Full auditability of what the agent did
- Tool call history (what files it read/wrote, what commands it ran)
- Potential context for future agents (e.g., the propose agent could review past sessions)

For persistent pi sessions (where the agent may need multiple turns), ABREKA uses `--session <path>` pointing to a file in the experiment's artifacts directory.

---

## CLI Reference

### Project Commands

```bash
abreka init .                          # Initialize ABREKA in current directory
abreka goal                            # View the research goal (abreka/goal.md)
abreka status                          # Project summary: experiment counts, best result
```

### Experiment Lifecycle

```bash
abreka exp new [--hypothesis "..."] [--parent <id>]
                                    # Create new experiment, set up directory, return ID

abreka exp update <id> [OPTIONS]       # Update experiment metadata
    --hypothesis "..."              #   Set hypothesis text
    --method "..."                  #   Set method text
    --status <status>               #   Set status manually
    --parent <id>                   #   Set parent IDs (replaces existing)
    --metric <spec>                 #   Set metric: [split:]key=value
    --tag <tag>                     #   Set tags (replaces existing)

abreka exp finish <id>                 # Mark completed, set completed_at timestamp
abreka exp fail <id> --reason "..."    # Mark failed with explanation
```

### Experiment Querying

```bash
abreka exp list [OPTIONS]              # List experiments with filtering
    --status <status>               #   Filter by status
    --parent <id>                   #   Filter by parent experiment
    --sort <spec>                   #   Sort by: time | metric:<split>:<key>
    --where <clause>                #   Filter: metric:<split>:<key> <op> <value>

abreka exp show <id>                   # Full detail view of one experiment
abreka exp search <query>              # Keyword search over hypotheses and findings
abreka exp diff <id1> <id2>            # Side-by-side comparison of two experiments
abreka exp tree                        # Show the experiment DAG
```

### Agent-Driven Commands

These commands invoke pi-rpc with the corresponding prompt from `abreka/prompts/`:

```bash
abreka propose                         # Agent analyzes history, proposes next experiment
                                    # Creates a new experiment in "proposed" status

abreka implement <id>                  # Agent writes code for a proposed experiment
                                    # Transitions status: proposed → implementing → implementing done

abreka test <id>                       # Agent validates code, runs tests
                                    # Transitions status: implementing → testing → testing done
                                    # Blocks run if tests fail

abreka evaluate <id>                   # Agent reads outputs, writes findings + metrics
                                    # Transitions status: running → completed (or failed)
```

### Orchestration

```bash
abreka run [duration]                  # Autonomous loop for the given duration
                                    # Example: abreka run 24h, abreka run 2h, abreka run 30m

abreka run --step                      # Single iteration: propose → implement → test → run → evaluate
                                    # Then stop (human-in-the-loop mode)
```

---

## The Autonomous Loop (`abreka run`)

### Loop Structure

```
while time_remaining > 0:
    1. abreka propose
       → Agent reviews goal.md and all prior experiments
       → Creates new experiment in "proposed" status
       → Writes hypothesis, method, parent links

    2. abreka implement <id>
       → Agent writes run.py, code/, and modifies src/<lib> if safe
       → Writes tests for new code

    3. abreka test <id>
       → Runs full test suite (uv run pytest)
       → If tests fail: agent attempts to fix
       → If still failing after retries: abreka exp fail, continue loop

    4. uv run abreka/experiments/<id>/run.py
       → Executed with timeout from config
       → Stdout captured, METRIC: lines parsed
       → If crash/timeout: abreka exp fail, continue loop

    5. abreka evaluate <id>
       → Agent reads artifacts and stdout
       → Writes structured metrics via abreka exp update
       → Writes narrative findings in findings.md
       → Marks experiment completed

    6. Save pi session transcript to artifacts/pi_session.json
```

### Error Recovery

- If any step fails, the experiment is marked `failed` with a reason, and the loop continues with a new proposal.
- If pi-rpc crashes, ABREKA restarts the subprocess and retries.
- If the same failure pattern repeats 3+ times, ABREKA logs a warning and adjusts its proposal context to include the failure pattern.

### Duration Tracking

`abreka run 24h` tracks wall-clock time. Individual experiment timeouts are separate (from `config.toml`). The loop stops when total elapsed time exceeds the requested duration, finishing the current experiment first.

---

## Experiment Queries — Detail

### `abreka exp list` Examples

```bash
# All experiments, default sort (by ID)
abreka exp list

# Only completed experiments, sorted by validation accuracy (descending)
abreka exp list --status completed --sort metric:val:accuracy

# Experiments with val accuracy above 0.95
abreka exp list --where "metric:val:accuracy > 0.95"

# Children of experiment 0003
abreka exp list --parent 0003

# Failed experiments, sorted by time (most recent first)
abreka exp list --status failed --sort time
```

### `abreka exp search` Examples

```bash
# Keyword search across hypotheses, methods, and findings
abreka exp search "augmentation"
abreka exp search "ResNet dropout"
abreka exp search "learning rate schedule"
```

Search is keyword-based (with potential for embedding-based search in the future). It searches across `hypothesis`, `method`, and `findings.md` content.

### `abreka exp diff` Example

```
┌──────────────────────────────────────────────────────────┐
│ Diff: 0003 vs 0005                                       │
├──────────────┬──────────────────────┬────────────────────┤
│ Field        │ Exp 0003             │ Exp 0005           │
├──────────────┼──────────────────────┼────────────────────┤
│ Status       │ completed            │ completed          │
│ Hypothesis   │ Random augmentation  │ CutMix augmentation│
│ Method       │ Random rotate ±15°   │ CutMix α=1.0       │
│ Parents      │ 0001                 │ 0003               │
│ val:accuracy │ 0.9934 (red)         │ 0.9961 (green)     │
│ val:loss     │ 0.0280 (red)         │ 0.0190 (green)     │
│ train:loss   │ 0.0050               │ 0.0043             │
└──────────────┴──────────────────────┴────────────────────┘
```

### `abreka exp tree` Example

```
Experiments
├── 0001 completed Vanilla 2-layer CNN baseline (val:accuracy=0.9912)
│   ├── 0002 completed Add dropout p=0.5 (val:accuracy=0.9923)
│   │   └── 0005 completed Dropout + CutMix (val:accuracy=0.9961)
│   ├── 0003 completed Random augmentation ±15° (val:accuracy=0.9934)
│   │   └── 0005 completed Dropout + CutMix (val:accuracy=0.9961)
│   └── 0004 failed Replace with ResNet-18
└── 0006 running MLP baseline for comparison
```

---

## Index File

`abreka/index.json` is a lightweight cache of all experiment metadata for fast listing and search without reading individual `experiment.toml` files. It is updated on every `abreka exp` mutation. Structure:

```json
{
  "0001": { "id": "0001", "status": "completed", "hypothesis": "...", "metrics": {...}, ... },
  "0002": { "id": "0002", "status": "proposed", ... }
}
```

If the index becomes out of sync, `abreka exp list` can rebuild it by scanning `abreka/experiments/*/experiment.toml`.

---

## Future Considerations

These are not part of the initial implementation but are worth keeping in mind:

- **Embedding-based search**: Replace keyword search in `abreka exp search` with vector embeddings over hypotheses and findings for semantic similarity.
- **Parallel experiments**: Run multiple experiments concurrently when they don't share dependencies.
- **Cost tracking**: Track pi-rpc token usage and compute costs per experiment.
- **Web dashboard**: A local web UI for browsing the experiment DAG, viewing findings, and comparing metrics.
- **Git integration**: Optionally commit each experiment's code changes to a branch for precise reproducibility.
- **Experiment templates**: Pre-built experiment structures for common patterns (hyperparameter sweep, ablation study, architecture search).
