# Evaluate Experiment {{exp_id}}

## CLI Reference

Run all abreka commands with `uv run abreka <command>`. Examples:
- `uv run abreka exp update {{exp_id}} --metric val:accuracy=0.99`
- `uv run abreka exp finish {{exp_id}}`
- `uv run abreka exp fail {{exp_id}} --reason "..."`

**Project Wiki** (durable cross-experiment knowledge — see "Wiki Maintenance" section below):
- `uv run abreka wiki list` / `uv run abreka wiki show <slug>` — browse and read pages
- `uv run abreka wiki search "<query>"` — keyword/regex search
- `uv run abreka wiki add --title "..." --tag <tag> --content "..." --exp {{exp_id}}` — create a new page
- `uv run abreka wiki append <slug> --content "..." --exp {{exp_id}}` — add a dated note under the page's `## History` section
- `uv run abreka wiki edit <slug> --summary "..." --content "..." --exp {{exp_id}}` — rewrite a page (use to consolidate)
- `uv run abreka wiki rename <old> <new> --exp {{exp_id}}` — rename a slug for clarity
- `uv run abreka wiki merge <src> <dst> --exp {{exp_id}}` — fold a duplicate page into another

You may NOT call `wiki delete` — that is a user-only operation.

{{wiki_index}}

## Research Goal

{{goal}}

## Experiment Details

- **ID**: {{exp_id}}
- **Hypothesis**: {{hypothesis}}
- **Method**: {{method}}
- **Primary metric**: {{primary_metric}} ({{metric_direction}})

## Run Output

```
{{run_output}}
```

## Artifacts

{{artifacts_list}}

## Parent Comparison

{{parent_context}}

## Branch Summary

{{branch_summary}}

## Global Position

{{global_context}}

## Working Directory

Your working directory is the **project root**. All paths are relative to it. Do NOT use `..` to navigate to a parent directory.

## Your Task

Evaluate experiment {{exp_id}} as **both**:
1. a run result, and
2. a branch update in the overall experiment graph.

You must:

1. **Extract metrics** from the run output above and set them:
   ```bash
   uv run abreka exp update {{exp_id}} --metric train:loss=<value> --metric val:accuracy=<value>
   ```

2. **Write findings** to `abreka/experiments/{{exp_id}}/findings.md`. Your findings file must explicitly include these sections:

   ```markdown
   # Findings

   ## Result Summary
   What happened in the run. Keep it factual.

   ## Hypothesis Assessment
   Was the hypothesis supported, refuted, or inconclusive? Why?

   ## Comparison to Parent(s)
   Better / worse / neutral vs each parent on the primary metric, with magnitudes.

   ## Branch Assessment
   Is this a new branch best? Does the branch now look like it's improving,
   plateauing, confounded, or losing steam?

   ## Global Position
   How does this result compare to the global best? Is the branch competitive?

   ## Recommended Next Step
   One of: continue (exploit), discriminate (isolate a confound), validate
   (reproduce / robustness check), pause, or stop. Be specific about *what*
   the next step should be.

   ## Confidence
   high / medium / low — and why.
   ```

3. **Mark the experiment as completed**:
   ```bash
   uv run abreka exp finish {{exp_id}}
   ```

4. **Log backlog ideas** (optional): Add backlog items especially when:
   - the branch should pause but a future revisit is justified,
   - a discriminative follow-up is identified but not highest priority now,
   - a broader exploration idea was surfaced by the result.

   ```bash
   uv run abreka backlog add --title "Short idea description" --description "Why this is worth trying" --source-exp {{exp_id}} --tag <tag>
   ```

   Only add ideas that are genuinely concrete and justified by the evidence. Do not add vague or low-confidence ideas.

5. **Wiki Maintenance** (after findings — this is the durable knowledge layer for *future* experiments):

   The wiki holds non-obvious facts that should outlive this run: data quirks, methodology lessons, surprising failure modes, distilled EDA, domain knowledge. Single-experiment results and per-run logs belong in `findings.md`, NOT the wiki.

   On every evaluate, do *each* of these to the extent the evidence justifies it:

   - **Add new pages** for genuinely new durable knowledge:
     ```bash
     uv run abreka wiki add --title "..." --summary "..." --tag <tag> --content "$(cat <<'EOF'
     # Title
     ...body...
     EOF
     )" --exp {{exp_id}}
     ```
     If this experiment generated EDA artifacts under `abreka/experiments/{{exp_id}}/artifacts/eda/`, distill durable findings from them into wiki pages. Do not copy raw EDA wholesale — promote only what will inform future experiments.

   - **Append `## History` notes** to existing pages whose claims this experiment confirmed, contradicted, or refined:
     ```bash
     uv run abreka wiki append <slug> --content "what this run adds" --exp {{exp_id}}
     ```

   - **Consolidate** ONE existing page if you notice redundancy, drift, or stale claims (rewrite the body to be tighter):
     ```bash
     uv run abreka wiki edit <slug> --content "..." --exp {{exp_id}}
     ```
     Use `wiki rename` for clarity and `wiki merge` when two pages cover overlapping ground. Do at most one consolidation per evaluate — keep the change small and clearly justified.

   Be conservative. The wiki is a curated reference, not a journal. Trivia, single-run anomalies, and uncertain claims do NOT belong here.

If the run failed or produced no useful results, mark it as failed:
```bash
uv run abreka exp fail {{exp_id}} --reason "Explanation of what went wrong"
```

Do not search for metrics with grep/rg. The run output is provided above — just read the METRIC: lines from it.
