# Whisper & Recall Eval System Design

**Date:** 2026-03-22
**Status:** Approved
**Context:** Whisper and recall precision are ormah's make-or-break feature. Without a rigorous eval system guiding development, improvements are guesswork and regressions go undetected.

---

## Goals

- Measure whisper injection precision and recall against a known corpus of memories and prompts
- Catch regressions automatically in CI
- Support eval-driven development: every pipeline change is validated against metrics before merging
- Grow ground truth over time via real session replays and affinity signal (Phase 2)

**Primary metric priority:** False negative rate (missed relevant memories) is the existential metric. Precision, F1, MRR, and injection rate are all tracked equally after that.

---

## Architecture: Standalone `eval/` Module with CLI

The eval system lives in `eval/` at repo root — a self-contained package independent of `src/ormah/`. It uses the same `MemoryEngine` code as production but points to a dedicated isolated SQLite database (`eval/eval_db/`), never touching the user's real memory store.

The eval DB is rebuilt fresh for each eval case at runtime (see Seeding Strategy). It is gitignored. The corpus files are version-controlled.

---

## Directory Layout

```
eval/
├── corpus/
│   ├── golden/
│   │   └── golden.jsonl          # Hand-crafted cases — version-controlled, ~50-100 cases
│   ├── synthetic/
│   │   └── synthetic.jsonl       # LLM-generated corpus — version-controlled, regenerable
│   └── sessions/
│       └── *.jsonl               # Real session replays — gitignored, user-captured
│
├── results/
│   ├── latest.json               # Latest run results (overwritten each run)
│   └── history.jsonl             # Appended after each run — metric trend history
│
├── eval_db/                      # Isolated SQLite eval database (gitignored)
│
├── runner.py                     # Core: loads corpus, runs pipeline, computes metrics
├── corpus_builder.py             # Seeds eval_db per-case; LLM synthetic generation
├── judge.py                      # Export/import labeling workflow for Claude Code judge
├── metrics.py                    # Precision@k, Recall@k, F1, MRR, injection rate
├── report.py                     # Formats stdout report and writes JSON results
└── cli.py                        # CLI entry points wired into `ormah eval *`
```

---

## Corpus & Ground Truth Format

Each entry in `golden.jsonl` and `synthetic.jsonl` is a self-contained evaluation case:

```json
{
  "id": "golden-001",
  "memories": [
    {
      "node_id": "golden-001-mem-0",
      "title": "Ormah uses SQLite with FTS5",
      "content": "The index layer uses SQLite with FTS5 for full-text search and sqlite-vec for vector similarity.",
      "type": "fact",
      "tier": "working",
      "tags": ["architecture", "search"],
      "space": "ormah"
    }
  ],
  "prompts": [
    {
      "text": "how does the search layer work in ormah?",
      "expected": {
        "should_inject": ["golden-001-mem-0"],
        "should_not_inject": []
      },
      "notes": "Direct factual match — should always surface"
    }
  ]
}
```

### Node IDs

Each memory object carries an explicit stable `node_id` (e.g. `"golden-001-mem-0"`). The corpus builder forces this ID into the eval DB at seed time, overriding the engine's auto-generated UUID. This makes `should_inject` lists stable across runs and machines. Node IDs must be unique within a corpus file.

### Corpus Tiers

- **Golden cases** are small and hand-verified: ~5–15 memories per case, 1–3 prompts each. They are the regression canary. Maintained manually.
- **Synthetic cases** are larger batches: 50–200 memories, many prompts. Generated by LLM once, version-controlled. The file includes a header object (first line) with `generated_at` and `generator_version` fields for audit. Regenerated manually when the pipeline changes significantly.
- **Session replay cases** have no pre-labeled expected nodes. They go through the labeling workflow before contributing to eval.

---

## Seeding Strategy & Eval DB Lifecycle

`ormah eval run` always rebuilds the eval DB as its first step — no separate `build-corpus` call required in normal usage. (`ormah eval build-corpus` exists as a standalone convenience command for inspection and debugging only.)

**Per-case isolation:** The eval DB is cleared and reseeded for each individual case. Only that case's memories are present in the DB when its prompts are run. This is the only design that makes precision@k and recall@k meaningful — without it, memories from case A contaminate the retrieval results for case B.

Seeding sequence per case:
1. Drop all nodes from eval DB
2. Insert each memory from the case, forcing the corpus `node_id` into the DB
3. Run whisper pipeline for each prompt in the case
4. Record which node IDs were injected, filtered, or missed
5. Compute per-case metrics
6. Clear DB and move to next case

---

## Runner & Metrics

The runner instantiates `MemoryEngine` pointed at the eval DB (no HTTP, no CLI overhead) and for each case executes the seeding + pipeline sequence above.

### Metrics

| Metric | Definition |
|---|---|
| `recall@k` | Of `should_inject` nodes, fraction that appeared in top-k results |
| `precision@k` | Of top-k injected, fraction in `should_inject` |
| `F1@k` | Harmonic mean of recall@k and precision@k |
| `MRR` | Mean Reciprocal Rank of first relevant result |
| `injection_rate` | Fraction of prompts where whisper fired at all |
| `false_negative_rate` | Fraction of `should_inject` nodes completely missed |

Default `k=8` (matching the `max_nodes=8` whisper injection cap). `k` is configurable via `--k` flag. All metrics are reported per-case and aggregated. Golden and synthetic corpora are reported separately so regressions in the canary set are immediately visible.

### Degenerate Cases

- **Missing corpus file:** `ormah eval run --corpus golden` when `golden.jsonl` does not exist → error and exit with a clear message.
- **Empty corpus:** corpus file exists but has zero cases → warning, no metrics computed, no threshold flags applied.
- **All-unlabeled cases:** all cases in scope have empty `should_inject` / `should_not_inject` → report skips precision/recall/F1, logs a warning ("No labeled cases — run `ormah eval export-for-labeling`"), threshold flags are not applied.
- **First run with no `history.jsonl`:** regression comparison is skipped. The run is recorded as the baseline. `--fail-on-regression` is a no-op on first run.

---

## Ground Truth: Labeling Workflow

Ground truth comes from corpus-file labels. In the current scope, the eval DB never reads from the production memory store or affinity table — this ensures eval results are fully reproducible across machines and fresh environments.

### Claude Code as Judge

When corpus cases have no labels yet (new session replays, new synthetic cases), export them for labeling:

```
ormah eval export-for-labeling
```

This writes `eval/corpus/pending_labels.jsonl` where each line is:

```json
{
  "case_id": "session-003",
  "prompt_idx": 0,
  "memory_id": "session-003-mem-2",
  "prompt_text": "how do we handle auth tokens?",
  "memory_title": "JWT token storage",
  "memory_content": "Auth tokens are stored in httpOnly cookies with 24h expiry..."
}
```

`memory_id` is the stable round-trip key used by `import-labels` to write scores back into the corpus. All other fields are for the human judge's context only.

The user then asks Claude Code to process `pending_labels.jsonl` — Claude reads the file, scores each pair, and writes `eval/corpus/labels.jsonl`:

```json
{"case_id": "session-003", "prompt_idx": 0, "memory_id": "session-003-mem-2", "score": 2, "reason": "Directly addresses the prompt — auth token storage is clearly relevant"}
```

Scoring scale:
- **2** = clearly relevant → `should_inject`
- **1** = borderline → excluded from precision/recall computation (ambiguous)
- **0** = not relevant → `should_not_inject`

Labels are imported back and persisted into the corpus files:

```
ormah eval import-labels
```

Labels survive across runs. Claude Code is only invoked for new unlabeled cases, not re-scored every time.

### Affinity Table (Phase 2 — Out of Scope)

Once the adaptive feedback loop (Phase 1) is live and has accumulated sufficient signal, production affinity data (`signal=+1/-1`) can be manually exported and added to session replay corpus labels via `import-labels`. The eval DB itself never reads directly from the production database — this boundary is intentional and must not be crossed, as it would make eval results non-reproducible.

---

## Capture & Labeling: Session Replay Flow

```
ormah eval capture-session <transcript_path>
```

Copies the file at `<transcript_path>` (a Claude Code session `.jsonl` transcript) into `eval/corpus/sessions/`. It then parses the transcript to extract user prompt turns and creates an unlabeled corpus entry: each prompt is added to the case's `prompts` list with empty `expected.should_inject` and `expected.should_not_inject` arrays. No LLM is called.

**Session replay cases have no embedded memories.** They do not carry their own memory snapshot. When the runner evaluates session replay cases, the eval DB is seeded with the full golden + synthetic corpus (all memories from both tiers). This tests whether real-world prompts would correctly surface the right memories from the eval corpus. The labeling workflow then exports each `(session_prompt, corpus_memory)` pair — Claude judges which corpus memories should have been injected for each real session prompt.

The session becomes available for labeling via `export-for-labeling`.

Expected input: a Claude Code session JSONL file where each line is a message object with `"type": "user"` or `"type": "assistant"` and a `"message"` field.

Full workflow:
```
ormah eval capture-session ~/.claude/projects/foo/session.jsonl
ormah eval export-for-labeling     # exports pending (prompt, memory) pairs
# ask Claude Code to process pending_labels.jsonl and write labels.jsonl
ormah eval import-labels           # merges labels into corpus
ormah eval run                     # sessions now included in eval
```

---

## Synthetic Corpus Generation

```
ormah eval generate-synthetic
```

Calls the configured LLM to generate new synthetic cases and rewrites `synthetic.jsonl` entirely (preserving existing cases and appending new ones). The first line is always a header object (not an eval case):

```json
{"_header": true, "generated_at": "2026-03-22T14:30:00Z", "generator_version": "0.3.6"}
```

The header is updated on each regeneration so regressions can be correlated with regeneration events. Because `generate-synthetic` rewrites the whole file, the header is always current and no in-place line editing is needed. The runner skips lines where `_header: true`.

---

## Regression Detection

Every run appends to `results/history.jsonl` (timestamp + all metric values). The runner compares current metrics against the previous run and flags changes in the report. Changes below 0.01 are suppressed as noise and shown as `→`.

### Report Output

```
═══ Ormah Eval Report ══════════════════════
Corpus: golden (42 cases) | 2026-03-22 14:30

  Recall@8        0.81  ████████░░  ▼ -0.06 vs last run  ← regression
  Precision@8     0.74  ███████░░░  ▲ +0.02
  F1@8            0.77  ████████░░  ▼ -0.03
  MRR             0.83  ████████░░  →
  Injection rate  0.91  █████████░  →
  False neg rate  0.19  ██░░░░░░░░  ▲ +0.06 (worse)

Worst cases:
  golden-007  recall=0.00  "how do we handle auth tokens?"
  golden-023  recall=0.50  "what's the DB schema for nodes?"
```

Results are also written to `results/latest.json` (machine-readable) and appended to `results/history.jsonl`.

### CI Gate Modes

Two complementary failure modes, combinable:

```bash
# Absolute floor — fail if metric drops below threshold
ormah eval run --corpus golden --fail-below recall@8=0.70,precision@8=0.60

# Regression gate — fail if any metric drops more than N vs previous run
ormah eval run --corpus golden --fail-on-regression delta=0.05
```

Only the golden corpus runs in CI — it is fast (no LLM calls, deterministic) and acts as the regression canary. Full corpus (synthetic + sessions) is run manually for deeper analysis.

---

## CLI Commands

```
ormah eval build-corpus                           # Seed eval DB (debugging/inspection only)
ormah eval run                                    # Run full eval suite, print report, write results/
ormah eval run --corpus golden                    # Run only golden set
ormah eval run --k 5                              # Override default k (default: 8)
ormah eval run --fail-below recall@8=0.70         # Absolute floor (CI mode)
ormah eval run --fail-on-regression delta=0.05    # Regression gate
ormah eval export-for-labeling                    # Export unlabeled pairs to pending_labels.jsonl
ormah eval import-labels                          # Merge labels.jsonl into corpus ground truth
ormah eval capture-session <path>                 # Copy session transcript into corpus/sessions/
ormah eval generate-synthetic                     # LLM-generate new synthetic cases
```

---

## CI Integration

```yaml
- name: Eval regression gate
  run: ormah eval run --corpus golden --fail-below recall@8=0.70 --fail-on-regression delta=0.05
```

The eval suite must pass before any whisper/recall pipeline change is merged.

---

## Out of Scope

- Evaluating `get_context` (agent-driven) separately from whisper — whisper is the priority
- Real-time eval during whisper execution
- Automated synthetic corpus regeneration on every CI run (regenerate manually when pipeline changes significantly)
- Cross-encoder fine-tuning pipeline (Phase 2 — after affinity data matures)
- Direct production DB reads from eval runner (eval must remain reproducible and isolated)
