# Drift Report — Tranche J (abi-evals)

## How abi-evals Currently Runs

The eval harness follows a three-stage pipeline:

1. **Load** — `runner.load_eval_suite(path)` reads an `eval-suite/1.0` JSON file
   containing a `suite_id` and a non-empty `cases[]` array.

2. **Execute** — `runner.run_suite(suite, evidence_pack, run_id, timestamp)`:
   - Parses cases into `EvalCase` dataclasses.
   - Sorts by `case_id` for deterministic ordering.
   - Resolves each case's actual value from `case.input` or from the
     evidence pack via `case.config["evidence_key"]` (dot-notation traversal).
   - Dispatches to one of seven comparators (`exact`, `golden_exact`,
     `golden_structural`, `numeric_tolerance`, `regex_assert`,
     `presence_assert`, `schema_validate`).
   - Aggregates per-case `EvalResult` objects into an `eval-result/1.0` dict
     with a `summary` block (total / passed / failed / errors / skipped).

3. **Persist** — `runner.write_results(results, path)` writes deterministic
   JSON (`indent=2, sort_keys=True`) with a trailing newline.

CLI entry point: `python -m abi_evals run --eval-suite <path> [--evidence-pack <path>] [--out <path>]`

## Where Eval Artifacts Are Stored

| Artifact              | Location                                   |
|-----------------------|--------------------------------------------|
| Eval suite (input)    | Any path; passed via `--eval-suite`        |
| Evidence pack (input) | Any dir or JSON file; `--evidence-pack`    |
| Eval results (output) | Written to `--out <path>` or stdout        |
| Golden baselines      | `<repo>/goldens/v<N>/manifest.json`        |

The eval harness does **not** create run directories; the orchestrator owns
directory layout and passes pre-built paths to the harness.

Transient/scratch directories (`runs/`, `.tmp/`, `artifacts/`) are gitignored
per `docs/HYGIENE_POLICY.md`.

## How It Locates Run / Evidence-Pack Directories

The CLI resolves `--evidence-pack <path>` as follows:

- **Directory**: looks for `<path>/manifest.json`; loads it as the evidence
  dict. Falls back to `{"dir": str(path)}` if no manifest exists.
- **File**: loads JSON directly as the evidence dict.

Evidence keys inside eval cases use dot-notation (`artifacts.greeting`) to
traverse into the evidence-pack dict.

There is no automatic discovery — every path is explicitly provided by the
caller.

## Dependency Profile (pre-J3)

- **Runtime**: zero external dependencies (pure stdlib).
- **Dev**: `pytest>=8.0`, `ruff>=0.8` (uv dependency-groups).
- **Build**: hatchling.

## Gap Identified — Promotion Readiness

No mechanism currently exists to:
- Detect a `promotion_candidate.json` artifact in evidence output.
- Invoke `abi-promotion verify-candidate` as a post-eval step.
- Store the resulting `promotion_receipt.json` alongside other eval artifacts.

Tranche J3 adds an **optional, report-only** promotion-readiness evaluation
that fills this gap via subprocess invocation (no hard import of
`abi_promotion`).
