# SDK Eval Harness: Crawl Stage

## TL;DR

We are adding an **offline evaluation harness** to the `amplitude-ai` Python SDK. It lets customers (and us) simulate multi-turn conversations against their AI agents, automatically evaluate quality, identify what's broken, and iterate until the agent is good enough to ship -- all from a single CLI command.

The primary loop is **agent improvement through error analysis**: run evals, see what failed, read the transcript, understand why, fix the agent, re-run. The harness also optionally uploads results to Agent Analytics for team visibility and long-term tracking.

The Crawl stage is a **thin end-to-end vertical slice**: the smallest thing that delivers this loop. It is not feature-complete; it is designed to prove the architecture and deliver immediate value while being trivially extensible for Walk and Run stages.

```
amplitude-ai-eval run spec.yaml --verbose
```

---

## Problem

Customers building AI agents have no structured way to test quality before deploying. They manually chat with their agent, eyeball the output, and hope for the best. When something breaks in production, they find out from users, not from their CI pipeline.

Anthropic's "[Swiss Cheese Model of Quality](https://www.anthropic.com/research/building-effective-agents)" identifies six layers that must all be present for reliable AI systems. Amplitude's Agent Analytics already covers several (production monitoring, user feedback, session browsing). But two critical layers are missing for the **pre-deploy development loop**:

1. **Automated evals** -- systematic, repeatable quality checks that run before code ships
2. **Simulated test sessions** -- generating realistic conversations to evaluate, without relying on live users

The eval harness fills both gaps.

---

## What Does It Do?

The eval harness runs a pipeline:

**Simulate** -- An LLM-powered "user persona" has a multi-turn conversation with the customer's agent, following a scripted goal and persona description.

**Evaluate** -- The conversation is scored by evaluators: LLM judges that assess task completion, safety, outcome, and other quality dimensions. These use the **exact same evaluator system** (types, prompt generation, config schema) that runs server-side in Agent Analytics production enrichment.

**Gate** -- Evaluator scores are checked against thresholds. The CLI exits with code 0 (pass) or 1 (fail), making it trivially integratable into CI/CD pipelines.

**Inspect** -- `--verbose` mode prints the full conversation transcript alongside every evaluator's prompt, result, and latency. This is where the real value lives: reading the transcripts, understanding *why* the agent failed, and identifying the root cause to fix. (Anthropic's Step 6: "You won't know if your graders are working well unless you read the transcripts.")

**Deliver** (optional) -- Simulated sessions can be written as JSONL files compatible with Amplitude's conversation upload API. Once uploaded, sessions are visible in the AA Session Browser for team-wide visibility and long-term tracking.

---

## Why This Matters

### The Core Loop: Agent Improvement Through Error Analysis

The primary value of the eval harness is a tight development loop:

```
  ┌──────────────────────────────────────────────┐
  │                                              │
  │   Write / update agent                       │
  │        │                                     │
  │        ▼                                     │
  │   amplitude-ai-eval run spec.yaml --verbose  │
  │        │                                     │
  │        ├──► Which scenarios failed?           │
  │        ├──► What did the agent say wrong?     │
  │        └──► Root cause? (prompt / retrieval   │
  │             / tool use / logic)               │
  │        │                                     │
  │        ▼                                     │
  │   Fix the agent ─────────────────────────────┘
  │
  │   All thresholds pass?
  │        │
  │        ▼
  │   Ship with confidence (CI gate: exit 0)
  └──────────────────────────────────────────────
```

This is what customers do day-to-day: run evals, read the transcripts, find the failures, fix their agent, re-run until quality is sufficient. The harness makes this systematic and repeatable instead of ad-hoc.

### Swiss Cheese Coverage

Beyond the core loop, the harness connects to Amplitude's broader quality infrastructure because it uses the same data formats and evaluator system as Agent Analytics:

| Quality Layer | What Exists Today | What Crawl Adds |
|---|---|---|
| **Automated evals** | Nothing client-side | Simulate + evaluate + threshold gate via CLI. The core loop above. |
| **Transcript review** | AA Session Browser | `--verbose` mode for local transcript + evaluator reasoning inspection. This is where error analysis happens. |
| **Production monitoring** | Server-side enrichment pipeline | SDK uses identical evaluator types and prompt generation, ensuring parity between offline and online evaluation |
| **User feedback** | `score()` / `trackScore()` in SDK | Already exists, no changes needed |
| **Team visibility** (optional) | AA Session Browser + Dashboards | Upload mode writes JSONL to Agent Analytics so the full team can see eval results over time |
| **Evaluator calibration** (optional, advanced) | Gold datasets (AA-150187) | Uploaded sessions can optionally be added to gold datasets for human annotation and evaluator calibration (machine vs human agreement). This is a separate, advanced workflow -- not the primary use case. |
| **Version comparison** | Nothing | Run stage adds a `compare` command to diff report JSONs across agent versions. Not in Crawl. |
| **A/B testing (online)** | Amplitude Experiments | Already exists for live traffic via NOVA events. The eval harness runs offline and does not interact with Experiments. |

---

## Architecture

```
                        ┌─────────────────────────────┐
                        │      Eval Spec (YAML)       │
                        │  target + scenarios +        │
                        │  evaluators + thresholds     │
                        └──────────┬──────────────────┘
                                   │
                        ┌──────────▼──────────────────┐
                        │       EvalRunner             │
                        │  (orchestrates the pipeline) │
                        └──┬───────┬──────────┬───────┘
                           │       │          │
              ┌────────────▼┐  ┌───▼────┐  ┌──▼──────────┐
              │  Simulator   │  │ Engine │  │  Delivery   │
              │  (LLM user   │  │ (eval  │  │  (JSONL     │
              │   persona)   │  │  each  │  │   upload)   │
              └──────┬───────┘  │session)│  └──────┬──────┘
                     │          └───┬────┘         │
              ┌──────▼───────┐     │        ┌──────▼──────┐
              │    Target    │     │        │  Amplitude  │
              │  (customer's │     │        │  Platform   │
              │    agent)    │  ┌──▼────┐   │  (Sessions, │
              └──────────────┘  │ Gates │   │ Gold Sets)  │
                                │(pass/ │   └─────────────┘
                                │ fail) │
                                └──┬────┘
                                   │
                                ┌──▼─────┐
                                │ Report │
                                │(console│
                                │+ JSON) │
                                └────────┘
```

Every box in this diagram is behind an **extension point** (a protocol or base class). Crawl implements one concrete adapter per extension point. Walk and Run add more adapters with zero changes to the orchestration layer:

| Extension Point | Crawl | Walk | Run |
|---|---|---|---|
| **Target** (how we talk to the agent) | `CallableTarget` (Python function) | `HttpTarget` (REST endpoint) | `CliTarget` (subprocess) |
| **Delivery** (where results go) | `UploadDelivery` (JSONL file) | -- | `LiveDelivery` (real-time SDK sessions) |
| **Eval Method** (how evaluators execute) | LLM judge only | -- | CODE (deterministic) + HYBRID |
| **Granularity** (what gets evaluated) | Whole session | -- | Individual turns |
| **Gates** (pass/fail logic) | Numeric thresholds | pass@k / pass^k (statistical) | Failure labels, aggregate rules |

---

## Design Principles

These principles govern not just Crawl, but the entire harness. They ensure Walk and Run extend naturally rather than requiring rearchitecture.

**One evaluator system, many surfaces.** The SDK uses the same `EvaluatorDefinition` schema and prompt generation logic as Agent Analytics server-side enrichment. An evaluator config that works in the SDK harness works identically in production monitoring, and vice versa.

**Prompts are generated, not written.** Evaluators are defined as structured fields (scale descriptions, categories, detection signals). The prompt generator -- ported from the server-side code -- assembles the actual LLM judge prompt at runtime. Customers describe *what* to evaluate; the system handles *how* to ask the LLM.

**Defaults in code, overrides in config.** Crawl ships 3 battle-tested default evaluators (see below). Walk expands to 11. Customers can override defaults or add their own via YAML configuration.

**Extension points, not features.** See the table above. Every subsystem has a pluggable interface from day one.

**CI/CD native.** Exit code 0 means all thresholds passed. Exit code 1 means a threshold was breached. This is all a CI pipeline needs.

---

## Crawl Scope: What Ships

### 3 Default Evaluators

Crawl ships one evaluator of each type to prove the architecture works across all evaluator primitives:

| Evaluator | Type | What It Measures |
|---|---|---|
| `task_completion` | RUBRIC (scored 0/1/2) | Did the agent complete what the user asked for? |
| `session_safety` | CLASSIFIER (categorical) | Content safety classification |
| `outcome` | CLASSIFIER (categorical) | What happened: response_provided / clarification_requested / unable_to_complete / escalated / abandoned |

These are ported from the same evaluator definitions running in Agent Analytics production enrichment today.

Walk-1 adds the remaining 8 defaults (response_quality, user_satisfaction, agent_confusion, request_complexity, task_failure, behavioral_patterns, negative_feedback, turn_categories).

### CLI

```bash
# Run an eval suite -- exits 0 (pass) or 1 (fail)
amplitude-ai-eval run spec.yaml

# Same, but print full transcripts and evaluator reasoning
amplitude-ai-eval run spec.yaml --verbose
```

### Eval Spec (YAML)

A single YAML file defines the entire eval run:

```yaml
target:
  type: callable
  module: "my_agent"
  function: "chat"

simulator:
  model: "gpt-4o-mini"

scenarios:
  - id: "pricing-inquiry"
    persona:
      description: "Small business owner evaluating plans"
    goal: "Get pricing information for the Pro plan"
    first_message: "How much does the Pro plan cost?"
    max_turns: 8

evaluators:
  defaults: true    # uses the 3 built-in evaluators

thresholds:
  numeric:
    task_completion: 0.6    # fail if score below this

settings:
  mode: upload
  agent_name: "my-agent"
  agent_version: "v1.0"
  upload:
    add_to_gold_dataset: "regression-suite-v1"
```

### Programmatic API

```python
from amplitude_ai.eval import EvalRunner, EvalSpec

spec = EvalSpec.from_yaml("spec.yaml")
runner = EvalRunner(amplitude_ai=ai, spec=spec)
report = await runner.run()
assert report.passed  # True if all thresholds met
```

### Output

**Console (default):**

```
Scenario: pricing-inquiry
  task_completion:  1.5 / 2.0  (threshold: 0.6)  PASS
  session_safety:   safe                           PASS
  outcome:          response_provided              --

Result: PASS (1/1 scenarios passed)
```

**Console (--verbose):** adds full conversation transcript, the exact prompt sent to each LLM judge, the raw judge response, and per-evaluator latency.

**JSON file:** machine-readable report for programmatic consumption and archival.

**JSONL upload file:** conversations in OpenAI Chat Completions format, compatible with the conversation upload API for gold dataset ingestion.

---

## Crawl Scope: What Does NOT Ship

These are explicitly deferred to Walk or Run. The architecture supports all of them from day one via extension points.

| Feature | Deferred To | Why |
|---|---|---|
| HTTP targets (testing deployed agents) | Walk-2 | CallableTarget is sufficient for initial development-loop testing |
| Multi-trial with pass@k/pass^k | Walk-3 | Single trial proves the pipeline; statistical rigor adds complexity |
| `validate` / `preview` CLI commands | Walk-4 | Useful but not essential for first end-to-end demo |
| Custom evaluators via YAML (Format A) | Walk-5 | Config builder has 35+ validation rules to port correctly |
| Turn-level evaluation | Run-1 | Session-level is the higher-value default |
| Code/deterministic evaluators | Run-1 | LLM judges cover the important cases first |
| Live delivery (real-time SDK sessions) | Run-2 | Upload delivery is sufficient; live delivery is a convenience optimization |
| Regression comparison across versions | Run-3 | Requires two completed eval runs to compare |
| Concurrent execution | Run-4 | Sequential is correct first; concurrency is optimization |
| Node/TypeScript SDK | Run-5 | Python proves the design; TypeScript mirrors it |

---

## How It Connects to the Broader System

```
  Developer writes / updates agent code
           │
           ▼
  amplitude-ai-eval run spec.yaml --verbose
           │
           ├──► Threshold gate: exit 0 (pass) / exit 1 (fail)
           │
           ├──► Console: per-scenario scores + pass/fail
           │
           └──► --verbose: full transcripts + evaluator reasoning
                       │
                       ▼
                 Error analysis:
                 - Which scenarios failed?
                 - What did the agent say wrong?
                 - Root cause in prompt / retrieval / tools?
                       │
                       ▼
                 Fix agent ──► Re-run evals (repeat until passing)
                       │
                       ▼
                 Ship with confidence (CI gate passes)


  ─── Optional: Upload to Amplitude ───

  JSONL uploaded to Agent Analytics
           │
           ├──► Sessions visible in AA Session Browser
           │    (team can review eval history over time)
           │
           └──► Optionally added to Gold Dataset
                (for evaluator calibration -- advanced workflow,
                 not the primary use case)
```

The eval harness is primarily a **developer tool for agent improvement**. The Amplitude upload path provides team visibility and connects to the broader quality infrastructure, but the core value -- run evals, find failures, fix the agent -- works entirely locally.

---

## Dependencies

| Dependency | Status | Impact if Not Ready |
|---|---|---|
| `amplitude-ai` SDK (Python) | Exists | Core dependency, no blockers |
| Conversation Upload API (AA-150202) | In development | Crawl can generate JSONL files locally regardless; upload is a delivery-time concern |
| Gold Dataset Backend (AA-150187) | In development | `add_to_gold_dataset` field is metadata in the JSONL; actual ingestion depends on backend availability |
| LLM provider access (OpenAI/Anthropic) | Customer-provided | Simulator and LLM evaluators require API keys; documented in spec |

---

## Success Criteria for Crawl

1. `amplitude-ai-eval run spec.yaml` runs end-to-end against a callable Python agent
2. 3 default evaluators produce scored results matching server-side AA output format
3. `--verbose` prints full transcript + evaluator prompt/result/latency for inspection
4. JSONL output file is validated against the conversation upload API format
5. Threshold gate returns exit code 0 when scores pass, exit code 1 when they fail
6. Unit tests cover all modules; integration test runs with `MockAmplitudeAI`

---

## Crawl / Walk / Run Roadmap (Summary)

| | Crawl | Walk | Run |
|---|---|---|---|
| **What it proves** | End-to-end pipeline works (simulate, evaluate, upload, gate, inspect) | Production-ready for external customers | Full feature set + version comparison + TypeScript + Langley migration |
| **Evaluators** | 3 defaults (LLM, session-level) | 11 defaults + custom (Format A YAML) + validate/preview | + turn-level + deterministic code evaluators |
| **Targets** | Callable (Python function) | + HTTP endpoints | + CLI subprocess |
| **Delivery** | JSONL upload | Same | + Live SDK sessions |
| **Gates** | Numeric thresholds | + pass@k / pass^k (statistical) | + failure labels, aggregate rules, regression diff |
| **CLI** | `run [--verbose]` | + `validate`, `preview`, `defaults` | + `compare`, `export` |
| **SDKs** | Python | Python | + Node TypeScript |
