# Sentient Evals Harness (OSS) — Tasks

## Purpose

Build `sentient-evals`: a plug-and-play evaluation harness for AI agents (any framework) that can run locally/offline, and can also be embedded by the Sentient platform as the shared eval engine.

This repo is intentionally **separate** from the Sentient monorepo so:
- The Sentient CLI (`sen`) can remain private.
- The eval harness can be adopted by anyone without platform lock-in.
- Release cadence, CI, licensing, and community contributions are cleanly isolated.

## Repo + integration plan

- **Separate git repo**: `sentient-evals` (recommended)
  - In the Sentient monorepo, consume it as an **external dependency** (pinned version) in `packages/api` and the private CLI packaging.
  - Local dev: editable install (e.g. `pip install -e ../sentient-evals`).

## Execution model (local vs hosted)

Two modes, one shared harness:

- **Local/offline mode (default OSS path)**:
  - Runs fully on the developer machine / CI.
  - No Sentient API required.
  - Produces a Harbor-style job directory with per-trial artifacts (config/result/transcript/outcome) for debuggability.

- **Hosted/platform mode (Sentient backend + worker fleet)**:
  - **Sentient backend (API)** is the control plane: auth, RBAC, dataset ownership, quotas, run creation, status, results queries.
  - **Eval workers** are the data plane: execute trials in parallel, run graders, write artifacts.
  - Start as a separate **worker service** (ECS/Fargate or similar) pulling jobs from a queue; avoid a separate “eval backend” initially.
  - Keep the external contract stable so a future split can happen behind the API gateway if needed.

## Data + artifacts model (Harbor-inspired)

- **Local runs**: write to a `jobs/` folder by default:
  - `jobs/<run_id>/run_config.json`
  - `jobs/<run_id>/run_result.json`
  - `jobs/<run_id>/trials/<trial_id>/...` (transcript, verifier outputs, etc.)
- **Hosted runs**:
  - Datasets + large artifacts in object storage (S3/GCS).
  - Structured results in DB (run/task/trial/grader rows).
  - Optional: store “debug bundle” as a Harbor-like tarball for download.

## Platform API surface (hosted mode)

These APIs live in the existing Sentient backend for now:
- Dataset CRUD with presigned uploads/downloads
- Run creation: “create run from suite + dataset + model config”
- Run status + streaming logs (optional)
- Results queries: per-run/per-suite summaries + drill-down to trial artifacts

The `sentient-evals` OSS CLI may include an optional plugin/adapter to submit runs to Sentient, but keep it off-by-default.

## Core abstractions (must-have)

- **Task**: inputs + environment spec + success criteria
- **Trial**: one attempt with deterministic seed/config snapshot
- **Transcript**: messages + tool calls + intermediate artifacts
- **Outcome**: final environment state (files, DB rows, HTTP responses, etc)
- **Grader**: pluggable scoring logic
- **Suite**: collection of tasks + shared config
- **Run**: orchestration metadata (timestamps, model, harness version, git SHA, etc)

### Additional must-have clarifications (production-grade)

These align with the evaluation definitions and practical guidance for agent evals in Anthropic’s guide:
https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

- **Agentic system scope**: explicitly support evaluating both workflows and agents (e.g., prompt-chaining, routing, parallel voting, orchestrator-workers, evaluator-optimizer loops), not only “single prompt → single response”.
- **Reproducibility**: record model/provider identifiers, prompts/judge prompts, tool schemas, harness version, and configuration snapshots; support deterministic replays where possible.
- **Transcript vs outcome**: graders must be able to score either the full transcript/trajectory or the final outcome state (and often both).
- **Noise/variance**: multi-trial is default; report confidence/variance and avoid overfitting to single-run outcomes.

What can make Sentient meaningfully “more than Harbor” (and a moat)
Technique breadth + composability: a single harness that supports many eval families (LLM-as-judge, deterministic/unit-test, rubric + tool-use constraints, budget/latency, security policies, multi-run stats, etc.) in a consistent schema.
Managed execution: queueing, concurrency, caching, sandbox orchestration, artifact retention, reproducibility, and cost controls at scale.
Workflow + governance: datasets/benchmarks versioning, eval baselines, regression gates for PRs, audit trails, RBAC, approvals.
Product integration: “run eval suites on every agent change” as a first-class part of your AgentOps dashboard (telemetry + evals + deployments).
So: we’re not inherently building a carbon copy, but we could end up there unless the SaaS layer and the “all techniques, one interface” layer become the center of gravity.

## Task list (stepwise)

- [x] 1. Define public schemas + contracts
  - Task/trial/transcript/outcome/result JSON schema
  - Stable IDs and reproducibility fields (seed, model, harness version)
  - Export formats: JSON + JSONL; optional JUnit for CI
  - Include explicit versions for: task schema, grader schema, and judge prompt schema
  - Include “transcript/trajectory” schema with tool calls + observations as first-class
  - Include “outcome pointers” schema (files/DB/etc) so outcomes can be validated without bloating artifacts
  - Implementation notes (current scaffold):
    - `sentient_evals.models` includes `schema_version="v1"` plus `PromptSpec`/`JudgeSpec` and `OutcomePointer`
    - Versioned JSON Schemas are in `sentient-evals/schemas/v1/*.schema.json` (export via `sentient-evals schema export`)
    - JUnit XML export for CI is available via `sentient-evals junit <run_dir> --out junit.xml`

- [x] 2. Job/trial filesystem format + result writer (Harbor-style)
  - Implemented canonical run-level `run_config.json` + `run_result.json` (replaces run/summary).
  - Trials now guarantee `trial_config.json`, `trial_result.json`, `trajectory.json`, `outcome.json` plus `judge/` and `verifier/` dirs; graders receive an artifact sink to write judge/verifier outputs.
  - Artifact writer uses atomic writes; schemas exported include `RunConfigFile`.
  - Standard on-disk layout (`jobs/<run_id>/...`) for local runs
  - Trial subfolders include:
    - `trial_config.json` (trial config)
    - `trial_result.json` (grader outputs + metrics)
    - `trajectory.json` (ATIF trajectory)
    - `outcome.json` (final state snapshot pointer + summaries)
  - Optional: `verifier/` folder outputs for deterministic graders (tests, stdout/stderr, rewards)
  - Add `judge/` folder for model-based graders (judge prompt, raw judge outputs, disagreement summaries)

- [x] 3. Harness runner (local mode)
  - Multi-trial execution with concurrency controls, resume/cancel, and adapter env wiring (tools via harness).
  - Deterministic replays: record/replay tool outputs per trial (`replay/tool_calls.jsonl`) with strict matching.
  - Capture transcripts + outcomes reliably; judge/verifier artifacts retained.
  - Run-level aggregation now includes pass@k, pass^k, per-task rates, and CI on pass rate.
  - Provenance recorded in run config (runtime/platform digest; ready for task-dir hashing extensions).

- [x] 4. Environment runners
  - Local runner (python function-based tasks)
  - Container runner (Docker) for sandboxed multi-turn agent evals + outcome grading via tests
  - Define sandboxing defaults (timeouts, cpu/mem limits, network policy) and document security boundaries
  - Implementation notes:
    - **Task bundle format**: Directory-based tasks (`task.toml`, `instruction.md`, `environment/`, `tests/`, `files/`) with deterministic SHA256 digest for reproducibility. Loader in `sentient_evals.task_bundles`.
    - **Environment abstraction**: `BaseEnvironment` interface with factory pattern (`sentient_evals.environments`). Supports pluggable backends: `local_python`, `docker_cli`, `docker_sdk`, `podman_cli`, `daytona`.
    - **Local Python runner**: Fast in-process execution via `LocalPythonEnvironment` for unit-style tasks.
    - **Docker CLI runner**: Production default using `docker` CLI with image caching (tagged by task digest), security hardening (`--cap-drop ALL`, `--security-opt no-new-privileges`), resource limits (CPU/memory), and network isolation.
    - **Docker SDK runner**: Optional backend using `docker` Python SDK for better control/streaming (requires `sentient-evals[docker]`).
    - **Podman support**: Rootless-friendly via `DockerCLIEnvironment` with `engine="podman"`.
    - **Daytona cloud runner**: Optional backend for high-concurrency (100s+ parallel trials) using async SDK with shared client manager, snapshot/image caching, and recursive file operations (requires `sentient-evals[daytona]`).
    - **ToolExecutor bridge**: `EnvironmentToolExecutor` adapts `BaseEnvironment` to existing `ToolExecutor` protocol, keeping adapters unchanged.
    - **Provenance**: Environment type, task bundle digest, container image refs, and resource policies recorded in `TrialConfig.provenance` and `RunConfigFile.provenance`.
    - **CLI integration**: `--tasks-dir` for task bundles, `--env` for backend selection, backend-specific flags (`--docker-image-tag-prefix`, `--daytona-snapshot-template`).
    - **Security defaults**: Untrusted task posture with capability dropping, network isolation, resource caps, and documented boundaries in README.

- [x] 5. Agent adapters
  - Minimal adapter interface: `AgentAdapter.run(task, seed, env: ToolExecutor) -> transcript, outcome`
  - **ATIF v1.5 trajectory output**: Trials now write `trajectory.json` (ATIF) as the canonical trace artifact (converted from `TranscriptEvent[]`), with custom metrics preserved under `metrics.extra`.
  - Built-in adapters for common patterns:
    - **Installed/CLI base**: `BaseInstalledAdapter` runs one-or-more `exec` commands (with optional `install`) and has a stdout-first fallback transcript.
    - **Workflow adapter interface**: `WorkflowAdapter` protocol plus a `WorkflowStubAdapter` demonstrating workflow metadata via `TranscriptEvent(kind="metric")`.
  - Optional integrations (behind extras): LangChain, CrewAI, AutoGen
    - Implemented as **thin wrappers** that delegate framework-specific wiring to a user-supplied factory/runner, while recording tool usage via a shared `ToolExecutorRecorder`.

- [x] 6. Graders (baseline set)
  - Code-based:
    - Outcome verification (unit tests, file diffs, DB assertions)
    - Static analysis hooks (lint/type checks)
    - Tool-usage verification (required/forbidden tools, argument constraints)
  - Model-based:
    - LLM-as-judge rubric grading (structured output, reference-free and reference-based)
    - Multi-LLM judge voting (majority/unanimous/average) + disagreement reporting
    - Calibration mode (sampled human review; store rubric + judge prompt version; track drift)
  - Budget graders:
    - Cost/tokens/latency ceilings + scoring
  - Add statistical reporting helpers:
    - paired comparisons for regressions
    - pass@k where applicable
    - confidence intervals / bootstrap for noisy tasks
  - **Implementation notes**:
    - Modular grader architecture with three tiers: deterministic (`graders/deterministic/`), model-based (`graders/judge/`), and human review (`graders/human/`).
    - Deterministic graders: `ExactMatchGrader`, `StateCheckGrader` (with operator support: `$eq`, `$regex`, etc.), `StaticAnalysisGrader`, `ToolUsageGrader`, `VerifierScriptGrader` (Harbor-style `tests/test.sh` execution).
    - Model-based graders: `LLMJudgeGrader` (using `JudgeClient` protocol), `MultiLLMJudgeGrader` (aggregation), `PairwiseJudgeGrader` (A/B comparison).
    - Human graders: `HumanReviewGrader` for creating review packets.
    - Budget graders: `BudgetGrader` for token/cost/latency limits.
    - CLI-driven configuration: graders can be specified via `--config` (TOML/JSON) or `--graders-file` (JSON array), enabling plug-and-play grader selection without writing Python code.
    - Environment-aware grading: `EnvironmentGrader` protocol allows graders to access the trial workspace (`env: ToolExecutor`) for outcome verification (e.g., `VerifierScriptGrader`, `StaticAnalysisGrader`).
    - Registry system (`registry.py`) supports built-in grader types and dynamic imports via `import_path`.
  - **Approach recap (what shipped / why)**:
    - Graders are configured via TOML/JSON and run per-trial, writing artifacts under `trials/<trial_id>/{verifier,judge}/` for debuggability.
    - Environment-aware graders (`VerifierScriptGrader`, `StaticAnalysisGrader`) execute inside the sandbox via `ToolExecutor`, so results reflect the final trial workspace state.
    - Harbor-style `tests/test.sh` verifiers are supported, with a safe fallback to run via `sh` when executable bits are not preserved by host↔container copying.

- [x] 6.5. Built-in adapters for common CLI agents 
  - Ship built-in adapters for commercial CLI agents so users can run evals with only a config file (no Python glue code).
  - Target agents (inspired by Harbor's installed agent pattern):
    - **Claude Code** (`claude-code`): Anthropic's CLI coding agent
    - **Codex CLI** (`codex`): OpenAI's CLI coding agent
    - **OpenCode** (`opencode`): Open-source CLI coding agent
    - Additional candidates: `cursor-cli`, `cline-cli`, `aider`, `gemini-cli`, `goose`, `qwen-coder`
  - Implementation approach (Harbor-inspired):
    - Extend `BaseInstalledAdapter` pattern (`adapters.py`) with agent-specific subclasses.
    - Each adapter handles:
      - Installation detection/verification (check if CLI tool is installed)
      - Command construction (map task input to CLI invocation with proper args/env vars)
      - Output parsing (extract transcript/outcome from CLI stdout/stderr)
      - Model parameter passing (via CLI flags/env vars)
    - Registry integration: register adapters in `registry.py` so they're available via `--adapter <name>` or config files.
    - CLI convenience: users can run `sentient-evals run --tasks-dir ./tasks --adapter claude-code --model anthropic/claude-opus-4-1` without writing adapter code.
    - Configuration via TOML/JSON:
      ```toml
      [adapter]
      type = "claude-code"
      config = { model = "anthropic/claude-opus-4-1", timeout_s = 300 }
      ```
    - Installation scripts: optional `install-{agent}.sh` helpers (similar to Harbor's `install-{agent-name}.sh.j2`) for CI/setup automation.
    - Fallback behavior: if agent CLI not found, provide clear error with installation instructions.
  - Success criteria:
    - Users can evaluate Claude Code, Codex CLI, and OpenCode with zero Python adapter code.
    - Adapters handle common CLI patterns (stdin/stdout, file-based I/O, environment variables for API keys).
    - Registry auto-discovers built-in adapters; external adapters via `import_path` still supported.
  - **Approach recap (what shipped / why)**:
    - Implemented a Harbor-style `BaseInstalledAdapter` + per-agent subclasses with `install-*.sh` scripts, registered in `registry.py` so users can select agents via `--adapter` or config files.
    - Docker-backed runs were hardened for real-world usage:
      - `--jobs-dir` is resolved to an absolute path (required for Docker bind-mounts).
      - Docker image existence checks correctly trigger builds.
      - Container keepalive command is properly quoted so the container stays running for `exec` operations.
    - Cursor CLI adapter was aligned with current Cursor CLI docs:
      - Uses `agent --print` with `--api-key`/`CURSOR_API_KEY` auth, and supports Cursor-native models (e.g. `composer-1`).
      - Supports `linux/amd64` platform override in task bundles (critical for Apple Silicon users running amd64-only agent CLIs).

- [x] 6.6. Built-in benchmarks + dataset registry (Harbor-style plug-and-play)
  - Ship a small set of **built-in** benchmark/task packs (smoke + starter suites) and a registry mechanism for larger community datasets.
  - UX goal: users can run evals without authoring task folders:
    - `sentient-evals datasets list`
    - `sentient-evals run --dataset <name>@<version> --env docker_cli --adapter cursor-cli --config eval.toml`
    - `sentient-evals datasets pull <name>@<version>` (cache locally)
  - Also ship a scaffolder for custom tasks (Harbor parity):
    - `sentient-evals tasks create <task-name>` → generates `task.toml`, `instruction.md`, `environment/Dockerfile`, `tests/test.sh`
  - Implementation sketch:
    - `datasets/` module: registry index (JSON) + downloader (git/tarball) + local cache dir
    - CLI commands: `datasets list/pull/path`, `tasks create`
    - Keep the harness universal: registry is optional; local `--tasks-dir` remains supported.
  - **Done**: `datasets list/pull/path` + `tasks create` implemented. Docker environment dynamically detects WORKDIR from task Dockerfiles for Harbor compatibility (`/app`, `/testbed`, `/workspace`). Tests mounted at `/tests`, logs at `/logs`.

- [x] 7. Reporting + aggregation
  - Aggregate over trials (mean/variance, pass@k where relevant)
  - Per-suite dashboards in terminal output (summary + failure drill-down)
  - “Regression suite” mode (compare two runs; highlight deltas)
  - Add “judge reliability” reporting: agreement rates, tie rates, and examples of judge disagreement for calibration
  - **Done**: Added score variance/stddev, run/report dashboards with failure drill-down, diff regression view, and judge reliability metrics (agreement/tie/examples).

- [x] 8. OSS CLI (local mode)
  - `sentient-evals run --suite ...`
  - `sentient-evals report --run ...`
  - `sentient-evals diff --baseline ... --candidate ...`
  - **Done**: Implemented `run`, `report`, `diff` (suite via `--suite-id`/`--config`, report/diff take run dirs).

- [ ] 9. Hosted mode: worker execution contract (Sentient backend + workers)
  - Same work as platform Task 17.5 (integration).
  - Define a minimal “run spec” and “trial spec” payload
  - Queue contract (job message schema; idempotency keys)
  - Worker writes:
    - artifact bundle to object storage (optional)
    - trial results to Sentient DB via internal API or direct DB session (choose one)
  - Concurrency controls + quotas at run level (avoid unbounded fan-out)

- [ ] 10. Hosted mode: Sentient backend endpoints (control plane)
  - Same work as platform Task 17.5 (integration). Dataset upload/download (presigned URLs)
  - Create run, list runs, get run, cancel run
  - Results endpoints: summary + drill-down to trials + artifact links

- [x] 11. cloud sandbox provider adapters (Harbor-like scaling) — **Done**
  - Pluggable “sandbox backend” interface
  - First provider: pick one (e.g., Daytona/Modal/E2B), support single-container tasks initially
  - Document limitations (multi-container vs single-container)
  - **Completed (Feb 2026):**
    - Cloud provider abstraction: `CloudSandboxProvider` + `CloudSandboxEnvironment` in `sentient_evals.environments.cloud` and `environments.providers.base`.
    - **Daytona**: `--env daytona`, full lifecycle, exec, fs, snapshot/image caching, shared client manager; optional extra `sentient-evals[daytona]`.
    - **E2B**: `--env e2b`, provider lifecycle, exec, fs upload/download, event logs, env metadata; template build from Dockerfile/image; `E2B_API_KEY`.
    - **Modal**: `--env modal`, `ModalProvider` + `ModalEnvironment`; image from Dockerfile/registry/debian_slim; app lookup, secrets, volumes, network allowlist; `MODAL_TOKEN_ID`/`MODAL_TOKEN_SECRET`; optional `sentient-evals[modal]`.
    - Factory and runner wire all three (`EnvironmentType.daytona`, `EnvironmentType.e2b`, `EnvironmentType.modal`) with backend-specific flags (`--daytona-*`, `--modal-*`, cloud API key env).
    - Provider-aware concurrency gating via task-level `environment.provider_concurrency` for Daytona, E2B, and Modal.
    - Limitation: single-container/single-sandbox task execution; multi-container not in scope for v1.

- [ ] 12. CI templates + examples
  - Minimal example suites (toy + realistic)
  - GitHub Actions template running a small regression suite
  - Documentation: “How to add a task”, “How to write a grader”, “How to run multi-trial”

- [ ] 13. RL evaluation support (after baseline graders + harness are stable)
  - Standardize “reward” reporting + trajectory capture suitable for RL-style evaluation
  - Add RL-specific metrics hooks (episode reward, success rate, step budget, cost/latency per episode)
  - Keep RL training/optimization out of v1; focus on evaluation harness contracts first
  - **Clarifications (what “RL eval support” means in v1)**:
    - **Canonical reward artifact**: require a machine-readable scalar reward per trial at `trials/<trial_id>/verifier/reward.json` (even if derived from `test.sh` exit code). Prefer schema:
      - `reward` (float), `passed` (bool), `components` (dict of named sub-rewards), `reason` (string|optional)
      - `episode_metrics` (steps/tool_calls/wall_time_ms/tokens/cost_usd/latency_ms as available)
      - `task_id`, `trial_id`, `run_id`, `seed`, `harness_version`, `task_bundle_digest` (when using bundles)
    - **Trajectory contract**: treat `trajectory.json` (ATIF) as the primary RL trajectory artifact; ensure events carry stable step indices + timestamps, and include tool-call timing/error metadata where available.
    - **Run-level aggregation**: surface RL-friendly aggregates in `run_result.json` (mean/stddev of reward, success rate, reward histograms/quantiles, mean steps/tool calls, mean cost/latency per episode).
    - **Rollout dataset export (no training)**: add an export path that converts completed runs into RL-consumable datasets:
      - **rollouts**: JSONL of \{task, trajectory, reward, episode_metrics, provenance\}
      - **preferences** (optional): derive (winner, loser) pairs from multi-trial tasks (useful for DPO-style training), but keep model training out of scope.
    - **Reward shaping via graders**: define a consistent way to derive reward from multiple graders (e.g., weighted sum + floor/ceiling + required graders gate), and persist the exact reward function config in trial/run provenance for reproducibility.

## What the Sentient private CLI should do

Two modes (recommended):
- **Local mode**: `sen evals run` shells out to `sentient-evals` (no Sentient backend required).
- **Hosted mode**: `sen evals trigger` calls Sentient backend endpoints to run eval pipelines against a deployment and store results for the dashboard.

The private CLI should not re-implement harness logic; it should depend on `sentient-evals` for local mode and use Sentient API for hosted mode.
