# evalkit — Developer Notes

## What is evalkit?

evalkit is a simple, provider-agnostic LLM/RAG evaluation framework for teams. It lets you define test cases, run them against any LLM or RAG pipeline, and judge the outputs using configurable metrics (faithfulness, relevancy, hallucination, completeness, etc.). Results are stored and can be viewed via a lightweight web UI.

## Project Structure

```
evalkit/
├── pyproject.toml              # Build config, deps, entry points
├── CLAUDE.md                   # This file
├── src/
│   └── evalkit/
│       ├── __init__.py         # Public API re-exports
│       ├── py.typed            # PEP 561 marker
│       ├── cli.py              # Click CLI (evalkit run, evalkit serve, evalkit report)
│       ├── models.py           # Pydantic models (TestCase, JudgeResponse, MetricScore, etc.)
│       ├── engine.py           # EvaluationEngine (sync + async with concurrency)
│       ├── client.py           # EvalKitClient SDK (typed server client, stdlib only)
│       ├── judges/             # Judge implementations
│       │   ├── __init__.py
│       │   ├── base.py         # BaseJudge ABC
│       │   ├── openai.py       # OpenAIJudge + AzureOpenAIJudge (structured outputs)
│       │   ├── anthropic.py    # AnthropicJudge (structured outputs via messages.parse)
│       │   └── ollama.py       # OllamaJudge (local, ollama package with structured outputs)
│       ├── metrics/            # Evaluation metrics
│       │   ├── __init__.py
│       │   ├── base.py         # BaseMetric ABC
│       │   ├── faithfulness.py
│       │   ├── answer_relevancy.py
│       │   ├── contextual_relevancy.py
│       │   ├── hallucination.py
│       │   ├── completeness.py
│       │   ├── tool_selection.py
│       │   ├── tool_parameter.py
│       │   ├── tool_parameter_similarity.py
│       │   ├── exact_match.py       # Deterministic (no judge)
│       │   ├── string_containment.py
│       │   ├── string_similarity.py # Levenshtein distance
│       │   ├── rouge.py             # evalkit[rouge]
│       │   └── bleu.py              # evalkit[bleu]
│       ├── prompts/            # Prompt templates (separated for easy review)
│       │   ├── __init__.py     # Re-exports SYSTEM_PROMPT
│       │   ├── system.py       # Shared judge system prompt
│       │   ├── faithfulness.py
│       │   ├── answer_relevancy.py
│       │   ├── contextual_relevancy.py
│       │   ├── hallucination.py
│       │   ├── completeness.py
│       │   └── tool_parameter_similarity.py
│       ├── suites/             # Pre-built metric bundles
│       │   ├── __init__.py
│       │   ├── base.py         # EvaluationSuite dataclass
│       │   ├── rag_qa.py
│       │   ├── document_search.py
│       │   ├── conversational.py
│       │   ├── agent_tool_call.py
│       │   └── live_qa.py      # LiveQASuite (no ground truth: answer + context relevancy)
│       ├── contrib/            # Framework trackers
│       │   ├── __init__.py     # Re-exports BaseTracker, ToolCallTracker
│       │   ├── tracker.py      # BaseTracker ABC, TrackerRun, ToolCallTracker
│       │   ├── openai.py       # OpenAIAgentTracker (evalkit[openai-agents])
│       │   ├── langchain.py    # LangChainTracker (evalkit[langchain])
│       │   └── pydanticai.py   # PydanticAITracker (evalkit[pydanticai])
│       ├── reports/            # Report generation
│       │   ├── __init__.py
│       │   └── generator.py    # ReportGenerator — aggregation + fix suggestions
│       └── server/             # FastAPI web UI (optional, evalkit[server])
│           ├── __init__.py
│           ├── app.py          # FastAPI application
│           ├── db/             # SQLAlchemy ORM + session
│           ├── routes/         # dashboard, projects, evaluations, REST API
│           │   ├── dashboard.py    # GET / (project list)
│           │   ├── projects.py     # Project CRUD + HTML form evaluation trigger
│           │   ├── evaluations.py  # Evaluation detail + JSON report download
│           │   └── api.py          # REST API (evaluate, projects, config)
│           └── templates/      # Jinja2 + Tailwind CSS + HTMX
├── tests/
│   ├── conftest.py             # MockJudge, shared fixtures
│   └── unit/                   # Unit tests (no API keys needed)
└── examples/
    ├── openai_agent.py         # OpenAI Agents SDK (agent + RAG via OpenAIAgentTracker)
    ├── langchain_agent.py      # LangChain (agent + RAG via LangChainTracker)
    ├── pydantic_ai_agent.py    # PydanticAI (agent + RAG via PydanticAITracker)
    ├── server_rag.py           # Server-side RAG evaluation via SDK
    ├── server_agent.py         # Local agent evaluation + server upload
    └── server_combined.py      # Combined RAG + agent server workflow
```

## Running Tests

```bash
pytest tests/unit/ -v
```

To run with coverage:

```bash
pytest tests/unit/ -v --cov=src/evalkit --cov-report=term-missing
```

Currently 485 unit tests, all passing. No API keys needed for unit tests.

## Starting the Server

```bash
evalkit serve
```

By default the server starts on `http://localhost:8000`. Use `--port` and `--host` flags to override.

### Server Configuration

The server's default judge provider, model, and evaluation suite can be set via CLI flags or environment variables:

| CLI Flag | Env Var | Default | Description |
|---|---|---|---|
| `--judge` / `-j` | `EVALKIT_JUDGE_PROVIDER` | `openai` | Judge provider (openai, azure, anthropic, ollama) |
| `--model` / `-m` | `EVALKIT_JUDGE_MODEL` | Provider default | Model name |
| `--suite` / `-s` | `EVALKIT_SUITE` | `rag_qa` | Default suite (rag_qa, document_search, conversational, agent_tool_call, live_qa, answer_similarity) |

Per-request fields in the `/api/v1/evaluate` payload override server defaults when provided.

The `GET /api/v1/config` endpoint returns current server defaults for client discovery.

### REST API

All REST endpoints live under `/api/v1/` prefix (defined in `routes/api.py`):

- `GET /api/v1/config` — server defaults and available suites/providers
- `GET /api/v1/projects` — list projects
- `POST /api/v1/projects` — create project `{name, description}`
- `POST /api/v1/evaluate` — submit evaluation (returns 202, runs in background)
- `POST /api/v1/results` — upload locally-generated results (returns 201, immediately complete)
- `GET /api/v1/evaluations/{id}` — poll evaluation status and results

The evaluate endpoint runs `EvaluationEngine.run_async()` via FastAPI `BackgroundTasks`. It builds the judge from the request's `judge_provider` field (falling back to server defaults), then persists results to the DB as `EvalResultModel` + `MetricScoreModel` rows.

The results endpoint accepts pre-computed evaluation results (from a local `EvaluationEngine` run) and stores them directly — no server-side LLM calls needed.

### Server SDK (`evalkit.client`)

`EvalKitClient` is a typed Python client for the REST API. It uses only stdlib (`urllib`) so it works with a bare `pip install evalkit-llm` — no server extras required.

```python
from evalkit import EvalKitClient

client = EvalKitClient("http://localhost:8000")

# Server-side evaluation: send test cases, server runs the judge
project = client.create_project("My App")
evaluation = client.evaluate(project.id, test_cases, suite="rag_qa")
result = client.wait_for_evaluation(evaluation.id)

# Local evaluation: run locally, upload results for the dashboard
from evalkit import EvaluationEngine, RAGQASuite
report = engine.run(test_cases, suite=RAGQASuite())
evaluation = client.upload_results(project.id, report)
```

Models: `Project`, `ServerConfig`, `Evaluation`, `EvalKitAPIError` — all re-exported from `evalkit`.

## Key Architecture Decisions

### src layout
Source lives under `src/evalkit/` rather than at the top level. This prevents accidentally importing the package from the repo root without installing it, catching packaging issues early.

### Structured outputs
`OpenAIJudge` and `AzureOpenAIJudge` use `chat.completions.parse(response_format=JudgeResponse)`. `AnthropicJudge` uses `messages.parse(output_format=JudgeResponse)`. `OllamaJudge` uses the `ollama` package with `format=JudgeResponse.model_json_schema()` and `json.loads()` for structured output parsing. All four guarantee type-safe responses.

Agent tool-call evaluation uses the `ToolCall(name, parameters)` model. `TestCase` has optional `expected_tool_calls` and `actual_tool_calls` fields for agent evaluation. The `tool_selection` and `tool_parameter` metrics are deterministic (no LLM call); `tool_parameter_similarity` uses the judge to check semantic equivalence of parameter values.

### Single judge base class
All judges inherit from `BaseJudge` (ABC). Subclass it, implement `evaluate()`, and async support is provided for free via `run_in_executor`. No protocol or multiple base classes.

### Framework trackers
`BaseTracker` (ABC) in `evalkit.contrib.tracker` defines the shared protocol for all framework trackers. Properties: `question`, `tool_calls`, `context`, `answer`. Convenience methods: `to_test_case()`, `evaluate()` (local with judge), `upload()` (server-side, defaults to `live_qa` suite).

Framework-specific implementations use native hook/callback systems:
- `OpenAIAgentTracker` extends `RunHooksBase` — passed as `hooks=` to `Runner.run_sync()`
- `LangChainTracker` extends `BaseCallbackHandler` — passed as `callbacks=[tracker]`
- `PydanticAITracker` performs post-hoc extraction from `AgentRunResult`

`ToolCallTracker` is the framework-agnostic fallback using `@tracker.wrap` decorator + `tracker.capture()` context manager.

`LiveQASuite` evaluates without ground truth (answer relevancy + contextual relevancy only). Auto-selected by `evaluate()` when no `expected_tool_calls` or `expected_answer` are provided.

### Extras pattern
Heavy dependencies (openai, anthropic, fastapi, etc.) are gated behind optional extras so that a basic install stays lightweight. Users install only what they need:
- `pip install evalkit-llm[openai]` — adds OpenAI + Azure OpenAI judge support
- `pip install evalkit-llm[openai-agents]` — adds OpenAI Agents SDK tracker
- `pip install evalkit-llm[anthropic]` — adds Anthropic support
- `pip install evalkit-llm[ollama]` — adds Ollama support
- `pip install evalkit-llm[langchain]` — adds LangChain tracker
- `pip install evalkit-llm[pydanticai]` — adds PydanticAI tracker
- `pip install evalkit-llm[server]` — adds the web UI
- `pip install evalkit-llm[dev]` — adds the full dev toolchain

### Storage abstraction
The server uses SQLAlchemy 2.0 ORM with SQLite as the default database. Set `DATABASE_URL` env var for Postgres or another backend. SQLite requires `check_same_thread=False` for FastAPI compatibility (set in `db/session.py`).

## Documentation

`README.md` is the user-facing documentation. Any change to public API, CLI flags, environment variables, configuration options, or judge/metric/tracker behavior **must** be reflected in the README. If you add a CLI option, add it to the CLI Reference section. If you add an env var, add it to the Environment Variables table. If you add a metric or suite, add it to the relevant section.

## Code Style

- Comments only where the code is hard to follow. No docstrings on private methods, no section separator comments.
- Module-level docstrings are one line.
- Class docstrings kept only when they explain scoring semantics (metrics) or non-obvious behavior.
- Tests have no section separators or label comments between test classes.
