---
title: EZVals
description: Unit Testing for LLMs and Agents
---

<img
  className="block dark:hidden"
  src="/assets/hero-light.svg"
  alt="EZVals Hero"
/>
<img
  className="hidden dark:block"
  src="/assets/hero-dark.svg"
  alt="EZVals Hero"
/>

EZVals is a lightweight, code-first evaluation framework for testing AI agents and LLM applications. Inspired by Pytest and LangSmith, EZVals lets you
write evaluations like you write unit tests!


## Install

<CodeGroup>

```bash uv
uv add ezvals --dev
```

```bash pip
pip install ezvals
```

```bash poetry
poetry add ezvals --group dev
```

<CardGroup cols={2}>
  <Card title="Version-Control" icon="code-branch">
    Aligned with your local env. Branch to experiment!
  </Card>
  <Card title="Pytest-Inspired" icon="flask-vial">
    Use `assert`, `cases`, and simple decorators.
  </Card>
  <Card title="Full Flexibility" icon="sliders">
    Use the same evaluator across a dataset or case-by-case
  </Card>
  <Card title="Agent-Friendly" icon="robot">
    Powerful and compact CLI and SDK for your coding agent to use
  </Card>
</CardGroup>


</CodeGroup>

## Quick Example

Evaluating a simple sentiment analyzer against a ground truth dataset:

```python
from ezvals import eval, EvalContext

@eval(
  input="I love this product!",  # Prompt
  reference="positive",          # Ground Truth
  dataset="sentiment"            # Label for filtering
)
async def test_sentiment_analysis(ctx: EvalContext):
    # Run test and store output in context
    ctx.output = await analyze_sentiment(ctx.input)

    # Evaluate!
    assert ctx.output == ctx.reference, f"Expected {ctx.reference}, got {ctx.output}"
```

Or use `cases=` to apply eval to a dataset:

```python
@eval(
  dataset="sentiment",
  cases=[
      {"input": "I love this product!", "reference": "positive"},
      {"input": "This is terrible", "reference": "negative"},
      {"input": "It's okay I guess", "reference": "neutral"},
  ],
)
async def test_sentiment_batch(ctx: EvalContext):
    # Case data is injected into the ctx
    ctx.output = await analyze_sentiment(ctx.input)

    assert ctx.output == ctx.reference, f"Expected {ctx.reference}, got {ctx.output}"
```

Start the web UI to run evals and review results:

```bash
ezvals serve sentiment_evals.py

# Or run all in a dir
ezvals serve evals/
```

### Web UI

EZVals spins up a local Web UI that makes it easy to filter, run, and rerun evals. Do deep analysis on the results

<img
  className="block dark:hidden"
  src="/assets/ui-screenshot-light.png"
  alt="EZVals Web UI"
/>
<img
  className="hidden dark:block"
  src="/assets/ui-screenshot-dark.png"
  alt="EZVals Web UI"
/>

All results are stored locally in a `.json` file for further analysis.

## Agent Mode

The CLI and SDK make it easy for your coding agent to run, analyze, and iterate on the evals!

> Can you run `test_sentiment_batch` evals and tell me why the scores are so low?

Your coding agent would run:

```bash
ezvals run sentiment_evals.py::test_sentiment_batch --session sentiment-failed-results

// Eval results saved to ./ezvals/cool-cloud-2025.json
```

The agent could review the json results before presenting a solution to you. It could even implement that solution and rerun the evals to compare
before and after!

## Existing eval frameworks are frustrating:

<CardGroup cols={3}>
  <Card title="Too Opinionated" icon="lock" color="#9ca3af">
    One function per dataset, rigid patterns. No way to run different logic per
    test case.
  </Card>
  <Card title="Cloud-Based" icon="cloud" color="#9ca3af">
    Datasets in the cloud. No version control. Code and data live in different
    places.
  </Card>
  <Card title="UI-Based" icon="display" color="#9ca3af">
    Your coding agent can't run evals, analyze results, or iterate on datasets.
  </Card>
</CardGroup>

**Pytest isn't the answer either.** Tests are pass/fail—evals are for _analysis_. You want to see all results, latency, cost, comparisons over time. Pytest doesn't do that.

**EZVals is different:** minimal, flexible, agent-friendly. Everything lives locally as code and JSON.

<Card title="Ready to start?" icon="play" href="/quickstart">
  Follow our quickstart guide to set up EZVals in under 5 minutes.
</Card>
