---
title: CLI Reference
description: Command-line interface for running evaluations
---

## Commands

EZVals has three main commands:

- `ezvals serve` - Start the web UI to browse and run evaluations interactively
- `ezvals run` - Run evaluations headlessly (for CI/CD pipelines)
- `ezvals export` - Export a run to various formats (JSON, CSV, Markdown)

## ezvals serve

Start the web UI to discover and run evaluations interactively.

```bash
ezvals serve PATH [OPTIONS]
```

Where `PATH` can be:
- A directory: `ezvals serve evals/`
- A file: `ezvals serve evals/customer_service.py`
- A specific function: `ezvals serve evals.py::test_refund`
- A run JSON file: `ezvals serve .ezvals/sessions/default/run_123.json`

The UI opens automatically in your browser. Evaluations are discovered and displayed but not run until you click the Run button.

### Loading Previous Runs

You can load a previous run by passing the JSON file path directly:

```bash
ezvals serve .ezvals/sessions/default/sleek-wolf_1705312200.json
```

This opens the UI with that run's results loaded. If the original eval source file still exists, you can rerun evaluations. If the source file was moved or deleted, the UI works in view-only mode.

### Options

<ParamField path="-d, --dataset" type="string" default="all">
  Filter evaluations by dataset.
</ParamField>

```bash
ezvals serve evals/ --dataset customer_service
```

<ParamField path="-l, --label" type="string" default="all">
  Filter evaluations by label. Can be specified multiple times.
</ParamField>

```bash
ezvals serve evals/ --label production -l critical
```

<ParamField path="--results-dir" type="string" default=".ezvals/runs">
  Directory for JSON results storage.
</ParamField>

```bash
ezvals serve evals/ --results-dir ./my-results
```

<ParamField path="--port" type="integer" default="8000">
  Port for web UI server.
</ParamField>

```bash
ezvals serve evals/ --port 3000
```

<ParamField path="--session" type="string">
  Name for this evaluation session. Groups related runs together.
</ParamField>

```bash
ezvals serve evals/ --session model-comparison
```

<ParamField path="--run" type="flag">
  Automatically run all evaluations on startup. Same as clicking the Run button immediately.
</ParamField>

```bash
ezvals serve evals/ --run
```

## ezvals run

Run evaluations headlessly. Outputs minimal text by default (optimized for LLM agents). Use `--visual` for rich table output.

```bash
ezvals run PATH [OPTIONS]
```

Where `PATH` can be:
- A directory: `ezvals run evals/`
- A file: `ezvals run evals/customer_service.py`
- A specific function: `ezvals run evals.py::test_refund`
- A case variant: `ezvals run evals.py::test_math[2-3-5]`

### Filtering Options

<ParamField path="-d, --dataset" type="string" default="all">
  Filter evaluations by dataset. Can be specified multiple times.
</ParamField>

```bash
ezvals run evals/ --dataset customer_service
ezvals run evals/ -d customer_service -d technical_support
```

<ParamField path="-l, --label" type="string" default="all">
  Filter evaluations by label. Can be specified multiple times.
</ParamField>

```bash
ezvals run evals/ --label production
ezvals run evals/ -l production -l critical
```

<ParamField path="--limit" type="integer">
  Limit the number of evaluations to run.
</ParamField>

```bash
ezvals run evals/ --limit 10
```

### Execution Options

<ParamField path="-c, --concurrency" type="integer" default="1">
  Number of concurrent evaluations. `1` means sequential execution.
</ParamField>

```bash
# Run 4 evaluations in parallel
ezvals run evals/ --concurrency 4
ezvals run evals/ -c 4
```

<ParamField path="--timeout" type="float">
  Global timeout in seconds for all evaluations.
</ParamField>

```bash
ezvals run evals/ --timeout 30.0
```

### Output Options

<ParamField path="-v, --verbose" type="flag">
  Show stdout from eval functions (print statements, logs).
</ParamField>

```bash
ezvals run evals/ --verbose
ezvals run evals/ -v
```

<ParamField path="--visual" type="flag">
  Show rich progress dots, results table, and summary. Without this flag, output is minimal.
</ParamField>

```bash
ezvals run evals/ --visual
```

<ParamField path="-o, --output" type="string">
  Override the default results path. When specified, results are saved only to this path (not to `.ezvals/runs/`).
</ParamField>

```bash
ezvals run evals/ --output results.json
ezvals run evals/ -o results.json
```

<ParamField path="--no-save" type="flag">
  Skip saving results to file. Outputs JSON to stdout instead.
</ParamField>

```bash
# Get results as JSON without writing to disk
ezvals run evals/ --no-save | jq '.passed'
```

### Session Options

<ParamField path="--session" type="string">
  Name for this evaluation session. Groups related runs together.
</ParamField>

```bash
ezvals run evals/ --session model-comparison
```

<ParamField path="--run-name" type="string">
  Name for this specific run. Used as file prefix.
</ParamField>

```bash
ezvals run evals/ --session model-comparison --run-name gpt4-baseline
```

## ezvals export

Export a run file to various formats. Useful for sharing results, generating reports, or integrating with other tools.

```bash
ezvals export RUN_PATH [OPTIONS]
```

Where `RUN_PATH` is the path to a run JSON file (e.g., `.ezvals/sessions/default/run_123.json`).

### Options

<ParamField path="-f, --format" type="choice" default="json">
  Export format: `json`, `csv`, or `md`.
</ParamField>

```bash
ezvals export run.json -f md
ezvals export run.json -f csv
```

<ParamField path="-o, --output" type="string">
  Output file path. Defaults to `{run_name}.{format}`.
</ParamField>

```bash
ezvals export run.json -f md -o report.md
```

### Export Formats

| Format | Description |
|--------|-------------|
| `json` | Copy the raw JSON file |
| `csv` | Flat CSV with all results |
| `md` | Markdown with ASCII bar charts and results table |

### Examples

```bash
# Export to Markdown
ezvals export .ezvals/sessions/default/run_123.json -f md

# Export to CSV
ezvals export run.json -f csv -o results.csv
```

## Examples

### Start the Web UI

```bash
# Discover evals in a directory and open the UI
ezvals serve evals/

# Start UI on a custom port
ezvals serve evals/ --port 8080

# Filter what's shown in the UI
ezvals serve evals/ --dataset qa --label production

# Load a previous run to view or continue
ezvals serve .ezvals/sessions/default/sleek-wolf_1705312200.json
```

### Run All Evaluations

```bash
ezvals run evals/
```

### Run Specific File

```bash
ezvals run evals/customer_service.py
```

### Run Specific Function

```bash
ezvals run evals/customer_service.py::test_refund
```

### Run Parametrized Variant

```bash
ezvals run evals/math.py::test_addition[2-3-5]
```

### Filter by Dataset and Label

```bash
ezvals run evals/ --dataset qa --label production
```

### Run with Concurrency and Timeout

```bash
ezvals run evals/ -c 8 --timeout 60.0
```

### Export Results

```bash
# Results auto-save to .ezvals/runs/ by default
ezvals run evals/

# Override output path
ezvals run evals/ -o results.json
```

### Verbose Debug Run

```bash
# Show eval stdout and rich output
ezvals run evals/ -v --visual --limit 5
```

### Production CI Pipeline

```bash
# Minimal output for LLM agents/CI
ezvals run evals/ -c 16 --timeout 120
```

### Session Tracking

```bash
# Group runs under a session
ezvals run evals/ --session model-comparison --run-name baseline

# Continue the session with another run
ezvals run evals/ --session model-comparison --run-name improved
```

## Configuration File

EZVals supports a `ezvals.json` config file for persisting default CLI options. The file is auto-generated in your project root on first run.

### Default Config

```json
{
  "concurrency": 1,
  "results_dir": ".ezvals/runs"
}
```

### Supported Options

| Option | Type | Description | Used by |
|--------|------|-------------|---------|
| `concurrency` | integer | Number of concurrent evaluations | `run` |
| `timeout` | float | Global timeout in seconds | `run` |
| `verbose` | boolean | Show stdout from eval functions | `run` |
| `results_dir` | string | Directory for results storage | `serve` |
| `port` | integer | Web UI server port | `serve` |

### Precedence

CLI flags always override config values:

```bash
# Config has concurrency: 1, but this uses 4
ezvals run evals/ -c 4
```

### Editing via UI

Click the settings icon in the web UI header to view and edit config values. Changes are saved to `ezvals.json`.

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | Evaluations completed (regardless of pass/fail) |
| Non-zero | Error during execution (bad path, exceptions, etc.) |

<Note>
The CLI does not currently set non-zero exit codes for failed evaluations—only for execution errors. Check the JSON output or summary for pass/fail status.
</Note>

## Environment Variables

| Variable | Description |
|----------|-------------|
| `EZVALS_CONCURRENCY` | Default concurrency level |
| `EZVALS_TIMEOUT` | Default timeout in seconds |

## Output Format

### Minimal Output (Default)

By default, `ezvals run` outputs minimal text optimized for LLM agents and CI pipelines:

```
Running...
Results saved to .ezvals/runs/swift-falcon_2024-01-15T10-30-00Z.json
```

### Visual Output (--visual)

Use `--visual` for rich progress dots, results table, and summary:

```
Running...
customer_service.py ..F

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                     customer_service                           ┃
┣━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┫
┃ Name                ┃ Status   ┃ Score    ┃ Latency           ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ test_refund         │ ✓ passed │ 1.0      │ 0.23s             │
│ test_complaint      │ ✗ failed │ 0.0      │ 0.45s             │
└─────────────────────┴──────────┴──────────┴───────────────────┘

Summary: 1/2 passed (50.0%)
```

### JSON File Output

Results are always saved as JSON to `.ezvals/runs/` (or custom path via `-o`):

```json
{
  "run_id": "2024-01-15T10-30-00Z",
  "total": 2,
  "passed": 1,
  "failed": 1,
  "results": [...]
}
```
