---
title: "Datasets"
description: "Define Cases and Datasets for systematic evaluation; run programmatically or from files."
sidebarTitle: "Datasets"
icon: "table"
keywords: ["dataset","case","yaml","json","parallel"]
---

### API

- `Case[Input, Output, Metadata]`
- `Dataset[Input, Output, Metadata]`

Source: [datasets.py](https://github.com/lastmile-ai/mcp-eval/blob/main/src/mcp_eval/datasets.py)

### Programmatic

```python
from mcp_eval import Case, Dataset, ToolWasCalled, ResponseContains

cases = [
  Case(
    name="fetch_example",
    inputs="Fetch https://example.com",
    evaluators=[ToolWasCalled("fetch"), ResponseContains("Example Domain")],
  )
]

dataset = Dataset(name="Fetch Suite", cases=cases)
report = await dataset.evaluate(lambda inputs, agent, session: agent.generate_str(inputs))
report.print(include_input=True, include_output=True)
```

Parallel evaluation:

```python
report = await dataset.evaluate(
  lambda inputs, agent, session: agent.generate_str(inputs),
  max_concurrency=4,
)
```

### YAML/JSON

Save/load via `Dataset.to_file` and `Dataset.from_file`. Schema: [mcpeval.config.schema.json](https://github.com/lastmile-ai/mcp-eval/blob/main/schema/mcpeval.config.schema.json).

YAML example (from [basic_fetch_dataset.yaml](https://github.com/lastmile-ai/mcp-eval/blob/main/examples/mcp_server_fetch/datasets/basic_fetch_dataset.yaml)):

```yaml
name: "Basic Fetch Dataset"
server_name: "fetch"
cases:
  - name: "simple_fetch"
    inputs: "Fetch https://example.com"
    expected_output: "Example Domain"
    evaluators:
      - ToolWasCalled:
          tool_name: "fetch"
      - ResponseContains:
          text: "Example Domain"
```

### Concurrency

`Dataset.evaluate(..., max_concurrency=N)` runs cases in parallel.

### Examples

- [test_dataset_style.py](https://github.com/lastmile-ai/mcp-eval/blob/main/examples/mcp_server_fetch/tests/test_dataset_style.py)
- [datasets/](https://github.com/lastmile-ai/mcp-eval/tree/main/examples/mcp_server_fetch/datasets/)

{/* TODO: Add a screenshot showing dataset progress UI and a snippet of the printed table. */}


