What it is
A Python library that lets you write test scenarios for your MCP server and measure how reliably Claude picks the right tool. You write a prompt and declare which tool you expect to be called. The library runs it N times, calculates hit rates, and tells you where Claude is getting confused.

The problem it solves
MCP server developers write tool descriptions blind. They ship, and discover in production that Claude keeps calling list_users when the user says "find John" — because search_users and list_users have descriptions that are too similar. There's no pre-ship way to catch this. mcpbench is that pre-ship check.

What using it looks like
python# test_server.py
from mcpbench import MCPBench, expect

bench = MCPBench(server=mcp, model="claude-sonnet-4-20250514")

@bench.scenario("user wants to search by name")
async def test_search():
    result = await bench.run("find all users named John")
    expect(result).called_tool("search_users")
    expect(result).tool_param("query", contains="John")

@bench.scenario("user wants a specific record")
async def test_get():
    result = await bench.run("get me user with id 42")
    expect(result).called_tool("get_user")
    expect(result).tool_param("user_id", equals=42)
bash$ mcpbench run tests/ --runs 5

Running 2 scenarios × 5 runs each...

✓ user wants to search by name     ████████░░  4/5  (80%)
✗ user wants a specific record     ██████░░░░  3/5  (60%) ← below threshold

1 scenario below threshold (default: 80%)
```

---

## Build order

### Friday night — prove the core works (2hrs)

Just one script, no framework. Goal: get Claude to tell you which tool it called.

**Steps:**
1. `pip install anthropic fastmcp`
2. Write the dummy server with three similar tools: `search_users`, `get_user`, `list_users`
3. Write `experiment.py` that:
   - Spins up the dummy server in-memory with FastMCP
   - Calls `client.list_tools()` to get tool definitions
   - Converts them to Anthropic API format
   - Sends a prompt to Claude with those tools
   - Prints which tool Claude picked and with what params
4. Run it with 3-4 different prompts and confirm the output makes sense

You're done Friday when you can print `{"tool_called": "search_users", "params": {"query": "John"}}` reliably.

---

### Saturday morning — the runner (2-3hrs)

Turn your experiment script into a proper runner that executes N times and collects results.

**Steps:**
1. Create `mcpbench/runner.py`
2. Write `async def run_scenario(prompt, server, n, model)` that:
   - Calls Claude N times in a loop
   - Collects `{tool_called, params}` each time
   - Returns a `ScenarioResult` object with all N results
3. Add basic error handling — what happens if Claude doesn't call any tool? (it can happen, treat it as a miss)
4. Test it manually: run your search prompt 5 times, confirm you get 5 results back

---

### Saturday afternoon — the scenario decorator + expect API (2-3hrs)

Build the developer-facing interface — the part people actually write.

**Steps:**
1. Create `mcpbench/bench.py` — the `MCPBench` class
   - `__init__` takes `server` and `model`
   - `scenario(description)` is a decorator that registers the test function
   - `run(prompt)` calls the runner and returns a result object
2. Create `mcpbench/expect.py` — the `expect()` assertion API
   - `called_tool(name)` — checks the most common tool across N runs matches expected
   - `tool_param(key, contains=None, equals=None)` — checks params on successful runs
   - On failure, raise `AssertionError` with a clear human-readable message like:
     `Expected search_users (got list_users 3/5 times)`
3. Wire them together and run your two test scenarios end to end

---

### Saturday evening — hit rate stats + thresholds (1hr)

**Steps:**
1. Add hit rate calculation to `ScenarioResult`:
   - `hit_rate` = number of times expected tool was called / total runs
   - `confusion` = dict of what tool was called instead and how often
2. Add `--threshold` flag (default 0.8) — scenarios below it are marked as failing
3. Print the confusion breakdown when a scenario fails:
```
   ✗ user wants a specific record  3/5 (60%)
     called get_user:   3x ✓
     called list_users: 2x ✗ ← confused here

Sunday morning — CLI (1-2hrs)
Steps:

pip install click
Create mcpbench/cli.py with one command: mcpbench run <path> --runs N --threshold F
The CLI should:

Import the test file
Discover all registered scenarios
Run them all
Print the results table
Exit with code 1 if any scenario is below threshold (important for CI)


Register the CLI in pyproject.toml under [project.scripts]


Sunday afternoon — packaging + publish (1-2hrs)
Steps:

Set up pyproject.toml properly:

toml   [project]
   name = "mcpbench"
   version = "0.1.0"
   dependencies = ["anthropic", "fastmcp", "click"]
```
2. Write the README with:
   - One paragraph explaining the problem
   - Installation (`pip install mcpbench`)
   - The full usage example from this spec
   - A note that you need an `ANTHROPIC_API_KEY` env variable
3. Push to GitHub with a proper repo description
4. `pip install build && python -m build && pip install twine && twine upload dist/*`
5. Verify `pip install mcpbench` works from scratch

---

## What's explicitly out of scope for this weekend

- Multiple model comparison (Claude vs GPT)
- HTML/JSON report output
- Statistical confidence intervals
- Async parallel runs (run scenarios concurrently)
- GitHub Actions integration

These are all v2 features. Resist the urge.

---

## File structure when done
```
mcpbench/
├── __init__.py          # exports MCPBench, expect
├── bench.py             # MCPBench class, scenario decorator
├── runner.py            # core Claude calling logic
├── expect.py            # assertion API
└── cli.py               # mcpbench CLI command
tests/
└── dummy_server.py      # the test server you build against
experiment.py            # your Friday night proof of concept (keep it)
pyproject.toml
README.md

The one thing that will trip you up
FastMCP's client is async. Everything in runner.py will be async def. When you wire it to the CLI (which is sync), you'll need asyncio.run() to bridge them. Keep that in one place — in cli.py — and keep everything else async. Don't fight it.