Metadata-Version: 2.4
Name: mcp-testing
Version: 0.1.0a0
Summary: Local testing framework for MCP (Model Context Protocol) servers
Author-email: Antoni Gmitruk <founders@golf.dev>
License: Apache-2.0
Project-URL: Homepage, https://github.com/golf-mcp/golf-testing
Project-URL: Documentation, https://github.com/golf-mcp/golf-testing#readme
Project-URL: Repository, https://github.com/golf-mcp/golf-testing
Project-URL: Bug Tracker, https://github.com/golf-mcp/golf-testing/issues
Keywords: mcp,testing,ai,claude,protocol,cli,local
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Environment :: Console
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0.0
Requires-Dist: anthropic>=0.65.0
Requires-Dist: openai>=1.0.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: click>=8.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: PyYAML>=6.0.0
Requires-Dist: mcp>=1.15.0
Requires-Dist: PyJWT>=2.10.1
Requires-Dist: packaging>=21.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-httpx>=0.21.0; extra == "dev"
Requires-Dist: ruff>=0.13.0; extra == "dev"
Requires-Dist: mypy>=1.18.0; extra == "dev"
Requires-Dist: pre-commit>=4.0.0; extra == "dev"
Provides-Extra: ci
Requires-Dist: pytest>=8.0.0; extra == "ci"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "ci"
Requires-Dist: pytest-cov>=4.0.0; extra == "ci"
Requires-Dist: pytest-xdist>=3.0.0; extra == "ci"
Dynamic: license-file

<div align="center">
  <h1>🧪 MCP Testing</h1>
  <p><strong>AI-powered testing framework for MCP servers</strong></p>
  <p>Test your MCP servers with real AI agents conducting conversations and LLM judges evaluating results</p>
</div>

<div align="center">
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License"></a>
  <a href="https://github.com/golf-mcp/golf-testing/pulls"><img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg" alt="PRs Welcome"></a>
  <a href="https://pypi.org/project/mcp-testing/"><img src="https://img.shields.io/pypi/v/mcp-testing" alt="PyPI"></a>
  <a href="#"><img src="https://img.shields.io/badge/python-3.12+-blue.svg" alt="Python"></a>
</div>

## Why MCP Testing?

Traditional testing doesn't work for MCP servers. You can't write unit tests for natural language interactions. MCP Testing solves this with:

- **🤖 Real AI Agents** - Claude and GPT-4 actually use your MCP server
- **👤 User Simulation** - AI simulates realistic multi-turn user behavior  
- **⚖️ LLM-as-a-Judge** - Intelligent evaluation instead of brittle assertions
- **🎭 Comprehensive Testing** - Security, compliance and performace all in one framework


## Quick Start

Get testing in 3 steps:

1. **Install & Setup**
   ```bash
   pip install mcp-testing
   export ANTHROPIC_API_KEY="sk-ant-..."  # For AI agents
   export OPENAI_API_KEY="sk-..."         # For LLM judge
   ```

2. **Interactive Onboarding**
   ```bash
   mcp-t quickstart  # Creates your first server & test suite
   ```

3. **Run Tests**
   ```bash
   mcp-t run <suite-id> <server-id>
   # Example: mcp-t run example_suite_001 hackernews_mcp_server
   ```

## Core Concepts

### Test Flow
```
Your Test Case → AI Agent (Claude/GPT-4) → Your MCP Server
      ↓                    ↓                      ↓
 User Message         Tool Calls            Server Response
      ↓                    ↓                      ↓
User Simulator      Conversation Loop         More Tools
      ↓                    ↓                      ↓
   LLM Judge       Complete Transcript      Pass/Fail + Reasoning
```

### Configuration Files

**Server Config** (`examples/server.json`):
```json
{
  "url": "https://no-auth-server.courier-mcp.authed-qukc4.ryvn.run/mcp/",
  "name": "hackernews_mcp_server"
}
```

**Test Suite** (`examples/suite.json`):
```json
{
  "suite_id": "example_suite_001",
  "name": "Hacker News MCP Server Tests",
  "test_cases": [
    {
      "test_id": "hackernews_greeting",
      "user_message": "Hello! Can you help me browse Hacker News?",
      "success_criteria": "Agent should respond politely and explain Hacker News capabilities",
      "max_turns": 5
    }
  ]
}
```

### Test Types
- **💬 Conversational** - Real user workflows
- **🔒 Security** - Authentication & vulnerabilities  
- **✅ Compliance** - MCP protocol validation

## Commands

### Test Execution
```bash
mcp-t run <suite-id> <server-id>           # Run specific suite
mcp-t run example_suite_001 hackernews_mcp_server -v   # Verbose output
```

### Configuration Management  
```bash
mcp-t create server              # Interactive server setup
mcp-t create suite               # Create test suite
mcp-t create test-case           # Add test to suite
mcp-t list                       # Show all configs
mcp-t show suite example_suite_001   # View specific config
mcp-t quickstart                 # Complete onboarding
```

## Test Results

### Understanding Evaluation
```json
{
  "test_id": "hackernews_stories",
  "verdict": "PASS",
  "confidence_score": 0.89,
  "judge_reasoning": "The agent successfully fetched and displayed Hacker News stories. Good use of available tools and clear presentation of results.",
  "conversation_quality": 0.87,
  "tool_calls": [
    {"tool": "get_top_stories", "args": {}},
    {"tool": "get_story_details", "args": {"story_id": 123}}
  ]
}
```

## Support

- [Documentation](https://docs.golf.dev)
- [Contributing Guide](CONTRIBUTING.md)

---

<div align="center">
  <p>Built with ❤️ for the MCP ecosystem</p>
  <p><sub>Made in San Francisco, CA</sub></p>
</div>
