# Scrapy MCP Middleware

Scrapy middleware for tracking requests and responses with MCP (Model Context Protocol) integration.

## Installation

```bash
pip install scrapy-mcp-middleware
```

## Usage

Add the middleware to your Scrapy settings:

```python
SPIDER_MIDDLEWARES = {
    'scrapy_mcp_middleware.SessionTrackerMiddleware': 100,
}

# Optional: Configure the session store URL (defaults to http://127.0.0.1:8765)
SESSION_STORE_URL = 'http://127.0.0.1:8765'
```

## Features

- Tracks all requests and responses
- Hierarchical request tree tracking
- Base64 encoding for request/response bodies
- Configurable session store URL
- Automatic storage cleanup on spider start

## Requirements

- Python >= 3.8
- Scrapy >= 2.0.0
- requests >= 2.25.0
