# HTTP MCP Transport Implementation Spec

**Status:** DRAFT
**Created:** 2025-01-20
**Priority:** HIGH
**Complexity:** MEDIUM-HIGH

## Overview

Implement full support for the Model Context Protocol (MCP) Streamable HTTP transport as defined in the [MCP Transports Specification](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports).

Currently, Kollabor only supports the **stdio** transport. HTTP transport will enable:
- Remote MCP servers (cloud-hosted or LAN)
- Server-initiated notifications via SSE
- Streaming responses from long-running tools
- Multi-client server scenarios

## Current State

**What Works (stdio):**
- `MCPServerConnection` class handles stdio transport
- JSON-RPC protocol over stdin/stdout
- Server process lifecycle management
- Tool discovery and execution

**What's Missing (HTTP):**
```python
# kollabor/llm/mcp_integration.py:542-559
async def _execute_http_tool(self, server_config: Dict, tool_name: str,
                            params: Dict) -> Dict[str, Any]:
    """Execute a tool via HTTP MCP server."""
    # This would implement HTTP-based MCP tool calls
    # For now, return a placeholder
    return {
        "status": "not_implemented",
        "message": "HTTP MCP servers not yet implemented"
    }
```

**Dead Code to Remove/Refactor:**
- `_execute_stdio_tool()` (lines 479-540) - Old pre-JSON-RPC implementation
- `_execute_http_tool()` (lines 542-559) - Current stub
- `_validate_server()` (lines 372-383) - Only used by dead code
- `_get_server_capabilities()` (lines 385-410) - Only used by dead code
- `_execute_server_command()` (lines 561-584) - Only used by dead code

## Requirements

### Functional Requirements

1. **HTTP Transport Support**
   - Support both HTTP and HTTPS endpoints
   - Single MCP endpoint handling GET and POST methods
   - JSON-RPC over HTTP with proper headers
   - Server-Sent Events (SSE) for streaming responses

2. **Session Management**
   - Handle `Mcp-Session-Id` header for session tracking
   - Session initialization and renewal
   - Session termination (DELETE method)
   - Handle session expiry (404 responses)

3. **Message Exchange**
   - POST requests for client-to-server messages
   - GET requests to open SSE streams
   - Accept both `application/json` and `text/event-stream` responses
   - Handle batch JSON-RPC messages

4. **Server Discovery**
   - Parse `type: "http"` from `mcp_settings.json`
   - Initialize HTTP servers via POST
   - Discover tools via `tools/list`
   - Maintain connection pool for HTTP servers

5. **Security**
   - Validate `Origin` header (DNS rebinding protection)
   - Support authentication headers
   - HTTPS/TLS support
   - localhost-only binding preference for local servers

### Configuration Format

**Existing `mcp_settings.json` (stdio):**
```json
{
  "servers": {
    "filesystem": {
      "type": "stdio",
      "command": "npx -y @modelcontextprotocol/server-filesystem /path",
      "enabled": true
    }
  }
}
```

**New HTTP Support:**
```json
{
  "servers": {
    "remote-llm": {
      "type": "http",
      "url": "https://api.example.com/mcp",
      "enabled": true,
      "headers": {
        "Authorization": "Bearer ${MCP_API_TOKEN}",
        "X-Custom-Header": "value"
      },
      "options": {
        "timeout": 30,
        "verify_ssl": true,
        "origin": "https://myapp.example.com"
      }
    },
    "local-http": {
      "type": "http",
      "url": "http://localhost:3000/mcp",
      "enabled": true
    }
  }
}
```

### Non-Functional Requirements

1. **Performance**
   - Connection pooling/reuse
   - Configurable timeouts (default: 30s)
   - Async/non-blocking I/O

2. **Reliability**
   - Automatic reconnection on failure
   - Session renewal on expiry
   - SSE reconnection with `Last-Event-ID`

3. **Compatibility**
   - Backwards compatible with existing stdio transport
   - Support deprecated HTTP+SSE transport (2024-11-05)
   - Graceful degradation for older servers

## Architecture Design

### Class Structure

```
MCPIntegration (existing)
├── MCPServerConnection (stdio - existing)
│   ├── connect() - Start subprocess
│   ├── initialize() - Send initialize request
│   ├── list_tools() - Get tools from server
│   ├── call_tool() - Execute tool
│   └── close() - Terminate subprocess
│
└── MCPHTTPConnection (NEW)
    ├── connect() - Verify HTTP endpoint
    ├── initialize() - POST initialize request
    ├── list_tools() - POST tools/list
    ├── call_tool() - POST tools/call
    ├── open_sse_stream() - GET for server messages
    ├── close() - DELETE session
    └── _handle_sse_events() - Process SSE stream
```

### Key Components

**1. MCPHTTPConnection Class**

```python
class MCPHTTPConnection:
    """Manages HTTP-based MCP server connection."""

    def __init__(self, server_name: str, url: str,
                 headers: Optional[Dict[str, str]] = None,
                 options: Optional[Dict[str, Any]] = None):
        self.server_name = server_name
        self.base_url = url
        self.headers = headers or {}
        self.options = options or {}

        # Session state
        self.session_id: Optional[str] = None
        self.initialized = False

        # HTTP client (aiohttp for async)
        self._client: Optional[aiohttp.ClientSession] = None
        self._sse_stream: Optional[aiohttp.ClientResponse] = None

    async def connect(self) -> bool:
        """Verify HTTP endpoint is accessible."""

    async def initialize(self) -> bool:
        """Send initialize request, establish session."""

    async def list_tools(self) -> List[Dict[str, Any]]:
        """Request tools list via POST."""

    async def call_tool(self, tool_name: str, arguments: Dict) -> Dict:
        """Execute tool via POST, handle SSE response."""

    async def open_sse_stream(self, callback: Callable) -> None:
        """Open SSE stream for server messages."""

    async def close(self) -> None:
        """Terminate session with DELETE."""

    async def _handle_sse_events(self, response: aiohttp.ClientResponse):
        """Process SSE event stream."""

    async def _post_request(self, method: str, params: Dict) -> Dict:
        """Send JSON-RPC request via POST."""

    def _build_headers(self, accept_sse: bool = True) -> Dict[str, str]:
        """Build HTTP headers with Accept types."""
```

**2. Updated MCPIntegration**

```python
class MCPIntegration:
    """MCP server and tool integration (stdio + HTTP)."""

    def __init__(self):
        self.mcp_servers: Dict[str, Dict] = {}  # Server configs
        self.tool_registry: Dict[str, Dict] = {}  # Tool metadata

        # Connections by transport type
        self.stdio_connections: Dict[str, MCPServerConnection] = {}
        self.http_connections: Dict[str, MCPHTTPConnection] = {}

    async def discover_mcp_servers(self) -> Dict[str, Any]:
        """Discover both stdio and HTTP servers."""

    async def _connect_stdio_server(self, name: str, config: Dict) -> bool:
        """Connect to stdio server (existing logic)."""

    async def _connect_http_server(self, name: str, config: Dict) -> bool:
        """Connect to HTTP server (NEW)."""

    async def call_mcp_tool(self, tool_name: str, params: Dict) -> Dict:
        """Route to appropriate connection type."""
```

### Protocol Flow

**Initialization Flow:**
```
Client -> POST /mcp (InitializeRequest)
        Accept: application/json, text/event-stream

Server <- 200 OK
        Content-Type: application/json
        Mcp-Session-Id: uuid-1234
        { "jsonrpc": "2.0", "id": "...", "result": {...} }

Client -> POST /mcp (tools/list)
        Mcp-Session-Id: uuid-1234

Server <- 200 OK
        { "jsonrpc": "2.0", "id": "...", "result": { "tools": [...] } }
```

**Tool Execution Flow:**
```
Client -> POST /mcp (tools/call)
        Mcp-Session-Id: uuid-1234

Server <- 200 OK
        Content-Type: text/event-stream

Server -> SSE event: data={...partial result...}
Server -> SSE event: data={...final result...}
Server -> closes stream
```

**Server-Initiated Messages (GET):**
```
Client -> GET /mcp
        Accept: text/event-stream
        Mcp-Session-Id: uuid-1234

Server <- 200 OK
        Content-Type: text/event-stream

Server -> SSE event: data={...notification...}
(stream stays open for more messages)
```

### HTTP Client Choice

**Recommendation: aiohttp**

- Native async/await support
- Built-in SSE handling via `async def content()` iterator
- Connection pooling
- SSL/TLS support
- WebSocket support (future-proofing)
- Widely used and maintained

**Alternative: httpx**
- Modern HTTP/2 support
- Async and sync APIs
- Connection pooling

**Decision:** Use `aiohttp` for better async SSE support.

### Dependencies

**New dependency:**
```toml
# pyproject.toml
dependencies = [
    "aiohttp>=3.9.0",  # HTTP client with SSE support
    # ... existing dependencies
]
```

## Implementation Plan

### Phase 1: Foundation (2-3 files)

**File: `kollabor/llm/mcp_http_connection.py`** (NEW)

```python
"""HTTP-based MCP server connection."""

import aiohttp
import asyncio
import json
import logging
import uuid
from typing import Any, AsyncIterator, Callable, Dict, List, Optional

logger = logging.getLogger(__name__)


class MCPHTTPConnection:
    """Manages HTTP-based MCP server connection with SSE support."""

    def __init__(self, server_name: str, url: str,
                 headers: Optional[Dict[str, str]] = None,
                 options: Optional[Dict[str, Any]] = None):
        # ... (full implementation)

    async def connect(self) -> bool:
        """Verify endpoint accessibility with OPTIONS or GET."""

    async def initialize(self) -> bool:
        """Initialize MCP session via HTTP POST."""

    async def list_tools(self) -> List[Dict[str, Any]]:
        """Get tools list via POST."""

    async def call_tool(self, tool_name: str, arguments: Dict) -> Dict[str, Any]:
        """Call tool, handle SSE streaming response."""

    async def open_sse_stream(self, callback: Callable[[Dict], None]) -> None:
        """Open SSE stream for server-initiated messages."""

    async def close(self) -> None:
        """Close session and HTTP client."""

    async def _post_jsonrpc(self, method: str, params: Dict) -> Dict:
        """Send JSON-RPC POST request."""

    async def _read_sse_stream(self, response: aiohttp.ClientResponse) -> AsyncIterator[Dict]:
        """Read and parse SSE events."""

    def _validate_origin(self) -> bool:
        """Validate Origin header if configured."""
```

**Update: `kollabor/llm/mcp_integration.py`**

1. Remove dead code methods
2. Add HTTP connection management
3. Update `discover_mcp_servers()` to handle both types
4. Update `call_mcp_tool()` to route to correct connection

### Phase 2: Integration (update existing files)

**Update: `kollabor/llm/tool_executor.py`**

- Already calls `mcp_integration.call_mcp_tool()`
- No changes needed (handles both transports transparently)

**Update: `kollabor/config/loader.py`**

- Ensure HTTP server configs are loaded correctly
- Support environment variable expansion in headers (`${MCP_API_TOKEN}`)

### Phase 3: Testing (new test files)

**File: `tests/test_mcp_http_connection.py`** (NEW)

```python
"""Test HTTP MCP connection functionality."""

import asyncio
import unittest
from unittest.mock import AsyncMock, Mock, patch

from kollabor.llm.mcp_http_connection import MCPHTTPConnection


class TestMCPHTTPConnection(unittest.IsolatedAsyncioTestCase):

    async def test_connect_valid_endpoint(self):
        """Test connecting to valid HTTP endpoint."""
        # ... implementation

    async def test_initialize_session(self):
        """Test session initialization."""
        # ... implementation

    async def test_list_tools(self):
        """Test tools listing."""
        # ... implementation

    async def test_call_tool_simple(self):
        """Test simple tool call (JSON response)."""
        # ... implementation

    async def test_call_tool_streaming(self):
        """Test tool call with SSE streaming."""
        # ... implementation

    async def test_session_expiry_renewal(self):
        """Test session expiry handling and renewal."""
        # ... implementation

    async def test_sse_server_messages(self):
        """Test receiving server-initiated messages."""
        # ... implementation

    async def test_origin_validation(self):
        """Test Origin header validation."""
        # ... implementation
```

**File: `tests/integration/test_http_mcp_e2e.py`** (NEW)

```python
"""End-to-end tests for HTTP MCP with mock server."""

import asyncio
import unittest
from aiohttp import web

from kollabor.llm.mcp_integration import MCPIntegration


class TestHTTPMCPIntegration(unittest.IsolatedAsyncioTestCase):
    """Integration tests with real HTTP server."""

    async def setup_async(self):
        """Start mock MCP HTTP server."""
        # ... setup aiohttp server with MCP endpoints

    async def test_full_http_mcp_flow(self):
        """Test: connect -> initialize -> list tools -> call tool."""
        # ... full flow test

    async def test_sse_streaming_response(self):
        """Test streaming tool response via SSE."""
        # ... SSE streaming test
```

### Phase 4: Documentation

**Update: `CONFIGURATION.md`**

Add HTTP MCP configuration section with examples.

**Update: `README.md`**

Document HTTP transport support.

**Create: `docs/reference/http-mcp-transport.md`**

Detailed implementation reference.

## Security Considerations

### DNS Rebinding Protection

**Server Responsibility:**
- Validate `Origin` header on all requests
- Bind to localhost only for local servers

**Client Responsibility:**
- Send `Origin` header if configured
- Validate server certificates (HTTPS)

### Implementation

```python
def _validate_origin(self, response: aiohttp.ClientResponse) -> bool:
    """Validate response matches expected origin."""
    expected_origin = self.options.get("origin")
    if not expected_origin:
        return True  # No validation configured

    actual_origin = response.headers.get("Origin")
    if actual_origin != expected_origin:
        logger.warning(
            f"Origin mismatch: expected {expected_origin}, "
            f"got {actual_origin}"
        )
        return False

    return True
```

### Session ID Security

- Generate cryptographically secure session IDs
- Never log session IDs
- Validate session ID format (ASCII 0x21-0x7E)
- Clear session IDs from memory on close

### Authentication

```python
# Support various auth methods via headers
headers = {
    # Bearer token
    "Authorization": "Bearer ${MCP_API_TOKEN}",

    # API key
    "X-API-Key": "${MCP_API_KEY}",

    # Custom auth
    "X-Custom-Auth": "custom_value"
}
```

## Error Handling

### HTTP Status Codes

| Status | Meaning | Action |
|--------|---------|--------|
| 200 OK | Success (JSON response) | Process response |
| 202 Accepted | Request accepted (no body) | Continue waiting |
| 400 Bad Request | Invalid request | Log error, retry |
| 404 Not Found | Session expired | Re-initialize |
| 405 Method Not Allowed | Transport not supported | Fallback to old transport |
| 500 Server Error | Server error | Retry with backoff |

### Retry Logic

```python
async def _post_with_retry(self, method: str, params: Dict,
                           max_retries: int = 3) -> Dict:
    """POST with exponential backoff retry."""
    for attempt in range(max_retries):
        try:
            return await self._post_jsonrpc(method, params)
        except aiohttp.ClientError as e:
            if attempt == max_retries - 1:
                raise
            wait_time = 2 ** attempt  # 1s, 2s, 4s
            await asyncio.sleep(wait_time)
            logger.warning(f"Retry {attempt + 1}/{max_retries} after error: {e}")
```

## Testing Strategy

### Unit Tests

1. **Connection Management**
   - Connect/disconnect
   - Session initialization
   - Session expiry handling

2. **Message Exchange**
   - POST requests
   - JSON-RPC formatting
   - SSE parsing

3. **Error Handling**
   - Network failures
   - Invalid responses
   - Session timeouts

### Integration Tests

1. **Mock Server**
   - aiohttp test server
   - Full MCP protocol implementation
   - SSE streaming support

2. **End-to-End**
   - Discovery flow
   - Tool execution
   - Server notifications

### Manual Testing

**Test with real HTTP MCP server:**
```json
// ~/.kollabor-cli/mcp/mcp_settings.json
{
  "servers": {
    "test-http": {
      "type": "http",
      "url": "http://localhost:3000/mcp",
      "enabled": true
    }
  }
}
```

Run Kollabor and verify:
- Server discovery
- Tool listing
- Tool execution
- Session management

## Migration Path

### Backwards Compatibility

- Existing stdio servers continue to work
- No breaking changes to API
- Optional feature (disabled by default)

### Rollout Strategy

1. **Phase 1** - Implement core HTTP support (no breaking changes)
2. **Phase 2** - Add SSE streaming support
3. **Phase 3** - Add server notification support (GET stream)
4. **Phase 4** - Add deprecated transport fallback

## Performance Considerations

### Connection Pooling

```python
# Reuse HTTP client across connections
self._client = aiohttp.ClientSession(
    connector=aiohttp.TCPConnector(
        limit=10,  # Max 10 connections
        limit_per_host=2,  # Max 2 per server
        ttl_dns_cache=300,  # Cache DNS for 5 min
    ),
    timeout=aiohttp.ClientTimeout(total=30)
)
```

### SSE Stream Management

- Single SSE connection per server
- Background task to read stream
- Callback mechanism for server messages

### Memory Management

- Limit SSE buffer size
- Clear session data on close
- Timeout idle connections

## Open Questions

1. **aiohttp vs httpx?**
   - aiohttp has better SSE support
   - httpx has HTTP/2 support
   - **Recommendation:** Start with aiohttp

2. **Should we support deprecated HTTP+SSE (2024-11-05)?**
   - Spec provides backwards compat guidance
   - **Recommendation:** Implement fallback in Phase 4

3. **How to handle long-running SSE streams?**
   - Background task with asyncio.create_task()
   - Stop on application shutdown
   - **Recommendation:** Use background task with cancellation

4. **Connection pooling strategy?**
   - One ClientSession per server?
   - Shared ClientSession?
   - **Recommendation:** Shared ClientSession with connection limits

## Success Criteria

- [ ] HTTP MCP servers can be configured in `mcp_settings.json`
- [ ] Discovery successfully initializes HTTP servers
- [ ] Tools from HTTP servers are registered and callable
- [ ] SSE streaming responses are handled correctly
- [ ] Session management works (init, renew, terminate)
- [ ] Security measures in place (Origin validation, HTTPS)
- [ ] Unit tests cover all major code paths
- [ ] Integration tests verify end-to-end functionality
- [ ] Documentation is complete
- [ ] No breaking changes to existing stdio transport

## References

- [MCP Transports Specification](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports)
- [aiohttp Documentation](https://docs.aiohttp.org/)
- [Server-Sent Events (W3C)](https://html.spec.whatwg.org/multipage/server-sent-events.html)
- [JSON-RPC 2.0 Specification](https://www.jsonrpc.org/specification)

## Appendix: Example HTTP MCP Server

For testing purposes, here's a simple Node.js HTTP MCP server:

```javascript
// test-mcp-http-server.js
const express = require('express');
const bodyParser = require('body-parser');
const cors = require('cors');

const app = express();
app.use(cors());
app.use(bodyParser.json());

const sessions = new Map();
let sessionIdCounter = 0;

app.post('/mcp', async (req, res) => {
  const { jsonrpc, id, method, params } = req.body;

  // Initialize
  if (method === 'initialize') {
    const sessionId = `session-${sessionIdCounter++}`;
    sessions.set(sessionId, { tools: [...] });

    res.setHeader('Mcp-Session-Id', sessionId);
    return res.json({
      jsonrpc: '2.0',
      id,
      result: {
        protocolVersion: '2025-03-26',
        capabilities: {},
        serverInfo: { name: 'test-server', version: '1.0.0' }
      }
    });
  }

  // List tools
  if (method === 'tools/list') {
    const session = sessions.get(req.headers['mcp-session-id']);
    return res.json({
      jsonrpc: '2.0',
      id,
      result: { tools: session.tools }
    });
  }

  // Call tool
  if (method === 'tools/call') {
    // Return SSE stream
    res.setHeader('Content-Type', 'text/event-stream');
    res.write(`data: ${JSON.stringify({ jsonrpc: '2.0', id, result: {...} })}\n\n`);
    res.end();
    return;
  }
});

app.get('/mcp', (req, res) => {
  // SSE stream for server messages
  res.setHeader('Content-Type', 'text/event-stream');
  // Send periodic notifications
});

app.listen(3000, () => console.log('MCP HTTP server on :3000'));
```

Run with:
```bash
node test-mcp-http-server.js
```

---

**Document Status:** Ready for review
**Next Steps:** Awaiting approval to begin implementation
