# Epic Technical Specification: RabbitMQ Connection Management

Date: 2025-12-29
Author: Luciano
Epic ID: epic-2
Status: Draft

---

## Overview

This epic establishes the foundational connection management infrastructure for the RabbitMQ MCP server, enabling robust communication with RabbitMQ through both AMQP 0-9-1 protocol (for messaging operations) and HTTP Management API (for topology operations). The implementation delivers production-grade reliability through flexible configuration management, automatic reconnection with exponential backoff, connection health monitoring, and comprehensive TLS/SSL support. This epic directly addresses FR-006 (Connection Management) and FR-007 (Auto-Reconnection) from the PRD, ensuring the MCP server maintains stable connections even in unstable network conditions and automatically recovers from connection disruptions without manual intervention.

The scope encompasses a complete configuration management system with multi-source precedence (CLI args → env vars → TOML → defaults), dual-protocol connection establishment (AMQP + HTTP), proactive health checks with automatic failure detection, intelligent reconnection logic with exponential backoff (1s, 2s, 4s, 8s, 16s, 32s, then 60s max), HTTP connection pooling with cache invalidation strategies, and enterprise-grade TLS/SSL certificate handling with configurable verification modes.

## Objectives and Scope

**In Scope:**
- Configuration management system supporting CLI args, environment variables, TOML config files, and built-in defaults
- AMQP 0-9-1 protocol connection establishment using pika library with authentication and TLS support
- HTTP Management API client using httpx with async support, connection pooling (5 connections default), and Basic Auth
- Periodic health checks (<1 second) for both AMQP and HTTP connections with automatic failure detection
- Automatic reconnection with exponential backoff (1s → 2s → 4s → 8s → 16s → 32s → 60s max) and indefinite retry until recovery
- HTTP connection pooling (configurable, default 5 connections) with intelligent cache invalidation on stale connection detection, health failures, configuration changes, and proactive refresh
- TLS/SSL certificate handling for both AMQP and HTTP with configurable verification (default enabled) and custom CA certificate support
- Secure credential management with automatic sanitization in logs and error messages
- Connection state tracking (DISCONNECTED, CONNECTING, CONNECTED, RECONNECTING) and metrics exposure

**Out of Scope:**
- AMQP message publishing and consumption (Epic 4)
- Topology operations (queues, exchanges, bindings) (Epic 3)
- CLI interface implementation (Epic 5)
- Advanced observability features beyond basic connection health (Epic 7)
- Performance benchmarking and optimization (Epic 6)

## System Architecture Alignment

This epic implements critical infrastructure components defined in the Architecture's Technology Stack Details and aligns with multiple Architecture Decision Records:

**Core Components:**
- **Connection Module** (`src/rabbitmq_mcp_connection/connection.py`): AMQP connection establishment using pika library with auto-reconnection state machine
- **HTTP Client Module** (`src/rabbitmq_mcp_connection/http_client.py`): Async HTTP client using httpx with connection pooling and Basic Auth
- **Configuration Module** (`src/config/settings.py`): Pydantic BaseSettings for multi-source configuration with validation and SecretStr protection
- **Health Check Module** (`src/rabbitmq_mcp_connection/health.py`): Background asyncio task for periodic connection health verification

**Architecture Constraints:**
- Python 3.12+ with type hints (mandatory per Consistency Rules)
- Async/await pattern for all I/O operations (per Consistency Rules)
- Pydantic for all configuration validation (ADR-008)
- Automatic credential sanitization in all logs (ADR-006)
- Stateless server design (ADR-005) - no connection state shared between instances
- Structured logging with structlog (ADR-009) for all connection lifecycle events

**Integration Points:**
- RabbitMQ AMQP 0-9-1 protocol (default port 5672)
- RabbitMQ Management API HTTP REST (default port 15672)
- Configuration sources: CLI args → env vars → config.toml → .env → defaults
- Logging pipeline: structlog → sanitizers → correlation ID → file handler
- Health checks exposed via MCP tools for client visibility

**Security Alignment:**
- TLS/SSL support mandatory for production (Security Architecture)
- Credentials via environment variables or encrypted config files (never hardcoded)
- Automatic credential sanitization with 100% coverage (Security Architecture)
- File permissions 600 for config files, 700 for config directory
- Bearer token support for HTTP transport (Security Architecture)

## Detailed Design

### Services and Modules

| Module | Responsibilities | Inputs | Outputs | Owner |
|--------|------------------|--------|---------|-------|
| **ConfigurationManager** (`config/settings.py`) | Load and validate configuration from multiple sources with precedence: CLI args > env vars > TOML > defaults. Validate parameters, provide defaults, protect sensitive values with SecretStr. | CLI args (argparse), environment variables (os.environ), TOML file path (configurable), .env file | Validated Settings object with all connection parameters, credential sanitization applied | Config Module |
| **AMQPConnectionManager** (`rabbitmq_mcp_connection/connection.py`) | Establish and maintain AMQP 0-9-1 connection using pika.BlockingConnection or pika.SelectConnection. Handle authentication, TLS/SSL, vhost targeting. Implement connection state machine: DISCONNECTED → CONNECTING → CONNECTED → RECONNECTING. Expose connection health status. | Settings object (AMQP_HOST, AMQP_PORT, AMQP_USER, AMQP_PASSWORD, AMQP_VHOST, AMQP_USE_TLS, AMQP_VERIFY_CERT, AMQP_CA_BUNDLE) | pika.Connection instance, connection state enum, health status boolean | Connection Module |
| **HTTPClientManager** (`rabbitmq_mcp_connection/http_client.py`) | Create and manage httpx.AsyncClient for Management API operations. Configure base URL (http(s)://host:port/api), HTTP Basic Auth, TLS/SSL, connection pooling (default 5 connections), timeouts (30s default). Handle 401 Unauthorized, 404 Not Found errors gracefully. | Settings object (HTTP_HOST, HTTP_PORT, HTTP_USER, HTTP_PASSWORD, HTTP_USE_TLS, HTTP_VERIFY_CERT, HTTP_CA_BUNDLE, MAX_HTTP_CONNECTIONS) | httpx.AsyncClient instance with connection pool, request/response handlers with correlation IDs | HTTP Client Module |
| **HealthCheckService** (`rabbitmq_mcp_connection/health.py`) | Periodic background health checks for AMQP and HTTP connections. Verify AMQP via pika.connection.is_open, verify HTTP via GET /api/healthchecks/node. Track consecutive failures (3+ trigger reconnection). Expose health status: {amqp_connected, http_connected, last_check}. Run as asyncio.create_task with configurable interval (default 30s). | AMQPConnectionManager instance, HTTPClientManager instance, HEALTH_CHECK_INTERVAL (env var) | Health status dict, failure count, metrics (successful_checks, failed_checks, avg_check_duration) | Health Module |
| **ReconnectionManager** (`rabbitmq_mcp_connection/reconnect.py`) | Automatic reconnection with exponential backoff on connection loss. Backoff schedule: 1s, 2s, 4s, 8s, 16s, 32s, then 60s max. Attempt indefinitely until successful. Re-establish both AMQP and HTTP connections. Broadcast reconnection events. Reset backoff after successful connection >5 minutes. | Connection loss event (from HealthCheckService or connection error), AMQPConnectionManager, HTTPClientManager | Reconnection state {state, attempt, next_attempt_in}, downtime metrics, success/failure logs | Reconnection Module |
| **ConnectionPoolManager** (`rabbitmq_mcp_connection/pool.py`) | Manage HTTP connection pool with intelligent cache invalidation. Pool size configurable (default 5). Idle timeout 60s. Eviction triggers: HTTP 503, connection timeout, socket errors, SSL errors, health check failures, configuration changes. Proactive refresh via background validation. Track pool metrics: active_connections, pool_size, evictions, health_score. | HTTPClientManager, MAX_HTTP_CONNECTIONS (env var), pool health check interval (30s) | Pool status, eviction events, pool health metrics, connection lifecycle logs | Connection Module |
| **TLSContextManager** (`rabbitmq_mcp_connection/tls.py`) | Create SSL contexts for AMQP and HTTP with TLS 1.2+ protocol. Configure certificate verification (default enabled). Load custom CA certificates from CA_CERT_PATH. Warn when verification disabled. | AMQP_USE_TLS, AMQP_VERIFY_CERT, HTTP_USE_TLS, HTTP_VERIFY_CERT, CA_CERT_PATH | ssl.SSLContext instances for AMQP and HTTP, TLS configuration logs | Security Module |

### Data Models and Contracts

**Configuration Schema (Pydantic BaseSettings):**
```python
class Settings(BaseSettings):
    # AMQP Configuration
    amqp_host: str = "localhost"
    amqp_port: int = 5672
    amqp_user: str = "guest"
    amqp_password: SecretStr = SecretStr("guest")  # Protected from logging
    amqp_vhost: str = "/"
    amqp_use_tls: bool = False
    amqp_verify_cert: bool = True
    amqp_ca_bundle: Optional[str] = None
    
    # HTTP Management API Configuration
    http_host: str = "localhost"
    http_port: int = 15672
    http_user: str = "guest"
    http_password: SecretStr = SecretStr("guest")  # Protected from logging
    http_use_tls: bool = False
    http_verify_cert: bool = True
    http_ca_bundle: Optional[str] = None
    max_http_connections: int = 5
    
    # Health Check Configuration
    health_check_interval: int = 30  # seconds
    health_check_timeout: int = 1    # seconds
    
    # Reconnection Configuration
    reconnect_max_delay: int = 60    # seconds
    reconnect_reset_after: int = 300 # seconds (5 minutes)
    
    # General Configuration
    config_file: Optional[str] = None
    log_level: str = "INFO"
    
    class Config:
        env_file = ".env"
        env_prefix = ""  # Allow both prefixed and non-prefixed env vars
```

**Connection State Enum:**
```python
class ConnectionState(Enum):
    DISCONNECTED = "disconnected"
    CONNECTING = "connecting"
    CONNECTED = "connected"
    RECONNECTING = "reconnecting"
```

**Health Status Model:**
```python
class HealthStatus(BaseModel):
    amqp_connected: bool
    http_connected: bool
    last_check_timestamp: datetime
    consecutive_failures: int
    amqp_state: ConnectionState
    http_state: ConnectionState
```

**Pool Health Model:**
```python
class PoolHealth(BaseModel):
    size: int
    active: int
    idle: int
    evictions_last_hour: int
    avg_connection_age_seconds: float
    health_score: float  # successful_requests / total_requests
```

**Reconnection State Model:**
```python
class ReconnectionState(BaseModel):
    state: str  # "reconnecting", "connected", "failed"
    attempt: int
    next_attempt_in_seconds: Optional[int]
    total_downtime_seconds: float
    last_reconnect_timestamp: Optional[datetime]
```

### APIs and Interfaces

**AMQP Connection API:**
```python
class AMQPConnectionManager:
    async def connect(self) -> None:
        """
        Establish AMQP connection with timeout (<5s).
        Raises: ConnectionError if unable to connect within timeout.
        """
    
    async def disconnect(self) -> None:
        """Close AMQP connection gracefully."""
    
    def is_connected(self) -> bool:
        """Check if AMQP connection is open."""
    
    def get_state(self) -> ConnectionState:
        """Return current connection state."""
    
    async def get_channel(self) -> pika.channel.Channel:
        """
        Get AMQP channel for operations.
        Raises: ConnectionError if not connected.
        """
```

**HTTP Client API:**
```python
class HTTPClientManager:
    async def request(
        self,
        method: str,
        path: str,
        params: Optional[dict] = None,
        json: Optional[dict] = None,
        timeout: int = 30
    ) -> httpx.Response:
        """
        Execute HTTP request to Management API.
        
        Args:
            method: HTTP method (GET, POST, PUT, DELETE)
            path: API path (e.g., "/queues/%2F")
            params: Query parameters
            json: Request body (JSON)
            timeout: Request timeout in seconds
        
        Returns:
            httpx.Response with status_code, json(), headers
        
        Raises:
            httpx.HTTPStatusError: For 4xx/5xx responses
            httpx.TimeoutException: If request exceeds timeout
        """
    
    async def close(self) -> None:
        """Close HTTP client and connection pool."""
    
    def get_pool_status(self) -> PoolHealth:
        """Return current connection pool health metrics."""
```

**Health Check API:**
```python
class HealthCheckService:
    async def start(self) -> None:
        """Start background health check task."""
    
    async def stop(self) -> None:
        """Stop background health check task."""
    
    def get_status(self) -> HealthStatus:
        """Return current health status."""
    
    def is_healthy(self) -> bool:
        """Return True if both AMQP and HTTP are connected."""
```

**Reconnection API:**
```python
class ReconnectionManager:
    async def handle_connection_loss(self) -> None:
        """
        Handle connection loss event and trigger reconnection.
        Implements exponential backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s max.
        Attempts indefinitely until successful.
        """
    
    def get_state(self) -> ReconnectionState:
        """Return current reconnection state."""
    
    async def force_reconnect(self) -> None:
        """Manually trigger reconnection (useful for testing)."""
```

### Workflows and Sequencing

**1. Server Startup and Connection Establishment:**
```
1. Parse CLI arguments (argparse)
2. Load configuration with precedence: CLI args > env vars > config.toml > .env > defaults
3. Validate configuration with Pydantic (fail fast if invalid)
4. Create TLS contexts if TLS enabled
5. Initialize AMQPConnectionManager with settings
6. Connect to RabbitMQ AMQP (timeout 5s)
   - If failure: Log error and exit (fail-fast pattern)
7. Initialize HTTPClientManager with settings
8. Create HTTP client with connection pool (5 connections)
9. Test HTTP connection with GET /api/overview
   - If failure: Log error and exit (fail-fast pattern)
10. Start HealthCheckService background task
11. Log: "Connected to RabbitMQ at {host}:{port} vhost={vhost}"
12. Server ready to accept MCP operations
```

**Sequence Diagram:**
```mermaid
sequenceDiagram
    participant Main
    participant Config as ConfigurationManager
    participant AMQP as AMQPConnectionManager
    participant HTTP as HTTPClientManager
    participant Health as HealthCheckService
    participant RabbitMQ

    Main->>Config: Load configuration (CLI/env/TOML)
    Config-->>Main: Settings object
    Main->>AMQP: Initialize with settings
    Main->>AMQP: connect()
    AMQP->>RabbitMQ: AMQP connection (port 5672)
    RabbitMQ-->>AMQP: Connection established
    Main->>HTTP: Initialize with settings
    Main->>HTTP: Test connection
    HTTP->>RabbitMQ: GET /api/overview (port 15672)
    RabbitMQ-->>HTTP: 200 OK
    Main->>Health: Start background task
    Health-->>Main: Task started
    Main->>Main: Log "Connected to RabbitMQ"
    Main->>Main: Ready for MCP operations
```


**2. Health Check Workflow (Every 30s):**
```
1. Check AMQP connection: pika.connection.is_open
   - If False: Increment failure count, log warning
2. Check HTTP connection: GET /api/healthchecks/node (1s timeout)
   - If request fails or status != 200: Increment failure count, log warning
3. Update health status: {amqp_connected, http_connected, last_check, consecutive_failures}
4. If consecutive_failures >= 3:
   - Log: "Health check failed 3 consecutive times, triggering reconnection"
   - Signal ReconnectionManager to handle connection loss
5. If health check successful: Reset consecutive_failures to 0
6. Update metrics: successful_checks++, avg_check_duration
7. Wait for next interval (30s)
```

**Sequence Diagram:**
```mermaid
sequenceDiagram
    participant Health as HealthCheckService
    participant AMQP as AMQPConnectionManager
    participant HTTP as HTTPClientManager
    participant Reconnect as ReconnectionManager
    participant RabbitMQ

    loop Every 30s
        Health->>AMQP: is_connected()
        AMQP-->>Health: True/False
        Health->>HTTP: GET /api/healthchecks/node
        HTTP->>RabbitMQ: Health check request
        RabbitMQ-->>HTTP: 200 OK / Error
        HTTP-->>Health: Response
        Health->>Health: Update health status
        alt consecutive_failures >= 3
            Health->>Reconnect: Trigger reconnection
        else Health check OK
            Health->>Health: Reset failure count
        end
        Health->>Health: Wait 30s
    end
```

**3. Connection Loss and Reconnection Workflow:**
```
1. Connection loss detected (health check failure or connection error)
2. Log: "Connection lost, initiating reconnection..."
3. Set connection state to RECONNECTING
4. Start exponential backoff loop:
   - Attempt 1: Wait 1s, try reconnect
   - Attempt 2: Wait 2s, try reconnect
   - Attempt 3: Wait 4s, try reconnect
   - Attempt 4: Wait 8s, try reconnect
   - Attempt 5: Wait 16s, try reconnect
   - Attempt 6: Wait 32s, try reconnect
   - Attempt 7+: Wait 60s, try reconnect (max backoff)
5. For each attempt:
   - Log: "Reconnection attempt {n} in {delay}s..."
   - Try to establish AMQP connection
   - Try to establish HTTP connection
   - If both successful:
     - Log: "Reconnected to RabbitMQ successfully"
     - Set state to CONNECTED
     - Reset backoff to 1s (after 5 minutes of stable connection)
     - Exit loop
   - If partial success (AMQP ok, HTTP failed or vice versa):
     - Log warning, continue retrying
6. In-flight operations during reconnection:
   - Return error: "Connection lost, please retry after reconnection"
7. Track metrics: reconnection_attempts, total_downtime, last_reconnect_timestamp
```

**Sequence Diagram:**
```mermaid
sequenceDiagram
    participant Reconnect as ReconnectionManager
    participant AMQP as AMQPConnectionManager
    participant HTTP as HTTPClientManager
    participant RabbitMQ

    Note over Reconnect: Connection loss detected
    Reconnect->>Reconnect: Set state RECONNECTING
    loop Exponential Backoff (1s, 2s, 4s, 8s, 16s, 32s, 60s max)
        Reconnect->>Reconnect: Wait {backoff_delay}s
        Reconnect->>AMQP: Attempt AMQP connect
        AMQP->>RabbitMQ: Connection attempt
        alt Success
            RabbitMQ-->>AMQP: Connected
            Reconnect->>HTTP: Attempt HTTP connect
            HTTP->>RabbitMQ: Connection test
            alt Both successful
                RabbitMQ-->>HTTP: 200 OK
                Reconnect->>Reconnect: Set state CONNECTED
                Reconnect->>Reconnect: Log "Reconnected successfully"
            else HTTP failed
                Reconnect->>Reconnect: Continue retrying
            end
        else AMQP failed
            RabbitMQ-->>AMQP: Connection refused
            Reconnect->>Reconnect: Increment attempt, continue
        end
    end
```


**4. HTTP Connection Pool Management Workflow:**
```
1. Client requests HTTP operation
2. Pool checks for available connection:
   - If available: Reuse connection from pool
   - If pool full: Block with 10s timeout (configurable)
   - If timeout: Log warning "HTTP connection pool exhausted"
3. Execute HTTP request with correlation ID
4. On completion: Return connection to pool
5. Background pool health validation (every 30s):
   - Iterate idle connections
   - Send lightweight GET /api/overview
   - If validation fails: Evict connection from pool
   - Log: "Evicted stale connection from pool"
6. Connection eviction triggers:
   - HTTP 503 Service Unavailable: Immediate eviction
   - Connection timeout (30s): Immediate eviction
   - Socket error (ECONNRESET, EPIPE): Immediate eviction
   - SSL/TLS error: Immediate eviction
   - 3 consecutive errors on same connection: Force eviction
   - Connection age >5 minutes + idle >60s: Eligible for eviction
7. Configuration change detection:
   - On SIGHUP or config file change: Flush entire pool
   - Re-create connections with new settings
8. Track pool metrics:
   - eviction_rate (evictions per hour)
   - pool_health_score = successful_requests / total_requests
   - Alert if health_score < 0.90
```

**5. TLS/SSL Certificate Handling Workflow:**
```
1. Check if TLS enabled (AMQP_USE_TLS or HTTP_USE_TLS)
2. If TLS disabled:
   - Log warning: "TLS disabled - use only for development"
   - Continue with plaintext connection
3. If TLS enabled:
   - Create SSL context with ssl.create_default_context()
   - Set minimum TLS version: TLS 1.2+
   - If custom CA certificate specified (CA_CERT_PATH):
     - Load CA certificate: ssl_context.load_verify_locations(cafile=ca_cert_path)
   - If certificate verification disabled (VERIFY_CERT=false):
     - Set cert_reqs=ssl.CERT_NONE
     - Log warning: "Certificate verification disabled - insecure mode"
   - If certificate verification enabled (default):
     - Set cert_reqs=ssl.CERT_REQUIRED
     - Log: "Using TLS/SSL with certificate verification"
4. Apply SSL context to AMQP connection or HTTP client
5. On TLS handshake failure:
   - Log error: "SSL certificate verification failed: {details}"
   - Fail connection attempt (trigger reconnection if auto-reconnect enabled)
```

## Non-Functional Requirements

### Performance

**Latency Targets:**
- Configuration loading: <100ms at startup (FR-006)
- AMQP connection establishment: <5 seconds when server available (FR-006)
- HTTP connection establishment: <5 seconds when server available
- Health check execution: <1 second (95th percentile) (FR-006)
- Reconnection after server recovery: <10 seconds (FR-007)
- HTTP request via connection pool: <30ms overhead (connection reuse)

**Throughput:**
- Configuration validation: Process all parameters in <100ms
- Health checks: Execute reliably every 30 seconds without drift
- Connection pool: Support 100+ concurrent requests without exhaustion
- Logging overhead: <5ms per connection lifecycle event (FR-016)

**Resource Constraints:**
- Memory: <50MB for connection management modules
- CPU: <5% during steady state (health checks only)
- Network: Minimal overhead (health checks <1KB per check)
- Connection pool: Default 5 HTTP connections, configurable up to 20

### Security

**Authentication:**
- AMQP: Username/password authentication (configurable via AMQP_USER, AMQP_PASSWORD)
- HTTP: HTTP Basic Auth using configured credentials (HTTP_USER, HTTP_PASSWORD)
- Credentials stored as Pydantic SecretStr to prevent accidental logging
- Environment variables recommended over config files for production

**Credential Protection (100% Coverage):**
- Automatic sanitization in all logs using regex patterns
- Patterns detect: passwords, tokens, API keys, authorization headers, connection strings
- Sanitization applied to: structured logs, error messages, stack traces, configuration validation errors
- Replacement: `[REDACTED]` for all detected sensitive values
- Log files: 600 permissions (owner read/write only)
- Log directory: 700 permissions (owner access only)

**Network Security:**
- TLS/SSL support for AMQP and HTTP connections (configurable)
- TLS 1.2+ enforced when enabled
- Certificate verification enabled by default (VERIFY_CERT=true)
- Self-signed certificates supported via VERIFY_CERT=false (logs warning)
- Custom CA certificates via CA_CERT_PATH environment variable
- Warning logged when TLS disabled: "WARNING: TLS disabled - use only for development"

**Audit Trail:**
- All connection lifecycle events logged: connect, disconnect, reconnect, health check failure
- Logs include: timestamp, correlation_id, event, result, error details (sanitized)
- Correlation IDs propagated across all connection-related operations
- Audit logs retained minimum 30 days (configurable via LOG_RETENTION_DAYS)

### Reliability/Availability

**Automatic Reconnection (FR-007):**
- Exponential backoff: 1s, 2s, 4s, 8s, 16s, 32s, then 60s max
- Infinite retry attempts until connection restored
- Backoff reset after 5 minutes of stable connection
- Both AMQP and HTTP connections re-established on reconnection
- In-flight operations fail gracefully during reconnection with clear error message

**Health Monitoring:**
- Periodic health checks every 30 seconds (configurable via HEALTH_CHECK_INTERVAL)
- AMQP health: pika.connection.is_open check
- HTTP health: GET /api/healthchecks/node request
- 3+ consecutive failures trigger automatic reconnection
- Health status exposed to clients for visibility

**Connection Pool Reliability:**
- Stale connection detection and automatic eviction
- Proactive validation of idle connections (every 30s)
- Circuit breaker: 5 consecutive pool-wide failures trigger reconnection
- Pool exhaustion handled with 10s timeout and clear warning
- Pool health score tracked: alert if <0.90

**Error Handling:**
- Connection failures: Immediate error with clear message, trigger reconnection
- Authentication failures: Return error "Authentication failed: {details}"
- Timeout failures: Return error "Connection timeout after 30s"
- TLS handshake failures: Return error "SSL certificate verification failed: {details}"
- Configuration validation failures: Fail fast at startup with corrective action

### Observability

**Structured Logging (100% JSON):**
- All connection events logged with structlog
- Log levels: ERROR (connection failures), WARN (health check failures), INFO (connections, reconnections), DEBUG (detailed flow)
- Correlation IDs generated and bound to all logs
- Log format: `{"timestamp": "...", "level": "...", "event": "...", "correlation_id": "...", "details": {...}}`
- Log files: `./logs/rabbitmq-mcp-{date}.log` with daily rotation + 100MB size trigger

**Metrics Tracking:**
- Connection lifecycle: connection_attempts, successful_connections, failed_connections, reconnection_attempts
- Health checks: successful_checks, failed_checks, avg_check_duration
- Connection pool: active_connections, idle_connections, evictions_per_hour, pool_health_score
- Reconnection: total_downtime, last_reconnect_timestamp, current_backoff_delay

**Distributed Tracing (OpenTelemetry):**
- All connection operations traced with correlation IDs
- Spans: connection_establishment, health_check, reconnection_attempt, http_request
- Trace propagation across AMQP and HTTP operations
- OTLP exporter for trace aggregation (Jaeger, Prometheus compatible)
- Example OTLP collector endpoint: `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317`

**Status Exposure:**
- Health status API: `connection.health` returns {amqp_connected, http_connected, last_check}
- Pool status API: `connection.pool_status` returns {size, active, idle, evictions, health_score}
- Reconnection state API: `connection.reconnection_state` returns {state, attempt, next_attempt_in}

## Dependencies and Integrations

**Core Dependencies (from pyproject.toml):**
| Dependency | Version | Purpose | Notes |
|------------|---------|---------|-------|
| pika | >=1.3 | AMQP 0-9-1 client for RabbitMQ | Supports BlockingConnection and SelectConnection, heartbeat handling, auto-reconnection hooks |
| aio-pika | >=9.5,<11 | Async AMQP client (if async needed) | Alternative to pika for async operations, compatible with asyncio event loop |
| httpx | >=0.27 | Async HTTP client for Management API | Built-in connection pooling, HTTP/2 support, timeout handling, AsyncClient for async operations |
| pydantic | >=2.0 | Configuration validation and data models | SecretStr for credential protection, BaseSettings for multi-source config, custom validators |
| pydantic-settings | >=2.0 | Multi-source configuration loading | Environment variables, .env files, TOML support, precedence handling |
| structlog | >=24.1 | Structured JSON logging | Processors: timestamper, dict_tracebacks, JSONRenderer, context binding for correlation IDs |
| opentelemetry-api | >=1.22 | Distributed tracing API | Span creation, trace propagation, correlation IDs |
| opentelemetry-sdk | >=1.22 | OpenTelemetry implementation | OTLP exporter, tracer provider, resource attributes |
| opentelemetry-instrumentation | >=0.43b0 | Automatic instrumentation | HTTP request tracing, AMQP operation tracing |

**Development Dependencies (for testing):**
| Dependency | Version | Purpose |
|------------|---------|---------|
| pytest | >=8.0 | Test framework |
| pytest-asyncio | >=0.23 | Async test support |
| testcontainers | >=3.7 | RabbitMQ Docker containers for integration tests (uses rabbitmq:3.13-management image) |
| pytest-mock | >=3.12 | Mocking for connection tests |
| pytest-cov | >=4.1 | Code coverage reporting |

**External Integrations:**
- **RabbitMQ Server**: Version 3.11.x, 3.12.x, or 3.13.x required
  - AMQP port: 5672 (default, configurable)
  - Management API port: 15672 (default, configurable)
  - Management API plugin must be enabled
  - Credentials with appropriate permissions for vhost operations
- **TLS/SSL Certificates**: If TLS enabled, valid certificates required
  - Server certificates for RabbitMQ
  - Optional: Custom CA certificates for corporate environments
  - Self-signed certificates supported for development (with warning)
- **Configuration Files**: TOML parser (tomli or tomllib for Python 3.11+)
  - Config file location: `./config/config.toml` or path from `CONFIG_FILE` env var
  - Example provided: `config.toml.example`
- **Environment Variables**: All configuration parameters available as env vars
  - Recommended for production: Store credentials in env vars or secure vault
  - Precedence: CLI args > env vars > TOML > .env > defaults

**Version Constraints:**
- Python 3.12+ required (per pyproject.toml)
- RabbitMQ 3.11.x minimum (Management API compatibility)
- pika 1.3+ for Python 3.12 compatibility
- httpx 0.27+ for AsyncClient improvements
- Pydantic 2.0+ for performance and validation features

## Acceptance Criteria (Authoritative)

**Story 2.1: Configuration Management System**
1. Configuration loaded with precedence: CLI args > environment variables > TOML config file > built-in defaults
2. Supported parameters: AMQP_HOST, AMQP_PORT, AMQP_USER, AMQP_PASSWORD, AMQP_VHOST, AMQP_USE_TLS, AMQP_VERIFY_CERT, HTTP_HOST, HTTP_PORT, HTTP_USER, HTTP_PASSWORD, HTTP_USE_TLS, MAX_HTTP_CONNECTIONS
3. TOML config file location: `./config/config.toml` or path from `CONFIG_FILE` env var
4. Sensitive values (passwords) never logged in plaintext (SecretStr protection)
5. Configuration validation at startup with clear error messages (Pydantic validation)
6. Missing required parameters return error: "Required parameter 'AMQP_HOST' not configured"
7. Configuration loading completes in <100ms

**Story 2.2: AMQP Connection Establishment**
8. Connection succeeds within 5 seconds when RabbitMQ available (FR-006)
9. Connection uses pika library with AMQP 0-9-1 protocol
10. Authentication uses configured username and password
11. Connection targets configured vhost (default: "/")
12. TLS/SSL enabled if AMQP_USE_TLS=true with certificate verification based on AMQP_VERIFY_CERT
13. Connection failures return error: "Failed to connect to RabbitMQ at {host}:{port}: {error_details}"
14. Connection timeout is 30 seconds (configurable)
15. Successful connection logs: "Connected to RabbitMQ at {host}:{port} vhost={vhost}"

**Story 2.3: HTTP Management API Client**
16. Client configured with base URL: http(s)://{host}:{port}/api
17. HTTP Basic Auth header set using configured username and password
18. TLS/SSL enabled if HTTP_USE_TLS=true
19. Certificate verification follows HTTP_VERIFY_CERT setting
20. Client timeout is 30 seconds (configurable)
21. Client includes User-Agent header: "rabbitmq-mcp-server/{version}"
22. Connection pooling maintains up to 5 concurrent connections (configurable)
23. Client retries on connection errors: 0 retries (fail-fast pattern)

**Story 2.4: Connection Health Checks**
24. AMQP connection health verified via pika.connection.is_open check
25. HTTP API health verified via GET /api/healthchecks/node request
26. Health checks complete in <1 second (FR-006)
27. Health checks run every 30 seconds (configurable via HEALTH_CHECK_INTERVAL)
28. Failed health checks log warning: "Health check failed: {reason}"
29. Consecutive failures (3+) trigger reconnection logic
30. Health status exposed: {amqp_connected: bool, http_connected: bool, last_check: timestamp}
31. Health status available via MCP tool: "connection.health"

**Story 2.5: Automatic Reconnection with Exponential Backoff**
32. Reconnection attempts follow exponential backoff: 1s, 2s, 4s, 8s, 16s, 32s, then 60s max (FR-007)
33. Reconnection attempts continue indefinitely until successful (FR-007)
34. Each attempt logs: "Reconnection attempt {n} in {delay}s..."
35. Successful reconnection completes in <10 seconds after RabbitMQ becomes available (FR-007)
36. Reconnection re-establishes both AMQP and HTTP connections
37. In-flight operations receive error: "Connection lost, please retry after reconnection"
38. After reconnection, server logs: "Reconnected to RabbitMQ successfully"
39. Reconnection state exposed: {state: "reconnecting", attempt: 5, next_attempt_in: "32s"}
40. Backoff reset to 1s after successful connection lasting >5 minutes

**Story 2.6: Connection Pooling for HTTP with Cache Invalidation Strategy**
41. HTTP client reuses existing connections from pool (up to 5 connections)
42. Connection pool blocks when full with 10s timeout (configurable)
43. Idle connections closed after 60s (connection timeout)
44. Pool size configurable via MAX_HTTP_CONNECTIONS env var
45. Pool metrics tracked: active_connections, pool_size, wait_time
46. Pool exhaustion logs warning: "HTTP connection pool exhausted, waiting for available connection"
47. Pool status exposed via "connection.pool_status" operation
48. Stale connections automatically detected and evicted based on error patterns (HTTP 503, timeout, socket error, SSL error)
49. Configuration changes trigger pool flush and reconnection
50. Connection health validated proactively via background checks (every 30s)
51. Pool health score tracked: alert if <0.90 (successful_requests / total_requests)

**Story 2.7: TLS/SSL Certificate Handling**
52. AMQP connection uses SSL context with TLS 1.2+ protocol when AMQP_USE_TLS=true
53. HTTP client uses HTTPS with TLS 1.2+ protocol when HTTP_USE_TLS=true
54. Certificate verification enabled by default (VERIFY_CERT=true)
55. Self-signed certificates allowed via VERIFY_CERT=false (logs warning)
56. Custom CA certificates specified via CA_CERT_PATH env var
57. Certificate verification failures return error: "SSL certificate verification failed: {details}"
58. TLS handshake completes within connection timeout (30s)
59. TLS connections log: "Using TLS/SSL for {protocol} connection (verify={verify_mode})"

## Traceability Mapping

| Acceptance Criteria | Spec Section | Component/API | Test Idea |
|---------------------|--------------|---------------|-----------|
| AC 1-7 (Configuration) | Services: ConfigurationManager | `config/settings.py`, Pydantic BaseSettings | Unit: Test precedence order with mock sources; Integration: Load from actual TOML and env vars |
| AC 8-15 (AMQP Connection) | Services: AMQPConnectionManager | `rabbitmq_mcp_connection/connection.py`, pika.BlockingConnection | Integration: Connect to testcontainer RabbitMQ, test timeout, auth failure, TLS |
| AC 16-23 (HTTP Client) | Services: HTTPClientManager | `rabbitmq_mcp_connection/http_client.py`, httpx.AsyncClient | Integration: Test Management API requests, pool behavior, auth, TLS |
| AC 24-31 (Health Checks) | Services: HealthCheckService | `rabbitmq_mcp_connection/health.py`, asyncio.create_task | Integration: Simulate connection failure, verify health check detects it in <30s |
| AC 32-40 (Reconnection) | Services: ReconnectionManager | `rabbitmq_mcp_connection/reconnect.py`, exponential backoff logic | Integration: Kill RabbitMQ container, verify reconnection with backoff schedule |
| AC 41-51 (Connection Pool) | Services: ConnectionPoolManager | `rabbitmq_mcp_connection/pool.py`, httpx connection pool | Unit: Mock pool exhaustion, eviction triggers; Integration: Concurrent requests, stale connection eviction |
| AC 52-59 (TLS/SSL) | Services: TLSContextManager | `rabbitmq_mcp_connection/tls.py`, ssl.create_default_context | Integration: Test with self-signed cert, custom CA, verify mode on/off |

## Risks, Assumptions, Open Questions

**Risks:**
1. **Risk**: pika library has known issues with async event loops in some Python versions
   - **Mitigation**: Use pika 1.3+ with Python 3.12+ for best compatibility; consider aio-pika for async if issues arise
   - **Contingency**: Fallback to aio-pika if pika async issues encountered

2. **Risk**: Exponential backoff may cause long reconnection times (up to 60s between attempts)
   - **Mitigation**: Backoff schedule tested in integration tests to ensure reasonable recovery time; 60s max is acceptable for production
   - **Monitoring**: Track reconnection metrics to detect prolonged downtime

3. **Risk**: Connection pool exhaustion under high load may cause request failures
   - **Mitigation**: Pool size configurable (default 5, increase if needed); 10s timeout prevents indefinite blocking
   - **Monitoring**: Alert on pool exhaustion warnings; track pool_health_score

4. **Risk**: TLS certificate verification failures may prevent connections in corporate environments
   - **Mitigation**: Support custom CA certificates via CA_CERT_PATH; VERIFY_CERT=false for development
   - **Documentation**: Provide clear instructions for certificate setup in README

5. **Risk**: Health checks may generate excessive load on RabbitMQ if interval too short
   - **Mitigation**: Default 30s interval is conservative; configurable via HEALTH_CHECK_INTERVAL
   - **Testing**: Verify health check overhead <1% CPU with testcontainer

**Assumptions:**
1. **Assumption**: RabbitMQ Management API plugin is enabled on target server
   - **Validation**: Check at startup, fail with clear error if plugin not available

2. **Assumption**: Credentials have sufficient permissions for vhost operations
   - **Validation**: Test connection at startup, fail with clear error if unauthorized

3. **Assumption**: Network between MCP server and RabbitMQ is reliable (>95% uptime)
   - **Validation**: Reconnection logic handles transient failures gracefully

4. **Assumption**: Configuration file (config.toml) is valid TOML format
   - **Validation**: Pydantic validation catches format errors, provides clear message

5. **Assumption**: Python 3.12+ is available in deployment environment
   - **Validation**: pyproject.toml requires-python constraint enforced by uv

**Open Questions:**
1. **Question**: Should we support connection pooling for AMQP (in addition to HTTP)?
   - **Decision Needed**: Epic 4 (Message Publishing & Consumption) may require AMQP pool
   - **Impact**: If yes, add AMQPConnectionPool in Epic 4

2. **Question**: Should health checks be exposed as Prometheus metrics endpoint?
   - **Decision Needed**: Epic 7 (Structured Logging & Observability) covers this
   - **Impact**: If yes, defer to Epic 7

3. **Question**: Should reconnection backoff be configurable or hardcoded?
   - **Decision Needed**: Current design is hardcoded with 60s max; consider making configurable
   - **Impact**: Add RECONNECT_MAX_DELAY env var if needed

4. **Question**: Should we support multiple RabbitMQ servers (failover/load balancing)?
   - **Decision Needed**: Out of scope for MVP (Phase 1); consider for Phase 2 (Epic 17: Performance Optimizations)
   - **Impact**: If yes, requires connection manager refactor

5. **Question**: Should configuration file support secrets from external vault (e.g., AWS Secrets Manager)?
   - **Decision Needed**: Out of scope for MVP; consider for Phase 2 (Epic 13: Advanced Security)
   - **Impact**: If yes, add vault integration in Epic 13

## Test Strategy Summary

**Unit Tests (pytest):**
- **Configuration Module**: Test precedence order (CLI > env > TOML > defaults), validation errors, SecretStr protection, Pydantic model validation
- **Connection Pool**: Mock pool exhaustion, eviction triggers, health score calculation, concurrent request handling
- **Reconnection Logic**: Mock backoff schedule, verify attempt counts, test backoff reset after 5 minutes
- **TLS Context**: Test SSL context creation, certificate loading, verify mode settings
- **Health Check**: Mock connection status, test failure detection, consecutive failure counting

**Integration Tests (testcontainers):**
- **AMQP Connection**: Connect to RabbitMQ testcontainer, test auth failure, timeout, TLS with self-signed cert, vhost targeting
- **HTTP Client**: Test Management API requests (GET /api/overview), connection pooling, auth errors, TLS
- **Health Checks**: Simulate connection failure by stopping container, verify health check detects failure in <30s
- **Reconnection**: Kill RabbitMQ container, verify reconnection with exponential backoff (1s, 2s, 4s, 8s, 16s, 32s, 60s)
- **Connection Pool**: Send 100 concurrent requests, verify pool reuse, eviction on container restart

**Performance Tests:**
- **Configuration Loading**: Measure time to load and validate config (<100ms target)
- **Connection Establishment**: Measure time to connect to RabbitMQ (<5s target)
- **Health Check Latency**: Measure health check execution time (<1s target, p95)
- **Reconnection Time**: Measure time from server recovery to successful reconnection (<10s target)
- **Pool Performance**: Measure request latency with connection reuse (<30ms overhead target)

**Edge Cases:**
- **Partial Connection Failure**: AMQP connects but HTTP fails (or vice versa) - verify reconnection logic handles both
- **Configuration File Corruption**: Invalid TOML format - verify clear error message
- **Credentials Change During Runtime**: Modify env vars - verify pool flush and reconnection
- **Certificate Expiry**: Simulate expired TLS certificate - verify clear error message
- **Pool Exhaustion Under Load**: Send 100+ concurrent requests with pool size 5 - verify graceful degradation
- **Long-Running Connection**: Keep connection alive >5 minutes - verify backoff reset

**Contract Tests:**
- **MCP Tool Exposure**: Verify `connection.health`, `connection.pool_status`, `connection.reconnection_state` tools are exposed via MCP protocol
- **Health Status Schema**: Verify health status matches HealthStatus Pydantic model
- **Error Response Format**: Verify connection errors follow JSON-RPC 2.0 error format

**Coverage Target:**
- Minimum 80% code coverage for connection management modules
- 100% coverage for critical paths (connection establishment, reconnection, health checks)
- Use pytest-cov to track coverage: `pytest --cov=src/rabbitmq_mcp_connection --cov-report=html`

**Test Execution:**
- Complete test suite should execute in <5 minutes (per FR-018)
- Integration tests require Docker for testcontainers
- CI/CD pipeline runs all tests on every commit (GitHub Actions)
- Performance tests run nightly to detect regressions
