# Step 05: Create Health Check System

## Prerequisites
- Step 04 completed (import dependencies resolved) ✅
- All server modules can be created ✅
- Main server creation works ✅

## Current System Status
🎉 **SYSTEM IS 100% OPERATIONAL AND READY FOR HEALTH CHECKS!**

### ✅ **All Prerequisites Met:**
- **7/7 Server modules**: Working perfectly
- **17+ Tools**: Available across all servers
- **49+ Functions**: Available in context logic modules
- **Main Server**: Creating successfully
- **FastMCP API**: All compatibility issues resolved

## Objectives
1. Create a comprehensive health check system
2. Implement individual server health checks (add to 7 working servers)
3. Create master health check coordinator
4. Create CLI tool for easy health checking
5. Prepare for test.sh integration

## Implementation Details

### Step 1: Create Health Check Framework
Create `_mcp/health_check.py`:

```python
"""
Storm Logger MCP Health Check System
Comprehensive health monitoring for all MCP servers
"""

import time
import traceback
from typing import Dict, List, Optional
from dataclasses import dataclass
from enum import Enum

class HealthStatus(Enum):
    HEALTHY = "healthy"
    WARNING = "warning"
    ERROR = "error"
    UNKNOWN = "unknown"

@dataclass
class HealthCheckResult:
    service: str
    status: HealthStatus
    message: str
    timestamp: str
    details: Optional[Dict] = None
    response_time: Optional[float] = None

class MCPHealthChecker:
    def __init__(self):
        self.servers = {}
        self.last_check = None
    
    def register_server(self, name: str, server):
        """Register a server for health monitoring."""
        self.servers[name] = server
    
    def check_server_health(self, name: str) -> HealthCheckResult:
        """Check health of a specific server."""
        start_time = time.time()
        
        try:
            server = self.servers.get(name)
            if not server:
                return HealthCheckResult(
                    service=name,
                    status=HealthStatus.ERROR,
                    message=f"Server {name} not found",
                    timestamp=self._get_timestamp()
                )
            
            # Basic health checks
            checks = {
                "server_exists": server is not None,
                "has_tools": len(server.tools) > 0,
                "has_name": hasattr(server, 'name') and server.name,
            }
            
            # Advanced health checks if available
            if hasattr(server, 'health_check'):
                health_result = server.health_check()
                checks.update(health_result.get('checks', {}))
            
            # Determine overall status
            if all(checks.values()):
                status = HealthStatus.HEALTHY
                message = f"Server {name} is healthy"
            elif any(checks.values()):
                status = HealthStatus.WARNING
                message = f"Server {name} has some issues"
            else:
                status = HealthStatus.ERROR
                message = f"Server {name} is unhealthy"
            
            response_time = time.time() - start_time
            
            return HealthCheckResult(
                service=name,
                status=status,
                message=message,
                timestamp=self._get_timestamp(),
                details=checks,
                response_time=response_time
            )
            
        except Exception as e:
            return HealthCheckResult(
                service=name,
                status=HealthStatus.ERROR,
                message=f"Health check failed: {str(e)}",
                timestamp=self._get_timestamp(),
                details={"error": str(e), "traceback": traceback.format_exc()}
            )
    
    def check_all_servers(self) -> List[HealthCheckResult]:
        """Check health of all registered servers."""
        results = []
        for name in self.servers:
            results.append(self.check_server_health(name))
        self.last_check = time.time()
        return results
    
    def get_overall_status(self) -> HealthStatus:
        """Get overall system health status."""
        if not self.servers:
            return HealthStatus.UNKNOWN
        
        results = self.check_all_servers()
        
        # If any server is in error state, system is error
        if any(r.status == HealthStatus.ERROR for r in results):
            return HealthStatus.ERROR
        
        # If any server has warnings, system has warnings
        if any(r.status == HealthStatus.WARNING for r in results):
            return HealthStatus.WARNING
        
        # If all servers are healthy, system is healthy
        return HealthStatus.HEALTHY
    
    def _get_timestamp(self) -> str:
        """Get current timestamp in ISO format."""
        return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
```

### Step 2: Add Health Check to Main Server
Update `_mcp/main.py`:

```python
from .health_check import MCPHealthChecker, HealthStatus

# Global health checker
health_checker = MCPHealthChecker()

def create_storm_mcp_server():
    """Create the main Storm Logger MCP server with health monitoring."""
    server = FastMCP(
        name="Storm Logger Enterprise MCP Server",
        instructions="Enterprise-grade MCP server for Storm Logger development",
        tags={"storm_logger", "enterprise", "main"}
    )
    
    # Register all servers for health monitoring
    servers = {
        "coverage": create_coverage_server(),
        "testing": create_testing_server(),
        "security": create_security_server(),
        "patterns": create_patterns_server(),
        "gitlab": create_gitlab_server(),
        "python": create_python_server(),
        "core": create_core_server()
    }
    
    for name, srv in servers.items():
        health_checker.register_server(name, srv)
    
    @server.tool(tags={"health", "admin"})
    def health_check() -> Dict:
        """Comprehensive health check for all MCP servers."""
        results = health_checker.check_all_servers()
        overall_status = health_checker.get_overall_status()
        
        return {
            "overall_status": overall_status.value,
            "timestamp": health_checker._get_timestamp(),
            "servers": [
                {
                    "name": r.service,
                    "status": r.status.value,
                    "message": r.message,
                    "response_time": r.response_time,
                    "details": r.details
                }
                for r in results
            ],
            "summary": {
                "total_servers": len(results),
                "healthy": sum(1 for r in results if r.status == HealthStatus.HEALTHY),
                "warnings": sum(1 for r in results if r.status == HealthStatus.WARNING),
                "errors": sum(1 for r in results if r.status == HealthStatus.ERROR)
            }
        }
    
    return server
```

### Step 3: Add Health Check to Individual Servers
Update each server module to include health check:

```python
# Example for coverage server
@server.tool(tags={"health", "internal"})
def health_check() -> Dict:
    """Health check for coverage server."""
    checks = {
        "coverage_analysis_available": True,
        "coverage_reporting_available": True,
        "dependencies_ok": True
    }
    
    try:
        # Test critical functionality
        from _mcp.context_logic import coverage_analysis
        checks["coverage_analysis_available"] = hasattr(coverage_analysis, 'analyze_coverage_gaps')
    except ImportError:
        checks["coverage_analysis_available"] = False
    
    return {
        "status": "healthy" if all(checks.values()) else "warning",
        "checks": checks,
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
    }
```

### Step 4: Create Health Check CLI Tool
Create `_mcp/health_cli.py`:

```python
#!/usr/bin/env python3
"""
Storm Logger MCP Health Check CLI
Command-line tool for checking MCP server health
"""

import sys
import json
from pathlib import Path

# Add project root to path
project_root = Path(__file__).parent.parent
sys.path.insert(0, str(project_root))

from _mcp.main import create_storm_mcp_server

def main():
    """Main health check CLI function."""
    try:
        print("⚡ Storm Logger MCP Health Check")
        print("=" * 50)
        
        # Create server
        server = create_storm_mcp_server()
        
        # Get health check tool
        health_tool = None
        for tool in server.tools:
            if tool.name == "health_check":
                health_tool = tool
                break
        
        if not health_tool:
            print("❌ Health check tool not found")
            return 1
        
        # Run health check
        result = health_tool.function()
        
        # Display results
        print(f"Overall Status: {result['overall_status'].upper()}")
        print(f"Timestamp: {result['timestamp']}")
        print(f"Total Servers: {result['summary']['total_servers']}")
        print(f"Healthy: {result['summary']['healthy']}")
        print(f"Warnings: {result['summary']['warnings']}")
        print(f"Errors: {result['summary']['errors']}")
        
        print("\nServer Details:")
        for server_info in result['servers']:
            status_icon = "✅" if server_info['status'] == 'healthy' else "⚠️" if server_info['status'] == 'warning' else "❌"
            print(f"  {status_icon} {server_info['name']}: {server_info['message']}")
        
        # Return appropriate exit code
        if result['overall_status'] == 'healthy':
            return 0
        elif result['overall_status'] == 'warning':
            return 1
        else:
            return 2
            
    except Exception as e:
        print(f"❌ Health check failed: {e}")
        return 3

if __name__ == "__main__":
    sys.exit(main())
```

## Testing/Validation

### Step 1: Test Health Check Framework
```bash
# Test health check CLI
python _mcp/health_cli.py
```

### Step 2: Test Individual Server Health
```bash
# Test server creation and health checks
python -c "
import sys
sys.path.append('.')
from _mcp.main import create_storm_mcp_server
server = create_storm_mcp_server()
print('✓ Server created')
for tool in server.tools:
    if tool.name == 'health_check':
        result = tool.function()
        print(f'Health check: {result[\"overall_status\"]}')
        break
"
```

### Step 3: Test Integration with Test Runner
```bash
# Test health check integration
./test.sh --health-check
```

## Success Criteria
- [ ] Health check system implemented
- [ ] All servers have health check functions
- [ ] Main server health check works
- [ ] CLI health check tool works
- [ ] Integration with test runner

## JSON Update Instructions

When starting this step:
```json
{
  "steps": {
    "05": {
      "status": "in_progress",
      "progress": 25,
      "issues": ["Implementing health check framework"],
      "next_action": "Create health check system"
    }
  }
}
```

When 50% complete (framework implemented):
```json
{
  "steps": {
    "05": {
      "status": "in_progress",
      "progress": 50,
      "issues": ["Adding health checks to servers"],
      "next_action": "Add health check to all server modules"
    }
  }
}
```

When complete:
```json
{
  "steps": {
    "05": {
      "status": "completed",
      "progress": 100,
      "issues": [],
      "next_action": "Proceed to Step 06"
    }
  },
  "current_step": "06",
  "overall_progress": 50,
  "completed_features": [
    "All 7 server modules implemented",
    "Health check functions added",
    "Server creation working",
    "Import dependencies resolved",
    "Complete health check system"
  ]
}
```

## Troubleshooting

### Common Issues

1. **Health check tool not found**
   - Check tool registration in main server
   - Verify tool name is correct
   - Check tool tags are set properly

2. **Server health check fails**
   - Check individual server health check functions
   - Verify imports are working
   - Check for missing dependencies

3. **CLI tool fails**
   - Check Python path setup
   - Verify server creation works
   - Check for missing imports

### Debug Commands
```bash
# Test individual server health
python -c "
import sys
sys.path.append('.')
from _mcp.servers.coverage import create_coverage_server
server = create_coverage_server()
for tool in server.tools:
    if tool.name == 'health_check':
        result = tool.function()
        print(f'Coverage health: {result}')
"
```

## Next Steps
After completing this step:
1. Update JSON status to "completed"
2. Proceed to Step 06: Test Individual Server Modules
3. Use health check system to validate testing

## Notes
- Health check system is critical for production monitoring
- Each server should have specific health validation
- CLI tool provides easy testing interface
- Integration with test runner enables automated checking