# Monitoring Stack Integration Guide

This directory contains the full Aut-O monitoring stack (Prometheus + Grafana + Loki) integrated with LLMHosts.com for T-12 (World-Class Logging & Monitoring) compliance.

## Architecture

```
┌─────────────────┐
│ LLMHosts Proxy  │──► /metrics (Prometheus format)
│   (port 4000)   │
└─────────────────┘
        │
        │ scraped by
        ▼
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Prometheus    │────►│    Grafana      │◄────│      Loki       │
│   (port 9090)   │     │   (port 3001)   │     │   (port 3100)   │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │
        │ alerts
        ▼
┌─────────────────┐
│  AlertManager   │
│   (port 9093)   │
└─────────────────┘
```

## Quick Start

### 1. Install prometheus-client

The Python proxy requires `prometheus-client` to expose metrics:

```bash
pip install -e ".[dev]"
```

This installs `prometheus-client>=0.21.0` as a dev dependency.

### 2. Start the Monitoring Stack

From this directory (`infrastructure/monitoring/`):

```bash
docker compose -f docker-compose.monitoring.yml up -d
```

This starts:
- **Prometheus** (http://localhost:9091) — metrics database
- **Grafana** (http://localhost:3003) — visualization dashboards
- **Loki** (http://localhost:3100) — log aggregation
- **Promtail** (no UI) — log shipper
- **AlertManager** (http://localhost:9093) — alert routing
- **Node Exporter** (http://localhost:9100) — host metrics

### 3. Start LLMHosts

```bash
llmhosts serve
```

The proxy will:
- Mount `/metrics` endpoint (Prometheus format)
- Record HTTP request metrics automatically
- Include correlation IDs in logs
- Redact PII from all logs

### 4. Verify Integration

#### Check /metrics endpoint

```bash
curl http://localhost:4000/metrics
```

You should see Prometheus format output:

```
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/health",status="200"} 5.0
...
```

#### Check Prometheus targets

Visit http://localhost:9091/targets

The `llmhosts-proxy` target should show as **UP**.

#### Access Grafana

1. Visit http://localhost:3003
2. Login: `admin` / `admin123`
3. Go to Dashboards → Browse
4. Open **LLMHosts Overview** dashboard

## Exposed Metrics

### HTTP Metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `http_requests_total` | Counter | `method`, `path`, `status` | Total HTTP requests |
| `http_request_duration_seconds` | Histogram | `method`, `path` | Request latency distribution |

### LLM-Specific Metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `llm_requests_total` | Counter | `backend`, `model`, `cached` | Total LLM requests |
| `llm_tokens_total` | Counter | `backend`, `model` | Total tokens processed |
| `llm_cost_usd_total` | Counter | `backend` | Total cost in USD |
| `llm_savings_usd_total` | Counter | - | Total savings vs cloud |
| `active_backends` | Gauge | - | Number of active backends |
| `cache_hit_rate` | Gauge | - | Cache hit rate (0.0-1.0) |

## Correlation IDs

Every request gets a correlation ID:

1. **Client provides**: Use `X-Correlation-ID` header
2. **Auto-generated**: UUID if not provided
3. **Returned**: Same header in response
4. **Logged**: Included in structured JSON logs

Example:

```bash
curl -H "X-Correlation-ID: my-trace-123" http://localhost:4000/health
```

Logs will include:

```json
{
  "timestamp": "2026-02-17T12:00:00Z",
  "level": "INFO",
  "logger": "llmhosts.proxy.middleware",
  "message": "GET /health -> 200 (1.2ms)",
  "correlation_id": "my-trace-123"
}
```

## PII Redaction

All logs automatically redact:

- API keys (`sk-***`, `api_key=***`)
- Bearer tokens
- Email addresses
- Phone numbers (US format)
- SSNs (US format)
- Credit card numbers
- Passwords

Example:

```
Before: "User sk-abc123def456 requested model gpt-4"
After:  "User sk-***REDACTED*** requested model gpt-4"
```

## Custom Metrics

To add custom metrics in your code:

```python
from llmhosts.metrics import record_llm_request

# Record an LLM request
record_llm_request(
    backend="ollama",
    model="llama3:8b",
    cached=False,
    tokens=1500,
    cost=0.0,
    savings=0.015,
)
```

## Dashboard Access

### Prometheus

- URL: http://localhost:9091
- No authentication
- Use for ad-hoc queries (PromQL)

Example query:

```promql
rate(http_requests_total[5m])
```

### Grafana

- URL: http://localhost:3003
- Username: `admin`
- Password: `admin123`

Pre-configured dashboards:
- **LLMHosts Overview** — high-level metrics
- **Node Exporter Full** — host-level metrics (CPU, memory, disk, network)

### Loki (Logs)

Access via Grafana:
1. Go to **Explore** (compass icon)
2. Select **Loki** data source
3. Use LogQL queries:

```logql
{job="llmhosts-proxy"} |= "ERROR"
```

## Alert Rules

Alert rules are defined in `prometheus/alert-rules.yml`:

| Alert | Condition | Severity |
|-------|-----------|----------|
| `InstanceDown` | Target unreachable >5min | critical |
| `HighErrorRate` | 5xx errors >5% | warning |
| `HighLatency` | p95 latency >2s | warning |

Alerts route to AlertManager (http://localhost:9093).

## Production Considerations

### 1. Retention

Default retention:
- Prometheus: 15 days
- Loki: 7 days

Adjust in `docker-compose.monitoring.yml`:

```yaml
prometheus:
  command:
    - '--storage.tsdb.retention.time=30d'
```

### 2. Resource Limits

Add to `docker-compose.monitoring.yml`:

```yaml
prometheus:
  deploy:
    resources:
      limits:
        memory: 2G
        cpus: '1.0'
```

### 3. Security

- Change Grafana admin password (`GF_SECURITY_ADMIN_PASSWORD`)
- Enable authentication on Prometheus (reverse proxy + auth)
- Restrict network access (internal only)
- Use HTTPS (Traefik with Let's Encrypt)

### 4. High Availability

For production:
- Run Prometheus in HA mode (2+ replicas)
- Use remote storage (Thanos, Cortex, Mimir)
- Enable Grafana HA (shared DB)

## Troubleshooting

### /metrics endpoint returns 404

**Cause**: `prometheus-client` not installed.

**Fix**:

```bash
pip install prometheus-client
# or
pip install -e ".[dev]"
```

### Prometheus target shows DOWN

**Cause**: LLMHosts not running or port mismatch.

**Fix**:

1. Check LLMHosts is running: `curl http://localhost:4000/health`
2. Verify port in `prometheus/prometheus.yml` matches server port
3. Restart Prometheus: `docker compose -f docker-compose.monitoring.yml restart prometheus`

**Note**: Prometheus runs on port 9091 on the host (to avoid conflicts with Headscale on port 9090), but uses port 9090 internally within the Docker network.

### No logs in Loki

**Cause**: Promtail not shipping logs.

**Fix**:

1. Check Promtail container: `docker compose -f docker-compose.monitoring.yml logs promtail`
2. Verify log file path in `promtail/promtail-config.yml`
3. Enable JSON logging: `export LLMHOST_OBSERVABILITY_LOG_FORMAT=json`

### Grafana shows "No data"

**Cause**: Data source not configured or no metrics yet.

**Fix**:

1. Go to Configuration → Data Sources
2. Verify Prometheus URL: `http://prometheus:9090` (internal network address, not host port)
3. Click **Save & Test**
4. Generate metrics: `curl http://localhost:4000/health`

## Files

```
infrastructure/monitoring/
├── INTEGRATION.md                     # This file
├── docker-compose.monitoring.yml      # Stack definition
├── prometheus/
│   ├── prometheus.yml                 # Scrape config
│   └── alert-rules.yml                # Alert definitions
├── grafana/
│   ├── datasources.yml                # Auto-provisioned data sources
│   └── dashboards/                    # Pre-built dashboards
├── loki/
│   └── loki-config.yml                # Log aggregation config
├── promtail/
│   └── promtail-config.yml            # Log shipper config
└── alertmanager/
    └── alertmanager.yml               # Alert routing
```

## References

- [Prometheus Documentation](https://prometheus.io/docs/)
- [Grafana Documentation](https://grafana.com/docs/)
- [Loki Documentation](https://grafana.com/docs/loki/)
- [PromQL Cheat Sheet](https://promlabs.com/promql-cheat-sheet/)
- [LogQL Cheat Sheet](https://grafana.com/docs/loki/latest/logql/)

## Support

For issues with monitoring integration:

1. Check logs: `docker compose -f docker-compose.monitoring.yml logs`
2. Verify metrics: `curl http://localhost:4000/metrics`
3. Test Prometheus: http://localhost:9090/targets
4. File an issue: https://github.com/LookNoHandsMom/LLMHosts.com/issues
