# Monitoring Stack — LLMHosts.com (T-12)

Prometheus + Grafana + Loki + AlertManager monitoring for LLMHosts.

## Quick Start

```bash
# Start the LLMHosts app first (creates the llmhostscom_llmhost-internal network)
docker compose -f docker-compose.prod.yml -f docker-compose.external-proxy.yml up -d

# Start monitoring stack
docker compose -f monitoring/docker-compose.monitoring.yml up -d

# Verify all services are healthy
docker compose -f monitoring/docker-compose.monitoring.yml ps
```

## Services

| Service | Port | URL | Purpose |
|---------|------|-----|---------|
| Prometheus | 9090 | http://localhost:9090 | Metrics collection and alerting engine |
| Grafana | 3001 | http://localhost:3001 | Dashboards and visualization |
| Loki | 3100 | http://localhost:3100 | Log aggregation |
| Promtail | — | — | Log collection agent |
| AlertManager | 9093 | http://localhost:9093 | Alert routing and notification |
| Node Exporter | 9100 | http://localhost:9100 | Host-level metrics |

All ports bind to `127.0.0.1` only (not exposed to the internet).

## Default Credentials

- **Grafana**: admin / changeme (set `GRAFANA_ADMIN_PASSWORD` env var to override)

## What Gets Monitored

### Metrics (Prometheus)

| Source | Endpoint | Metrics |
|--------|----------|---------|
| LLMHosts proxy | `llmhost:4000/metrics` | HTTP request rate, latency histograms, active backends, cache hit rate |
| LLMHosts custom | `llmhost:4000/metrics/prometheus` | LLM cost/savings, tokens, backend health, uptime |
| Node Exporter | `host:9100/metrics` | CPU, memory, disk, network |

### Logs (Loki via Promtail)

- Docker container logs (llmhost, llmhost-frontend, all monitoring containers)
- System logs (/var/log/syslog, /var/log/auth.log)

### Alerts

| Alert | Condition | Severity |
|-------|-----------|----------|
| HighCPUUsage | CPU > 80% for 5m | warning |
| CriticalCPUUsage | CPU > 95% for 2m | critical |
| HighMemoryUsage | Memory > 85% for 5m | warning |
| DiskSpaceLow | Disk < 20% free for 5m | warning |
| DiskSpaceCritical | Disk < 10% free for 2m | critical |
| ServiceDown | Any scrape target unreachable for 1m | critical |
| LLMHostProxyDown | Proxy unreachable for 30s | critical |
| HighErrorRate | 5xx rate > 5% for 5m | warning |
| SlowResponses | P95 latency > 2s for 5m | warning |
| LowCacheHitRate | Cache < 10% hits (after 100 requests) for 15m | warning |
| NoActiveBackends | Zero backends for 1m | critical |

## Grafana Dashboard

A pre-built "LLMHosts Overview" dashboard is provisioned automatically with panels for:
- Request rate and error rate
- P50/P95/P99 latency
- Cache hit rate gauge
- Active backends count
- LLM requests by backend
- Cost savings (USD)
- CPU, memory, and disk usage
- Total tokens processed and uptime

## Configuring Alert Notifications

Edit `alertmanager/alertmanager.yml` to add real notification endpoints:

```yaml
# Email
global:
  smtp_smarthost: "smtp.resend.com:587"
  smtp_from: "alerts@llmhosts.com"
  smtp_auth_username: "resend"
  smtp_auth_password: "re_..."

receivers:
  - name: "default-webhook"
    email_configs:
      - to: "you@example.com"
        send_resolved: true
```

Reload without restart: `curl -X POST http://localhost:9093/-/reload`

## Data Retention

| Component | Retention | Config |
|-----------|-----------|--------|
| Prometheus | 30 days | `--storage.tsdb.retention.time=30d` |
| Loki | 30 days | `retention_period: 720h` in loki-config.yml |
| Grafana | Persistent | grafana_data volume |

## Stopping

```bash
docker compose -f monitoring/docker-compose.monitoring.yml down

# To also remove data volumes:
docker compose -f monitoring/docker-compose.monitoring.yml down -v
```
