# Monitoring Guide

LLMHosts ships a complete monitoring stack based on Prometheus, Grafana, and Loki. This guide covers how to set it up, use the pre-built dashboards, and configure alerting.

## Quick Start

```bash
# Start the monitoring stack
docker compose -f monitoring/docker-compose.monitoring.yml up -d

# Verify all services are running
docker compose -f monitoring/docker-compose.monitoring.yml ps

# Access Grafana
open http://localhost:3001
# Default credentials: admin / admin (change on first login)
```

The monitoring stack runs alongside your LLMHosts deployment and auto-discovers the proxy's metrics endpoints.

## Architecture

```
LLMHosts Proxy (:4000)
├── /metrics              ← Standard prometheus_client metrics
└── /metrics/prometheus   ← LLMHosts-specific metrics
     │
     ▼ (scrape every 15s)
Prometheus (:9090) ──────► AlertManager (:9093) → Email/Slack/Webhook
     │
     ▼
Grafana (:3001) ──► 3 Pre-built Dashboards

Docker Containers
     │
     ▼ (log collection)
Promtail (:9080) ──► Loki (:3100) ──► Grafana (Explore)

Host System
     │
     ▼
Node Exporter (:9100) ──► Prometheus
```

## Pre-Built Dashboards

Three Grafana dashboards are auto-provisioned when the stack starts.

### 1. LLMHosts Overview

**File**: `monitoring/grafana/dashboards/llmhosts-overview.json`

A high-level operational dashboard with 13 panels:

| Panel | What It Shows |
|-------|--------------|
| Request Rate | Total requests per second across all endpoints |
| Error Rate (5xx) | Percentage of server errors with color thresholds |
| Request Latency | P50, P95, P99 response times |
| Cache Hit Rate | Current cache effectiveness (gauge) |
| Active Backends | Number of healthy inference backends |
| LLM Requests by Backend | Stacked bar chart of request distribution |
| Cost Savings | Cumulative cost vs savings from local inference |
| CPU Usage | Host CPU utilization with warning thresholds |
| Memory Usage | Host memory utilization with warning thresholds |
| Disk Usage | Root filesystem usage (gauge) |
| Tokens Processed | Total tokens served |
| Uptime | LLMHosts server uptime |

### 2. LLMHosts Cache Performance

**File**: `monitoring/grafana/dashboards/llmhosts-cache.json`

Deep dive into the three-tier semantic cache (exact hash → entity namespace → vCache):

| Panel | What It Shows |
|-------|--------------|
| Cache Hit Rate (Overall) | Current hit rate as a gauge |
| Cache Hits / Misses | Total counts (stat panels) |
| Hits vs Misses Over Time | Rate of hits and misses as time series |
| Cache Hit Rate Over Time | Hit rate trend with threshold shading |
| Savings from Cache (USD) | Money saved by serving cached responses |
| Savings Percentage | Percentage of costs avoided (gauge) |
| Cached vs Uncached Requests | Stacked bar showing cache vs passthrough |
| Requests Per Minute | Overall traffic to correlate with cache behavior |

### 3. LLMHosts Backend Health & Cost

**File**: `monitoring/grafana/dashboards/llmhosts-backends.json`

Backend-level health, cost, and performance breakdown:

| Panel | What It Shows |
|-------|--------------|
| Backend Health Status | Per-backend UP/DOWN indicators |
| Active Backends | Count of healthy backends |
| Total Cost / Savings / Tokens | Summary stats |
| Requests by Backend | Which backends handle traffic |
| Requests by Model | Which models are most used |
| Tokens by Backend | Token throughput per backend |
| Cost by Backend (USD) | Hourly cost rate per backend |
| Request Latency (P50/P95/P99) | End-to-end latency percentiles |
| Error Rate by Status Code | 4xx and 5xx breakdown |
| CPU / Memory / Uptime | Infrastructure health |

## Importing Dashboards Manually

If you're not using the Docker Compose monitoring stack, you can import dashboards into any Grafana instance.

### Via Grafana UI

1. Open Grafana → **Dashboards** → **New** → **Import**
2. Click **Upload dashboard JSON file**
3. Select one of the files from `monitoring/grafana/dashboards/`:
   - `llmhosts-overview.json`
   - `llmhosts-cache.json`
   - `llmhosts-backends.json`
4. Select your Prometheus datasource when prompted
5. Click **Import**

### Via Grafana API

```bash
# Import a dashboard via the HTTP API
curl -X POST http://localhost:3001/api/dashboards/db \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_GRAFANA_API_KEY" \
  -d "{
    \"dashboard\": $(cat monitoring/grafana/dashboards/llmhosts-overview.json),
    \"overwrite\": true
  }"
```

### Via Provisioning (Docker Compose)

The dashboards are auto-loaded by Grafana's provisioning system. The config is at `monitoring/grafana/provisioning/dashboards/dashboards.yml`:

```yaml
apiVersion: 1
providers:
  - name: "llmhosts"
    orgId: 1
    folder: "LLMHosts"
    type: file
    editable: true
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards
```

The Docker Compose stack mounts the dashboard directory:

```yaml
volumes:
  - ./grafana/dashboards:/var/lib/grafana/dashboards
  - ./grafana/provisioning:/etc/grafana/provisioning
```

## Metrics Reference

LLMHosts exports metrics on two endpoints.

### `/metrics` (prometheus_client)

Standard Python prometheus_client library metrics:

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `http_requests_total` | Counter | method, path, status | Total HTTP requests |
| `http_request_duration_seconds` | Histogram | method, path | Request latency (11 buckets: 5ms–10s) |
| `active_backends` | Gauge | — | Number of healthy backends |
| `cache_hit_rate` | Gauge | — | Overall cache hit rate (0.0–1.0) |
| `llm_requests_total` | Counter | backend, model, cached | LLM inference requests |
| `llm_tokens_total` | Counter | backend, model | Tokens processed |
| `llm_cost_usd_total` | Counter | backend | Cumulative cost in USD |
| `llm_savings_usd_total` | Counter | — | Cumulative savings vs cloud pricing |

### `/metrics/prometheus` (Custom export)

LLMHosts-specific business metrics:

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `llmhost_requests_total` | Counter | backend | Requests per backend |
| `llmhost_request_duration_seconds` | Gauge | — | Average request duration |
| `llmhost_cache_hits_total` | Counter | — | Total cache hits |
| `llmhost_cache_misses_total` | Counter | — | Total cache misses |
| `llmhost_tokens_total` | Counter | direction | Total tokens (direction=total) |
| `llmhost_cost_dollars_total` | Counter | — | Cumulative cost |
| `llmhost_savings_dollars_total` | Counter | — | Cumulative savings |
| `llmhost_backend_healthy` | Gauge | backend | Backend health (0=down, 1=up) |
| `llmhost_uptime_seconds` | Gauge | — | Server uptime |
| `llmhost_requests_per_minute` | Gauge | — | Current request rate |
| `llmhost_cache_hit_rate` | Gauge | — | Current cache hit rate |
| `llmhost_savings_percent` | Gauge | — | Percentage of costs saved |

## Alerting

The monitoring stack includes 11 pre-configured alert rules in `monitoring/prometheus/alert-rules.yml`.

### Alert Summary

| Alert | Severity | Condition |
|-------|----------|-----------|
| HighCPUUsage | warning | CPU >80% for 5m |
| CriticalCPUUsage | critical | CPU >95% for 2m |
| HighMemoryUsage | warning | Memory >85% for 5m |
| DiskSpaceLow | warning | Disk <20% free for 5m |
| DiskSpaceCritical | critical | Disk <10% free for 2m |
| ServiceDown | critical | Any scrape target down for 1m |
| LLMHostProxyDown | critical | Proxy unreachable for 30s |
| HighErrorRate | warning | 5xx rate >5% for 5m |
| SlowResponses | warning | P95 latency >2s for 5m |
| LowCacheHitRate | warning | <10% hit rate after 100+ requests for 15m |
| NoActiveBackends | critical | Zero healthy backends for 1m |

### Configuring Notifications

Edit `monitoring/alertmanager/alertmanager.yml` to route alerts:

```yaml
# Email notifications
receivers:
  - name: 'email'
    email_configs:
      - to: 'ops@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'password'

# Slack notifications
receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T.../B.../xxx'
        channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'

# Webhook (generic)
receivers:
  - name: 'webhook'
    webhook_configs:
      - url: 'https://your-service.example.com/alerts'
```

## Customizing Dashboards

The provisioned dashboards allow UI edits (`allowUiUpdates: true`). Changes made in the Grafana UI persist in the Docker volume but will be overwritten if the container is recreated.

To make permanent changes:

1. Edit the dashboard in the Grafana UI
2. Go to **Dashboard settings** → **JSON Model**
3. Copy the JSON and save it back to the corresponding file in `monitoring/grafana/dashboards/`
4. Commit the updated JSON to version control

## Adding Custom Panels

All dashboards use the `${DS_PROMETHEUS}` variable for the datasource. Use any of the metrics from the reference tables above in your PromQL queries.

Example — adding a panel for tokens per model:

```json
{
  "title": "Tokens by Model",
  "type": "timeseries",
  "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
  "targets": [
    {
      "expr": "sum by (model) (rate(llm_tokens_total[5m]))",
      "legendFormat": "{{ model }}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "short",
      "custom": { "drawStyle": "line", "fillOpacity": 15 }
    }
  }
}
```

## Log Exploration

Loki collects logs from all Docker containers and syslog. Access logs via Grafana → **Explore** → select **Loki** datasource.

Useful LogQL queries:

```logql
# All LLMHosts proxy logs
{container_name="llmhost"}

# Error-level logs only
{container_name="llmhost"} |= "error"

# Frontend logs
{container_name="llmhosts-frontend"}

# Slow requests (>1s)
{container_name="llmhost"} | json | duration > 1s
```

## Stack Resource Usage

The monitoring stack is designed to be lightweight:

| Service | Memory (min–max) | CPU | Disk (30d retention) |
|---------|-----------------|-----|---------------------|
| Prometheus | 128–512 MB | Low | ~500 MB |
| Grafana | 64–256 MB | Low | ~50 MB |
| Loki | 64–256 MB | Low | ~200 MB |
| Promtail | 32–128 MB | Low | Negligible |
| AlertManager | 16–64 MB | Low | Negligible |
| Node Exporter | Minimal | Low | None |

**Total**: ~1–1.5 GB RAM, ~750 MB disk for 30 days of data.

## Troubleshooting

| Issue | Fix |
|-------|-----|
| Grafana shows "No data" | Check Prometheus targets at `http://localhost:9090/targets` — all should be UP |
| Dashboards not loading | Verify provisioning config mounts: `docker compose logs llmhost-grafana` |
| Prometheus can't reach LLMHosts | Both must be on the same Docker network (`llmhosts-internal`) |
| Node Exporter metrics missing | Node Exporter runs on host network — check `http://localhost:9100/metrics` |
| Alerts not firing | Check AlertManager at `http://localhost:9093` and verify receiver config |
| High memory usage | Reduce retention: `--storage.tsdb.retention.time=7d` in Prometheus |
