# Monitoring

Citadel uses the kube-prometheus-stack (Prometheus, Grafana, AlertManager) for cluster and application monitoring. The lifecycle operator has pre-positioned PrometheusRule alerts for node lifecycle events.

## Setup

```bash
./scripts/03-install-monitoring.sh
```

This installs the kube-prometheus-stack Helm chart into the `monitoring` namespace, creating:
- Prometheus (metrics scraping)
- Grafana (dashboards)
- AlertManager (alert routing)
- node-exporter DaemonSet (host metrics)
- kube-state-metrics (K8s object metrics)

## Access

```bash
# Grafana
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
# http://localhost:3000 — login: admin / <password printed during install>

# Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
# http://localhost:9090
```

## Recommended Dashboards

Import via Grafana UI (+ → Import → Enter ID):

| Dashboard | Import ID | Purpose |
|-----------|-----------|---------|
| Cilium Metrics | 16611 | CNI health, policy enforcement, BPF maps |
| Hubble Metrics | 16613 | Network flow visibility, DNS, HTTP |

## PrometheusRule Alerts

File: `monitoring/node-lifecycle-alerts.yaml`

> **Important:** The operator does not yet export Prometheus metrics. These alerts are pre-positioned for when the `citadel_node_lifecycle_*` metrics are implemented. The PrometheusRule is valid and will activate automatically once the operator exposes the expected metrics.

### Expected Metrics

The operator should export these metrics under the `citadel_node_lifecycle_` prefix:

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `citadel_node_lifecycle_node_age_hours` | gauge | `node` | Current age of each node |
| `citadel_node_lifecycle_drain_result` | counter | `node`, `status` | Drain outcomes (success/failed) |
| `citadel_node_lifecycle_policy_evaluation_errors_total` | counter | `policy` | Policy evaluation errors |
| `citadel_node_lifecycle_pdb_blocking_duration_seconds` | gauge | `node`, `namespace`, `pdb` | How long a PDB has been blocking |
| `citadel_node_lifecycle_replacement_in_progress` | gauge | `node` | 1 while replacing a node |
| `citadel_node_lifecycle_replacement_start_time` | gauge | `node` | Unix timestamp when replacement started |

### Alerts

| Alert | Severity | For | Trigger |
|-------|----------|-----|---------|
| **NodeAgeExceeded** | warning | 10m | Any node running longer than 48 hours |
| **DrainFailed** | critical | 0m | A drain operation failed (immediate) |
| **PolicyEvaluationError** | warning | 5m | Errors evaluating NodeTerminationPolicy resources |
| **PDBBlockingDrain** | warning | 0m | PDB blocking drain for >15 minutes |
| **NodeReplacementStalled** | critical | 5m | Node replacement in progress for >1 hour |

### Alert Details

#### NodeAgeExceeded

```yaml
expr: citadel_node_lifecycle_node_age_hours > 48
for: 10m
severity: warning
```

Fires when a node has exceeded the 48-hour lifecycle window. The operator should have already initiated replacement. Investigate whether the operator is running and whether replacement is blocked.

#### DrainFailed

```yaml
expr: increase(citadel_node_lifecycle_drain_result{status="failed"}[10m]) > 0
for: 0m
severity: critical
```

A drain operation failed — pods couldn't be evicted. Blocks node replacement. Check controller logs for eviction errors and PDB status.

#### PolicyEvaluationError

```yaml
expr: increase(citadel_node_lifecycle_policy_evaluation_errors_total[15m]) > 0
for: 5m
severity: warning
```

The lifecycle controller is encountering errors evaluating policies. May indicate CRD schema issues, RBAC problems, or controller bugs.

#### PDBBlockingDrain

```yaml
expr: citadel_node_lifecycle_pdb_blocking_duration_seconds > 900
for: 0m
severity: warning
```

A PodDisruptionBudget has been blocking drain for >15 minutes. Consider enabling `forceAfterTimeoutSeconds` in the policy, or adjusting the PDB.

#### NodeReplacementStalled

```yaml
expr: (time() - citadel_node_lifecycle_replacement_start_time) > 3600
      and citadel_node_lifecycle_replacement_in_progress == 1
for: 5m
severity: critical
```

VM replacement has been in progress for >1 hour. The VM may have failed to create, or K3s agent may have failed to join. Check Multipass VM status and K3s agent logs.

## Hubble Observability

Hubble provides L3/L4/L7 network flow monitoring:

```bash
# Live flow monitoring
hubble observe --follow

# Filter by namespace
hubble observe -n production

# Show only dropped traffic
hubble observe --verdict DROPPED

# Show HTTP traffic with status codes
hubble observe -t l7 --protocol http

# Open Hubble UI
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
# http://localhost:12000
```

## Cilium Metrics

Cilium exports Prometheus metrics on ports 9962 and 9965:

```bash
# Check Cilium Prometheus targets
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
# Navigate to Status → Targets, look for cilium-agent
```

Key Cilium metrics:
- `cilium_policy_verdict` — Policy enforcement decisions
- `cilium_forward_count_total` — Forwarded packets
- `cilium_drop_count_total` — Dropped packets (by reason)
- `cilium_endpoint_count` — Number of managed endpoints
