# Scripts Reference

All setup scripts are in the `scripts/` directory. They share common utilities from `scripts/common.sh`.

## Execution Order

```
01-install-k3s.sh          Required — K3s bootstrap (root)
02-install-cilium.sh        Required — Cilium CNI + Hubble
03-install-monitoring.sh    Optional — Prometheus + Grafana
04-deploy-demo.sh           Optional — 3-tier demo app
05-test-policies.sh         Optional — Policy verification (needs demo)
06-enable-wireguard.sh      Optional — WireGuard encryption (root)
07-enable-audit-logging.sh  Optional — API audit logs (root, restart)
08-enable-encryption.sh     Optional — etcd encryption (root, restart)
09-install-cert-manager.sh  Optional — TLS certificates
10-install-dex.sh           Optional — SSO/OIDC (needs cert-manager, restart)
```

Dependencies: 01 → 02 → (03-10 mostly independent, except 10 requires 09).

## Shared Utilities: `common.sh`

All scripts source this file for shared functions.

### Environment

```bash
export KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}
```

### Functions

| Function | Purpose |
|----------|---------|
| `log()` | Timestamped message |
| `log_info()` | Blue `[INFO]` prefix |
| `log_warn()` | Yellow `[WARN]` prefix |
| `log_error()` | Red `[ERROR]` to stderr |
| `log_success()` | Green `[OK]` prefix |
| `require_kubectl()` | Verify kubectl installed and cluster accessible |
| `require_helm()` | Verify Helm installed |
| `require_cilium()` | Verify Cilium pods running in kube-system |
| `require_root()` | Verify running as root (EUID == 0) |
| `get_repo_root()` | Get repo root from any script location |

---

## Script Details

### `01-install-k3s.sh` — K3s Bootstrap

| Property | Value |
|----------|-------|
| Prerequisites | 2+ CPU, 4GB+ RAM, 20GB+ disk |
| Requires root | Yes |
| Idempotent | No (fails if K3s already installed) |
| Requires restart | No |

**Creates:**
- K3s systemd service
- Kubeconfig at `/etc/rancher/k3s/k3s.yaml` (mode 644)
- Copy at `$HOME/.kube/config` for `SUDO_USER`

**K3s flags:**
```bash
--flannel-backend=none          # Cilium provides CNI
--disable-kube-proxy            # Cilium replaces kube-proxy
--disable-network-policy        # Cilium provides enhanced policies
--disable servicelb             # User supplies own ingress
--disable traefik               # User deploys own ingress
--write-kubeconfig-mode 644     # Readable by non-root
```

After running, nodes appear `NotReady` until Cilium is installed.

---

### `02-install-cilium.sh` — Cilium CNI + Hubble

| Property | Value |
|----------|-------|
| Prerequisites | K3s running |
| Requires root | No |
| Idempotent | Yes (helm upgrade --install) |
| Requires restart | No |

**Cilium version:** 1.18.6

**Creates:**
- Cilium DaemonSet (kube-system)
- Hubble Relay + UI deployments (kube-system)
- SPIRE server + agents (cilium-spire namespace) for mTLS
- Cilium CLI + Hubble CLI in `/usr/local/bin`

**Key settings:**
- `kubeProxyReplacement=true`
- `hubble.enabled=true` with relay and UI
- `prometheus.enabled=true`
- `authentication.mutual.spire.enabled=true`
- Pod CIDR: `10.42.0.0/16`

**Verification:**
```bash
cilium status --wait
cilium connectivity test
```

---

### `03-install-monitoring.sh` — Prometheus + Grafana

| Property | Value |
|----------|-------|
| Prerequisites | K3s + Cilium running |
| Requires root | No |
| Idempotent | Yes (helm upgrade --install) |
| Requires restart | No |

**Creates:**
- `monitoring` namespace
- Prometheus, Grafana, AlertManager, node-exporter, kube-state-metrics

**Access:**
```bash
# Grafana (admin / auto-generated password printed to stdout)
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

# Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
```

**Recommended dashboards:** Cilium (16611), Hubble (16613).

---

### `04-deploy-demo.sh` — 3-Tier Demo App

| Property | Value |
|----------|-------|
| Prerequisites | Cilium running |
| Requires root | No |
| Idempotent | Yes (kubectl apply) |
| Requires restart | No |

**Creates:**
- `demo` namespace
- Pods: frontend (nginx), backend (Flask), database (Redis), test-client
- Default-deny + system policies
- Application-specific policies

Architecture: `test-client → frontend:80 → backend:5678 → database:6379`

---

### `05-test-policies.sh` — Network Policy Verification

| Property | Value |
|----------|-------|
| Prerequisites | Demo app deployed (04) |
| Requires root | No |
| Idempotent | N/A (test runner) |
| Requires restart | No |

**Tests:**
- test-client → frontend (allowed)
- test-client → backend (allowed)
- frontend → backend (allowed)
- backend → database (allowed)
- test-client → database (blocked)
- frontend → database (blocked)
- database → backend (egress blocked)

Exit code 0 = all passed, 1 = any failed.

---

### `06-enable-wireguard.sh` — WireGuard Encryption

| Property | Value |
|----------|-------|
| Prerequisites | Cilium running, WireGuard kernel module |
| Requires root | Yes |
| Idempotent | Yes (helm upgrade --reuse-values) |
| Requires restart | No |

**Enables:** Transparent pod-to-pod encryption via WireGuard. Creates test pods for verification.

**Verification:** `hubble observe --namespace demo` shows encrypted flows.

---

### `07-enable-audit-logging.sh` — Kubernetes Audit Logs

| Property | Value |
|----------|-------|
| Prerequisites | K3s running |
| Requires root | Yes |
| Idempotent | Partial (checks for duplicates) |
| Requires restart | **Yes** (`systemctl restart k3s`) |

**Creates:**
- Audit policy: `/etc/rancher/k3s/audit-policy.yaml`
- Audit log dir: `/var/log/kubernetes/audit/`
- Log rotation: `/etc/logrotate.d/kubernetes-audit` (daily, 30-day retention)
- Fluent Bit config for optional forwarding

**K3s flags added:**
```yaml
kube-apiserver-arg:
  - "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
  - "audit-log-path=/var/log/kubernetes/audit/audit.log"
  - "audit-log-maxage=30"
  - "audit-log-maxbackup=10"
  - "audit-log-maxsize=100"
```

---

### `08-enable-encryption.sh` — Data-at-Rest Encryption

| Property | Value |
|----------|-------|
| Prerequisites | K3s running |
| Requires root | Yes |
| Idempotent | Yes (checks for existing config) |
| Requires restart | **Yes** (`systemctl restart k3s`) |

**Creates:**
- AES-256 encryption key (32 bytes, base64)
- Encryption config: `/etc/rancher/k3s/encryption-config.yaml`
- Key backup: `/etc/rancher/k3s/encryption-keys/citadel-key-1.<timestamp>.key`

**Encrypts:** Secrets, ConfigMaps, ServiceAccounts, PVCs, CRDs in etcd.

**Verification:** `k8s:enc:aesgcm:v1:citadel-key-1` prefix on stored secrets.

---

### `09-install-cert-manager.sh` — TLS Certificates

| Property | Value |
|----------|-------|
| Prerequisites | K3s running, Helm |
| Requires root | No |
| Idempotent | Yes (helm upgrade --install) |
| Requires restart | No |

**cert-manager version:** v1.14.4

**Creates:**
- `cert-manager` namespace
- cert-manager, webhook, cainjector deployments (2 replicas each)
- 4 ClusterIssuers: `letsencrypt-staging`, `letsencrypt-prod`, `citadel-selfsigned-bootstrap`, `internal-ca`
- Internal CA certificate and root CA secret

**Certificate defaults:** 90-day duration, 30-day renewal, ECDSA P-256.

---

### `10-install-dex.sh` — SSO/OIDC Provider

| Property | Value |
|----------|-------|
| Prerequisites | cert-manager running |
| Requires root | No |
| Idempotent | Yes (helm upgrade --install) |
| Requires restart | **Yes** (K3s OIDC flags) |

**Dex version:** 0.18.0

**Creates:**
- `dex` namespace
- Dex deployment (2 replicas)
- TLS certificate for `dex.citadel.local`
- RBAC bindings for OIDC groups
- K3s API server OIDC configuration

**OIDC clients:** `kubernetes` (public), `citadel-dashboard` (authz code + PKCE), `citadel-cli` (device code).

---

### `node-lifecycle.sh` — Node Age Checker

| Property | Value |
|----------|-------|
| Prerequisites | K3s running |
| Requires root | No |
| Idempotent | N/A (operational) |
| Requires restart | No |

**Commands:**
```bash
node-lifecycle.sh status              # Show all node ages
node-lifecycle.sh check [node]        # Check specific or all nodes
node-lifecycle.sh cycle               # Find and drain aged nodes
node-lifecycle.sh drain <node>        # Drain specific node
```

**Options:** `--max-age HOURS`, `--dry-run`

**Environment:** `NODE_MAX_AGE_HOURS` (default: 48), `DRAIN_TIMEOUT` (default: 300), `DRY_RUN=true`.
