# Troubleshooting Runbook

Organized by failure mode. Each entry: symptoms, diagnostic commands, resolution.

---

## Operator Won't Start

**Symptoms:** Deployment stays at 0/1 ready. Pod in CrashLoopBackOff or ImagePullBackOff.

**Diagnose:**
```bash
kubectl -n citadel-system get pods
kubectl -n citadel-system describe pod -l app.kubernetes.io/name=citadel-operator
kubectl -n citadel-system logs -l app.kubernetes.io/name=citadel-operator --tail=50
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| CRDs not installed | `citadel install` (re-run, idempotent) or `kubectl apply -f crds/` |
| ClusterRole missing | Check `kubectl get clusterrole citadel-operator` |
| Image pull failure | Verify image exists and is accessible from the cluster |
| kopf not installed | Use `pip install 'citadel-operator[operator]'` |
| Python import error | Check logs for traceback; verify package version |

---

## Drain Stuck

**Symptoms:** Node stays cordoned. Pods not evicted. Policy shows matched but not terminated.

**Diagnose:**
```bash
# Check what's on the node
kubectl get pods --field-selector spec.nodeName=<node> -A -o wide

# Check PDBs
kubectl get pdb -A
kubectl describe pdb <name> -n <namespace>

# Check operator logs for eviction errors
kubectl -n citadel-system logs -l app.kubernetes.io/name=citadel-operator | grep -i "evict\|drain\|pdb"

# Check node labels
kubectl get node <node> --show-labels | tr ',' '\n' | grep citadel
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| PDB blocking eviction | Add `forceAfterTimeoutSeconds` to policy drain config |
| Pod with long `terminationGracePeriodSeconds` | Wait or reduce grace period |
| Finalizer preventing pod deletion | `kubectl edit pod <name>` and remove finalizer |
| Local storage and `deleteEmptydirData: false` | Set `deleteEmptydirData: true` in policy |
| DaemonSet pod (expected) | DaemonSet pods are correctly skipped |

**Manual recovery:**
```bash
# Force-uncordon if drain should be cancelled
kubectl uncordon <node>

# Clear termination labels
kubectl label node <node> citadel.dev/terminate-reason- citadel.dev/terminate-after- citadel.dev/policy-name- citadel.dev/managed-by-
```

---

## Reconciliation Failing (Namespace Infra)

**Symptoms:** NamespaceInfrastructure stuck in Provisioning or Failed phase.

**Diagnose:**
```bash
kubectl get nsinfra -A
kubectl describe nsinfra <name> -n <namespace>
kubectl -n citadel-system logs -l app.kubernetes.io/name=citadel-operator | grep -i "namespace\|infra\|reconcil"
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| Target namespace doesn't exist | `kubectl create ns <target>` |
| CiliumNetworkPolicy CRD missing | Cilium not installed — run `./scripts/02-install-cilium.sh` |
| RBAC missing cilium.io permissions | Check `kubectl get clusterrole citadel-operator -o yaml` |
| Invalid spec (e.g., bad resource values) | Check conditions in status for error message |

---

## Certificate Renewal Failing

**Symptoms:** Certificate stuck in `Issuing` state. Secret not updated. Certificate expired.

**Diagnose:**
```bash
kubectl get certificates -A
kubectl describe certificate <name> -n <namespace>
kubectl get certificaterequests -A
kubectl describe clusterissuer <issuer-name>

# Check cert-manager logs
kubectl -n cert-manager logs -l app=cert-manager --tail=50
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| Webhook not ready | `kubectl rollout status deploy/cert-manager-webhook -n cert-manager` |
| ClusterIssuer misconfigured | `kubectl describe clusterissuer <name>` for status |
| ACME challenge failing | Check HTTP-01 ingress or DNS-01 provider |
| Rate limit (Let's Encrypt) | Switch to `letsencrypt-staging` for testing |
| Expired CA certificate | Rotate the self-signed bootstrap cert |

**Manual rotation:**
```bash
# Delete the backing Secret to force cert-manager to re-issue
kubectl delete secret <cert-secret-name> -n <namespace>
# cert-manager will detect the missing Secret and re-issue
```

---

## Secret Sync Failing

**Symptoms:** ExternalSecret shows `SecretSyncedError`. Target Secret not created or stale.

**Diagnose:**
```bash
kubectl get externalsecrets -n <namespace>
kubectl describe externalsecret <name> -n <namespace>
kubectl get secretstores -n <namespace>
kubectl describe secretstore <name> -n <namespace>
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| SecretStore provider unreachable | Check network connectivity to Vault/AWS/Azure |
| Authentication failure | Verify ServiceAccount token or static credentials |
| Remote key doesn't exist | Check key path in the external store |
| ESO not running | `kubectl get pods -n external-secrets` |

---

## Compliance Scan Not Running

**Symptoms:** No kube-bench results. ConfigMaps empty in citadel-compliance namespace.

**Diagnose:**
```bash
kubectl get cronjobs -n citadel-compliance
kubectl get jobs -n citadel-compliance
kubectl logs -n citadel-compliance -l app.kubernetes.io/name=kube-bench --tail=50
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| CronJob suspended | `kubectl patch cronjob kube-bench-scan -n citadel-compliance -p '{"spec":{"suspend":false}}'` |
| Image not accessible | Check image pull policy and registry access |
| Node selector mismatch | kube-bench needs to run on control plane nodes |

**Trigger manual scan:**
```python
from citadel_operator.k3s.compliance import ComplianceScanner
scanner = ComplianceScanner()
job = scanner.trigger_cis_scan()
print(f"Manual scan job: {job}")
```

---

## Nodes Not Ready After K3s Install

**Symptoms:** `kubectl get nodes` shows NotReady status.

**Diagnose:**
```bash
kubectl get nodes
kubectl describe node <name>
kubectl get pods -n kube-system
```

**Most likely cause:** Cilium not yet installed. K3s was configured with `--flannel-backend=none`, so there's no CNI until Cilium is deployed.

**Fix:**
```bash
./scripts/02-install-cilium.sh
cilium status --wait
```

---

## WireGuard Not Working

**Symptoms:** Traffic not encrypted. Hubble doesn't show encrypted flows.

**Diagnose:**
```bash
# Check WireGuard kernel module
lsmod | grep wireguard

# Check Cilium encryption status
cilium encrypt status

# Check node WireGuard interfaces
kubectl exec -n kube-system ds/cilium -- wg show
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| WireGuard module not loaded | `modprobe wireguard` or install `wireguard-tools` |
| Cilium not configured | Re-run `06-enable-wireguard.sh` |
| Kernel too old | WireGuard requires Linux 5.6+ (or backported module) |

---

## Audit Logs Missing

**Symptoms:** No audit log file. `/var/log/kubernetes/audit/audit.log` doesn't exist or is empty.

**Diagnose:**
```bash
# Check K3s config
grep -A5 "audit" /etc/rancher/k3s/config.yaml

# Check log file
ls -la /var/log/kubernetes/audit/

# Check K3s was restarted after config change
systemctl status k3s
```

**Fix:** K3s must be restarted after audit configuration changes:
```bash
sudo systemctl restart k3s
# Wait for cluster to be ready
kubectl get nodes
# Check audit log
tail -f /var/log/kubernetes/audit/audit.log
```

---

## Dex SSO Authentication Failing

**Symptoms:** OIDC token validation fails. `kubectl` with OIDC token returns 401.

**Diagnose:**
```bash
# Check Dex is running
kubectl get pods -n dex

# Check Dex TLS certificate
kubectl get certificate dex-tls -n dex

# Check K3s OIDC flags
grep -i oidc /etc/rancher/k3s/config.yaml

# Test Dex discovery endpoint
curl -sk https://dex.citadel.local/.well-known/openid-configuration
```

**Common causes and fixes:**

| Cause | Fix |
|-------|-----|
| K3s not restarted after OIDC config | `sudo systemctl restart k3s` |
| Dex TLS cert expired | Delete `dex-tls` secret, cert-manager re-issues |
| Issuer URL mismatch | Ensure K3s `oidc-issuer-url` matches Dex's `issuer` config |
| DNS not resolving `dex.citadel.local` | Add to `/etc/hosts` or configure CoreDNS |
