# Citadel Operations Reference

How the system runs in production — deployment topology, CLIs, scripts, CRDs, monitoring, and troubleshooting.

## Deployment Topology

The operator runs as a single-replica Deployment in `citadel-system` namespace:

| Property | Value |
|----------|-------|
| Replicas | 1 |
| ServiceAccount | `citadel-operator` |
| Security context | Non-root, uid 1000 |
| CPU requests/limits | 100m / 500m |
| Memory requests/limits | 128Mi / 256Mi |
| Liveness probe | HTTP GET `/healthz` on port 8081 |
| Entrypoint | `kopf run --standalone --liveness=http://0.0.0.0:8081/healthz` |
| Modules loaded | `citadel_operator.k3s.lifecycle_operator`, `citadel_operator.k3s.namespace_infra_operator` |

---

## Two CLIs

Citadel has two CLIs serving different purposes:

### `citadel` — Operator Deployment CLI

Installed via pip (`pip install jaymd96-citadel-operator`). Manages the operator lifecycle within an existing cluster.

```
citadel install   [--namespace NS] [--image IMG] [--with-default-policies] [--no-default-policies] [--timeout SECS]
citadel uninstall [--namespace NS] [--delete-crds]
citadel status    [--namespace NS]
citadel manifests [--namespace NS] [--image IMG]
```

Creates: Namespace, 2 CRDs, ServiceAccount, ClusterRole (14 rules), ClusterRoleBinding, Deployment, and optionally 6 default NodeTerminationPolicies.

### `./citadel-ctl` — Infrastructure Setup CLI

Executable Python script in repo root. Wraps shell scripts and kubectl commands for bootstrapping the cluster infrastructure.

```
./citadel-ctl cluster create <name> [--nodes N]   Create K3s cluster via Rancher Desktop
./citadel-ctl cluster delete <name>               Destroy cluster VMs
./citadel-ctl cluster list                        List all VMs

./citadel-ctl node status                         Show node ages and health
./citadel-ctl node list                           List nodes with lifecycle labels

./citadel-ctl lifecycle install                   Apply CRDs + RBAC + default policies
./citadel-ctl lifecycle policies                  Show active NodeTerminationPolicies

./citadel-ctl network install                     Install Cilium CNI (02-install-cilium.sh)
./citadel-ctl network policies                    Apply all policies/ YAML files
./citadel-ctl network test                        Run network policy tests (05-test-policies.sh)

./citadel-ctl monitor install                     Install Prometheus + Grafana (03-install-monitoring.sh)

./citadel-ctl security enable-wireguard           Enable WireGuard encryption (06-enable-wireguard.sh)
./citadel-ctl security enable-audit               Enable audit logging (07-enable-audit-logging.sh)
```

---

## Installation Procedure

### Full Setup Order

1. `sudo ./scripts/01-install-k3s.sh` — Bootstrap K3s (requires root)
2. `./scripts/02-install-cilium.sh` — Install Cilium CNI + Hubble
3. `./scripts/03-install-monitoring.sh` — Prometheus + Grafana (optional)
4. `./scripts/04-deploy-demo.sh` — Demo app (optional)
5. `./scripts/05-test-policies.sh` — Verify policies (optional, needs demo)
6. `sudo ./scripts/06-enable-wireguard.sh` — WireGuard encryption (optional)
7. `sudo ./scripts/07-enable-audit-logging.sh` — Audit logging (optional, requires restart)
8. `sudo ./scripts/08-enable-encryption.sh` — Data-at-rest encryption (optional, requires restart)
9. `./scripts/09-install-cert-manager.sh` — cert-manager + issuers (optional)
10. `./scripts/10-install-dex.sh` — Dex SSO (optional, requires cert-manager)
11. `pip install jaymd96-citadel-operator && citadel install` — Deploy the operator

### What `citadel install` Creates

1. Namespace `citadel-system`
2. CRD `nodeterminationpolicies.citadel.dev`
3. CRD `namespaceinfrastructures.citadel.dev`
4. ServiceAccount `citadel-operator`
5. ClusterRole `citadel-operator` (14 permission rules)
6. ClusterRoleBinding
7. Deployment `citadel-operator` (1 replica)
8. Default policies (if `--with-default-policies`):
   - `max-age-48h` (priority 10)
   - `disk-pressure` (priority 100)
   - `memory-pressure` (priority 100)
   - `not-ready` (priority 200)
   - `stuck-volume-detector` (priority 150, disabled)
   - `gpu-health-detector` (priority 150, disabled)

---

## Script Reference

All scripts source `scripts/common.sh` which exports `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` by default.

| Script | Purpose | Prerequisites | Requires Root | Idempotent | Requires Restart |
|--------|---------|---------------|---------------|------------|-----------------|
| `01-install-k3s.sh` | Bootstrap K3s with security defaults | 2+ CPU, 4GB+ RAM, 20GB+ disk | Yes | No | — |
| `02-install-cilium.sh` | Install Cilium CNI + Hubble + SPIRE mTLS | K3s running | No | Yes (helm upgrade) | No |
| `03-install-monitoring.sh` | Install kube-prometheus-stack | K3s + Cilium | No | Yes (helm upgrade) | No |
| `04-deploy-demo.sh` | Deploy 3-tier demo app (nginx → Flask → Redis) | Cilium | No | Yes (kubectl apply) | No |
| `05-test-policies.sh` | Verify network policy enforcement | Demo app deployed | No | N/A (tests) | No |
| `06-enable-wireguard.sh` | Enable WireGuard pod-to-pod encryption | Cilium, WireGuard module | Yes | Yes (helm upgrade) | No |
| `07-enable-audit-logging.sh` | Configure K3s audit logging | K3s running | Yes | Partial | Yes |
| `08-enable-encryption.sh` | Enable AES-GCM etcd encryption | K3s running | Yes | Yes | Yes |
| `09-install-cert-manager.sh` | Install cert-manager + 4 ClusterIssuers | K3s, Helm | No | Yes (helm upgrade) | No |
| `10-install-dex.sh` | Install Dex OIDC provider | cert-manager | No | Yes (helm upgrade) | Yes (K3s OIDC flags) |
| `node-lifecycle.sh` | Node age checking and manual drain | K3s | No | N/A (operations) | No |
| `common.sh` | Shared utilities (logging, prereq checks) | — | — | — | — |

### `common.sh` Shared Functions

| Function | Purpose |
|----------|---------|
| `log()` | Timestamped message |
| `log_info()` / `log_warn()` / `log_error()` / `log_success()` | Colored log levels |
| `require_kubectl()` | Verify kubectl and cluster connectivity |
| `require_helm()` | Verify Helm installed |
| `require_cilium()` | Verify Cilium pods running |
| `require_root()` | Verify EUID == 0 |
| `get_repo_root()` | Get repo root from any script location |

### K3s Configuration (01-install-k3s.sh)

```bash
--flannel-backend=none        # Cilium provides CNI
--disable-kube-proxy          # Cilium replaces kube-proxy
--disable-network-policy      # Cilium provides enhanced policies
--disable servicelb            # User supplies own ingress
--disable traefik              # User deploys own ingress
--write-kubeconfig-mode 644    # Readable by non-root
```

### Cilium Configuration (02-install-cilium.sh)

Version: 1.18.6

Key settings:
- `kubeProxyReplacement=true`
- `hubble.enabled=true` with relay and UI
- `prometheus.enabled=true`
- `authentication.mutual.spire.enabled=true` (mTLS via SPIRE)
- `ipam.operator.clusterPoolIPv4PodCIDRList=10.42.0.0/16`

---

## CRD Operations

### NodeTerminationPolicy

```yaml
apiVersion: citadel.dev/v1alpha1
kind: NodeTerminationPolicy
metadata:
  name: max-age-48h
spec:
  reason: "Node exceeded 48-hour maximum age"
  priority: 10          # 0-1000, higher = evaluated first
  enabled: true
  dryRun: false          # Label but don't drain
  selector:
    maxAgeHours: 48
    conditions:          # Match K8s node conditions
      - type: DiskPressure
        status: "True"
    customProbes:        # Custom failure detection
      - name: stuck-volume
        type: exec
        exec:
          command: ["/bin/sh", "-c", "timeout 5 ls /var/lib/kubelet/pods/*/volumes/"]
        failureThreshold: 3
        periodSeconds: 60
    nodeSelector:        # Label-based node targeting
      node-pool: gpu
  drain:
    timeoutSeconds: 300
    deleteEmptydirData: true
    forceAfterTimeoutSeconds: 600  # Force-delete after timeout
```

Short names: `ntp`, `ntpolicy`

Status subresource: `lastEvaluated`, `nodesMatched`, `nodesTerminated`, `lastTermination`, `conditions`

Printer columns: Reason, Priority, Enabled, Matched, Terminated, Age

### NamespaceInfrastructure

```yaml
apiVersion: citadel.dev/v1alpha1
kind: NamespaceInfrastructure
metadata:
  name: team-alpha-infra
  namespace: citadel-system
spec:
  targetNamespace: team-alpha
  networkPolicy:
    defaultDeny: true
    allowedPeers: ["shared-services"]
    allowKubeSystem: true
    allowMonitoring: true
  resourceQuota:
    cpu: "4"
    memory: "8Gi"
    pods: "20"
  limitRange:
    defaultCpu: "500m"
    defaultMemory: "512Mi"
    maxCpu: "4"
    maxMemory: "8Gi"
```

Short names: `nsi`, `nsinfra`

Status subresource: `phase` (Pending → Provisioning → Active/Failed), `ready`, `managedResources`, `conditions`, `policyHash`

Phase lifecycle:
1. **Pending** — Resource created, not yet reconciled
2. **Provisioning** — Reconciler applying resources
3. **Active** — All resources applied, ready=true
4. **Failed** — Error during provisioning

Drift detection: 5-minute timer recomputes policy hash. If hash differs, re-reconciles.

---

## Network Policy Architecture

8 policy files in `policies/`, applied in alphabetical order:

| File | Scope | Purpose |
|------|-------|---------|
| `01-default-deny.yaml` | Cluster-wide | Deny all + allow DNS + allow API server + allow health checks |
| `02-system-policies.yaml` | kube-system | Cilium agent, Hubble relay/UI, CoreDNS |
| `03-demo-app-policies.yaml` | demo | L7 HTTP policies for frontend→backend→database |
| `04-l7-demo-policies.yaml` | demo | HTTP method+path filtering |
| `05-fqdn-egress-policies.yaml` | demo | FQDN-based egress (github.com, stripe.com, AWS) |
| `06-pod-disruption-budgets.yaml` | demo, kube-system | PDBs for frontend, backend, database |
| `07-resource-quotas.yaml` | demo | ResourceQuota, LimitRange, PriorityClasses |
| `08-pod-security-standards.yaml` | demo, staging, production | PSS enforcement + metadata access blocking |

The namespace infra operator generates additional per-namespace CiliumNetworkPolicies (default-deny + tenant-allow + peer policies).

---

## Monitoring and Alerting

### Stack

Installed by `03-install-monitoring.sh` (kube-prometheus-stack Helm chart):
- Prometheus (metrics scraping)
- Grafana (dashboards, default login: admin / auto-generated)
- AlertManager (alert routing)
- node-exporter DaemonSet (host metrics)
- kube-state-metrics (K8s object metrics)

### Access

```bash
# Grafana
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
# http://localhost:3000

# Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
# http://localhost:9090
```

### PrometheusRule Alerts

File: `monitoring/node-lifecycle-alerts.yaml`

**IMPORTANT: The operator does not yet export Prometheus metrics.** These alerts are pre-positioned for when metrics are implemented. The expected metrics prefix is `citadel_node_lifecycle_`.

| Alert | Expression | For | Severity | Description |
|-------|-----------|-----|----------|-------------|
| NodeAgeExceeded | `citadel_node_lifecycle_node_age_hours > 48` | 10m | warning | Node exceeded 48h lifecycle |
| DrainFailed | `increase(citadel_node_lifecycle_drain_result{status="failed"}[10m]) > 0` | 0m | critical | Drain operation failed |
| PolicyEvaluationError | `increase(citadel_node_lifecycle_policy_evaluation_errors_total[15m]) > 0` | 5m | warning | Policy evaluation errors |
| PDBBlockingDrain | `citadel_node_lifecycle_pdb_blocking_duration_seconds > 900` | 0m | warning | PDB blocking drain >15min |
| NodeReplacementStalled | `(time() - citadel_node_lifecycle_replacement_start_time) > 3600 and replacement_in_progress == 1` | 5m | critical | Replacement stuck >1hr |

Expected metrics the operator should export:

| Metric | Type | Labels |
|--------|------|--------|
| `citadel_node_lifecycle_node_age_hours` | gauge | `node` |
| `citadel_node_lifecycle_drain_result` | counter | `node`, `status` (success/failed) |
| `citadel_node_lifecycle_policy_evaluation_errors_total` | counter | `policy` |
| `citadel_node_lifecycle_pdb_blocking_duration_seconds` | gauge | `node`, `namespace`, `pdb` |
| `citadel_node_lifecycle_replacement_in_progress` | gauge | `node` |
| `citadel_node_lifecycle_replacement_start_time` | gauge | `node` |

### Recommended Grafana Dashboards

| Dashboard | Import ID | Purpose |
|-----------|-----------|---------|
| Cilium Metrics | 16611 | CNI health, policy enforcement |
| Hubble Metrics | 16613 | Network flow visibility |

---

## Security Operations

### Certificates (cert-manager)

4 ClusterIssuers installed by `09-install-cert-manager.sh`:

| Issuer | Purpose | Trust |
|--------|---------|-------|
| `letsencrypt-staging` | ACME testing | Not browser-trusted |
| `letsencrypt-prod` | Production external | Browser-trusted (rate-limited) |
| `citadel-selfsigned-bootstrap` | Root CA generation | Internal only |
| `internal-ca` | Service-to-service mTLS | Self-signed CA |

Defaults: 90-day duration, 30-day renewal window, ECDSA P-256, always rotate key.

### Secrets (external-secrets-operator)

Providers: kubernetes, vault, aws, azure

Sync intervals: 5m (sensitive), 1h (standard), 24h (stable)

### Compliance (kube-bench + Trivy)

- kube-bench CronJob in `citadel-compliance` namespace → CIS Benchmark results in ConfigMaps
- Trivy Operator → VulnerabilityReports, ConfigAuditReports CRDs

### SSO (Dex)

OIDC group mappings:
- `oidc:platform-admins` → `cluster-admin`
- `oidc:platform-ops` → `citadel-operator`
- `oidc:developers` → `citadel-developer` (namespace: default)
- `oidc:viewers` → `view`

### mTLS (SPIRE)

Trust domain: `spiffe://citadel.local/ns/{namespace}/sa/{serviceaccount}`

Transparent at eBPF layer — no application changes.

### Data-at-Rest Encryption

AES-GCM encryption of etcd Secrets, ConfigMaps, ServiceAccounts, PVCs, CRDs.

Key stored at: `/etc/rancher/k3s/encryption-keys/`

### Audit Logging

Logs written to: `/var/log/kubernetes/audit/audit.log`

Rotation: Daily, 30-day retention, compressed.

---

## Troubleshooting Runbook

### Operator Won't Start

**Symptoms**: Deployment stays at 0/1 ready, pod in CrashLoopBackOff.

```bash
kubectl -n citadel-system get pods
kubectl -n citadel-system describe pod -l app.kubernetes.io/name=citadel-operator
kubectl -n citadel-system logs -l app.kubernetes.io/name=citadel-operator
```

**Common causes**:
- Missing CRDs → `citadel install` or `kubectl apply -f crds/`
- RBAC insufficient → check ClusterRole exists
- Image pull failure → verify image is accessible
- kopf not installed → use `citadel-operator[operator]` extra

### Drain Stuck

**Symptoms**: Node stays cordoned, pods not evicted.

```bash
kubectl get pods --field-selector spec.nodeName=<node> -A
kubectl get pdb -A
kubectl describe pdb <name> -n <ns>
```

**Common causes**:
- PDB blocking eviction → consider `forceAfterTimeoutSeconds` in policy
- Pod with long `terminationGracePeriodSeconds`
- Finalizer preventing pod deletion
- Local storage (emptyDir) and `deleteEmptydirData: false`

### Reconciliation Failing (Namespace Infra)

**Symptoms**: NamespaceInfrastructure stuck in Provisioning or Failed.

```bash
kubectl get nsinfra -A
kubectl describe nsinfra <name> -n <ns>
kubectl -n citadel-system logs -l app.kubernetes.io/name=citadel-operator | grep namespace-infra
```

**Common causes**:
- Target namespace doesn't exist
- CiliumNetworkPolicy CRD not installed (Cilium not running)
- RBAC missing CiliumNetworkPolicy permissions

### Certificate Renewal Failing

**Symptoms**: Certificate stuck in `Issuing` or expired.

```bash
kubectl get certificates -A
kubectl describe certificate <name> -n <ns>
kubectl get certificaterequests -A
kubectl describe clusterissuer <issuer-name>
```

**Common causes**:
- cert-manager webhook not ready → `kubectl rollout status deployment/cert-manager-webhook -n cert-manager`
- ClusterIssuer misconfigured
- ACME challenge failing (DNS/HTTP-01)

### Secret Sync Failing

**Symptoms**: ExternalSecret shows `SecretSyncedError`.

```bash
kubectl get externalsecrets -n <ns>
kubectl describe externalsecret <name> -n <ns>
kubectl get secretstores -n <ns>
```

**Common causes**:
- SecretStore provider unreachable
- Authentication failure (expired token, wrong credentials)
- Remote key path does not exist

### Compliance Scan Not Running

**Symptoms**: No kube-bench results in ConfigMaps.

```bash
kubectl get cronjobs -n citadel-compliance
kubectl get jobs -n citadel-compliance
kubectl logs -n citadel-compliance -l app.kubernetes.io/name=kube-bench
```

**Common causes**:
- CronJob suspended
- Node selector mismatch
- kube-bench image not accessible

---

## Operational Procedures

### Adding a New Termination Policy

```yaml
apiVersion: citadel.dev/v1alpha1
kind: NodeTerminationPolicy
metadata:
  name: my-custom-policy
spec:
  reason: "Custom policy triggered"
  priority: 50
  enabled: true
  dryRun: true    # Start with dry-run!
  selector:
    maxAgeHours: 24
  drain:
    timeoutSeconds: 300
```

1. Create with `dryRun: true`
2. Wait for evaluation cycle (60s)
3. Check `kubectl get ntp my-custom-policy` for matched count
4. If matches look correct, set `dryRun: false`

### Onboarding a Namespace

```yaml
apiVersion: citadel.dev/v1alpha1
kind: NamespaceInfrastructure
metadata:
  name: team-beta-infra
  namespace: citadel-system
spec:
  targetNamespace: team-beta
  networkPolicy:
    defaultDeny: true
    allowedPeers: ["shared-services"]
  resourceQuota:
    cpu: "8"
    memory: "16Gi"
    pods: "50"
```

1. Create the target namespace first: `kubectl create ns team-beta`
2. Apply the NamespaceInfrastructure resource
3. Wait for phase → Active: `kubectl get nsinfra -n citadel-system`
4. Verify resources: `kubectl get ciliumnetworkpolicies,resourcequotas,limitranges -n team-beta`

### Rotating Certificates

```python
from citadel_operator import CertificateManager

cm = CertificateManager()
# Check what needs renewal
expiring = cm.check_expiry(namespace="dex")
# Rotate by deleting the backing Secret (cert-manager re-issues)
cm.rotate_certificate("dex-tls", namespace="dex")
```

Or manually: `kubectl delete secret dex-tls -n dex` — cert-manager re-creates it.

### Upgrading the Operator

```bash
pip install --upgrade citadel-operator
citadel install --image citadel-operator:new-version
```

The install command is idempotent — it patches existing resources.

---

## Known Limitations

| Limitation | Impact | Mitigation |
|------------|--------|------------|
| No Prometheus metrics exported | Alerts in monitoring/ are aspirational | Metrics export planned for future release |
| Single-replica operator only | No HA, single point of failure | Kopf leader election via leases prevents split-brain |
| Rancher Desktop-only provider | Cannot replace nodes on cloud VMs | Extend `RancherDesktopProvider` or add cloud providers |
| No automated node replacement post-drain | Drained nodes stay cordoned | Requires manual or external automation to create replacement |
| No CRD validation webhooks | Invalid CRDs accepted by API server | Validated at reconciliation time |
| Remaining `citadel` naming | CRD group is `citadel.dev`, namespace is `citadel-system` | Breaking change deferred |
| No multi-tenancy in operator | One operator watches all namespaces | Acceptable for single-cluster deployments |

---

## Key Ports and Services

| Service | Port | Protocol | Namespace | Purpose |
|---------|------|----------|-----------|---------|
| CoreDNS | 53 | UDP/TCP | kube-system | DNS resolution |
| kube-apiserver | 6443 | TCP | (host) | K8s API |
| Cilium agent | 4244 | TCP | kube-system | Hubble relay endpoint |
| Hubble relay | 4245 | TCP | kube-system | Flow aggregation |
| Hubble UI | 8081 | TCP | kube-system | Network visualization |
| Prometheus | 9090 | TCP | monitoring | Metrics |
| Grafana | 3000 | TCP | monitoring | Dashboards |
| Dex | 5556 | HTTPS | dex | OIDC provider |

---

## Environment Variables

| Variable | Default | Purpose |
|----------|---------|---------|
| `KUBECONFIG` | `/etc/rancher/k3s/k3s.yaml` | K8s config path |
| `NODE_MAX_AGE_HOURS` | `48` | Max node age (node-lifecycle.sh) |
| `DRAIN_TIMEOUT` | `300` | Drain timeout seconds (node-lifecycle.sh) |
| `DRY_RUN` | `false` | Dry-run mode (node-lifecycle.sh) |
