# Architecture

Citadel is a lightweight, security-first Kubernetes platform built on K3s and Cilium. It provides hardened cluster infrastructure with zero-trust networking, ephemeral node lifecycle, and comprehensive observability.

## What Citadel Is (and Isn't)

**Citadel is infrastructure substrate.** It knows *how* to manage clusters — drain nodes, enforce network policies, provision namespace infrastructure, rotate certificates. It does not decide *when* or *what* to deploy. Application orchestration (release channels, deployment plans, canary rollouts) belongs to a separate control plane (e.g., Apollo).

**Citadel is not a cloud provider.** It runs on K3s clusters bootstrapped via Rancher Desktop for development. Production use requires adapting the provider layer for your infrastructure.

## Component Map

```
┌──────────────────────────────────────────────────────────────────────┐
│                          Citadel Platform                            │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────────┐  ┌─────────────────────┐                   │
│  │   citadel CLI        │  │   ./citadel-ctl CLI        │                  │
│  │   (pip install)      │  │   (repo script)      │                  │
│  │                      │  │                      │                   │
│  │  install/uninstall   │  │  cluster create/del  │                  │
│  │  status/manifests    │  │  network/monitor     │                  │
│  │                      │  │  security/lifecycle  │                   │
│  │  Deploys operator    │  │  Bootstraps infra    │                  │
│  └──────────┬───────────┘  └──────────┬───────────┘                  │
│             │                         │                              │
│             ▼                         ▼                              │
│  ┌──────────────────────────────────────────────────────────┐       │
│  │                    Kubernetes (K3s)                       │       │
│  │                                                          │       │
│  │  ┌─────────────────────────────────────────────┐        │       │
│  │  │         citadel-operator Deployment          │        │       │
│  │  │         (citadel-system namespace)            │        │       │
│  │  │                                              │        │       │
│  │  │  ┌──────────────────┐ ┌───────────────────┐ │        │       │
│  │  │  │  Lifecycle       │ │  Namespace Infra   │ │        │       │
│  │  │  │  Operator        │ │  Operator          │ │        │       │
│  │  │  │                  │ │                    │ │        │       │
│  │  │  │  Watches NTP CRD │ │  Watches NSI CRD  │ │        │       │
│  │  │  │  60s eval timer  │ │  300s drift check  │ │        │       │
│  │  │  │  Drain/label     │ │  Provision/cleanup │ │        │       │
│  │  │  └──────────────────┘ └───────────────────┘ │        │       │
│  │  └─────────────────────────────────────────────┘        │       │
│  │                                                          │       │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │       │
│  │  │ Cilium CNI   │  │ Prometheus   │  │ cert-manager │  │       │
│  │  │ + Hubble     │  │ + Grafana    │  │ + Dex SSO    │  │       │
│  │  │ + SPIRE mTLS │  │ + Alerts     │  │ + ESO        │  │       │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  │       │
│  └──────────────────────────────────────────────────────────┘       │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
```

## Two CLIs

| CLI | Install method | Purpose | Scope |
|-----|---------------|---------|-------|
| `citadel` | `pip install jaymd96-citadel-operator` | Deploy/manage the operator in an existing cluster | Operator lifecycle only |
| `./citadel-ctl` | Run from repo root | Bootstrap cluster infrastructure (K3s, Cilium, monitoring, security) | Full infrastructure setup |

See [CLI Reference](cli-reference.md) for complete documentation.

## Operators

### Node Lifecycle Operator

Reconciles `NodeTerminationPolicy` CRDs (group: `citadel.dev/v1alpha1`). Evaluates policies on a 60-second timer, matching nodes by age, conditions, custom probes, and label selectors. Matched nodes are labeled with termination intent, then drained using the Kubernetes Eviction API (respecting PDBs).

Key components:
- `LifecyclePolicyEvaluator` — Pure evaluation logic, no I/O
- `LifecycleController` — Wires evaluator to `NodeManager`
- `HealthChecker` — 4-state model (HEALTHY → WARNING → ERROR → TERMINAL)
- `CustomProbeChecker` — Exec and HTTP probes with failure thresholds

See [Node Lifecycle Operator](operators/node-lifecycle.md).

### Namespace Infrastructure Operator

Reconciles `NamespaceInfrastructure` CRDs. Provisions network policies (CiliumNetworkPolicy), resource quotas, and limit ranges per namespace. Tracks phase (Pending → Provisioning → Active/Failed) and detects configuration drift via policy hash on a 5-minute timer.

Key components:
- `NamespaceInfraReconciler` — Reconciliation logic
- `namespace_infra_builder` — Pure functions for resource generation

See [Namespace Infrastructure Operator](operators/namespace-infra.md).

## Module Map

```
src/citadel_operator/
├── __init__.py              # Package version + 26 public API exports
├── cli/                     # CLI for operator deployment
│   ├── __init__.py          # Argument parser, subcommand dispatch
│   ├── _install.py          # Create-or-patch deployment logic
│   ├── _uninstall.py        # Removal logic
│   ├── _status.py           # Health check
│   └── _manifests.py        # Pure manifest generation
├── k3s/                     # Operators + utilities
│   ├── __init__.py          # Re-exports from all modules
│   ├── lifecycle_operator.py    # Kopf handlers for NodeTerminationPolicy
│   ├── lifecycle_models.py      # CRD dataclasses
│   ├── namespace_infra_operator.py  # Kopf handlers for NamespaceInfrastructure
│   ├── namespace_infra_models.py    # CRD dataclasses
│   ├── namespace_infra_builder.py   # Pure resource builders
│   ├── node.py              # NodeManager (cordon, drain, label)
│   ├── health.py            # HealthChecker (4-state model)
│   ├── custom_probes.py     # CustomProbeChecker (exec + HTTP)
│   ├── probe_discovery.py   # ProbeDiscoverer (Apollo annotations)
│   ├── certificates.py      # CertificateManager (cert-manager)
│   ├── secrets.py           # SecretsManager (external-secrets-operator)
│   ├── compliance.py        # ComplianceScanner (kube-bench + Trivy)
│   ├── sso.py               # SSOManager (Dex OIDC)
│   └── provider.py          # RancherDesktopProvider (VM lifecycle)
└── crds/                    # Bundled in wheel
    ├── nodeterminationpolicy.yaml
    ├── namespaceinfrastructure.yaml
    └── default-policies.yaml
```

## Technology Stack

| Component | Technology | Version |
|-----------|-----------|---------|
| Kubernetes | K3s | Latest stable |
| CNI | Cilium | 1.18.6 |
| Observability | Hubble + Prometheus + Grafana | — |
| mTLS | SPIRE (via Cilium) | — |
| Encryption | WireGuard (transit) + AES-GCM (rest) | — |
| Certificates | cert-manager | v1.14.4 |
| SSO | Dex | 0.18.0 |
| Secrets | external-secrets-operator | — |
| Compliance | kube-bench + Trivy Operator | — |
| Operator framework | kopf | >= 1.37 |
| Language | Python | >= 3.11 |
| Build system | Hatch | — |

## Design Principles

- **Zero-trust by default** — All traffic denied unless explicitly allowed via CiliumNetworkPolicy
- **Identity-based security** — Policies use pod labels and SPIFFE identities, not IP addresses
- **Ephemeral infrastructure** — No node lives longer than 48 hours; immutable and replaced, never upgraded
- **Pure + stateful separation** — Configuration generation is pure (testable, offline); cluster operations are stateful (isolated, mockable)
- **Frozen dataclasses** — All data snapshots are immutable; safe to pass between layers
- **Idempotent operations** — Create-or-patch strategy everywhere; safe to re-run

## Relationship to Apollo

Citadel is a consumer of Apollo's orchestration model, not a dependency:
- Apollo health probes are discovered via pod annotations (`apollo.citadel.io/*`)
- Apollo can deploy workloads to Citadel-managed clusters
- Citadel's node lifecycle provides the infrastructure guarantee Apollo relies on
- Neither project imports the other's code
