# Apollo + Rubix Parity Analysis

How Rubix Citadel maps to Palantir's Apollo deployment platform and Rubix
infrastructure substrate, what we already cover, and what remains to build.

---

## 1. Platform Architecture Comparison

### Palantir (Commercial)

Palantir's infrastructure is a two-layer system: **Rubix** (infrastructure substrate)
managed by **Apollo** (deployment orchestration). Together they power Foundry/AIP
across cloud, on-prem, edge, and air-gapped environments.

```
+-----------------------------------------------------------+
| APOLLO HUB                                                 |
|   Orchestration Engine (Plans, Constraints, Rollbacks)     |
|   Release Channels (DEV → RC → RELEASE)                    |
|   Promotion Pipelines (Timed, Canary)                      |
|   Vulnerability Scanning + Automatic Recalls               |
|   Module Publishing + Secrets Management                   |
|   Multi-Hub Hierarchies                                    |
+-----------------------------------------------------------+
          │ encrypted outbound only
          ▼
+-----------------------------------------------------------+
| APOLLO SPOKE CONTROL PLANE                                 |
|   helm-chart-operator (lifecycle management)               |
|   expected-state-k8s (state reporting + health probes)     |
|   apollo-auth-broker (credential brokering)                |
+-----------------------------------------------------------+
          │ manages
          ▼
+-----------------------------------------------------------+
| PALANTIR RUBIX (Infrastructure Substrate)                  |
|                                                            |
|  COMPUTE          NETWORKING          SECURITY             |
|  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐  |
|  │ Ephemeral   │  │ Cilium eBPF  │  │ FIPS Envoy mTLS  │  |
|  │ 48h Nodes   │  │ Zero-Trust   │  │ Pod Security Std │  |
|  │ Demand-     │  │ Hubble       │  │ Audit Logging    │  |
|  │ Sensing     │  │ Observability│  │ FedRAMP/IL5/IL6  │  |
|  │ Autoscale   │  │ L7 Filtering │  │ CBAC/MAC/DAC     │  |
|  │ k8s-spark-  │  │ FQDN Egress  │  │ WAF              │  |
|  │ scheduler   │  │ Control      │  │ IDS/DLP          │  |
|  └─────────────┘  └──────────────┘  └──────────────────┘  |
|                                                            |
|  STORAGE          IDENTITY           COMPLIANCE            |
|  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐  |
|  │ Encrypted   │  │ Multipass    │  │ FedRAMP High     │  |
|  │ Persistent  │  │ JWT/SSO      │  │ DISA IL-5/IL-6   │  |
|  │ Volumes     │  │ MFA (FIDO2)  │  │ CMMC Level 2     │  |
|  │ HSM Keys    │  │ Central Auth │  │ SOC 2, HIPAA     │  |
|  └─────────────┘  └──────────────┘  │ GDPR, ITAR       │  |
|                                     └──────────────────┘  |
+-----------------------------------------------------------+
```

### Our Ecosystem

```
+-----------------------------------------------------------+
| APOLLO-CLONE (Separate Repo -- BUILT)                      |
|   Orchestration Engine (Plans, Constraints, Rollbacks)     |
|   Release Channels + Promotion (timed, canary, manual)     |
|   Helm Chart Operator (kopf-based CRD controller)          |
|   Expected-State Reporting (watchers, drift detection)     |
|   Vulnerability Scanning + Automatic Recalls               |
|   Secrets Management (Vault, encrypted storage)            |
|   SSO/OIDC (multi-provider) + SAML 2.0                    |
|   Certificate Management (X.509, HSM, rotation)            |
|   Compliance Governance (FedRAMP/IL5, audit trails)        |
+-----------------------------------------------------------+
          │ manages
          ▼
+-----------------------------------------------------------+
| RUBIX CITADEL (v0.2.0)                                     |
|                                                            |
|  COMPUTE          NETWORKING          SECURITY             |
|  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐  |
|  │ Ephemeral   │  │ Cilium eBPF  │  │ Pod Security Std │  |
|  │ 48h Nodes   │  │ Zero-Trust   │  │ mTLS (Cilium)    │  |
|  │ K3s         │  │ Default Deny │  │ Audit Logging    │  |
|  │ Multipass   │  │ Hubble       │  │ Data-at-Rest Enc │  |
|  │ Custom      │  │ Observability│  │ RBAC (lifecycle)  │  |
|  │ Probes      │  │ L7 Filtering │  │ cert-manager     │  |
|  │             │  │ FQDN Egress  │  │ Dex (infra)      │  |
|  └─────────────┘  └──────────────┘  └──────────────────┘  |
|                                                            |
|  NODE LIFECYCLE   POLICIES            MONITORING           |
|  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐  |
|  │ NodeManager │  │ NodeTermina- │  │ Prometheus +     │  |
|  │ HealthCheck │  │ tionPolicy   │  │ Grafana          │  |
|  │ DrainResult │  │ CRD          │  │ Cilium Metrics   │  |
|  │ Eviction    │  │ 4 Default    │  │ Alerting Rules   │  |
|  │ Force Drain │  │ Policies     │  │ ServiceMonitors  │  |
|  │ Probe Disc. │  │ Custom Probs │  │ kube-bench/Trivy │  |
|  └─────────────┘  └──────────────┘  └──────────────────┘  |
+-----------------------------------------------------------+
```

> **Key insight**: Apollo-clone is a fully-built control plane. Citadel's scope
> is **infrastructure substrate only** — the cluster foundation that Apollo
> deploys onto. Features like release channels, entity lifecycle, orchestration,
> and application-level auth belong in Apollo, not Citadel. Citadel installs
> infrastructure components (Dex, cert-manager, external-secrets-operator) that
> Apollo then consumes.
>
> **Infra boundary refactor**: Infrastructure concerns are progressively moving
> from Apollo to Citadel. Citadel now owns:
> - **Node lifecycle operator** — kopf operator reconciling NodeTerminationPolicy
>   CRDs, using NodeManager.drain() and HealthChecker. Ported from Apollo's
>   spoke/node_lifecycle/ (Apollo retains CRD models for status reads).
> - **Namespace infrastructure operator** — kopf operator reconciling
>   NamespaceInfrastructure CRDs (CiliumNetworkPolicies, ResourceQuotas,
>   LimitRanges). Apollo creates the CRD; Citadel provisions the resources.
>   Apollo retains namespace creation, tenant labels, and RBAC bindings.
> - **Certificates** — cert-manager integration (via CertificateProvider ABC)
> - **Secrets** — external-secrets-operator integration (via SecretProvider ABC)
> - **Vulnerability scanning** — Trivy Operator integration

---

## 2. Feature Parity Matrix

### Legend

| Symbol | Meaning |
|--------|---------|
| **=**  | Full parity -- equivalent capability exists |
| **~**  | Partial parity -- core present, gaps in scope or polish |
| **P**  | Planned -- architecture/design exists, implementation needed |
| **-**  | Not covered -- no equivalent, needs decision on whether to build |

### 2.1 Ephemeral Node Lifecycle

| Palantir Capability | Citadel Equivalent | Parity | Notes |
|--------------------|--------------------|--------|-------|
| **48-hour node maximum age** | `max-age-48h` NodeTerminationPolicy (priority 10) | **=** | Identical concept |
| **Policy-driven node selection** | NodeTerminationPolicy CRD with selector (maxAgeHours, conditions, nodeSelector) | **=** | CRD supports age, health conditions, and label selectors |
| **Health-based termination** (disk pressure, memory pressure, not ready) | 3 default health policies (disk-pressure priority 100, memory-pressure priority 100, not-ready priority 200) | **=** | Same signals, same approach |
| **Multidimensional termination pipeline** (age + health + bad states) | Priority-based policy evaluation (higher priority evaluated first) | **~** | Core mechanism present. Missing: ability to define custom "bad state" detectors beyond K8s conditions (e.g. Palantir's stuck-EBS-volume detector) |
| **Decision recording** (labels + annotations for audit) | Node label schema: `rubix.dev/terminate-reason`, `terminate-after`, `policy-name`, `managed-by` | **=** | Same pattern -- declarative intent recorded before destructive action |
| **Eviction API** (PDB-respecting drain) | NodeManager.drain() using Eviction API with PDB retry | **=** | Explicitly uses Eviction API, not `kubectl drain --force`. Retries on 429 (PDB blocking) |
| **Forced eviction after timeout** | DrainResult with timeout_seconds (30-3600s, default 300s) | **~** | Timeout exists. Unclear if force-delete happens after timeout or if drain just fails. Palantir force-deletes after a set time |
| **Graceful drain with PDB respect** | PodDisruptionBudgets defined (frontend minAvailable:2, backend maxUnavailable:1, database minAvailable:1) | **=** | Comprehensive PDB set for demo app. Production workloads would define their own |
| **Immutable infrastructure** (replace, don't patch) | MultipassProvider.replace_vm() -- destroy + create fresh + rejoin | **=** | Identical philosophy. Nodes are replaced, never upgraded in-place |
| **Demand-sensing autoscaling** | Not implemented | **-** | Palantir Rubix uses predictive algorithms to scale clusters. Citadel has no autoscaling -- VMs are manually provisioned |
| **GPU/CPU dynamic scaling** | Not implemented | **-** | Palantir supports heterogeneous GPU/CPU scaling. Not in scope for K3s edge deployments |
| **k8s-spark-scheduler** (gang scheduling) | Not implemented | **-** | Specialised for Spark workloads. Not needed unless running Spark on Citadel |
| **Node image-based updates** (kernel/OS via image swap) | VM replacement serves same purpose | **~** | VM replacement achieves the same outcome (fresh kernel/OS). Missing: pre-built node images with specific OS hardening |
| **dryRun mode** | NodeTerminationPolicy `spec.dryRun: true` | **=** | Labels nodes without draining -- observe before enforcing |
| **HealthChecker** (pure evaluation, no I/O) | HealthChecker class with HEALTHY/WARNING/ERROR/TERMINAL states | **=** | Identical design. Decoupled from action. Compatible with Apollo's Witchcraft health model |
| **Node health states** | 4 states: HEALTHY, WARNING, ERROR, TERMINAL | **=** | Maps directly to Apollo's 4 health states (HEALTHY, REPAIRING→WARNING, WARNING, ERROR→ERROR/TERMINAL) |

### 2.2 Networking & Zero-Trust

| Palantir Capability | Citadel Equivalent | Parity | Notes |
|--------------------|--------------------|--------|-------|
| **Cilium eBPF CNI** | Cilium installed via Helm, replaces Flannel + kube-proxy | **=** | Identical -- eBPF datapath, kube-proxy replacement |
| **Default-deny networking** | `default-deny-all` CiliumClusterwideNetworkPolicy | **=** | Same zero-trust foundation |
| **DNS exception** | `allow-dns` policy (kube-system port 53 UDP/TCP) | **=** | Minimal exception |
| **API server exception** | `allow-apiserver` policy (port 6443 TCP) | **=** | Required for cluster operation |
| **Health check exception** | `allow-health-checks` from host/remote-node | **=** | Kubelet probe traffic |
| **Identity-based policies** (pod labels, not IPs) | All policies use label selectors (tier=frontend, tier=backend, etc.) | **=** | Cilium identity model |
| **L7 HTTP filtering** (method + path restrictions) | `04-l7-demo-policies.yaml` -- GET/POST/PUT/DELETE method filtering, path regex (`/api/v1/*`, `/health`, `/ready`) | **=** | Kernel-level HTTP inspection via eBPF |
| **gRPC filtering** | gRPC example in L7 policies (POST to `/myservice.MyService/*`) | **=** | Demonstrated |
| **FQDN-based egress control** | `05-fqdn-egress-policies.yaml` -- per-service allowlists (github.com, stripe.com, amazonaws.com, etc.) | **=** | DNS-based egress filtering. Prevents data exfiltration |
| **SSRF prevention** (block metadata endpoints) | Blocked in Pod Security Standards (169.254.169.254) | **=** | Cloud metadata endpoint blocked |
| **Hubble network observability** | Cilium installed with Hubble + Hubble Relay + Hubble UI | **=** | Full network flow visibility |
| **FIPS-validated Envoy** (mTLS, end-to-end encryption) | Not implemented | **P** | Palantir uses FIPS Envoy for all intra-cluster TLS. Citadel has WireGuard (encryption) but not Envoy (L7 proxy + mTLS) |
| **Custom xDS control plane** (Envoy egress filtering) | Not implemented | **-** | Palantir built custom Envoy control plane for egress. Citadel uses Cilium FQDN policies instead (simpler, less flexible) |
| **WireGuard pod-to-pod encryption** | Script `06-enable-wireguard.sh` exists | **~** | Script provided, integration status unclear. Palantir uses Envoy for encryption; Citadel uses WireGuard (different mechanism, same goal) |
| **Web Application Firewall** (WAF) | Not implemented | **-** | Palantir deploys WAF for internet-facing services. Would need external WAF (e.g. ModSecurity, AWS WAF) or Envoy filter |
| **BCAP integration** (trusted network dynamic lists) | Not implemented | **-** | Palantir-specific. Boundary Condition Aware Proxy for dynamic trusted network lists |
| **Micro-segmentation** (host + container + network firewalls) | Container-level via Cilium. Host-level not explicitly configured | **~** | Cilium provides container/pod-level segmentation. Missing: explicit host-level firewall hardening (iptables/nftables on nodes) |
| **Network policy for system namespaces** | `02-system-policies.yaml` -- Cilium agent, Hubble relay, Hubble UI, CoreDNS policies | **=** | System components have explicit least-privilege network policies |

### 2.3 Pod Security & Workload Isolation

| Palantir Capability | Citadel Equivalent | Parity | Notes |
|--------------------|--------------------|--------|-------|
| **Restricted Pod Security Standard** | PSS enforcement on demo + production namespaces (restricted) | **=** | Non-root, no privilege escalation, read-only rootfs, dropped capabilities, seccomp |
| **Baseline PSS for staging** | Staging namespace with baseline enforcement | **=** | Less restrictive for development |
| **Resource quotas** (namespace limits) | `07-resource-quotas.yaml` -- CPU/memory/pod limits per namespace | **=** | Prevents noisy neighbours and resource exhaustion |
| **LimitRanges** (per-container defaults/max/min) | Default 500m/512Mi, max 4CPU/8Gi, min 50m/64Mi | **=** | Enforces sane container sizing |
| **PriorityClasses** | critical-workload (1M), high-priority-compute (100K), best-effort (1K) | **=** | Preemption ordering for resource contention |
| **Workload isolation by privilege** | Namespace-based with PSS tiers | **~** | Same concept. Palantir may have more granular isolation (dedicated node pools per sensitivity) |
| **No kubectl exec/SSH** | Not explicitly enforced | **P** | Palantir prohibits backend SSH and kubectl exec. Could enforce via admission webhook or OPA policy |
| **Compliance operator** (OS + K8s hardening) | kube-bench CronJob + Trivy Operator (`security/compliance/`), Grafana dashboard, remediation guide | **=** | CIS benchmarks + vulnerability scanning + compliance reporting |

### 2.4 Audit Logging & Compliance

| Palantir Capability | Citadel Equivalent | Parity | Notes |
|--------------------|--------------------|--------|-------|
| **Kubernetes audit logging** | `audit-policy.yaml` (120 lines) -- RequestResponse for secrets, RBAC, exec, network policies, nodes; Metadata for workloads; None for noise | **=** | Comprehensive policy targeting FedRAMP/SOC2 compliance |
| **Audit.3 schema** (standardised structured logs) | Not implemented as structured schema | **~** | K8s audit logs are structured JSON but don't follow Palantir's audit.3 schema with `categories`, `entities`, `eventId`, etc. |
| **Audit log API** (list-log-files, get-content) | Not implemented | **-** | Palantir exposes audit logs via REST API. Citadel logs go to K8s audit backend (file or webhook) |
| **SIEM integration** | Not implemented | **P** | Could forward audit logs to external SIEM via fluentd/fluentbit. Architecture supports it |
| **15-minute availability** | Depends on K8s audit backend configuration | **~** | K8s audit logs are near-real-time to file. Palantir guarantees 15 minutes |
| **Append-only immutable logs** | Not explicitly enforced | **P** | K8s audit logs are append-only by nature. Immutability depends on storage backend (could use WORM storage) |
| **FedRAMP High** | Audit policy designed for FedRAMP | **~** | Policy covers the right events. Full FedRAMP requires many more controls beyond audit (identity, encryption, incident response, etc.) |
| **DISA IL-5/IL-6** | Not specifically targeted | **-** | Requires classified network deployment, specific DoD controls. Specialised |
| **SOC 2** | Audit logging + RBAC + encryption contribute | **~** | Components present but no formal SOC 2 assessment |
| **CMMC Level 2** | Not specifically targeted | **-** | Requires NIST SP 800-171 implementation. Many controls overlap with what exists |

### 2.5 Identity & Access Management

| Palantir Capability | Citadel Equivalent | Apollo-Clone | Parity | Notes |
|--------------------|--------------------|-------------|--------|-------|
| **Multipass** (central authentication) | Dex installed as infra (`security/sso/`) | Full OIDC + SAML pipeline (`auth/oidc/`, `auth/_saml.py`) | **=** | **Apollo provides the auth platform. Citadel installs Dex as infrastructure prerequisite** |
| **SSO integration** (SAML 2.0, OIDC, Entra ID) | Dex connectors (SAML, LDAP, GitHub, OIDC) | Multi-provider OIDC, SAML 2.0 with group sync | **=** | Citadel configures K8s API server OIDC. Apollo handles application-level SSO |
| **MFA enforcement** (FIDO2, hardware tokens) | Not in Citadel | TOTP + WebAuthn in Apollo auth pipeline | **~** | Apollo handles MFA. Citadel delegates to IdP |
| **JWT-based sessions** | Not in Citadel (infra scope) | Bearer token + API key auth in Apollo | **=** | Application-level concern, correctly in Apollo |
| **RBAC for lifecycle controller** | `rbac.yaml` -- ServiceAccount, ClusterRole, ClusterRoleBinding | Casbin RBAC with 6-tuple model | **=** | Both have RBAC at their respective layers |
| **Leader election** (HA controller) | Lease-based leader election in RBAC permissions | — | **=** | Infrastructure concern, correctly in Citadel |
| **Classification-based access** (CBAC/MAC) | Not implemented | Multi-layer auth (Cross-Org → RBAC → MAC → MFA → ABAC) | **~** | Apollo has the framework. Full MAC requires org investment |
| **Central Auth** (managed IdP) | Dex as IdP infrastructure | Full auth chain with token blacklist, refresh | **=** | Complementary layers |

### 2.6 Apollo Deployment Orchestration

> **NOTE**: Apollo-clone (`../../apollo/`) is a fully-built deployment
> orchestration platform. Most capabilities listed below exist there, not in
> Citadel. Citadel provides the infrastructure substrate that Apollo deploys onto.

| Palantir Capability | Apollo-Clone Status | Citadel Role | Combined Parity | Notes |
|--------------------|--------------------|--------------|--------------------|-------|
| **Hub-Spoke architecture** | Full Hub + Spoke agents (`spoke/`) | Infrastructure host | **=** | Apollo provides Hub API, Spoke agents. Citadel provides the K8s cluster they run on |
| **Orchestration Engine** (Plans + Constraints) | `orchestration/engine.py` — full plan lifecycle, multi-stage approvals | None | **=** | **Apollo territory** |
| **Products + Releases** | `models/release.py` + `catalog/` + forge/release-hub | None | **=** | Apollo manages releases. forge builds them |
| **Release Channels** (DEV → RC → RELEASE) | `services/channels/` — full promotion strategies (timed, canary, manual, fast-track) | None | **=** | **Apollo territory — do not duplicate in Citadel** |
| **Promotion Pipelines** (timed, canary) | `services/channels/_promotion.py`, `_canary.py`, `_scheduler.py` | None | **=** | **Apollo territory** |
| **Canary evaluation** | `services/channels/_canary.py` — health thresholds | None | **=** | Apollo evaluates health for promotion decisions |
| **Blue-green deployments** | Supported via Helm rollback in `spoke/helm_chart_operator/` | None | **~** | Apollo handles via Helm. True blue-green needs traffic splitting |
| **Maintenance windows** | Constraint support in orchestration engine | None | **=** | **Apollo territory** |
| **Product dependencies** | Referenced in Apollo module constraints | None | **~** | Ordering exists. Runtime enforcement in orchestration |
| **Suppression windows** | Implicit via plan lifecycle (BLOCKED state) | None | **~** | Could be more explicit |
| **Release Recalls** | `models/release.py` — RecallStrategy (FREEZE, WARN, FORCE_ROLLBACK) | None | **=** | **Apollo territory** |
| **Roll-off strategies** | RecallStrategy enum in Apollo | None | **=** | **Apollo territory** |
| **Vulnerability scanning + automatic recalls** | `security/` — Trivy, ClamAV, AutoRecallEngine | kube-bench + Trivy Operator (infra scanning) | **=** | Apollo governs recalls. Citadel runs infrastructure-level scans |
| **Secrets management** | Vault integration, encrypted storage, audit logging | external-secrets-operator (infra install) | **=** | Apollo manages secrets lifecycle. Citadel installs the operator |
| **Disconnected operation** | Not fully implemented | K3s supports air-gapped | **~** | Both need work for true air-gap |
| **Module publishing** | Apollo CLI (`cli/`) with full command set | None | **=** | **Apollo territory** |
| **Helm chart operator** | `spoke/helm_chart_operator/` — kopf CRD controller, rollback handling | None | **=** | **Apollo territory — do not duplicate in Citadel** |
| **expected-state-k8s** | `spoke/state_reporter/` — watchers, drift detection, CRD watchers | None | **=** | **Apollo territory — do not duplicate in Citadel** |
| **apollo-auth-broker** | `spoke/` — auth broker for credential distribution | None | **~** | In Apollo spoke agents |
| **Health probe discovery** | `services/_health.py` — pod probes, real-time tracking | `k3s/probe_discovery.py` (infra-level probes) | **=** | Apollo does app-level. Citadel does node/infra-level. Complementary |
| **Multi-Hub hierarchies** | Not implemented | Not implemented | **-** | Enterprise scale feature, deferred |
| **Environment management** | Full environment model in Apollo | None | **=** | **Apollo territory** |
| **Authorization model** | Casbin RBAC, 6-tuple model, multi-layer auth | K8s RBAC for infra | **=** | **Apollo territory** |

### 2.7 Encryption

| Palantir Capability | Citadel Equivalent | Parity | Notes |
|--------------------|--------------------|--------|-------|
| **Data at rest encryption** (AES-256, HSM keys) | AES-GCM for etcd secrets (`security/encryption-config.yaml`), MinIO SSE-S3, LUKS2 guidance | **=** | Covers etcd, PVs, and disk layers |
| **Data in transit** (TLS 1.2+, ECDHE) | WireGuard (pod-to-pod) + Cilium mTLS (`security/mutual-auth.yaml`) | **=** | Both network encryption and mutual authentication |
| **FIPS-validated cryptography** | Not implemented | **-** | Palantir uses FIPS Envoy. Would need FIPS-compiled binaries |
| **mTLS** (mutual TLS for all services) | Cilium mutual authentication with SPIRE/SPIFFE (`security/mutual-auth.yaml`) | **=** | eBPF-enforced mTLS, automatic certificate rotation |
| **Certificate management** (90-day/1-year max) | cert-manager with Let's Encrypt + internal CA (`security/certificates/`). 90-day max, 30-day renewal | **=** | Full lifecycle: provisioning, rotation, monitoring |
| **HSM key storage** | Not in Citadel | Apollo has HSM integration (`auth/hsm/`) | **~** | Apollo provides HSM. Citadel uses standard K8s secrets for AES keys |
| **Application-level encryption** (Cipher) | Not implemented | **-** | Faraday-level concern, not infrastructure. See Faraday FOUNDRY-PARITY.md |

### 2.8 Monitoring & Observability

| Palantir Capability | Citadel Equivalent | Parity | Notes |
|--------------------|--------------------|--------|-------|
| **Prometheus metrics** | Prometheus + Grafana stack installed | **=** | Standard monitoring stack |
| **Cilium ServiceMonitors** | ServiceMonitors for Cilium agent + operator | **=** | Cilium metrics scraped into Prometheus |
| **Hubble flow visibility** | Hubble Relay + Hubble UI installed | **=** | Network flow observation |
| **DataDog integration** | Not implemented | **-** | Palantir bundles DataDog monitors. Could integrate |
| **Fleet-wide observability** | Not implemented | **-** | Requires Hub to aggregate across environments |
| **Custom dashboards** | Node lifecycle dashboard (`monitoring/node-lifecycle-dashboard.json`) + compliance dashboard (`security/compliance/compliance-dashboard.json`) | **=** | Grafana dashboards for node lifecycle and compliance |
| **Alerting** (AlertManager) | PrometheusRule with 5 alerts (`monitoring/node-lifecycle-alerts.yaml`) — drain failures, stale nodes, PDB blocking, policy errors | **=** | Alerting for all lifecycle events |

### 2.9 Infrastructure Provider

| Palantir Capability | Citadel Equivalent | Parity | Notes |
|--------------------|--------------------|--------|-------|
| **AWS** (including GovCloud, SECRET, TOP SECRET) | Not implemented | **-** | Palantir deploys across all AWS classifications |
| **Azure** (including GovCloud, SECRET) | Not implemented | **-** | Multi-cloud support |
| **GCP, Oracle Cloud** | Not implemented | **-** | Multi-cloud support |
| **On-premises datacenters** | K3s runs anywhere | **~** | K3s is designed for on-prem/edge. Missing: bare-metal provisioning automation |
| **VMware vSphere** | Not implemented | **-** | Enterprise virtualisation |
| **Bare metal** | Not implemented | **-** | Palantir supports bare metal deployment |
| **Multipass VMs** (development) | MultipassProvider (create, destroy, replace, join) | **=** | Primary development/testing provider |
| **Tactical edge** (single-node minimum) | K3s single-node supported | **=** | K3s was designed for edge. Single-node is native |
| **Cloud provider abstraction** | Not implemented | **P** | Citadel only has Multipass. Need AWS/Azure/GCP providers for production |
| **Dynamic Terraform controls** | Not implemented | **-** | Palantir uses Terraform for cloud resource provisioning |

---

## 3. Gap Priority Analysis

> **Updated 2026-02-28**: This analysis now accounts for apollo, which
> provides the full deployment orchestration control plane. Gaps previously
> attributed to Citadel that are actually Apollo's responsibility have been
> reclassified. Citadel gaps are limited to **infrastructure substrate** scope.

### Tier 1: Critical for Production Use — COMPLETED

All Tier 1 gaps have been addressed on the `feat/parity-gaps-tier1` branch.

| Gap | Status | Implementation |
|-----|--------|----------------|
| ~~Cloud providers (AWS, Azure, GCP)~~ | **Remaining** | Still only Multipass. Needs provider interface for AWS EC2, Azure VMs, GCP Compute |
| ~~Forced drain after timeout~~ | **Done** | `force_after_timeout` param on `NodeManager.drain()`, force-deletes after timeout |
| ~~mTLS or mutual authentication~~ | **Done** | Cilium mutual auth with SPIRE/SPIFFE (`security/mutual-auth.yaml`) |
| ~~Data at rest encryption~~ | **Done** | AES-GCM for etcd, MinIO SSE-S3, LUKS2 guidance (`security/encryption-config.yaml`) |
| ~~Alerting rules~~ | **Done** | 5 PrometheusRule alerts (`monitoring/node-lifecycle-alerts.yaml`) |
| ~~Custom bad-state detectors~~ | **Done** | `CustomProbeChecker` with exec/HTTP probes (`k3s/custom_probes.py`) |

### Tier 2: Previously "Apollo Clone" — NOW IN APOLLO-CLONE

These gaps were originally planned for Citadel but **already exist in apollo**.
They should NOT be duplicated in Citadel.

| Gap | Apollo-Clone Module | Status |
|-----|-------------------|--------|
| **expected-state-k8s** (state reporting) | `spoke/state_reporter/` — watchers, drift detection, bidirectional sync | **EXISTS IN APOLLO** |
| **Health probe discovery** (app-level) | `services/_health.py` — pod probes, real-time tracking | **EXISTS IN APOLLO** |
| **Helm chart operator** (entity lifecycle) | `spoke/helm_chart_operator/` — kopf CRD controller, rollback | **EXISTS IN APOLLO** |
| **Orchestration Engine** (Plans + Constraints) | `orchestration/engine.py` — plan lifecycle, approval workflows | **EXISTS IN APOLLO** |
| **Release Channels + Promotion** | `services/channels/` — timed, canary, manual, fast-track | **EXISTS IN APOLLO** |
| **Secrets management** (lifecycle) | `auth/_vault_integration.py`, `spec/_secrets.py` | **EXISTS IN APOLLO** |

### Tier 2 (Citadel-scoped): Infrastructure Installations — COMPLETED

These install infrastructure components that Apollo consumes. Not duplications —
they're the plumbing that Apollo's application-level features run on top of.

| Gap | Status | Citadel Installs | Apollo Consumes |
|-----|--------|-----------------|-----------------|
| ~~SSO/OIDC~~ | **Done** | Dex + K8s OIDC flags (`security/sso/`) | `auth/oidc/` authenticates through Dex |
| ~~Certificate management~~ | **Done** | cert-manager + ClusterIssuers (`security/certificates/`) | `auth/_certificate_manager.py` uses certs |
| ~~Secrets operator~~ | **Done** | external-secrets-operator (`security/secrets/`) | `auth/_vault_integration.py` manages secrets |
| ~~Compliance scanning~~ | **Done** | kube-bench + Trivy Operator (`security/compliance/`) | `security/` aggregates into governance |
| ~~Health probes (infra)~~ | **Done** | Node/infra-level probes (`k3s/probe_discovery.py`) | Complements Apollo's app-level probes |

### Tier 3: Enterprise & Compliance Features (Build When Needed)

Important for government/enterprise but not blocking core functionality.

| Gap | Impact | Effort | Approach |
|-----|--------|--------|----------|
| **FIPS-validated cryptography** | Required for FedRAMP/IL5. All crypto must use FIPS-validated implementations | High | Use FIPS-compiled Envoy, FIPS Go runtime, FIPS Python (via OpenSSL FIPS module). K3s has FIPS builds |
| **Compliance operator** (CIS/NIST benchmarks) | Automated compliance scanning and reporting | Medium | Deploy kube-bench for CIS benchmarks. Trivy operator for vulnerability scanning. Integrate results with audit logging |
| **Audit.3 structured schema** | Standardised audit format for SIEM integration and cross-service correlation | Medium | Define audit event schema matching Palantir's categories. Structured JSON with eventId, categories, entities, time fields |
| **SIEM export** | Enterprise SOCs need audit logs in their SIEM | Medium | Fluentbit/Fluentd forwarding from K8s audit logs to Splunk/Sentinel/Elastic |
| **WAF** (Web Application Firewall) | Internet-facing services need OWASP protection | Medium | ModSecurity with Envoy or Nginx. Or cloud-native WAF (AWS WAF, Azure WAF) |
| **SSO/OIDC integration** | Enterprise identity federation | Medium | Dex or Keycloak as OIDC provider for K3s. Supports SAML 2.0 upstream |
| **Classification-based access** (CBAC/MAC) | Government mandatory access controls | High | Deep integration with identity and data layers. Labeling system + policy enforcement |
| **Vulnerability scanning + recalls** | Automated CVE response | High | Trivy/Grype scanning in CI. Integration with release management to block/recall vulnerable releases |
| **Certificate management** | Automated certificate rotation | Medium | cert-manager with Let's Encrypt or internal CA. Configure max 90-day lifetimes |

### Tier 4: Scale & Multi-Environment (Future Vision)

| Gap | Impact | Effort | Approach |
|-----|--------|--------|----------|
| **Multi-Hub hierarchies** | Enterprise with many environments | Very High | Hubs managed by Hubs. Only needed at very large scale |
| **Fleet-wide observability** | Central monitoring across environments | High | Thanos or Mimir for multi-cluster Prometheus. Grafana dashboards per environment |
| **Disconnected Apollo** | Air-gapped environments | High | Cryptographic artifact signing, integrity validation, periodic sync |
| **Demand-sensing autoscaling** | Cost optimisation at scale | High | ML-based workload prediction. Cloud provider API for scaling node groups |
| **Dynamic Terraform** | Cloud resource provisioning automation | Medium | Terraform modules per cloud provider, triggered by platform events |

---

## 4. Concept Mapping Reference

### Core Abstractions

| Palantir Concept | Citadel Equivalent | Notes |
|-----------------|-------------------|-------|
| Rubix (substrate) | Rubix Citadel | Same name, same concept. Ours is K3s-based, lighter weight |
| Apollo (deployment) | Apollo Clone (planned, separate repo) | Not yet built. forge/sls-distribution handles build+publish |
| Hub | Not yet built | Central management plane |
| Spoke | Not yet built | Per-cluster agent set |
| Spoke Control Plane | Not yet built | helm-chart-operator + expected-state-k8s + auth-broker |
| Environment | K3s cluster | Same concept -- a managed Kubernetes cluster |
| Entity | Helm release | Same concept -- an instance of a Product in an Environment |
| Product | Helm chart (from forge) | Same concept -- a versioned software component |
| Release | Helm chart version | Same concept |
| Module | Apollo module YAML (rubix-platform.yml) | Module structure defined in workloads |
| Release Channel | Not yet built | DEV/RC/RELEASE progression |
| Plan | Not yet built | Unit of change in Apollo |
| Constraint | Not yet built | Rules that gate Plan execution |
| Recall | Not yet built | Mark release as bad |
| NodeTerminationPolicy | NodeTerminationPolicy CRD | **Same concept, already built** |
| Node health states | HealthChecker (HEALTHY/WARNING/ERROR/TERMINAL) | Identical 4-state model |
| Ephemeral compute | 48-hour node lifecycle | **Same concept, already built** |
| Eviction API | NodeManager.drain() | **Same mechanism, already built** |
| PodDisruptionBudget | PDB definitions | **Same concept, already built** |
| Cilium | Cilium (installed via Helm) | **Same technology** |
| Hubble | Hubble + Relay + UI | **Same technology** |
| FIPS Envoy | Not yet built | WireGuard provides encryption, not mTLS |
| Multipass (auth) | Not built (different from Multipass VMs) | Palantir's auth system, unrelated to Canonical Multipass |
| Compliance operator | Not yet built | CIS/NIST benchmark scanning |
| Audit.3 | K8s audit logging (similar intent) | Same goal, different schema |
| Maintenance window | Not yet built | Apollo constraint |
| Suppression window | Not yet built | Apollo safety mechanism |

### Architectural Decisions: Where We Diverge

| Aspect | Palantir Rubix | Citadel | Rationale |
|--------|---------------|---------|-----------|
| **K8s distribution** | Custom (open-source base + hardening) | K3s (lightweight single-binary) | K3s is designed for edge/small. Single binary, no external dependencies. Trade: fewer features, smaller footprint |
| **Node provisioning** | Cloud APIs (AWS/Azure/GCP/bare metal) | Multipass VMs (development) | Multipass for development. Production providers not yet built |
| **Encryption** | FIPS Envoy (mTLS) | WireGuard (transparent pod-to-pod) | WireGuard is simpler (kernel-level), lower overhead. Trade: no application-level mTLS, no FIPS validation |
| **Egress filtering** | Custom Envoy xDS control plane | Cilium FQDN policies | Cilium FQDN is simpler, no sidecar. Trade: less flexible than Envoy L7 filtering |
| **Autoscaling** | Demand-sensing ML algorithms | Manual (none) | Not in scope for edge/small deployments. Could add HPA/VPA/Karpenter later |
| **Deployment orchestration** | Apollo (full platform) | Not yet built (forge handles build/publish) | Apollo Clone is planned. Current: manual Helm installs |
| **Scale target** | Tens to thousands of nodes | 1-10 nodes (edge/small) | K3s is designed for small clusters. Different market segment |
| **Compliance** | FedRAMP High, IL5, IL6, CMMC L2, SOC 2 | Designed for compliance (audit policy, PSS) | Components present. Formal certification requires org-level investment |
| **Identity** | Multipass (full auth platform) | K8s RBAC only | No custom auth system. Relies on K8s native + external IdP |
| **Operator framework** | Go-based controllers | Python (kopf) | Matches ecosystem. Trade: performance at extreme scale |

---

## 5. What Citadel Already Does Well

These areas match or exceed Palantir's implementation for the target scale:

### Node Lifecycle (Core Strength)

Citadel's node lifecycle implementation is the most complete component and maps
directly to Palantir's ephemeral compute model:

```
Palantir Rubix                          Citadel
────────────                            ────────
Policy-driven selection        ═══════  NodeTerminationPolicy CRD
Decision recording             ═══════  Node label schema (rubix.dev/*)
Eviction API (PDB-aware)       ═══════  NodeManager.drain()
48-hour max age                ═══════  max-age-48h policy
Health-based termination       ═══════  disk-pressure, memory-pressure, not-ready
Immutable replacement          ═══════  MultipassProvider.replace_vm()
Dry-run mode                   ═══════  spec.dryRun: true
Health evaluation              ═══════  HealthChecker (4 states)
```

### Zero-Trust Networking (Comprehensive)

The Cilium policy set covers all layers of zero-trust:

```
Layer 3/4: Default deny + identity-based allow    ✓
Layer 7:   HTTP method + path filtering            ✓
DNS:       FQDN-based egress control               ✓
System:    Explicit policies for kube-system        ✓
SSRF:      Cloud metadata endpoint blocked          ✓
Demo:      3-tier app with complete policy set      ✓
```

### Security Posture (Strong Foundation)

```
Pod Security Standards:    Restricted enforcement     ✓
Resource Quotas:           Namespace-level limits      ✓
LimitRanges:               Per-container bounds        ✓
PriorityClasses:           Workload preemption order   ✓
PDBs:                      Availability during drain   ✓
Audit Logging:             FedRAMP-targeting policy     ✓
RBAC:                      Least-privilege lifecycle    ✓
Leader Election:           HA controller support        ✓
```

---

## 6. Implementation Roadmap (Revised)

> **Updated 2026-02-28**: With apollo providing the control plane, Citadel's
> roadmap is now focused purely on infrastructure substrate and Apollo integration.
> Phases 2-4 from the original roadmap are **replaced by Apollo integration work**.

### Phase 1: Production-Ready Citadel — MOSTLY COMPLETE (v0.2.0)

| Work Item | Status | Notes |
|-----------|--------|-------|
| ~~Forced drain after timeout~~ | **Done** | `force_after_timeout` on `NodeManager.drain()` |
| ~~Data at rest encryption~~ | **Done** | AES-GCM etcd + MinIO SSE-S3 + LUKS2 |
| ~~mTLS via Cilium mutual auth~~ | **Done** | SPIRE/SPIFFE + eBPF enforcement |
| ~~Alerting rules~~ | **Done** | 5 PrometheusRule alerts |
| ~~Grafana dashboards~~ | **Done** | Node lifecycle + compliance dashboards |
| ~~Custom bad-state detectors~~ | **Done** | exec + HTTP probes in CRD |
| ~~Compliance operator~~ | **Done** | kube-bench + Trivy Operator |
| ~~SSO/OIDC infrastructure~~ | **Done** | Dex + connectors + K8s OIDC |
| ~~Certificate management~~ | **Done** | cert-manager + ClusterIssuers |
| ~~Secrets operator~~ | **Done** | external-secrets-operator |
| AWS EC2 provider | **Remaining** | Critical for production |
| Host-level firewall hardening | **Remaining** | Low priority |

**Exit Criteria**: ~~Can deploy~~ **Can now deploy** a Citadel cluster with
encrypted storage, mutual authentication, monitoring alerts, compliance scanning,
SSO, and 48-hour node lifecycle. **Remaining: AWS provider for production infra.**

### Phase 2: Apollo Integration (v0.3.0)

Focus: connect Citadel infrastructure to apollo control plane.

| Work Item | Effort | Priority | Notes |
|-----------|--------|----------|-------|
| Apollo Spoke deployment on Citadel | Medium | Critical | Deploy apollo spoke agents onto Citadel K3s clusters |
| Citadel → Apollo state reporting integration | Medium | Critical | Ensure Citadel health/probe data flows into Apollo's `state_reporter` |
| Helm chart operator deployment | Low | Critical | Deploy Apollo's existing operator, not rebuild it |
| Auth chain integration (Dex → Apollo OIDC) | Medium | High | Configure Apollo to use Citadel's Dex instance |
| Secrets integration (ESO → Apollo Vault) | Medium | High | Apollo's Vault backend uses Citadel's external-secrets-operator |
| Compliance data flow | Low | Medium | Citadel scans → Apollo governance dashboard |

**Exit Criteria**: Apollo manages Citadel clusters end-to-end — deploys software
via Helm operator, monitors health, enforces release channels, and uses Citadel's
infrastructure security.

### Phase 3: Enterprise Compliance (v1.0.0)

Focus: meet formal compliance requirements.

| Work Item | Effort | Priority |
|-----------|--------|----------|
| FIPS-validated cryptography | High | For gov |
| Audit.3 structured schema | Medium | For gov |
| SIEM export (fluentbit → Splunk/Sentinel) | Medium | For gov |
| WAF integration | Medium | For internet-facing |
| Disconnected/air-gapped operation | High | For gov |
| AWS GovCloud / IL5 deployment | Very High | For gov |

---

## 7. Where Rubix Workloads Fits

Rubix Workloads (the FaaS platform) runs ON TOP of Citadel. The relationship:

```
Rubix Citadel (infrastructure substrate)
  ├── Provides: K3s cluster, Cilium networking, node lifecycle, security
  ├── Manages: nodes, network policies, pod security, audit logging
  │
  └── Hosts → Rubix Workloads (application platform)
                ├── Provides: @function, @transform, autoscaling, CLI
                ├── Manages: Function CRs, Knative Services, HTTPRoutes
                │
                └── Hosts → Faraday (data platform)
                              ├── Provides: datasets, ontology, engine, sync
                              └── Manages: data resources, pipelines, transforms
```

### Security Integration Points

| Citadel Provides | Workloads/Faraday Consumes |
|-----------------|--------------------------|
| Default-deny networking | Function pods must have explicit network policies |
| Pod Security Standards | Function containers must meet restricted PSS |
| Resource quotas | Function resource requests/limits enforced |
| Audit logging | Function invocations logged at K8s level |
| Node lifecycle | Functions must tolerate pod eviction (Knative handles this) |
| WireGuard encryption | All function-to-function traffic encrypted |
| FQDN egress control | Functions can only reach whitelisted external APIs |

### What Citadel Needs from Workloads

| Requirement | Why |
|------------|-----|
| PDBs on Function deployments | Knative manages replicas; PDBs prevent drain from killing all replicas |
| Health endpoints on Function pods | Citadel health probe discovery needs endpoints to query |
| Security context compliance | Function containers must run as non-root, read-only FS |
| Resource requests/limits | Every function pod must declare resources for quota enforcement |
| Network policy labels | Function pods must be labeled for Cilium identity |

---

## 8. What We Will Never Build (and Why)

| Feature | Reason to Skip |
|---------|---------------|
| **Demand-sensing ML autoscaler** | Palantir's Rubix scales clusters with thousands of nodes. Citadel targets 1-10 nodes. Kubernetes HPA/VPA + Karpenter is sufficient |
| **k8s-spark-scheduler** | Gang scheduling for Spark. Not running Spark on K3s |
| **Custom Envoy xDS control plane** | Cilium FQDN policies cover our egress needs without sidecar overhead |
| **Multi-Hub hierarchies** | Only needed at very large scale (hundreds of environments). Build when needed |
| **BCAP integration** | Palantir-specific. Dynamic trusted network lists for government networks |
| **AWS GovCloud/SECRET/TOP SECRET regions** | Requires FedRAMP High + specific DoD accreditations. Org-level investment, not a code problem |
| **Dynamic Terraform controls** | Premature. Manual or simple automation is sufficient at our scale |

---

## 9. Quick Reference: Palantir Component → Citadel Mapping

| Palantir Component | Citadel File/Module | Status |
|-------------------|-------------------|--------|
| Ephemeral 48h nodes | `src/citadel_operator/k3s/node.py` + `crds/default-policies.yaml` | Built |
| Node health evaluation | `src/citadel_operator/k3s/health.py` | Built |
| Node termination policy | `crds/nodeterminationpolicy.yaml` | Built |
| VM lifecycle management | `src/citadel_operator/k3s/provider.py` | Built |
| Eviction API drain | `src/citadel_operator/k3s/node.py` NodeManager.drain() | Built |
| Default-deny networking | `policies/01-default-deny.yaml` | Built |
| L3/L4 identity policies | `policies/03-demo-app-policies.yaml` | Built |
| L7 HTTP/gRPC filtering | `policies/04-l7-demo-policies.yaml` | Built |
| FQDN egress control | `policies/05-fqdn-egress-policies.yaml` | Built |
| Pod Disruption Budgets | `policies/06-pod-disruption-budgets.yaml` | Built |
| Resource quotas/limits | `policies/07-resource-quotas.yaml` | Built |
| Pod Security Standards | `policies/08-pod-security-standards.yaml` | Built |
| K8s audit logging | `config/audit-policy.yaml` | Built |
| RBAC (lifecycle controller) | `crds/rbac.yaml` | Built |
| Cilium CNI install | `scripts/02-install-cilium.sh` | Built |
| Prometheus/Grafana | `scripts/03-install-monitoring.sh` + `monitoring/` | Built |
| WireGuard encryption | `scripts/06-enable-wireguard.sh` | Script exists |
| K3s cluster creation | `scripts/01-install-k3s.sh` + CLI | Built |
| Apollo Spoke agents | **apollo** `spoke/` | **Built (in apollo)** |
| Apollo Hub | **apollo** `api/`, `catalog/` | **Built (in apollo)** |
| Orchestration Engine | **apollo** `orchestration/engine.py` | **Built (in apollo)** |
| Release management | **apollo** `services/channels/` | **Built (in apollo)** |
| mTLS (Cilium) | `security/mutual-auth.yaml` | **Built** |
| Data at rest encryption | `security/encryption-config.yaml` | **Built** |
| Certificate management | `security/certificates/` | **Built** |
| SSO/OIDC infrastructure | `security/sso/` | **Built** |
| Compliance scanning | `security/compliance/` | **Built** |
| Secrets operator | `security/secrets/` | **Built** |
| Custom probes | `src/citadel_operator/k3s/custom_probes.py` | **Built** |
| Health probe discovery | `src/citadel_operator/k3s/probe_discovery.py` | **Built** |
| Cloud providers (AWS/Azure/GCP) | Not yet built | Planned |
| FIPS cryptography | Not yet built | Future |

---

## 10. Summary Scorecard (Revised)

> **Updated 2026-02-28**: Scores now reflect Citadel's `feat/parity-gaps-tier1`
> branch AND apollo's control plane. The "Combined" column shows ecosystem
> parity when both are deployed together.

| Layer | Citadel | Apollo-Clone | Combined | Key Remaining Gap |
|-------|---------|-------------|----------|-------------------|
| **Ephemeral Nodes** | **98%** | — | **98%** | Cloud provider abstraction |
| **Zero-Trust Networking** | **95%** | — | **95%** | WAF, FIPS Envoy |
| **Pod Security** | **95%** | — | **95%** | kubectl-exec prevention |
| **Audit Logging** | **70%** | **90%** (compliance logging) | **90%** | Audit.3 schema, SIEM export |
| **Encryption** | **90%** | HSM integration | **95%** | FIPS validation |
| **Identity/Auth** | **60%** (infra) | **90%** (app) | **90%** | MFA hardware tokens |
| **Apollo Spoke** | Hosts it | **95%** | **95%** | Air-gapped sync |
| **Apollo Hub** | Hosts it | **90%** | **90%** | Multi-Hub hierarchies |
| **Cloud Providers** | **10%** | — | **10%** | **AWS, Azure, GCP — main remaining gap** |
| **Compliance** | **75%** | **85%** | **85%** | Formal FedRAMP certification (org effort) |
| **Monitoring** | **90%** | Prometheus integration | **90%** | Fleet-wide multi-cluster |
| **Overall (Citadel only)** | **~80%** | — | — | Cloud providers, FIPS |
| **Overall (Combined)** | — | — | **~85%** | Cloud providers, air-gap, FIPS, formal certs |

The combined ecosystem now covers the vast majority of Palantir's Apollo+Rubix
capabilities. The main remaining gaps are:

1. **Cloud providers** — Citadel only has Multipass (dev). Production needs AWS/Azure/GCP
2. **FIPS cryptography** — Required for government. Not a code problem, requires certified binaries
3. ~~**Apollo↔Citadel integration** — Both systems exist but aren't wired together yet~~ **DONE** — see below
4. **Air-gapped operation** — Cryptographic signing + offline sync
5. **Formal compliance certification** — FedRAMP/SOC2 requires org-level investment, not just code

---

## 11. Apollo↔Citadel CRD-Based Integration

> **Added 2026-02-28**: `feat/infra-boundary-refactor` branch on both repos.

### Architecture

Apollo declares infrastructure intent via provider abstractions; Citadel's
operators fulfill it through Kubernetes CRDs. When operators are absent,
Apollo falls back to standalone mode (local CA, direct Vault access,
subprocess scanning).

```
Apollo Hub                          Kubernetes Cluster
+----------------------------+      +----------------------------+
| infrastructure/             |      | Citadel Operators          |
|   _detection.py             |─────>| (CRD probe at startup)     |
|   _certificate_provider.py  |      |                            |
|   _secret_provider.py       |      | cert-manager               |
|   _factory.py               |      |   Certificate CRD ───> Secret (tls.crt/tls.key)
|                             |      |                            |
| CertManagerProvider ────────|─CRD─>| cert-manager controller    |
| ExternalSecretProvider ─────|─CRD─>| ESO controller             |
| OperatorScanner ────────────|─CRD─>| Trivy Operator             |
+----------------------------+      +----------------------------+
         │ fallback                          │ fulfills
         ▼                                   ▼
  LocalCertProvider              Operator lifecycle management
  VaultSecretProvider            (auto-renewal, continuous scan,
  TrivyScanner (subprocess)       sync monitoring)
```

### Integration Points

| Concern | CRD Provider | Standalone Fallback | CRD Group |
|---------|-------------|-------------------|-----------|
| TLS Certificates | `CertManagerCertificateProvider` | `LocalCertificateProvider` | `cert-manager.io/v1` |
| Secret Retrieval | `ExternalSecretProvider` | `VaultSecretProvider` | `external-secrets.io/v1beta1` |
| Vuln Scanning | `OperatorScanner` | `TrivyScanner` (subprocess) | `aquasecurity.github.io/v1alpha1` |

### Detection Flow

At Apollo startup (`_on_startup`):
1. `detect_capabilities()` probes `ApiextensionsV1Api` for CRD groups
2. Results stored as `InfrastructureCapabilities` (frozen dataclass)
3. Factory functions select provider implementations based on capabilities
4. Scanner auto-selection: `"auto"` default resolves to `"operator"` or `"trivy"`

### Citadel Changes

- `CertificateConfig.extra_labels` field allows Apollo to tag Certificate CRDs
  with `app.kubernetes.io/requested-by: apollo` for attribution tracking
- All Apollo-created CRDs include `app.kubernetes.io/managed-by: rubix-citadel`
  label to maintain operator ownership semantics

### Key Design Decisions

1. **Apollo never imports from `citadel_operator`** — self-contained CRD construction
2. **CRD shapes replicate Citadel's** — same YAML structure, Apollo just creates them
3. **Graceful degradation** — all providers have standalone fallbacks
4. **No runtime dependency** — `kubernetes` package is optional; missing = standalone mode
