# Citadel Development Reference

Complete technical inventory of every module in `src/citadel_operator/`.

## Package Overview

| Property | Value |
|----------|-------|
| Package name | `citadel-operator` |
| Version | `0.1.0` |
| Python | `>= 3.11` |
| License | MIT |
| Build system | Hatch (`hatchling`) |
| Entry point | `citadel = "citadel_operator.cli:main"` |
| Core deps | `kubernetes >= 29.0.0`, `pyyaml >= 6.0` |
| Optional deps | `kopf >= 1.37` (operator mode) |
| Dev deps | `pytest >= 8.0`, `pytest-cov >= 5.0`, `kopf >= 1.37` |
| CRDs bundled | `nodeterminationpolicy.yaml`, `namespaceinfrastructure.yaml`, `default-policies.yaml` |

---

## Public API Surface

26 exports from `citadel_operator/__init__.py` (re-exported from `k3s/`):

```python
# From k3s.certificates
CertificateConfig, CertificateManager

# From k3s.custom_probes
CustomProbeChecker, CustomProbeSpec, parse_custom_probes

# From k3s.health
HealthChecker, NodeCondition

# From k3s.lifecycle_models
CycleResult, LifecycleConfig, NodeTerminationPolicy, PolicyMatch

# From k3s.namespace_infra_builder
build_default_deny_policy, build_limit_range, build_peer_policies,
build_resource_quota, build_tenant_allow_policy

# From k3s.node
NodeManager

# From k3s.probe_discovery
DiscoveredProbe, EntityHealth, ProbeAnnotation, ProbeDiscoverer,
ProbeResult (aliased as DiscoveryProbeResult)

# From k3s.provider
RancherDesktopProvider

# From k3s.secrets
ExternalSecretConfig, SecretKeyMapping, SecretStoreConfig, SecretsManager
```

---

## CLI Architecture

Entry point: `citadel_operator.cli:main`

### Subcommands

```
citadel install   [--namespace NS] [--image IMG] [--with-default-policies] [--no-default-policies] [--timeout SECS]
citadel uninstall [--namespace NS] [--delete-crds]
citadel status    [--namespace NS]
citadel manifests [--namespace NS] [--image IMG]
```

### Defaults

| Setting | Default |
|---------|---------|
| Namespace | `citadel-system` |
| Image | `citadel-operator:latest` |
| Timeout | `120` seconds |
| Default policies | `True` |

### Module Layout

| Module | Purpose |
|--------|---------|
| `cli/__init__.py` | Argument parser, subcommand dispatch |
| `cli/_install.py` | Create-or-patch CRDs, RBAC, Deployment; wait for rollout; apply default policies |
| `cli/_uninstall.py` | Remove operator and optionally CRDs |
| `cli/_status.py` | Check operator installation health |
| `cli/_manifests.py` | Pure manifest generation (no I/O) |

### `_manifests.py` Functions

| Function | Returns |
|----------|---------|
| `build_namespace(name)` | v1/Namespace |
| `build_service_account(namespace)` | v1/ServiceAccount |
| `build_cluster_role()` | ClusterRole (14 permission rules) |
| `build_cluster_role_binding(namespace)` | ClusterRoleBinding |
| `build_deployment(namespace, image)` | Deployment (1 replica, non-root uid 1000, resource limits) |
| `load_crds()` | List of CRD dicts from bundled YAML |
| `load_default_policies()` | List of default NodeTerminationPolicy dicts |
| `all_manifests(namespace, image)` | Full deployment manifest list |

Common labels applied to all resources:
```python
{
    "app.kubernetes.io/name": "citadel-operator",
    "app.kubernetes.io/part-of": "citadel",
    "app.kubernetes.io/managed-by": "citadel-cli",
}
```

---

## Lifecycle Operator

Module: `k3s/lifecycle_operator.py`

### Metadata

```python
OPERATOR_NAME = "citadel-lifecycle"
OPERATOR_VERSION = "0.2.0"
CRD_GROUP = "citadel.dev"
CRD_VERSION = "v1alpha1"
CRD_PLURAL = "nodeterminationpolicies"
```

### Classes

#### `LifecyclePolicyEvaluator`

Pure evaluator — no state, no I/O.

- `__init__(health_checker: HealthChecker | None = None)`
- `evaluate(policies: list[NodeTerminationPolicy], nodes: list[NodeInfo]) -> list[PolicyMatch]`
  - Filters enabled policies, sorts by priority (descending)
  - Matches nodes by age, conditions, label selector
  - Returns one match per node (highest-priority policy wins)
- `_node_matches_policy(node, policy) -> bool`
- `_conditions_match(node_conditions, policy_conditions) -> bool`

#### `LifecycleController`

Stateful controller — wires evaluator to NodeManager.

- `__init__(config: LifecycleConfig | None = None)`
  - Initializes K8s clients, NodeManager, HealthChecker, evaluator
- `async run_evaluation_cycle() -> CycleResult`
  1. List policies from CRD API
  2. List nodes via NodeManager
  3. Evaluate matches
  4. For each match: label node for termination (auditable)
  5. If not dry-run: drain node
  6. Update policy status subresource
  7. Return CycleResult

### Kopf Handlers

| Handler | Trigger | Behavior |
|---------|---------|----------|
| `on_policy_create` | CRD create | Run evaluation cycle |
| `on_policy_update` | CRD update | Run evaluation cycle |
| `on_policy_delete` | CRD delete | Log deletion |
| `on_evaluation_timer` | 60s timer (10s initial delay) | Run evaluation cycle (deduped on first policy alphabetically) |
| `on_startup` | Operator start | Initialize controller |
| `on_cleanup` | Operator stop | Cleanup |
| `health_probe` | Liveness/readiness | Return health dict |

Singleton: `get_controller() -> LifecycleController`

---

## Namespace Infrastructure Operator

Module: `k3s/namespace_infra_operator.py`

### Metadata

```python
OPERATOR_NAME = "citadel-namespace-infra"
OPERATOR_VERSION = "0.1.0"
CRD_GROUP = "citadel.dev"
CRD_VERSION = "v1alpha1"
CRD_PLURAL = "namespaceinfrastructures"
MAX_CONDITIONS = 10
```

### `NamespaceInfraReconciler`

- `__init__()` — Initialize K8s clients
- `async reconcile(body, name, namespace) -> str | None`
  1. Parse spec into `NamespaceInfraSpec`
  2. Set phase → `Provisioning`
  3. Apply network policies (default-deny + tenant + peers) via CiliumNetworkPolicy API
  4. Apply ResourceQuota if specified
  5. Apply LimitRange if specified
  6. Compute policy hash (SHA-256, first 16 chars)
  7. Set phase → `Active`, ready=True
  8. Returns error message on failure, None on success
- `async handle_delete(body, name, namespace)` — Clean up in reverse order
- `_apply_cilium_policy(ns, body)` — Create-or-patch CiliumNetworkPolicy
- `_apply_namespaced_resource(ns, kind, body)` — Create-or-patch core resources
- `_compute_policy_hash(spec)` — SHA-256 of JSON-serialized config
- `_patch_status(name, ns, status_dict)` — Update status subresource

### Kopf Handlers

| Handler | Trigger | Behavior |
|---------|---------|----------|
| `on_reconcile` | create/update/resume | Run reconciliation |
| `on_delete` | CRD delete | Clean up resources |
| `on_drift_check` | 300s timer | Re-reconcile Active resources |
| `on_startup` | Operator start | Initialize reconciler |
| `health_probe` | Liveness/readiness | Return health dict |

Singleton: `get_reconciler() -> NamespaceInfraReconciler`

---

## Lifecycle Models

Module: `k3s/lifecycle_models.py`

### `ConditionMatch` (frozen)
- `type: str` — e.g., DiskPressure, MemoryPressure, Ready
- `status: str` — True, False, Unknown

### `DrainConfig` (frozen)
- `timeout_seconds: int = 300`
- `delete_emptydir_data: bool = True`
- `force_after_timeout: int | None = None`

### `PolicySelector` (frozen)
- `max_age_hours: float | None = None`
- `conditions: list[ConditionMatch] = []`
- `node_selector: dict[str, str] = {}`
- `from_dict(cls, data) -> PolicySelector`

### `PolicySpec` (mutable)
- `reason: str`
- `selector: PolicySelector`
- `priority: int = 10`
- `enabled: bool = True`
- `dry_run: bool = False`
- `drain: DrainConfig = DrainConfig()`
- `from_dict(cls, data) -> PolicySpec`

### `PolicyStatus` (mutable)
- `last_evaluated: datetime | None = None`
- `nodes_matched: int = 0`
- `nodes_terminated: int = 0`
- `last_termination: datetime | None = None`
- `to_dict() -> dict`
- `from_dict(cls, data) -> PolicyStatus`

### `NodeTerminationPolicy` (mutable)
- `name: str`
- `spec: PolicySpec`
- `status: PolicyStatus = PolicyStatus()`
- `labels: dict[str, str] = {}`
- `policy_type: str` — Inferred from labels or selector (property)
- `from_dict(cls, data) -> NodeTerminationPolicy`

### `PolicyMatch` (frozen)
- `node_name: str`
- `policy_name: str`
- `reason: str`
- `priority: int`
- `age_hours: float`
- `dry_run: bool`
- `drain_timeout: int`
- `delete_emptydir_data: bool = True`
- `force_after_timeout: int | None = None`

### `CycleResult` (mutable)
- `policies_evaluated: int`
- `nodes_checked: int`
- `nodes_matched: int`
- `nodes_drained: int`
- `nodes_failed: list[str] = []`
- `dry_run_only: list[str] = []`

### `LifecycleConfig` (mutable)
- `kubeconfig: str | None = None`
- `evaluation_interval_seconds: int = 60`
- `dry_run_override: bool = False`
- `max_age_hours: float = 48.0`

---

## Namespace Infrastructure Models

Module: `k3s/namespace_infra_models.py`

### `InfraPhase` (Enum)
```python
PENDING = "Pending"
PROVISIONING = "Provisioning"
ACTIVE = "Active"
FAILED = "Failed"
```

### `NetworkPolicyConfig` (frozen)
- `default_deny: bool = True`
- `ingress_rules: tuple[dict, ...] = ()`
- `egress_rules: tuple[dict, ...] = ()`
- `allowed_peers: tuple[str, ...] = ()`
- `allow_kube_system: bool = True`
- `allow_monitoring: bool = True`
- `from_dict()`, `to_dict()`

### `ResourceQuotaConfig` (frozen)
- `cpu: str | None`, `memory: str | None`, `pods: str | None`
- `storage: str | None`, `pvcs: str | None`
- `custom: tuple[tuple[str, str], ...] = ()`

### `LimitRangeConfig` (frozen)
- `default_cpu: str = "500m"`, `default_memory: str = "512Mi"`
- `default_request_cpu: str = "100m"`, `default_request_memory: str = "128Mi"`
- `max_cpu: str = "4"`, `max_memory: str = "8Gi"`

### `NamespaceInfraSpec` (frozen)
- `target_namespace: str`
- `network_policy: NetworkPolicyConfig = NetworkPolicyConfig()`
- `resource_quota: ResourceQuotaConfig | None = None`
- `limit_range: LimitRangeConfig | None = None`

### `ManagedInfraResources` (mutable)
- `default_deny_policy: bool = False`
- `tenant_policy: bool = False`
- `resource_quota: bool = False`
- `limit_range: bool = False`

### `InfraCondition` (mutable)
- `type: str`, `status: str`, `message: str`, `reason: str`
- `last_transition_time: datetime`

### `NamespaceInfraStatus` (mutable)
- `phase: InfraPhase = InfraPhase.PENDING`
- `ready: bool = False`
- `managed_resources: ManagedInfraResources`
- `conditions: list[InfraCondition] = []`
- `policy_hash: str | None = None`

### `NamespaceInfrastructure` (mutable)
- `name: str`, `namespace: str`
- `spec: NamespaceInfraSpec`
- `status: NamespaceInfraStatus`

---

## Node Management

Module: `k3s/node.py`

### Constants

```python
LABEL_TERMINATE_REASON = "citadel.dev/terminate-reason"
LABEL_TERMINATE_AFTER = "citadel.dev/terminate-after"
LABEL_POLICY_NAME = "citadel.dev/policy-name"
LABEL_MANAGED_BY = "citadel.dev/managed-by"
MANAGED_BY_VALUE = "citadel-lifecycle"
```

### `NodeInfo` (frozen)
- `name: str`
- `creation_timestamp: datetime`
- `age_hours: float`
- `ready: bool`
- `schedulable: bool`
- `conditions: dict[str, str] = {}`
- `labels: dict[str, str] = {}`
- `marked_for_termination: bool` (property)
- `terminate_reason: str | None` (property)

### `DrainResult` (frozen)
- `node_name: str`
- `success: bool`
- `pods_evicted: int`
- `pods_failed: list[str] = []`
- `pods_force_deleted: list[str] = []`
- `error: str | None = None`

### `NodeManager`
- `__init__(kubeconfig: str | None = None)`
- `list_nodes() -> list[NodeInfo]`
- `get_node(name) -> NodeInfo`
- `get_node_age_hours(name) -> float`
- `cordon(name)` — Mark unschedulable
- `uncordon(name)` — Mark schedulable
- `label_for_termination(name, reason, policy_name)` — Set audit labels
- `clear_termination_labels(name)` — Remove termination labels
- `drain(name, timeout_seconds=300, delete_emptydir_data=True, force_after_timeout=None) -> DrainResult`
  - Uses Kubernetes Eviction API (respects PDBs)
  - Skips DaemonSet pods and mirror pods
  - Retries evictions blocked by PDBs
  - Force-deletes after `force_after_timeout` if set

---

## Health Checking

Module: `k3s/health.py`

### `HealthState` (Enum)
```python
HEALTHY = "healthy"
WARNING = "warning"
ERROR = "error"
TERMINAL = "terminal"
```

### State Determination Logic

| State | Condition |
|-------|-----------|
| TERMINAL | Not ready AND unschedulable |
| ERROR | Not ready (any), OR 2+ pressure conditions, OR failed custom probes |
| WARNING | 1 pressure condition OR age > 80% of max |
| HEALTHY | Default |

### `NodeCondition` (frozen)
- `condition_type: str`, `status: str`, `healthy: bool`, `message: str`

### `NodeHealthReport` (frozen)
- `node_name: str`, `state: HealthState`
- `conditions: list[NodeCondition]`
- `age_hours: float`, `max_age_hours: float`, `age_exceeded: bool`
- `custom_probe_report: CustomProbeReport | None`
- `entity_health: list[EntityHealth] | None`
- `needs_termination: bool` — Age exceeded OR terminal OR failed probes (property)
- `has_failed_custom_probes: bool` (property)
- `failed_custom_probes: list[ProbeResult]` (property)
- `unhealthy_entities: list[EntityHealth]` (property)

### `HealthChecker`
- `__init__(max_age_hours: float = 48.0)`
- `check(node, custom_probe_report=None, entity_health=None) -> NodeHealthReport`
- `check_all(nodes, custom_probe_reports=None, entity_health_by_node=None) -> list[NodeHealthReport]`

---

## Custom Probes

Module: `k3s/custom_probes.py`

### Enums

```python
class ProbeType(Enum):
    EXEC = "exec"
    HTTP = "http"

class ProbeStatus(Enum):
    PASSING = "passing"
    FAILING = "failing"
    UNKNOWN = "unknown"
```

### `ExecProbeConfig` (frozen)
- `command: list[str]`, `timeout_seconds: int = 10`

### `HttpProbeConfig` (frozen)
- `path: str`, `port: int`, `scheme: str = "http"`, `host: str = "localhost"`
- `url: str` (property)

### `CustomProbeSpec` (frozen)
- `name: str`, `probe_type: ProbeType`
- `failure_threshold: int = 3`, `period_seconds: int = 60`
- `exec_config: ExecProbeConfig | None`, `http_config: HttpProbeConfig | None`
- Validates required config in `__post_init__`

### `ProbeResult` (frozen)
- `probe_name: str`, `probe_type: ProbeType`, `status: ProbeStatus`
- `consecutive_failures: int`, `failure_threshold: int`
- `message: str`, `timestamp: float`
- `threshold_exceeded: bool` (property)

### `CustomProbeReport` (frozen)
- `node_name: str`, `results: list[ProbeResult]`, `timestamp: float`
- `any_failed: bool` (property)
- `failed_probes: list[ProbeResult]` (property)
- `passing_probes: list[ProbeResult]` (property)

### `CustomProbeChecker`

Stateful — tracks consecutive failures per probe per node.

- `__init__()`
- `check(node_name, probes) -> CustomProbeReport` — Execute probes, track failures
- `reset(node_name=None)` — Reset failure state
- `get_failure_count(node_name, probe_name) -> int`

### `parse_custom_probes(raw_probes: list[dict]) -> list[CustomProbeSpec]`

---

## Probe Discovery

Module: `k3s/probe_discovery.py`

### Annotation Constants

```python
ANNO_HEALTH_PORT  = "apollo.citadel.io/health-port"    # Required
ANNO_HEALTH_PATH  = "apollo.citadel.io/health-path"    # Default: /health
ANNO_HEALTH_SCHEME = "apollo.citadel.io/health-scheme"  # Default: http
ANNO_ENTITY_NAME  = "apollo.citadel.io/entity-name"     # Required
```

### `ProbeAnnotation` (frozen)
- `port: int`, `path: str`, `scheme: str`, `entity_name: str`
- `from_annotations(cls, annotations) -> ProbeAnnotation | None`

### `DiscoveredProbe` (frozen)
- `pod_name: str`, `pod_ip: str`, `namespace: str`, `entity_name: str`
- `port: int`, `path: str`, `scheme: str = "http"`
- `url: str` (property)

### `ProbeResult` (frozen)
- `probe: DiscoveredProbe`, `healthy: bool`, `status_code: int | None`
- `response_time_ms: float`, `error: str | None`, `checked_at: str`

### `EntityHealth` (frozen)
- `entity_name: str`, `total_pods: int`, `healthy_pods: int`, `unhealthy_pods: int`
- `health_ratio: float`, `probe_results: tuple[ProbeResult, ...]`
- `fully_healthy: bool`, `fully_unhealthy: bool` (properties)

### `ProbeDiscoverer`
- `__init__(namespace="default", poll_interval=30, health_check_timeout=5)`
- `discover_probes() -> list[DiscoveredProbe]` — Find annotated pods via kubectl
- `check_probe(probe) -> ProbeResult` — Single HTTP health check via urllib
- `check_all() -> list[ProbeResult]` — Discover + check all
- `aggregate_health() -> list[EntityHealth]` — Group by entity name

---

## Namespace Infrastructure Builder

Module: `k3s/namespace_infra_builder.py`

Pure functions — no I/O, no state.

### Constants
```python
LABEL_MANAGED_BY = "app.kubernetes.io/managed-by"
MANAGED_BY_VALUE = "citadel-operator"
```

### Functions

| Function | Returns |
|----------|---------|
| `build_default_deny_policy(target_ns, config)` | CiliumNetworkPolicy (deny-all with DNS + API server exceptions) |
| `build_tenant_allow_policy(target_ns, config)` | CiliumNetworkPolicy (custom rules + kube-system/monitoring access) |
| `build_peer_policies(target_ns, config)` | List of CiliumNetworkPolicies for bidirectional peer namespaces |
| `build_resource_quota(target_ns, config)` | ResourceQuota dict or None |
| `build_limit_range(target_ns, config)` | LimitRange dict or None |

All resources are labeled with `managed-by: citadel-operator`.

---

## Certificates

Module: `k3s/certificates.py`

### Constants
```python
VALID_ISSUERS = {"letsencrypt-staging", "letsencrypt-prod", "internal-ca", "citadel-selfsigned-bootstrap"}
DEFAULT_DURATION_HOURS = 2160   # 90 days
DEFAULT_RENEW_BEFORE_HOURS = 720  # 30 days
```

### `CertificateConfig` (frozen)
- `issuer_name: str`, `common_name: str`
- `dns_names: tuple[str, ...] = ()`
- `duration_hours: int = 2160`, `renew_before_hours: int = 720`
- `is_ca: bool = False`, `secret_name: str | None = None`
- `namespace: str = "default"`
- `private_key_algorithm: str = "ECDSA"`, `private_key_size: int = 256`
- `usages: tuple[str, ...] = ("server auth", "digital signature", "key encipherment")`
- `extra_labels: dict[str, str] = {}`
- `effective_secret_name: str` (property) — Auto-gen from CN
- `effective_dns_names: tuple[str, ...]` (property) — Falls back to (CN,)

### `CertificateManager`
- `__init__(kubeconfig=None)`
- `@staticmethod create_certificate(config) -> dict` — Pure YAML generation
- `list_certificates(namespace="default") -> list[dict]`
- `check_expiry(namespace="default") -> list[tuple[str, datetime, bool]]`
- `rotate_certificate(name, namespace="default") -> bool` — Deletes backing Secret

---

## Secrets

Module: `k3s/secrets.py`

### Constants
```python
VALID_PROVIDERS = {"kubernetes", "vault", "aws", "azure"}
DEFAULT_REFRESH_INTERVAL = "1h"
```

### `SecretKeyMapping` (frozen)
- `remote_key: str`, `secret_key: str`, `property: str | None = None`

### `SecretStoreConfig` (frozen)
- `name: str`, `provider: str`, `namespace: str | None = None`
- `vault_url: str | None`, `aws_region: str | None`, `azure_vault_url: str | None`
- `is_cluster_scoped: bool` (property)

### `ExternalSecretConfig` (frozen)
- `name: str`, `namespace: str`, `store_name: str`
- `refresh_interval: str = "1h"`, `keys: tuple[SecretKeyMapping, ...] = ()`
- `target_secret_name: str | None`, `store_kind: str = "SecretStore"`
- `effective_target_name: str` (property)

### `SecretsManager`
- `__init__(kubeconfig=None)`
- `@staticmethod create_secret_store(config) -> dict` — Pure YAML generation
- `@staticmethod create_external_secret(config) -> dict` — Pure YAML generation
- `list_secret_stores(namespace=None) -> list[dict]`
- `list_external_secrets(namespace) -> list[dict]`
- `check_sync_status(name, namespace) -> dict`

---

## Compliance

Module: `k3s/compliance.py`

### Constants
```python
KUBE_BENCH_NAMESPACE = "citadel-compliance"
KUBE_BENCH_CRONJOB_NAME = "kube-bench-scan"
SEVERITY_CRITICAL = "CRITICAL"
SEVERITY_HIGH = "HIGH"
SEVERITY_MEDIUM = "MEDIUM"
SEVERITY_LOW = "LOW"
```

### `ComplianceFailure` (frozen)
- `check_id: str`, `description: str`, `severity: str`, `remediation: str`

### `ComplianceScanResult` (frozen)
- `benchmark: str`, `score: float`, `total_checks: int`
- `passed: int`, `failed: int`, `warnings: int`
- `failures: tuple[ComplianceFailure, ...]`, `scanned_at: str`
- `passing_rate: float`, `has_critical_failures: bool`, `has_high_failures: bool` (properties)
- `failures_by_severity: dict[str, list[ComplianceFailure]]` (property)

### `ComplianceScanner`
- `__init__(kubeconfig=None)`
- `run_cis_benchmark() -> ComplianceScanResult` — Parse kube-bench ConfigMap results
- `trigger_cis_scan() -> str` — Create one-off Job
- `get_vulnerability_report(namespace="default") -> list[dict]` — Trivy VulnerabilityReports
- `get_config_audit_reports(namespace="default") -> list[dict]` — Trivy ConfigAuditReports
- `get_compliance_summary() -> dict` — Aggregated CIS + Trivy
- `list_non_compliant_workloads(namespace="default") -> list[dict]` — HIGH+ violations

---

## SSO

Module: `k3s/sso.py`

### Constants
```python
VALID_CONNECTOR_TYPES = {"saml", "ldap", "github", "oidc", "static"}
DEFAULT_SCOPES = ("openid", "profile", "email", "groups")
```

### `OIDCConfig` (frozen)
- `issuer_url: str`, `client_id: str`, `client_secret: str | None`
- `scopes: tuple[str, ...] = DEFAULT_SCOPES`
- `groups_claim: str = "groups"`, `username_claim: str = "email"`

### `DexConnector` (frozen)
- `connector_type: str`, `name: str`, `connector_id: str = ""`
- `config: dict = {}`
- `effective_id: str` (property)
- `to_dex_config() -> dict`

### `SSOManager`
- `__init__(dex_url=None)`
- `@staticmethod generate_k8s_oidc_flags(config) -> dict[str, str]`
- `@staticmethod generate_kubeconfig(config, token, ...) -> dict`
- `list_connectors() -> list[dict]` — Query Dex API
- `@staticmethod create_rbac_binding(group, cluster_role, namespace=None) -> dict`

---

## Provider

Module: `k3s/provider.py`

### Defaults
```python
DEFAULT_CPUS = "2"
DEFAULT_MEMORY = "2G"
DEFAULT_DISK = "20G"
DEFAULT_IMAGE = "22.04"  # Ubuntu LTS
```

### `VMInfo` (frozen)
- `name: str`, `state: str`, `ipv4: list[str]`
- `cpus: str`, `memory: str`, `disk: str`

### `ReplaceResult` (frozen)
- `old_vm: str`, `new_vm: str`, `success: bool`
- `new_ip: str | None`, `error: str | None`

### `RancherDesktopProvider`
- `__init__(cpus="2", memory="2G", disk="20G", image="22.04")`
- `list_vms() -> list[VMInfo]`
- `get_vm(name) -> VMInfo | None`
- `create_vm(name, cloud_init=None) -> VMInfo | None`
- `destroy_vm(name, purge=True) -> bool`
- `join_cluster(vm_name, server_url, token) -> bool`
- `replace_vm(old_name, server_url, token, cloud_init=None) -> ReplaceResult`

---

## Design Patterns

### Frozen Dataclasses for Immutability
All snapshot types (NodeInfo, DrainResult, CertificateConfig, ProbeResult, etc.) are frozen. Safe to pass between layers, serialize, and cache.

### Pure Functions for Configuration
Static methods (`create_certificate()`, `create_secret_store()`, `generate_k8s_oidc_flags()`) perform no I/O. Snapshot-testable and offline-friendly.

### Stateful Managers for Operations
`NodeManager`, `CertificateManager`, `SecretsManager` hold K8s client state. Instance methods for cluster operations.

### Kopf-Based Reconciliation
Two independent operators (lifecycle + namespace infra). Event-driven handlers (create/update/delete) + periodic timers. Health probes for liveness/readiness.

### Create-or-Patch Idempotency
Both CLI install and operator reconciliation use 409 Conflict → patch strategy for idempotent resource management.

### Policy-Driven Node Termination
Two-phase: label with audit trail → drain. Highest-priority policy wins. Drain respects PDBs via Eviction API with optional force-delete.

---

## Data Flow Diagrams

### Lifecycle Operator

```
NodeTerminationPolicy CRD
  ↓
LifecycleController.run_evaluation_cycle()
  ├→ List policies (CRD API) + nodes (NodeManager)
  ├→ LifecyclePolicyEvaluator.evaluate()
  │   ├→ Filter enabled, sort by priority
  │   └→ Match nodes by age/conditions/labels
  ├→ For each match:
  │   ├→ NodeManager.label_for_termination()
  │   ├→ [DRY_RUN] log only
  │   └→ [LIVE] NodeManager.drain()
  │       ├→ Cordon
  │       ├→ Evict pods (Eviction API)
  │       └→ [timeout] Force-delete
  └→ CycleResult
```

### Namespace Infrastructure Operator

```
NamespaceInfrastructure CRD
  ↓
NamespaceInfraReconciler.reconcile()
  ├→ Parse spec → NamespaceInfraSpec
  ├→ Set phase → Provisioning
  ├→ Build + apply CiliumNetworkPolicies (default-deny + tenant + peers)
  ├→ Build + apply ResourceQuota
  ├→ Build + apply LimitRange
  ├→ Compute policy hash (SHA-256)
  └→ Set phase → Active, ready=True
```

### Health Assessment Pipeline

```
NodeManager.list_nodes() → list[NodeInfo]
  ↓
HealthChecker.check_all()
  ├→ Evaluate K8s conditions (Ready, Pressure)
  ├→ Check age vs max_age_hours
  ├→ [optional] CustomProbeChecker.check()
  ├→ [optional] ProbeDiscoverer.aggregate_health()
  └→ Compute state: HEALTHY → WARNING → ERROR → TERMINAL
```

---

## Testing Architecture

### Test Files

| File | Tests | Assertions |
|------|-------|------------|
| `test_lifecycle_operator.py` | Evaluator + controller logic, policy matching, dry-run | ~100 |
| `test_models.py` | Dataclass immutability, defaults | ~15 |
| `test_namespace_infra_builder.py` | Pure builder functions (dict in → dict out) | ~50 |
| `test_health.py` | State transitions, condition reporting | ~30 |
| `test_custom_probes.py` | Exec/HTTP probes, failure tracking, period skipping | ~80 |
| `test_probe_discovery.py` | Annotation parsing, HTTP checks, entity aggregation | ~80 |
| `test_certificates.py` | Config validation, YAML generation, cluster ops | ~60 |
| `test_force_drain.py` | PDB-blocking, force-delete, DaemonSet skipping | ~30 |
| `test_cli_install.py` | CLI install flow | ~10 |
| `test_cli_manifests.py` | Manifest generation | ~10 |

### E2E Tests

| File | Purpose |
|------|---------|
| `test_02_node_operations.py` | Live node cordon/uncordon/label |
| `test_03_health_assessment.py` | Live health checking |
| `test_05_controller_dry_run.py` | Full evaluation cycle in dry-run |
| `test_06_drain.py` | Live drain operations |

### Test Patterns

- **Pure logic under test**: All external I/O mocked (K8s API, subprocess, urllib)
- **Factory helpers**: `_make_node_info()`, `_make_pod_dict()`, `_make_policy()`
- **No cluster dependency**: Unit tests run without cluster access
- **E2E optional**: Skipped if kubeconfig unavailable

---

## Technical Debt

### Remaining `citadel` naming

| Location | Current | Target |
|----------|---------|--------|
| CRD group | `citadel.dev` | — (breaking change, defer) |
| Node labels | `citadel.dev/terminate-reason` | — |
| Operator namespace | `citadel-system` | — |
| Managed-by values | `citadel-lifecycle`, `citadel-operator` | — |
| Probe annotations | `apollo.citadel.io/*` | — |
| `./citadel-ctl` CLI script | Legacy wrapper | May rename to `./citadel-infra` |
| Default image | `citadel-operator:latest` | — |
