# Node Lifecycle Operator

Citadel enforces a **48-hour ephemeral node lifecycle** inspired by [Palantir's Rubix](https://blog.palantir.com/the-benefits-of-running-kubernetes-on-ephemeral-compute-436e5acf13fb). No node lives longer than 48 hours. Nodes are immutable and replaced, never upgraded in-place.

## Architecture

```
NodeTerminationPolicy CRD               Citadel Operator
┌──────────────────────────┐            ┌──────────────────────────┐
│ Policy evaluation:       │            │ NodeManager              │
│  - age > 48h?            │───────────>│  - cordon / uncordon     │
│  - DiskPressure?         │            │  - drain (Eviction API)  │
│  - NotReady?             │            │  - label nodes           │
│  - Custom probes?        │            │                          │
│                          │            │ HealthChecker            │
│ 60-second eval timer     │            │  - HEALTHY / WARNING     │
│ Dry-run mode             │            │  - ERROR / TERMINAL      │
│ Priority ordering        │            │                          │
│ Audit trail (labels)     │            │ RancherDesktopProvider        │
└──────────────────────────┘            │  - destroy old VM        │
     Decides WHEN and WHY              │  - create fresh VM       │
                                        │  - auto-join cluster     │
                                        └──────────────────────────┘
                                             Knows HOW
```

## Quick Start

```bash
# Install the operator with default policies
citadel install

# Check active policies
kubectl get nodeterminationpolicies

# Check node ages and health
kubectl get nodes -o custom-columns='NAME:.metadata.name,AGE:.metadata.creationTimestamp,READY:.status.conditions[?(@.type=="Ready")].status'

# Watch policy evaluations
kubectl get ntp -w
```

## NodeTerminationPolicy CRD

Policies are declarative Kubernetes resources in the `citadel.dev/v1alpha1` API group.

### Minimal Example

```yaml
apiVersion: citadel.dev/v1alpha1
kind: NodeTerminationPolicy
metadata:
  name: max-age-48h
spec:
  reason: "Node exceeded 48-hour maximum age"
  priority: 10
  selector:
    maxAgeHours: 48
  drain:
    timeoutSeconds: 300
```

### Full Spec

```yaml
apiVersion: citadel.dev/v1alpha1
kind: NodeTerminationPolicy
metadata:
  name: example-policy
spec:
  reason: "Human-readable reason (written to node labels)"  # Required
  priority: 10           # 0-1000, higher = evaluated first (default: 10)
  enabled: true          # Whether this policy is active (default: true)
  dryRun: false          # Label but don't drain (default: false)
  selector:              # Required — at least one criterion
    maxAgeHours: 48      # Terminate nodes older than N hours
    conditions:          # Match Kubernetes node conditions
      - type: DiskPressure
        status: "True"   # Must be "True", "False", or "Unknown"
    customProbes:        # Custom failure detection
      - name: stuck-volume
        type: exec       # "exec" or "http"
        exec:
          command: ["/bin/sh", "-c", "timeout 5 ls /var/lib/kubelet/pods/*/volumes/"]
          timeoutSeconds: 10     # 1-300, default: 10
        failureThreshold: 3      # 1-100, consecutive failures before failed (default: 3)
        periodSeconds: 60        # 5-3600, how often to run (default: 60)
      - name: gpu-health
        type: http
        http:
          path: /healthz
          port: 8476
          scheme: http           # "http" or "https" (default: "http")
          host: localhost        # Default: "localhost"
        failureThreshold: 2
        periodSeconds: 30
    nodeSelector:        # Match nodes by labels
      node-pool: gpu
  drain:
    timeoutSeconds: 300              # 30-3600, max eviction wait (default: 300)
    deleteEmptydirData: true         # Evict pods with emptyDir (default: true)
    forceAfterTimeoutSeconds: 600    # 30-3600, force-delete after timeout (optional)
```

### Default Policies

Installed by `citadel install --with-default-policies`:

| Policy | Priority | Trigger | Drain Timeout | Enabled |
|--------|----------|---------|---------------|---------|
| `max-age-48h` | 10 | Node older than 48 hours | 300s | Yes |
| `disk-pressure` | 100 | DiskPressure condition True | 120s | Yes |
| `memory-pressure` | 100 | MemoryPressure condition True | 120s | Yes |
| `not-ready` | 200 | Ready condition False | 60s | Yes |
| `stuck-volume-detector` | 150 | Exec probe: stuck kubelet volume mounts | 120s | **No** (opt-in) |
| `gpu-health-detector` | 150 | HTTP probe: GPU health endpoint on port 8476 | 180s | **No** (opt-in) |

Higher priority policies are evaluated first. When multiple policies match a node, the highest-priority policy wins.

### Policy Priority Convention

| Range | Use |
|-------|-----|
| 1-50 | Age-based policies |
| 51-99 | Custom operational policies |
| 100-149 | Standard health conditions |
| 150-199 | Custom probe-based policies |
| 200+ | Critical health (NotReady) |

## Evaluation Loop

The operator runs an evaluation cycle every 60 seconds (10-second initial delay):

1. List all `NodeTerminationPolicy` resources from the API
2. Filter to enabled policies, sort by priority (descending)
3. List all nodes via `NodeManager`
4. For each policy, match nodes by selector criteria (age, conditions, labels, custom probes)
5. For each match (highest-priority policy wins per node):
   a. Label node with termination intent (auditable)
   b. If `dryRun: true` → log only, add to `dry_run_only` list
   c. If `dryRun: false` → drain node
6. Update policy status subresource (`nodesMatched`, `nodesTerminated`, `lastEvaluated`)

Timer deduplication: When multiple policies exist, the timer fires for each CRD instance but the evaluation only runs once (deduped on the first policy alphabetically).

## Node Label Schema

Before any destructive action, the controller labels the node with its intent:

| Label | Purpose | Example |
|-------|---------|---------|
| `citadel.dev/terminate-reason` | Why the node is being terminated | `Node exceeded 48-hour maximum age` |
| `citadel.dev/terminate-after` | ISO 8601 timestamp of the termination decision | `2024-01-15T10:30:00Z` |
| `citadel.dev/policy-name` | Which policy triggered the termination | `max-age-48h` |
| `citadel.dev/managed-by` | Controller identifier | `citadel-lifecycle` |

Labels are set *before* drain begins, creating an audit trail even if the drain fails.

## Drain Behavior

Citadel drains nodes using the **Kubernetes Eviction API**, not `kubectl drain --force`. This ensures PodDisruptionBudgets are respected.

### Standard Drain

1. **Cordon** the node (mark unschedulable)
2. **List** all non-DaemonSet, non-mirror pods on the node
3. **Evict** each pod via the Eviction API
   - If a PDB blocks eviction, retry with backoff
   - DaemonSet pods and mirror pods are skipped
4. **Wait** for evicted pods to terminate
5. **Timeout** after `spec.drain.timeoutSeconds` (default 300s)

### Force Drain

When `forceAfterTimeoutSeconds` is set in the policy:

1. Standard drain proceeds as above
2. If pods remain after `timeoutSeconds`, wait `forceAfterTimeoutSeconds`
3. Force-delete remaining pods directly (bypasses PDB constraints)

Use `forceAfterTimeoutSeconds` carefully — it can violate PDB guarantees. Recommended for non-critical workloads or when replacement is more important than graceful shutdown.

### Drain Result

```python
@dataclass(frozen=True)
class DrainResult:
    node_name: str
    success: bool
    pods_evicted: int
    pods_failed: list[str] = []
    pods_force_deleted: list[str] = []
    error: str | None = None
```

## Health Assessment

The `HealthChecker` evaluates node health using a 4-state model:

| State | Condition |
|-------|-----------|
| **TERMINAL** | Not ready AND unschedulable |
| **ERROR** | Not ready (any), OR 2+ pressure conditions, OR failed custom probes |
| **WARNING** | 1 pressure condition, OR age > 80% of max |
| **HEALTHY** | Default |

A `NodeHealthReport` aggregates K8s conditions, age, custom probes, and entity health into a single assessment.

### Custom Probes

Custom probes detect failure modes beyond standard Kubernetes conditions:

- **Exec probes**: Run a command on the node. Non-zero exit = failure.
- **HTTP probes**: Check an endpoint. Non-2xx response = failure.

Probes only "fail" after exceeding `failureThreshold` consecutive failures. Failure count resets on success. Probes respect `periodSeconds` — they skip execution if not yet due.

### Entity Health Discovery

Pods annotated with Apollo health annotations are automatically discovered:

```yaml
annotations:
  apollo.citadel.io/health-port: "8081"      # Required
  apollo.citadel.io/entity-name: "my-service" # Required
  apollo.citadel.io/health-path: "/ready"     # Default: /health
  apollo.citadel.io/health-scheme: "http"     # Default: http
```

The `ProbeDiscoverer` aggregates results per entity, computing health ratios. This feeds into the `HealthChecker` for comprehensive node health assessment.

## Dry-Run Mode

Two levels of dry-run:

1. **Per-policy**: Set `spec.dryRun: true` — this policy labels nodes but doesn't drain them
2. **Global override**: `LifecycleConfig(dry_run_override=True)` — all policies become dry-run

Dry-run nodes appear in `CycleResult.dry_run_only` and are labeled with termination intent labels (for visibility) but not drained.

## Python API

```python
from citadel_operator import NodeManager, HealthChecker, RancherDesktopProvider

# Node operations
nm = NodeManager()
nodes = nm.list_nodes()
nm.cordon("worker-1")
result = nm.drain("worker-1", timeout_seconds=300)
nm.label_for_termination("worker-1", reason="max-age", policy_name="max-age-48h")

# Health assessment
hc = HealthChecker(max_age_hours=48)
report = hc.check(nodes[0])
print(report.state)              # HealthState.HEALTHY
print(report.needs_termination)  # False

# VM replacement
mp = RancherDesktopProvider(cpus="2", memory="2G", disk="20G")
result = mp.replace_vm("worker-1", server_url="https://10.0.0.1:6443", token="K3S_TOKEN")
```

## RBAC

The operator runs as `citadel-operator` ServiceAccount in `citadel-system` with ClusterRole permissions for:

- `citadel.dev` CRDs: full CRUD + status updates
- `cilium.io` CiliumNetworkPolicies: full CRUD
- Core API: nodes, pods, eviction, namespaces, quotas, limit ranges
- Policy API: PDB read access
- Coordination API: lease management (leader election)
- Events: create/patch (audit trail)

## Status Subresource

Each policy tracks its evaluation state:

```yaml
status:
  lastEvaluated: "2024-01-15T10:30:00Z"
  nodesMatched: 2
  nodesTerminated: 5
  lastTermination: "2024-01-15T09:15:00Z"
  conditions:
    - type: Evaluated
      status: "True"
      reason: CycleComplete
      message: "Evaluated 4 policies, 2 nodes matched"
```

## Kopf Event Handlers

| Event | Handler | Behavior |
|-------|---------|----------|
| CRD created | `on_policy_create` | Run evaluation cycle |
| CRD updated | `on_policy_update` | Run evaluation cycle |
| CRD deleted | `on_policy_delete` | Log deletion |
| Timer (60s) | `on_evaluation_timer` | Run evaluation cycle (deduped) |
| Startup | `on_startup` | Initialize controller |
| Shutdown | `on_cleanup` | Cleanup |
| Health probe | `health_probe` | Return health dict |
