# ilum CLI

CLI installer and management tool for the **Ilum Data Lakehouse** platform.

![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue)
![CI](https://img.shields.io/github/actions/workflow/status/ilum-cloud/ilum/cli-ci.yml?label=CI)

---

## Overview

The Ilum platform deploys as a **Helm umbrella chart** with 32 optional modules, dozens of `--set` flags, and complex interdependencies between sub-charts. Getting a single flag wrong can silently break an entire subsystem.

`ilum` is a typed Python CLI that tames this complexity:

- **Quickstart** — `ilum quickstart` detects or creates a cluster and installs Ilum with all defaults in a single command. Zero decisions required.
- **Interactive setup** — `ilum init` walks you through prerequisites, cluster selection, and module configuration in a single wizard.
- **Local clusters** — `ilum cluster create` spins up k3d/minikube/kind clusters with preset resource profiles.
- **Install and upgrade** — `ilum install` and `ilum upgrade` handle module resolution, `--atomic` Helm execution, stuck-release recovery, and breaking-change warnings.
- **Module management** — `ilum module enable sql` resolves the full dependency chain and runs a single `helm upgrade`. `ilum module disable` is shared-dep-aware.
- **Values safety** — every upgrade fetches live values, detects external drift, computes a diff, and shows it for confirmation before applying. No silent overwrites.
- **Doctor checks** — 13 health checks covering binaries, cluster connectivity, pod health, RBAC, PVCs, Helm release state, service endpoints, and health endpoint probing.
- **Configuration profiles** — Cross-platform YAML config (XDG on Linux/macOS, `%APPDATA%` on Windows) with named profiles, dot-notation get/set, and file locking.
- **Log streaming** — `ilum logs core` tails pod logs by module name, with `--follow`, `--tail`, `--container`, and `--previous` support.
- **Deployment presets** — `ilum install --preset production` applies curated module bundles and Helm flags. Four presets: `development`, `production`, `data-engineering`, `air-gapped`.
- **Air-gapped tooling** — `ilum airgap images` lists container images, `ilum airgap export` creates a portable bundle, `ilum airgap import` loads images into a private registry.
- **Machine-readable output** — `--output json`, `--output yaml`, or `--output csv` on all query commands for CI/CD-friendly structured output. Use `--no-headers` and `--field-selector` for precise field control. Combine with `--quiet` to suppress informational messages.
- **Shell completion** — `ilum --install-completion bash/zsh/fish` installs tab completion for module names, profile names, and doctor check names.
- **Operation history** — `ilum history` provides a full audit trail of all CLI operations with timestamps, filtering, and JSON export.
- **Profile management** — `ilum config export` and `ilum config import` enable sharing profiles across teams. `ilum config use` switches profiles in one command.
- **Doctor auto-fix** — `ilum doctor --fix` auto-fixes simple issues (e.g., missing Helm repo). `--failures-only` filters to just failures. Resource checks validate cluster capacity.
- **Module dependency tree** — `ilum module tree <name>` visualizes the full dependency graph with resource estimates. `ilum module list --search` filters modules by name or description.
- **Structured error codes** — every error includes a machine-readable code (e.g., `ILUM-020`) for programmatic handling in CI/CD pipelines.
- **Command preview** — every mutating command shows the exact `helm` command before execution, so you always know what runs on your cluster.
- **Connect to existing releases** — `ilum connect` auto-detects running Ilum releases, syncs modules and values for drift detection. Use `--no-switch` to avoid changing the active profile.
- **Status and teardown** — `ilum status` shows release info, pod readiness, enabled modules, and Kubernetes events. `--wait` blocks until all pods are ready (CI-friendly). `ilum uninstall` handles clean teardown with optional PVC/namespace deletion. `ilum cleanup` provides tiered full-environment teardown.
- **Values inspection** — `ilum values` views live Helm values with dot-path filtering, revision pinning, and YAML export. `ilum diff` compares values across sources (defaults, files, revisions, snapshots).
- **Rollback** — `ilum rollback` explicitly rolls back to a previous revision with values diff preview and `--dry-run` support.
- **Shell access** — `ilum exec core` opens an interactive shell in a module pod with auto-detection, multi-pod selection, and shell fallback.
- **Resource monitoring** — `ilum top` shows per-module CPU/memory usage with percentage utilization, sorting, and `--watch` mode.
- **Dependency management** — `ilum deps list` shows tool status; `ilum deps install` installs missing prerequisites with platform-aware methods.

---

## Quick Start

### Prerequisites

| Tool | Minimum Version |
|------|-----------------|
| Python | 3.12+ |
| Helm | 3.12+ |
| kubectl | 1.28+ |
| Docker | 24.0+ |

### Install

```bash
# One-line installer (macOS: brew → pipx → uv → binary; Linux: pipx → uv → binary)
curl -fsSL https://get.ilum.cloud/cli | bash

# Or install via Homebrew (macOS)
brew tap ilum-cloud/tap https://github.com/ilum-cloud/ilum
brew install ilum

# Or install via pip / pipx
pip install ilum
pipx install ilum

# Editable install (development)
pip install -e ".[dev]"
```

**Windows (PowerShell):**

```powershell
# One-line installer (winget → pip → binary)
irm https://get.ilum.cloud/cli/windows | iex

# Or install via pip
pip install ilum
```

### First run

```bash
# One command — detects/creates a cluster and installs Ilum with defaults
ilum quickstart

# Or step-by-step: interactive setup wizard first, then install
ilum init
ilum install

# Enable an optional module
ilum module enable sql

# Check release status
ilum status
```

---

## CLI Reference

### Global Options

| Flag | Short | Description |
|------|-------|-------------|
| `--output` | `-o` | Output format: `table`, `json`, `yaml`, `csv` (default: `table`) |
| `--quiet` | `-q` | Suppress informational output (errors still shown) |
| `--verbose` | `-v` | Enable verbose (debug) output |
| `--no-headers` | | Suppress table/CSV headers (useful for scripting) |
| `--field-selector` | | Comma-separated list of fields to include in output |
| `--version` | `-V` | Print version and exit |

### `ilum init`

Interactive setup wizard that walks through prerequisites, cluster selection, and module configuration.

```bash
ilum init                            # Interactive wizard
ilum init --yes                      # Non-interactive with all defaults
ilum init --profile staging          # Create a named profile
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--yes` | `-y` | `false` | Non-interactive mode with defaults |
| `--profile` | `-p` | `default` | Profile name to create |

**Wizard steps:**

1. **Preflight** — checks for helm, kubectl, docker; offers to install missing tools (Homebrew on macOS, winget on Windows, official scripts on Linux). Cluster providers (minikube, k3d, kind) are also auto-installed when needed.
2. **Cluster** — lists existing kubecontexts or creates a new local cluster (k3d/minikube/kind)
3. **Profile** — prompts for profile name, Helm release name, and namespace
4. **Modules** — checkbox selection grouped by category, with defaults pre-checked
5. **Confirm** — summary table for review
6. **Save** — writes to `~/.config/ilum/config.yaml`

### `ilum quickstart`

Install Ilum in one command with sensible defaults. Detects an existing Kubernetes cluster or creates a local one (minikube by default), then installs with all default modules enabled.

```bash
ilum quickstart                            # Auto-detect cluster, install with defaults
ilum quickstart --provider k3d             # Use k3d instead of minikube
ilum quickstart --preset production        # Use a deployment preset
ilum quickstart -m sql -m airflow          # Enable additional modules
ilum quickstart --profile staging          # Name the profile
ilum quickstart --timeout 20m             # Custom Helm timeout
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--provider` | | `minikube` | Local cluster provider if a new cluster is needed (`minikube`, `k3d`, `kind`) |
| `--preset` | | | Deployment preset (`development`, `production`, `data-engineering`, `air-gapped`) |
| `--profile` | `-p` | `default` | Profile name to create |
| `--module` | `-m` | | Additional module to enable (repeatable) |
| `--timeout` | | `15m` | Helm install timeout |

**Steps performed:**

1. **Preflight** — checks for helm, kubectl, docker; offers to install missing tools
2. **Cluster** — probes the current kubecontext; if no cluster is reachable, creates one with the `dev` preset (4 CPUs, 8 GB)
3. **Profile** — creates a profile with default modules + any extras from `-m`
4. **Install** — runs `helm install` with `--atomic`, auto-resolves NodePort conflicts

### `ilum cluster`

Manage local Kubernetes clusters. Three subcommands:

```bash
# Create a local cluster
ilum cluster create                         # k3d with dev preset (6 CPUs, 12 GB)
ilum cluster create --provider minikube     # Use minikube instead
ilum cluster create --preset full           # Full preset (8 CPUs, 18 GB)
ilum cluster create --name my-cluster       # Custom cluster name

# List tracked clusters
ilum cluster list

# Delete a cluster
ilum cluster delete ilum-dev
```

**`create` options:**

| Flag | Default | Description |
|------|---------|-------------|
| `--provider` | `k3d` | Cluster provider: `minikube`, `k3d`, `kind` |
| `--preset` | `dev` | Resource preset: `dev` (4 CPUs, 8 GB), `full` (6 CPUs, 12 GB) |
| `--name` | *(from preset)* | Custom cluster name |

**Presets:**

| Preset | CPUs | Memory | Default Name |
|--------|------|--------|-------------|
| `dev` | 6 | 12 GB | `ilum-dev` |
| `full` | 8 | 18 GB | `ilum` |

Clusters are tracked in the config file and their kubecontext is available for profile configuration.

### `ilum preset`

Manage deployment presets — curated module bundles for common deployment scenarios.

```bash
ilum preset list                            # List all available presets
```

**Available presets:**

| Preset | Modules | Extra `--set` flags |
|--------|---------|---------------------|
| `development` | core, ui, mongodb, postgresql, minio, jupyter, gitea, openldap (8) | — |
| `production` | + monitoring, loki (10) | `global.security.enabled=true`, `ilum-core.replicaCount=2` |
| `data-engineering` | + sql, airflow, superset, trino, hive-metastore (13) | — |
| `air-gapped` | core, ui, mongodb, postgresql, minio, jupyter, gitea (7) | `global.imageRegistry=<user-provided>` |

Presets can be used with `ilum install --preset`, `ilum quickstart --preset`, and in the `ilum init` wizard.

### `ilum airgap`

Air-gapped deployment tooling for disconnected environments. Three subcommands:

```bash
# List container images required by selected modules
ilum airgap images                          # Default modules
ilum airgap images --preset production      # Production preset
ilum airgap images --format plain           # One image per line (for scripting)

# Export images and chart into a portable bundle
ilum airgap export /path/to/bundle          # Default modules
ilum airgap export /tmp/bundle --preset development  # Preset modules

# Import a bundle into a private container registry
ilum airgap import /path/to/bundle --registry registry.local:5000
```

**`images` options:**

| Flag | Default | Description |
|------|---------|-------------|
| `--preset` | | Deployment preset |
| `--module` / `-m` | | Additional module (repeatable) |
| `--version` | *(latest)* | Chart version |
| `--format` | `table` | Output format: `table` or `plain` |

**`export` options:** Same as `images` plus a required output directory argument.

**`import` options:**

| Flag | Default | Description |
|------|---------|-------------|
| `--registry` | *(required)* | Target private registry URL |

**Workflow:**
1. `ilum airgap export /tmp/bundle --preset air-gapped` — pulls images and packages chart
2. Transfer `/tmp/bundle` to the air-gapped environment
3. `ilum airgap import /tmp/bundle --registry registry.local:5000` — loads, retags, and pushes all images
4. `ilum install --preset air-gapped --set global.imageRegistry=registry.local:5000` — installs using the private registry

### `ilum install`

Install the Ilum platform via Helm.

```bash
ilum install                                # Install with default modules
ilum install --preset production            # Install with a deployment preset
ilum install --module sql --module airflow   # Install with specific modules
ilum install --version 6.7.0                # Pin chart version
ilum install -f custom-values.yaml          # Use a custom values file
ilum install --set global.security.enabled=true  # Override a single value
ilum install --dry-run                      # Preview without executing
ilum install --yes                          # Skip confirmation prompt
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--version` | | *(latest)* | Chart version to install |
| `--values` | `-f` | | Values file (repeatable) |
| `--set` | | | Helm `--set` flag (repeatable) |
| `--module` | `-m` | | Module to enable (repeatable) |
| `--preset` | | | Deployment preset (`development`, `production`, `data-engineering`, `air-gapped`) |
| `--atomic` | | `true` | Use `--atomic` for Helm install |
| `--yes` | `-y` | `false` | Skip confirmation prompt |
| `--dry-run` | | `false` | Preview without executing |
| `--timeout` | | `10m` | Helm timeout |

**Module priority:** `--module` flags > profile `enabled_modules` > default-enabled modules (7).

The command displays a summary table and the exact Helm command before execution, resolves module dependencies automatically, detects NodePort conflicts and offers reassignment, and saves a values snapshot on success.

### `ilum upgrade`

Upgrade an existing Ilum installation.

```bash
ilum upgrade                                # Upgrade with current values
ilum upgrade --version 6.8.0               # Upgrade to specific version
ilum upgrade --module langfuse             # Enable a module during upgrade
ilum upgrade --set ilum-core.replicaCount=3  # Override a value
ilum upgrade --force-rollback              # Recover a stuck release first
ilum upgrade --reset-defaults              # Use new chart defaults (for breaking upgrades)
ilum upgrade --dry-run                     # Preview values diff
```

**Options:** Same as `ilum install` plus:

| Flag | Default | Description |
|------|---------|-------------|
| `--force-rollback` | `false` | Rollback a stuck release before upgrading |
| `--reset-defaults` | `false` | Reset to new chart defaults before applying user values. Use when upgrading across chart versions with breaking changes (uses Helm's `--reset-then-reuse-values`). |

**Values safety pipeline:**

1. Fetches live values from the running release
2. Detects drift (external changes since last CLI operation)
3. Computes a values diff (current vs. desired)
4. Shows the diff as a key-path table for review
5. Warns about known breaking changes between versions (tuple-based semver comparison)
6. Shows the exact Helm command that will be executed
7. Executes only after confirmation
8. Saves a values snapshot for future drift detection

When upgrading to a new minor version (even without explicit `--version`), the CLI checks an embedded compatibility matrix and warns about breaking changes. An informational notice links to the upgrade notes for any minor-version jump. If the upgrade fails due to values schema conflicts, the CLI suggests retrying with `--reset-defaults`.

### `ilum logs`

Stream or tail logs from Ilum modules by name.

```bash
ilum logs core                          # Tail last 100 lines from ilum-core
ilum logs jupyter --tail 500            # Tail last 500 lines
ilum logs core --follow                 # Stream logs in real time (Ctrl+C to stop)
ilum logs core --container spark        # Target a specific container
ilum logs core --previous               # Logs from the previous container instance
ilum logs core --tail 0                 # Show all available logs
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--follow` | `-f` | `false` | Stream logs in real time |
| `--tail` | | `100` | Number of lines to show (0 = all) |
| `--container` | `-c` | | Target a specific container |
| `--previous` | `-p` | `false` | Show logs from previous container instance |
| `--max-duration` | | `0` | Stop `--follow` after N seconds (0 = unlimited) |

**Module resolution:** The module name (e.g., `core`, `jupyter`, `sql`) is resolved to a Kubernetes pod label via the module registry. When multiple pods match, the first pod is selected with an informational message. Invalid container names are validated against the pod spec before fetching logs.

**Piping:** Output uses `typer.echo()` (not Rich) so logs can be piped to `grep`, `jq`, or files.

### `ilum module`

Manage Ilum platform modules. Six subcommands:

```bash
# Enable modules (resolves dependencies automatically)
ilum module enable sql
ilum module enable langfuse airflow

# Disable modules (shared-dep aware)
ilum module disable sql
ilum module disable langfuse --yes

# List all available modules
ilum module list
ilum module list --category sql
ilum module list --enabled
ilum module list --search kafka

# Show detailed module info
ilum module show sql
ilum module show langfuse

# Show dependency tree with resource estimates
ilum module tree sql
ilum module tree langfuse

# Show live module status on a running release
ilum module status
ilum module status --category orchestration
```

**`enable` options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--set` | | | Additional Helm `--set` flag (repeatable) |
| `--reset-defaults` | | `false` | Reset to new chart defaults before applying user values |
| `--yes` | `-y` | `false` | Skip confirmation prompt |
| `--dry-run` | | `false` | Preview without executing |
| `--timeout` | | `10m` | Helm timeout |

**`disable` options:** Same as `enable`. Warns when disabling default modules.

**Dry-run behavior:** Shows the values diff and the exact Helm command, then exits without applying changes. Note that "no values changes needed" takes priority over dry-run — if nothing would change, the command exits early regardless.

**`list` options:**

| Flag | Short | Description |
|------|-------|-------------|
| `--category` | `-c` | Filter by category |
| `--enabled` / `--disabled` | | Filter by default-enabled status |
| `--search` | `-s` | Search modules by name or description |

**`show`** takes a single module name and displays enable/disable flags, dependencies, conflicts, and the resolved enable chain.

**`tree`** takes a single module name and displays its full dependency tree with resource estimates (CPU and memory requests).

**`status`** shows live module status on a running release — enabled/disabled state, pod readiness, and health. Detects config drift between CLI config and the actual cluster state.

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--category` | `-c` | | Filter by module category |

### `ilum status`

Show release info, pod readiness, enabled modules, and Kubernetes events.

```bash
ilum status                           # Default release in default namespace
ilum status --release my-ilum         # Named release
ilum status --no-pods                 # Skip pod table
ilum status --no-modules              # Skip modules table
ilum status --wait                    # Block until all pods are ready (CI-friendly)
ilum status --wait --wait-timeout 600 # Custom timeout (default: 300s)
ilum status --events                  # Show recent Kubernetes events
ilum status --events --events-type Warning  # Filter to warning events only
ilum status --events --events-since 1h      # Events from the last hour
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--pods` / `--no-pods` | | `true` | Show pod readiness table |
| `--modules` / `--no-modules` | | `true` | Show enabled modules table |
| `--wait` | | `false` | Wait for all pods to be ready before exiting |
| `--wait-timeout` | | `300` | Seconds to wait for pods (with `--wait`) |
| `--events` | | `false` | Show recent Kubernetes events |
| `--events-type` | | | Filter events by type (e.g. `Warning`) |
| `--events-since` | | | Filter events by age (e.g. `1h`, `30m`, `2h30m`) |

**`--wait` behavior:** Polls pod readiness every 5 seconds with a progress spinner showing `Pods: X/Y ready`. Excludes completed and failed pods. Exit 0 = all pods ready, exit 1 = timeout. Useful in CI/CD pipelines to gate subsequent steps on cluster readiness.

**Config drift detection:** When modules are shown, the status command compares the live cluster state against the CLI config and warns about any drift (modules enabled on cluster but not tracked, or vice versa). On first run against an existing cluster, live modules are auto-synced to the CLI config.

### `ilum values`

View or export live Helm values from a running release.

```bash
ilum values                              # Show user-supplied values (YAML panel)
ilum values --all                        # Include chart defaults
ilum values --path ilum-core.replicaCount  # Filter to a specific key path
ilum values --revision 3                 # Values at a specific revision
ilum values --diff                       # Diff user-supplied vs computed values
ilum values --export values.yaml         # Export to a YAML file
ilum values --output json                # Machine-readable output
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--path` | | | Dot-notation path to filter values (e.g. `ilum-core.replicaCount`) |
| `--all` | `-a` | `false` | Include chart defaults (not just user-supplied) |
| `--diff` | | `false` | Show diff between user-supplied and computed values |
| `--revision` | | | Retrieve values at a specific revision |
| `--export` | `-e` | | Export values to a YAML file |

### `ilum diff`

Compare values across different sources to understand what changed and when.

```bash
ilum diff                                 # Default: user-supplied vs computed values
ilum diff --source file -f custom.yaml   # Compare local file vs live values
ilum diff --source revision --revision 2 # Compare revision 2 vs current
ilum diff --source snapshot              # Compare last CLI snapshot vs live values
ilum diff --path ilum-core               # Filter diff to a specific key path
ilum diff --output json                  # Machine-readable output
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--source` | `-s` | `defaults` | Comparison source: `defaults`, `file`, `revision`, `snapshot` |
| `--values-file` | `-f` | | Local YAML file to compare (with `--source file`) |
| `--revision` | | | Revision number to compare (with `--source revision`) |
| `--path` | | | Dot-notation path to filter diff |

**Sources:**
- `defaults` — user-supplied values vs computed values (chart defaults merged in)
- `file` — local YAML file vs live values on the cluster
- `revision` — a specific revision's values vs the current revision
- `snapshot` — last CLI snapshot vs live values (detects external drift)

### `ilum rollback`

Explicitly roll back to a previous Helm revision.

```bash
ilum rollback                             # Roll back to the previous revision
ilum rollback --revision 3               # Roll back to revision 3
ilum rollback --dry-run                  # Preview without executing
ilum rollback --yes                      # Skip confirmation prompt
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--revision` | | `0` | Target revision (0 = previous) |
| `--dry-run` | | `false` | Show rollback plan without executing |
| `--yes` | `-y` | `false` | Skip confirmation prompt |
| `--timeout` | | `10m` | Helm timeout for rollback operation |

**Rollback workflow:**
1. Fetches release history and validates the target revision exists
2. Shows a comparison table (current vs target revision, chart version, status, deploy time)
3. Computes and displays a values diff between the two revisions
4. Confirms with the user (unless `--yes`)
5. Executes `helm rollback`
6. Post-rollback: saves a new values snapshot and updates module config

### `ilum exec`

Open an interactive shell in a module pod.

```bash
ilum exec core                           # Shell into the ilum-core pod
ilum exec jupyter                        # Shell into the jupyter pod
ilum exec core --shell /bin/sh           # Use sh instead of bash
ilum exec core --container spark         # Target a specific container
ilum exec core --pod ilum-core-0         # Skip auto-detection, use specific pod
ilum exec core --command "ls -la /data"  # Run a single command (non-interactive)
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--shell` | `-s` | `/bin/bash` | Shell to use (falls back to `/bin/sh`) |
| `--container` | `-c` | | Target a specific container |
| `--pod` | `-p` | | Pod name (skip auto-detection) |
| `--command` | | | Command to execute instead of a shell |

**Pod selection:** The module name is resolved to a pod label via the module registry. When multiple pods match and a TTY is available, an interactive selector is shown (requires the `questionary` package). Without a TTY, use `--pod` to specify explicitly. A health warning is shown for pods with more than 5 restarts.

### `ilum top`

Show resource usage per Ilum module (requires metrics-server).

```bash
ilum top                                 # All modules
ilum top core                            # Single module
ilum top --sort-by cpu                   # Sort by CPU usage
ilum top --sort-by memory               # Sort by memory usage
ilum top --watch                         # Continuously refresh
ilum top --watch --interval 10          # Custom refresh interval
ilum top --output json                  # Machine-readable output
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--sort-by` | | `name` | Sort by: `name`, `cpu`, `memory` |
| `--watch` | `-w` | `false` | Continuously refresh |
| `--interval` | `-i` | `5` | Refresh interval in seconds (with `--watch`) |

**Output table:**

```
 Resource Usage
┌──────────┬──────┬──────────┬─────────┬───────┬──────────┬─────────┬───────┐
│ Module   │ Pods │ CPU Used │ CPU Req │ CPU % │ Mem Used │ Mem Req │ Mem % │
├──────────┼──────┼──────────┼─────────┼───────┼──────────┼─────────┼───────┤
│ core     │ 1    │ 250m     │ 500m    │ 50%   │ 512Mi    │ 1.0Gi   │ 50%   │
│ mongodb  │ 1    │ 100m     │ 250m    │ 40%   │ 256Mi    │ 512Mi   │ 50%   │
│ TOTAL    │ 2    │ 350m     │ 750m    │ 47%   │ 768Mi    │ 1.5Gi   │ 50%   │
└──────────┴──────┴──────────┴─────────┴───────┴──────────┴─────────┴───────┘
```

**Prerequisites:** Requires [metrics-server](https://github.com/kubernetes-sigs/metrics-server) installed in the cluster. If not available, the command provides an installation hint.

### `ilum uninstall`

Uninstall Ilum and optionally clean up data.

```bash
ilum uninstall                        # Uninstall release (idempotent)
ilum uninstall --yes                  # Skip confirmation
ilum uninstall --delete-data          # Also delete PVCs (requires typed name)
ilum uninstall --delete-namespace     # Also delete the namespace
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--yes` | `-y` | `false` | Skip confirmation prompt |
| `--delete-data` | | `false` | Delete PVCs after uninstall |
| `--delete-namespace` | | `false` | Delete namespace after uninstall |
| `--dry-run` | | `false` | Preview without executing |

**Safety:** `--delete-data` and `--delete-namespace` require typing the release name to confirm. Uninstall is idempotent (exits 0 if release not found).

**Dry-run:** Shows the uninstall summary and the exact Helm command, then exits. PVC deletion and namespace deletion are skipped in dry-run mode.

### `ilum cleanup`

Full environment teardown with tiered, opt-in destructiveness. Combines uninstall, cluster deletion, config removal, dependency removal, and CLI self-uninstall into a single command.

```bash
ilum cleanup --dry-run                   # Preview what would be removed
ilum cleanup --yes                       # Uninstall release only (Tier 1)
ilum cleanup --all --yes                 # Release + cluster + config + deps
ilum cleanup --all --self --yes          # Full teardown including CLI itself
ilum cleanup --cluster --yes             # Release + cluster only
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--cluster` | | `false` | Also delete CLI-managed local k8s cluster |
| `--config` | | `false` | Also remove CLI config/state/cache directories |
| `--deps` | | `false` | Also remove dependencies (helm, kubectl, minikube, k3d, kind, docker) |
| `--self` | | `false` | Also uninstall the CLI itself (always last, never included in `--all`) |
| `--all` | | `false` | Shorthand for `--cluster --config --deps` |
| `--release` | `-r` | `ilum` | Helm release name |
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--yes` | `-y` | `false` | Skip all confirmations |
| `--dry-run` | | `false` | Preview without executing |

**Tiers (executed in order):**

1. **Tier 1** (always): Uninstall Helm release + delete PVCs + delete namespace
2. **Tier 2** (`--cluster`): Delete CLI-managed local cluster
3. **Tier 3** (`--config`): Remove CLI directories and Helm repo
4. **Tier 4** (`--deps`): Remove dependencies
5. **Tier 5** (`--self`): Uninstall the CLI itself

Each tier is independent — if one fails, the next still runs. Non-`--yes` mode requires typed confirmation at each destructive tier.

### `ilum doctor`

Run health checks on the Ilum installation.

```bash
ilum doctor                          # Run all 13 checks
ilum doctor --namespace prod         # Target a specific namespace
ilum doctor --context staging-ctx    # Use a specific kubeconfig context
ilum doctor --release my-ilum        # Check a named Helm release
ilum doctor --check helm             # Run a single check by name
ilum doctor --failures-only          # Show only failures and warnings
ilum doctor --fix                    # Auto-fix simple issues
ilum doctor --output json            # Machine-readable output
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--namespace` | `-n` | `default` | Kubernetes namespace |
| `--context` | | *(current)* | Kubernetes context |
| `--release` | `-r` | `ilum` | Helm release name |
| `--check` | `-c` | *(all)* | Run a single check by name |
| `--failures-only` | | `false` | Only show failed and warning checks |
| `--fix` | | `false` | Auto-fix simple issues (e.g., add missing Helm repo) |

**Available checks:**

| Check | Phase | What it verifies |
|-------|-------|-----------------|
| `helm` | Binary | Helm >= 3.12 on PATH |
| `kubectl` | Binary | kubectl >= 1.28 on PATH |
| `docker` | Binary | Docker >= 24.0 on PATH |
| `helm-repo` | Repo | ilum Helm repo configured |
| `cluster` | Cluster | Kubernetes API connectivity |
| `namespace` | Cluster | Target namespace exists |
| `pods` | Cluster | Pod health (detects CrashLoopBackOff) |
| `pvcs` | Cluster | PVC binding status (detects Pending) |
| `rbac` | Cluster | Required RBAC permissions |
| `release` | Cluster | Helm release state (detects stuck/failed) |
| `compatibility` | Cluster | Kubernetes >= 1.28 |
| `service-endpoints` | Cluster | Service endpoints exist and have ready addresses |
| `health-endpoints` | Cluster | Health check endpoints respond (core, airflow, superset, minio) |

**Exit codes:** `0` all checks pass, `1` any check fails.

Cluster-dependent checks are automatically **skipped** when the cluster is unreachable, so binary checks always report results.

### `ilum config`

Manage CLI configuration. Subcommands:

```bash
# Show full config as YAML
ilum config show

# Show a specific key
ilum config show active_profile

# Set a value using dot-notation
ilum config set active_profile staging
ilum config set profiles.default.cluster.namespace production

# Print config file path
ilum config path

# Create default config (--force to overwrite)
ilum config init
ilum config init --force

# Open config in $EDITOR (falls back to vi on Unix, notepad on Windows)
ilum config edit

# Switch active profile
ilum config use staging

# List all profiles
ilum config list-profiles

# Validate configuration
ilum config validate

# Backup configuration
ilum config backup

# Export a profile for team sharing
ilum config export production --out prod-profile.yaml

# Import a shared profile
ilum config import prod-profile.yaml --name production
```

### `ilum history`

View the operation audit log.

```bash
ilum history                         # Show recent operations
ilum history --last 5                # Last 5 operations
ilum history --operation upgrade     # Filter by operation type
ilum history --output json           # Machine-readable output
```

**Options:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--last` | `-n` | `20` | Number of recent operations to show |
| `--operation` | | | Filter by operation type |
| `--release` | `-r` | `ilum` | Filter by Helm release name |

### `ilum deps`

Manage CLI dependencies (helm, kubectl, docker, k3d, etc.). Two subcommands:

```bash
# List dependencies and their status
ilum deps list

# Install all missing core tools + default provider
ilum deps install

# Install specific tools
ilum deps install helm kubectl

# Install with a specific cluster provider
ilum deps install --provider kind

# Preview without executing
ilum deps install --dry-run
```

**`install` options:**

| Flag | Default | Description |
|------|---------|-------------|
| `--provider` | `k3d` | Cluster provider to install: `k3d`, `minikube`, `kind` |
| `--dry-run` | `false` | Preview what would be installed without executing |

When no tool names are specified, `ilum deps install` installs all core tools (helm, kubectl, docker) plus the selected cluster provider. Tool names can be specified as positional arguments to install only specific tools.

---

## Architecture

### Directory Layout

```
src/ilum/
├── __init__.py                 # Package root
├── __main__.py                 # python -m ilum entry point
├── py.typed                    # PEP 561 type marker
├── compat.py                   # Cross-platform helpers (file locking)
├── constants.py                # Shared constants (versions, URLs, timeouts)
├── errors.py                   # Typed exception hierarchy
│
├── cli/                        # Command layer (Typer)
│   ├── main.py                 # Root app, --version, --verbose callback
│   ├── init_cmd.py             # ilum init
│   ├── cluster_cmd.py          # ilum cluster {create,delete,list}
│   ├── install_cmd.py          # ilum install
│   ├── upgrade_cmd.py          # ilum upgrade
│   ├── module_cmd.py           # ilum module {enable,disable,list,show}
│   ├── status_cmd.py           # ilum status (--wait, --events)
│   ├── values_cmd.py           # ilum values
│   ├── diff_cmd.py             # ilum diff
│   ├── rollback_cmd.py         # ilum rollback
│   ├── exec_cmd.py             # ilum exec
│   ├── top_cmd.py              # ilum top
│   ├── uninstall_cmd.py        # ilum uninstall
│   ├── config_cmd.py           # ilum config {show,set,path,init,edit}
│   ├── doctor_cmd.py           # ilum doctor
│   ├── logs_cmd.py             # ilum logs
│   ├── quickstart_cmd.py       # ilum quickstart
│   ├── preset_cmd.py           # ilum preset list
│   ├── airgap_cmd.py           # ilum airgap {images,export,import}
│   ├── cleanup_cmd.py          # ilum cleanup
│   ├── connect_cmd.py          # ilum connect
│   ├── deps_cmd.py             # ilum deps {install,list}
│   ├── defaults.py             # Profile-based default resolution
│   ├── formatters.py           # OutputFormat, CommandResult, CSV/JSON/YAML formatters
│   └── output.py               # IlumConsole — Rich wrapper
│
├── config/                     # Configuration management
│   ├── paths.py                # IlumPaths — cross-platform directory resolution (XDG / %APPDATA%)
│   ├── models.py               # Pydantic v2 models (IlumConfig, ProfileConfig, ClusterConfig, ClusterRecord)
│   └── manager.py              # ConfigManager — YAML persistence with file locking
│
├── core/                       # Platform abstractions
│   ├── helm.py                 # HelmClient — subprocess wrapper with JSON parsing
│   ├── kubernetes.py           # KubeClient — kubernetes-client wrapper
│   ├── modules.py              # Module registry (32 modules) and dependency resolver
│   ├── values.py               # Values file merging (ruamel.yaml round-trip)
│   ├── safety.py               # Values snapshots, drift detection, diff computation
│   ├── release.py              # ReleaseManager — orchestration facade
│   ├── versioning.py           # Version parsing and comparison
│   ├── presets.py              # DeploymentPreset registry (4 presets)
│   └── images.py               # Image discovery from Helm templates
│
├── api/                        # REST API (FastAPI sidecar)
│   ├── app.py                  # Application factory with lifespan
│   ├── auth.py                 # Dual auth (Nginx cookie + API key)
│   ├── chart_cache.py          # Cached helm show values
│   ├── deps.py                 # Singleton ReleaseManager injection
│   ├── errors.py               # IlumError → HTTP status mapping
│   ├── job_runner.py           # K8s Job builder for helm operations
│   ├── models.py               # Pydantic v2 request/response schemas
│   ├── operations.py           # In-memory operation store (FIFO, max 100)
│   ├── run.py                  # ilum-api console script entry point
│   ├── startup.py              # Auto-connect and recovery on boot
│   └── routers/
│       ├── modules.py          # Module CRUD, enable/disable, config, restart, logs
│       ├── operations.py       # Operation tracking and cancellation
│       ├── release.py          # Stuck release recovery
│       ├── status.py           # Health, release info, pods
│       └── values.py           # Values read/diff/update
│
├── wizard/                     # Interactive setup wizard
│   ├── deps.py                 # ToolInstaller — platform-aware detection + auto-install
│   ├── cluster.py              # ClusterManager — local cluster create/delete/list
│   └── flow.py                 # InitWizard — 6-step orchestrator
│
└── doctor/                     # Health check system
    ├── checks.py               # 13 individual check functions + CheckResult/CheckStatus
    └── runner.py               # DoctorRunner — orchestrator with phased execution
```

### Values Safety System

Every upgrade, enable, and disable operation follows a **fetch-merge-diff-apply** pipeline:

```
1. Fetch live values    →  helm get values <release> --output json
2. Detect drift         →  compare live values against last CLI snapshot
3. Compute merge        →  deep_merge(live_values, cli_overlay) + apply_set_flags
4. Show diff            →  key-path table of what changes
5. Confirm              →  user approves or aborts
6. Execute              →  helm upgrade with computed values
7. Save snapshot        →  persist applied values to state dir
```

**Why not `--reuse-values`:** It's a black box — the CLI can't preview the merge. During chart version upgrades, `--reuse-values` suppresses new chart defaults. Fetch-merge-diff-apply gives full control and enables pre-execution diffs.

**Drift detection:** Before any modification, the CLI compares the live release values against the last snapshot saved in `~/.local/state/ilum/snapshots/`. If someone ran `helm upgrade --set custom.value=foo` outside the CLI, the drift is shown as a warning. External changes are always preserved — CLI changes are applied on top.

**Diff display:**

```
 Values Changes
┌──────────┬──────────────────────────────┬───────────┬───────────┐
│ Action   │ Key                          │ Current   │ New       │
├──────────┼──────────────────────────────┼───────────┼───────────┤
│ + added  │ ilum-sql.enabled             │ —         │ true      │
│ ~ changed│ postgresql.enabled           │ false     │ true      │
└──────────┴──────────────────────────────┴───────────┴───────────┘
```

### Orchestration Layer

`ReleaseManager` sits between CLI commands and low-level clients:

```
CLI commands → ReleaseManager → HelmClient / KubeClient / ModuleResolver / ConfigManager / safety.py
```

Key abstractions:
- **`ReleasePlan`** — describes a planned operation before execution (action, release, chart, namespace, set_flags, values_files, modules, warnings, computed_values, effective_diff, drift)
- **`ReleaseInfo`** — summary of an existing release (name, status, chart_version, revision, last_deployed)
- **`ReleaseManager`** — facade with plan builders (`plan_install`, `plan_upgrade`, `plan_enable`, `plan_disable`), `execute()`, `rollback_if_stuck()`, and snapshot management

### Key Design Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| CLI framework | Typer (built on Click) | Type-hint-driven commands, auto help/completion, `CliRunner` for testing |
| Data validation | Pydantic v2 | `BaseModel` with defaults, serialization, strict typing |
| Module definitions | `@dataclass(frozen=True)` | Immutable, lightweight, no validation overhead for static registry |
| Helm integration | Subprocess with `--output json` | Full Helm feature parity, no version coupling, show exact commands |
| Values safety | Fetch-merge-diff-apply | Full control over merge, enables diffs, handles version upgrades properly |
| Drift detection | Snapshot-based (state dir) | Cannot detect external changes without baseline; snapshot is small YAML |
| Drift behavior | Warn + merge (preserve external changes) | Safest non-destructive default |
| Flag ordering | Module flags first, user `--set` last | User flags override module defaults |
| YAML library | ruamel.yaml | Round-trip preservation of comments and ordering |
| Terminal output | Rich | Tables, panels, colored status markers, stderr separation |
| Config paths | XDG (Linux/macOS), %APPDATA% (Windows) | Consistent with Helm, kubectl, and OS conventions |
| Stuck-release handling | Explicit `--force-rollback` opt-in | Prevents accidental rollback |
| Uninstall idempotency | Exit 0 if release not found | Safe for scripts |

---

## Module System

The CLI encodes every Ilum platform module as a `ModuleDefinition` — a frozen dataclass capturing:

```python
@dataclass(frozen=True)
class ModuleDefinition:
    name: str                          # e.g. "sql", "jupyter"
    description: str                   # One-line summary
    category: ModuleCategory           # Logical grouping
    enable_flags: tuple[str, ...]      # Helm --set flags to enable
    disable_flags: tuple[str, ...]     # Helm --set flags to disable
    requires: tuple[str, ...] = ()     # Dependency modules
    conflicts_with: tuple[str, ...] = ()  # Mutually exclusive modules
    chart_condition: str = ""          # helm_aio/Chart.yaml condition
    values_key: str = ""               # Top-level values.yaml key
    pod_label: str = ""                # Label selector for pod queries
    default_enabled: bool = False      # Enabled in default installation
    required: bool = False             # Mandatory module (cannot be deselected)
    crd_chart: str = ""                # Helm chart providing required CRDs
```

### Categories (10)

| Category | Count | Modules |
|----------|-------|---------|
| Core | 4 | `core`, `ui`, `livy-proxy`, `api` |
| Infrastructure | 5 | `mongodb`, `kafka`, `postgresql`, `gitea`, `clickhouse` |
| Storage | 1 | `minio` |
| Notebook | 3 | `jupyter`, `jupyterhub`, `zeppelin` |
| Orchestration | 5 | `airflow`, `kestra`, `n8n`, `nifi`, `mageai` |
| SQL | 5 | `sql`, `trino`, `hive-metastore`, `nessie`, `unity-catalog` |
| Analytics | 3 | `superset`, `streamlit`, `marquez` |
| Monitoring | 3 | `monitoring`, `loki`, `graphite` |
| AI | 2 | `mlflow`, `langfuse` |
| Security | 1 | `openldap` |

**Total: 32 modules**

### Default-Enabled Modules (7)

These modules are enabled in a standard Ilum installation:

| Module | Description |
|--------|-------------|
| `core` | Ilum backend API (Spring Boot on Spark) |
| `ui` | Ilum web frontend (React + Nginx reverse proxy) |
| `mongodb` | MongoDB document store for metadata |
| `minio` | MinIO S3-compatible object storage |
| `postgresql` | PostgreSQL relational database |
| `jupyter` | Jupyter notebook server with Sparkmagic |
| `gitea` | Gitea self-hosted Git service |

### Dependency Resolution

The `ModuleResolver` walks the `requires` graph in **topological order** (depth-first), collecting enable flags from dependencies before the module itself. A visited set prevents infinite loops.

**Enable example — `sql`:**
```
sql requires → postgresql, core
Result flags: postgresql.enabled=true, ilum-core.enabled=true,
              ilum-sql.enabled=true, ilum-core.sql.enabled=true
```

**Disable with shared-dep awareness:**
When disabling a module, the resolver checks which dependencies are still required by other active modules. Shared dependencies are **not** disabled.

**Conflict detection:**
`hive-metastore` and `nessie` declare a mutual conflict — enabling both produces a validation error because they both configure `ilum-core.metastore.type` with different values.

**Multi-flag modules:**
The `sql` module requires *two* enable flags (`ilum-sql.enabled=true` + `ilum-core.sql.enabled=true`). `hive-metastore` requires *three* (`ilum-hive-metastore.enabled=true` + `ilum-core.metastore.enabled=true` + `ilum-core.metastore.type=hive`). The registry captures these as tuples, eliminating the most common misconfiguration pattern.

**Transitive dependencies — `langfuse`:**
```
langfuse requires → postgresql, clickhouse
Enabling langfuse automatically enables both infrastructure dependencies.
```

---

## Configuration

### Config Paths

The CLI uses platform-native directories:

**Linux / macOS (XDG Base Directory):**

| Directory | Default | Env Override |
|-----------|---------|-------------|
| Config | `~/.config/ilum/` | `XDG_CONFIG_HOME` |
| Data | `~/.local/share/ilum/` | `XDG_DATA_HOME` |
| State | `~/.local/state/ilum/` | `XDG_STATE_HOME` |
| Cache | `~/.cache/ilum/` | `XDG_CACHE_HOME` |

**Windows:**

| Directory | Default | Env Override |
|-----------|---------|-------------|
| Config | `%APPDATA%\ilum\` | `APPDATA` |
| Data | `%APPDATA%\ilum\data\` | `APPDATA` |
| State | `%LOCALAPPDATA%\ilum\state\` | `LOCALAPPDATA` |
| Cache | `%LOCALAPPDATA%\ilum\cache\` | `LOCALAPPDATA` |

The configuration file lives at `<config_dir>/config.yaml`. Run `ilum config path` to see the exact location.

Values snapshots are stored in `<state_dir>/snapshots/`.

### Config File Structure

```yaml
active_profile: default
profiles:
  default:
    name: default
    cluster:
      kubecontext: ""
      namespace: default
      chart_version: ""
      chart_repo: https://charts.ilum.cloud
    enabled_modules: []
    communication_type: grpc
    auth_type: internal
    custom_values_path: ""
clusters:
  - name: ilum-dev
    provider: k3d
    kubecontext: k3d-ilum-dev
    created_at: "2025-01-15T10:30:00Z"
created_at: "2025-01-15T10:30:00Z"
updated_at: "2025-01-15T10:30:00Z"
```

Four Pydantic v2 models back this structure:

- **`IlumConfig`** — top-level: active profile name, profile map, cluster records, timestamps
- **`ProfileConfig`** — per-environment: name, cluster settings, enabled modules, communication/auth type
- **`ClusterConfig`** — per-cluster: kubecontext, namespace, chart version, chart repo URL
- **`ClusterRecord`** — tracks locally-managed clusters: name, provider, kubecontext, created timestamp

### Dot-Notation Access

```bash
ilum config set active_profile staging
ilum config set profiles.default.cluster.namespace production
ilum config show active_profile           # prints: staging
ilum config show profiles.default.name    # prints: default
```

---

## Development Guide

### Prerequisites

- Python 3.12+ (install via [uv](https://docs.astral.sh/uv/): `uv python install 3.12`)
- [uv](https://docs.astral.sh/uv/) (recommended) or pip

### Setup

```bash
cd ilum-cli

# Using uv (recommended)
uv venv --python 3.12 .venv
source .venv/bin/activate      # Linux/macOS
# .venv\Scripts\activate       # Windows
uv pip install -e ".[dev]"

# Or using pip directly (requires Python 3.12+ already available)
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

After installation, the `ilum` command is live-linked to your source code — any changes to `.py` files take effect immediately, no rebuild needed:

```bash
ilum --version   # prints current version
ilum status      # runs your local code
```

### Running Checks

```bash
# All-in-one
ruff format --check src tests && ruff check src tests && mypy src && pytest tests/unit -v

# Individual
ruff format --check src tests      # format check
ruff check src tests               # lint
mypy src                           # type check (strict mode)
pytest tests/unit -v               # unit tests
pytest tests/unit -v --cov=ilum --cov-report=term-missing  # with coverage
```

### Building and Publishing

```bash
# Build wheel + source tarball
uv build
# Output: dist/ilum-<version>-py3-none-any.whl
#         dist/ilum-<version>.tar.gz

# Install the wheel on any machine with Python 3.12+
pip install dist/ilum-*.whl

# Publish to PyPI (needs token or OIDC)
uv publish
```

The CI release pipeline (`cli-release.yml`) handles this automatically via `workflow_dispatch` — set the version and branch in the GitHub Actions UI.

### Building the API Docker Image

The Ilum API is a FastAPI service deployed as a sidecar in the `ilum-core` pod. Build from the **repository root** (`ilum-helm/`):

```bash
# From the repo root (ilum-helm/)
docker build -t ilum/api:<version> -f ilum-cli/docker/Dockerfile.api .
```

The Dockerfile copies both `ilum-cli/` (the Python package) and `helm_aio/` (the bundled chart for local upgrades), so the build context must be the repo root.

### Standalone Binaries (Optional)

For distributing self-contained executables (no Python required on target). Not needed for `pip install ilum`.

```bash
# Install PyInstaller
pip install -e ".[build]"

# Build using the spec file
pyinstaller ilum.spec --clean
# Output: dist/ilum (or dist/ilum.exe on Windows)

# Docker-based cross-platform builds (Linux/Windows)
make build-linux-amd64        # Linux x86_64
make build-linux-arm64        # Linux arm64 (QEMU, ~10-15 min)
make build-windows-amd64      # Windows x86_64 (Wine, experimental)
make release                  # All platforms + checksums
```

macOS binaries must be built natively on Intel and Apple Silicon machines (cannot be Dockerized). Run `make help` for all targets.

---

## Testing

1294 unit tests across 75 test files.

### Test Structure

```
tests/
├── conftest.py                  # 8 shared fixtures (includes console singleton reset)
└── unit/
    ├── test_cli_main.py         # Typer command invocation, --version, --verbose
    ├── test_compat.py           # Cross-platform file locking, atomic config writes
    ├── test_config.py           # ConfigManager: load, save, get/set, init, cross-platform paths
    ├── test_config_cmd.py       # Config CLI: editor fallback (notepad on Windows, vi on Unix)
    ├── test_config_export.py    # Config export/import: round-trip, portability, name override
    ├── test_config_ux.py        # Config use, list-profiles, validate, backup, config_version
    ├── test_console_singleton.py # Console singleton usage, --quiet/--output propagation
    ├── test_completers.py       # Shell completion: module names, profile names, check names
    ├── test_doctor.py           # DoctorRunner: all checks, single check, skip logic
    ├── test_doctor_v2.py        # Doctor: fixable field, to_dict, fix_simple_issues, storage-class
    ├── test_doctor_resources.py # Doctor resource check, memory/CPU parsing
    ├── test_error_codes.py      # Error code registry, IlumError codes, handle_error display
    ├── test_formatters.py       # OutputFormat enum, CommandResult, ResultFormatter dispatch
    ├── test_helm.py             # HelmClient: subprocess wrapping, JSON parsing, errors, preview
    ├── test_installer.py        # Installer scripts: version flag, PATH setup, verification
    ├── test_json_output.py      # JSON/YAML output for module list/show, quiet mode
    ├── test_modules.py          # ModuleResolver: enables, disables, conflicts, validation
    ├── test_module_cmd.py       # Module enable/disable/list/show, dry-run
    ├── test_module_ux.py        # Module search, tree, resource estimates in show
    ├── test_output_flags.py     # --output and --quiet flag parsing, console properties
    ├── test_progress.py         # SpinnerHandle, progress_spinner, quiet mode
    ├── test_values.py           # Values merging, overlay, round-trip preservation
    ├── test_versioning.py       # Version parsing, comparison, minor_version_changed
    ├── test_safety.py           # ValuesDiff, flatten, snapshot I/O, drift detection
    ├── test_safety_v2.py        # Context-aware snapshots, legacy migration, backups, retry
    ├── test_resources.py        # Resource estimates, totals, format, cluster capacity check
    ├── test_release.py          # ReleaseManager plan/execute/rollback, preview, breaking changes
    ├── test_audit.py            # Audit trail: AuditEntry, AuditLog JSONL, history command
    ├── test_cleanup.py          # Cleanup command (tiered teardown, dry-run, confirmations)
    ├── test_install.py          # Install command (dry-run, modules, summary, errors, command preview)
    ├── test_upgrade.py          # Upgrade command (drift, diff, stuck, breaking changes)
    ├── test_status.py           # Status command (release info, pods, modules, --wait, --events)
    ├── test_uninstall.py        # Uninstall command (idempotent, delete-data, namespace, dry-run)
    ├── test_logs.py             # Logs command (tail, follow, container, module resolution)
    ├── test_quickstart.py       # Quickstart command (cluster detect/create, install, modules)
    ├── test_defaults.py         # Profile-based default resolution
    ├── test_init.py             # Init wizard (non-interactive, config creation, modules)
    ├── test_cluster.py          # Cluster create/delete/list, presets, provider args
    ├── test_connect.py          # Connect command: discover, attach, multi-release
    ├── test_deps.py             # Tool detection, platform install commands
    ├── test_deps_cmd.py         # ilum deps install/list commands
    ├── test_gap_fixes.py        # Gap-fix tests (CRD pre-install, new flags, edge cases)
    ├── test_presets.py          # DeploymentPreset registry, module validation
    ├── test_preset_cmd.py       # ilum preset list command
    ├── test_images.py           # Image discovery from Helm templates
    ├── test_airgap_cmd.py       # ilum airgap images/export/import commands
    ├── test_csv_output.py       # CSV format, --no-headers, --field-selector
    ├── test_values_cmd.py       # ilum values command
    ├── test_diff_cmd.py         # ilum diff command
    ├── test_rollback.py         # ilum rollback command
    ├── test_exec.py             # ilum exec command
    ├── test_top.py              # ilum top command
    ├── test_wizard_flow.py      # Wizard preset step, module skip logic
    └── test_api/
        ├── conftest.py              # API fixtures (mock_api_manager, api_client, unauthed_client)
        ├── test_auth.py             # Dual auth: Nginx cookie + API key
        ├── test_config_schema.py    # Config schema endpoint, tabs, field types
        ├── test_errors.py           # IlumError → HTTP status mapping
        ├── test_job_runner.py       # K8s Job builder, concurrency guard, recovery
        ├── test_module_logs.py      # Module log retrieval endpoint
        ├── test_module_restart.py   # Module restart (pod deletion) endpoint
        ├── test_modules.py          # Module list/detail/enable/disable/config endpoints
        ├── test_operations.py       # Operation store, tracking, cancellation
        ├── test_release_recovery.py # Stuck release recovery endpoint
        ├── test_startup.py          # Auto-connect, recovery on boot
        ├── test_status.py           # Health, release info, pod list endpoints
        └── test_values.py           # Values read/diff/update endpoints
```

### Shared Fixtures (conftest.py)

| Fixture | What it provides |
|---------|-----------------|
| `_reset_console_singleton` | **(autouse)** Resets `output_mod.console` after every test to prevent `--quiet`/`--output` leaking between tests |
| `tmp_config_dir` | Temporary config directories (XDG on Unix, APPDATA on Windows), returns `IlumPaths` |
| `mock_helm` | `MagicMock(spec=HelmClient)` — never calls subprocess, returns `rc=0` |
| `mock_k8s` | `MagicMock(spec=KubeClient)` — connected, namespace exists, all healthy |
| `cli_runner` | Typer `CliRunner` for command testing |
| `sample_values` | Dict with core/ui/mongodb/minio/postgresql enabled, kafka/sql/airflow disabled |
| `resolver` | `ModuleResolver` instance for dependency tests |
| `mock_release_mgr` | `MagicMock(spec=ReleaseManager)` with pre-configured return values |

### API Test Fixtures (tests/unit/test_api/conftest.py)

| Fixture | What it provides |
|---------|-----------------|
| `mock_api_manager` | `MagicMock(spec=ReleaseManager)` with `ReleasePlan` return values for API tests |
| `mock_api_helm` | `MagicMock(spec=HelmClient)` for API-layer Helm mocking |
| `mock_api_k8s` | `MagicMock(spec=KubeClient)` with Job method defaults |
| `api_client` | `TestClient` with `X-API-Key` auth header and `ILUM_API_IMAGE`/`ILUM_SERVICE_ACCOUNT` env vars |
| `unauthed_client` | `TestClient` without auth headers (for 401 tests) |

### Running Specific Tests

```bash
# Single file
pytest tests/unit/test_modules.py -v

# Single test
pytest tests/unit/test_modules.py::test_resolve_enables_sql -v

# With coverage
pytest tests/unit -v --cov=ilum --cov-report=term-missing
```

### Writing New Tests

Use the existing fixtures — they handle all mocking and isolation:

```python
def test_my_feature(tmp_config_dir, mock_helm, mock_k8s):
    # tmp_config_dir: real filesystem, isolated XDG dirs
    # mock_helm: safe subprocess mock
    # mock_k8s: safe Kubernetes API mock
    ...
```

---

## CI Pipeline

### CI (`.github/workflows/cli-ci.yml`)

Triggers on pushes and PRs that touch `ilum-cli/**`. Also supports `workflow_call` for reuse by the release pipeline.

**Lint** (Python 3.12):
1. `ruff format --check src tests`
2. `ruff check src tests`
3. `mypy src`

**Test** (matrix: Python 3.12, 3.13):
1. `pytest tests/unit -v --cov=ilum --cov-report=xml`
2. Coverage upload to Codecov (Python 3.12 only, flag: `cli`)

### Release (`.github/workflows/cli-release.yml`)

Triggered via `workflow_dispatch` with `version` and `branch` inputs. Can be converted to `workflow_call` for integration with the Helm release pipeline.

**Jobs:**

1. **Test** — reuses the CI workflow
2. **Publish PyPI** — sets version via `sed`, builds with `uv build`, publishes via OIDC trusted publishing
3. **Create release** — creates GitHub Release with tag `cli-v{version}`

### Installer Scripts

**`ilum-cli/installer/install.sh`** (Linux / macOS) — A `curl | bash` installer with tiered fallback:

1. **Homebrew** (macOS only) — if available, `brew tap ilum-cloud/tap && brew install ilum`
2. **pipx** — if available, `pipx install ilum==$VERSION`
3. **uv** — if available, `uv tool install ilum==$VERSION`
4. **Binary download** — downloads platform tarball from GitHub Releases, verifies SHA-256 checksum, installs to `/usr/local/bin` (or `~/.local/bin` as fallback)

```bash
curl -fsSL https://get.ilum.cloud/cli | bash
# Or pin a version:
ILUM_VERSION=0.2.0 bash install.sh
```

**`ilum-cli/installer/install.ps1`** (Windows) — A PowerShell installer with tiered fallback:

1. **winget** — if available, `winget install ilum-cloud.ilum`
2. **pip** — if available, `pip install ilum==$VERSION`
3. **Binary download** — downloads `.zip` from GitHub Releases, verifies SHA-256 with `Get-FileHash`, installs to `%LOCALAPPDATA%\Programs\ilum\`

```powershell
irm https://get.ilum.cloud/cli/windows | iex
# Or pin a version:
$env:ILUM_VERSION = "0.2.0"; .\install.ps1
```

---

## Distribution Guide

### Automated Release (Primary Flow)

Go to **GitHub Actions → CLI Release → Run workflow**, enter version and branch. The pipeline:

1. Runs the full test suite (reuses `cli-ci.yml`)
2. Sets the version in `pyproject.toml` via `sed`
3. Builds wheel + sdist with `uv build`
4. Publishes to PyPI via OIDC trusted publishing
5. Creates a GitHub Release with tag `cli-v{version}`

### Manual Release

```bash
cd ilum-cli
VERSION=0.0.1

# 1. Run tests
ruff format --check src tests && ruff check src tests && mypy src && pytest tests/unit -v

# 2. Set version and build
sed -i "s/^version = \".*\"/version = \"${VERSION}\"/" pyproject.toml
uv build

# 3. Publish to PyPI
uv publish   # needs --token pypi-XXXX or TWINE_USERNAME/TWINE_PASSWORD

# 4. Verify
pip install ilum==${VERSION}
ilum --version
```

### Post-Release Verification

- [ ] `pip install ilum==${VERSION}` installs correctly
- [ ] `ilum --version` prints the correct version
- [ ] GitHub Release page exists with tag `cli-v{VERSION}`

---

## Adding New Features

### Adding a New Module

1. Add a `ModuleDefinition` entry to `MODULE_REGISTRY` in `src/ilum/core/modules.py`
2. Set `name`, `description`, `category`, `enable_flags`, `disable_flags`
3. Declare `requires` for dependency modules, `conflicts_with` for mutual exclusions
4. Set `default_enabled=True` if it should be on by default
5. Add tests in `tests/unit/test_modules.py` — verify `resolve_enables()` and `validate_combination()`

### Adding a New CLI Command

1. Create `src/ilum/cli/<name>_cmd.py`
2. Define a Typer app or command function
3. Register in `src/ilum/cli/main.py` with `app.add_typer()` or `app.command()`
4. Add tests in `tests/unit/test_<name>.py` using the `cli_runner` fixture

### Adding a New Doctor Check

1. Add a check function in `src/ilum/doctor/checks.py` returning `CheckResult`
2. Use `CheckStatus.OK`, `WARN`, `FAIL`, or `SKIP`
3. Include a `suggestion` string for `FAIL`/`WARN` results
4. Wire it into `DoctorRunner.run_all()` and `run_single()` in `src/ilum/doctor/runner.py`
5. Add tests in `tests/unit/test_doctor.py`

---

## Error Handling

### Exception Hierarchy

```
IlumError (base — carries .suggestion field)
├── PrerequisiteError          # Required tool or resource missing
├── ClusterConnectionError     # Cannot reach Kubernetes API
├── HelmError                  # Helm operation failed
│   ├── HelmTimeoutError       # Helm operation timed out
│   └── HelmReleaseError       # Release in bad state (pending-install, etc.)
├── ConfigError                # Config missing or invalid
├── ValuesError                # Problem with Helm values files
├── ModuleError                # Dependency or conflict error
├── ReleaseExistsError         # Release already exists (use upgrade instead)
└── ReleaseNotFoundError       # Release not found (use install instead)
```

### The `suggestion` Pattern

Every `IlumError` accepts an optional `suggestion` parameter — an actionable recovery message displayed to the user:

```python
raise ReleaseNotFoundError(
    "Release 'ilum' not found.",
    suggestion="Use 'ilum install' to create a new release."
)
```

The `IlumConsole.handle_error()` method renders the error in red and the suggestion in blue, giving users a clear next step instead of a raw traceback.

---

## Roadmap

### Completed

- **Phase 1 — Foundation:** Project skeleton, `ilum doctor` (11 checks), `ilum config`, module registry (32 modules), `HelmClient`, `KubeClient`, `ModuleResolver`, `ConfigManager`, `IlumConsole`. 95 tests.
- **Phase 2 — Core Install/Upgrade Flow:** `ilum install`, `ilum upgrade`, `ilum module {enable,disable,list,show}`, `ilum status`, `ilum uninstall`, values-safety system (fetch-merge-diff-apply), drift detection, breaking-change warnings. 125 additional tests (220 total).
- **Phase 3 — Init Wizard and Distribution:** `ilum init` interactive wizard (preflight tool checks with auto-install, local cluster creation via k3d/minikube/kind, module selection with dependency validation), `ilum cluster {create,delete,list}`, `curl|bash` installer script with tiered fallback (pipx/uv/binary), PyInstaller spec for standalone binaries, GitHub Actions release pipeline (4 platforms + PyPI). 68 additional tests (288 total).
- **Phase 4 — Polish and Production Hardening:** `ilum logs` command (tail/stream by module with `--follow`, `--tail`, `--container`, `--previous`), command preview showing exact Helm commands before execution across all mutating commands, `--dry-run` on `uninstall` and `module enable/disable`, upgrade-path intelligence with tuple-based semver comparison and auto-resolved version breaking-change detection, installer hardening (trap-based temp cleanup, post-install verification). 72 additional tests (360 total).
- **Phase 5 — UX and Documentation:** `ilum connect` command (auto-detect or pick from existing Helm releases, sync modules from live values, values snapshot for drift detection), `ilum quickstart` command (one-command install: auto-detect cluster or create local minikube, install with default modules, zero user decisions), comprehensive user guide and CLI reference documentation. 31 additional tests (391 total).
- **Phase 5b — Ecosystem Expansion:** Deployment presets (`development`, `production`, `data-engineering`, `air-gapped` with `--preset` flag on install/quickstart/init), air-gapped tooling (`ilum airgap images/export/import` for disconnected deployments), Homebrew formula for macOS distribution. 48 additional tests (439 total).
- **Phase 5c — Windows Support:** Cross-platform file locking (`msvcrt` on Windows, `fcntl` on Unix), Windows config paths (`%APPDATA%`/`%LOCALAPPDATA%`), winget-based tool installation for Helm/kubectl, platform-aware editor default (`notepad` on Windows), PowerShell installer (`install.ps1`), Windows CI test matrix and binary builds. 12 additional tests (451 total).
- **Phase 6 — Operations and Lifecycle:** `ilum cleanup` command (tiered teardown: release, cluster, config, deps, self-uninstall), `ilum deps` command (dependency listing and install), `ilum module status` (live module health with pod readiness and config drift detection), `--max-duration` for log streaming, `--no-switch` for connect, CRD pre-install support for monitoring module, preset updates (openldap in development/data-engineering), comprehensive gap fixes. 306 additional tests (757 total).
- **Phase 7 — Day-2 Operations:** Enhanced machine-readable output (`--output csv`, `--no-headers`, `--field-selector`), `ilum values` command (view/export live values with dot-path filtering, revision pinning, diff, YAML export), `ilum diff` command (compare values across defaults, files, revisions, snapshots), `ilum status --wait` (poll until all pods ready, CI-friendly exit codes) and `--events` (Kubernetes events with type/age filtering), `ilum doctor` health endpoint and service endpoint checks (13 checks total), `ilum rollback` command (explicit rollback with values diff preview and dry-run), `ilum exec` command (shell into module pods with auto-detection and multi-pod selection), `ilum top` command (per-module resource usage with metrics-server integration, watch mode). 206 additional tests (963 total).
- **Phase 8 — REST API:** FastAPI REST service (`ilum.api`) deployed as a sidecar in `ilum-core` pod. Dual auth (Nginx cookie + API key), Kubernetes Job-based helm operations (survive pod restarts), in-memory operation store with FIFO eviction, chart value caching, auto-recovery of operations on startup. Endpoints: health, release info, pods, module CRUD (list/detail/enable/disable/config/restart/logs), operations (list/detail/cancel), values (read/diff/update), release recovery. 331 additional tests (1294 total).

---
