# Ilum CLI User Guide

The Ilum CLI (`ilum`) is a command-line installer and management tool for the Ilum Data Lakehouse platform. It wraps Helm and kubectl to give you a streamlined workflow for deploying, configuring, and operating Ilum on any Kubernetes cluster. This guide walks you through installation, first-cluster setup, and day-to-day module management.

## Table of Contents

- [Ilum CLI User Guide](#ilum-cli-user-guide)
  - [Table of Contents](#table-of-contents)
  - [1. Installation](#1-installation)
    - [1.1 One-Line Installer](#11-one-line-installer)
    - [1.2 Package Managers](#12-package-managers)
    - [1.3 Verifying the Installation](#13-verifying-the-installation)
    - [1.4 Shell Completion](#14-shell-completion)
    - [1.5 Prerequisites (helm, kubectl, docker)](#15-prerequisites-helm-kubectl-docker)
  - [2. Your First Cluster](#2-your-first-cluster)
    - [2.1 Quickstart (One Command)](#21-quickstart-one-command)
    - [2.2 Deployment Presets](#22-deployment-presets)
    - [2.3 Air-Gapped Deployments](#23-air-gapped-deployments)
    - [2.4 Running ilum init](#24-running-ilum-init)
    - [2.5 Creating a Local Cluster](#25-creating-a-local-cluster)
    - [2.6 Installing Ilum](#26-installing-ilum)
    - [2.7 Checking Status](#27-checking-status)
    - [2.8 Accessing the UI](#28-accessing-the-ui)
    - [2.9 Running Doctor](#29-running-doctor)
  - [3. Working with Modules](#3-working-with-modules)
    - [3.1 Browsing Modules](#31-browsing-modules)
    - [3.2 Enabling Modules](#32-enabling-modules)
    - [3.3 Understanding Dependency Resolution](#33-understanding-dependency-resolution)
    - [3.4 Disabling Modules](#34-disabling-modules)
    - [3.5 Module Categories](#35-module-categories)
  - [4. Day-2 Operations](#4-day-2-operations)
    - [4.1 Upgrading Ilum](#41-upgrading-ilum)
    - [4.2 Breaking Change Warnings](#42-breaking-change-warnings)
    - [4.3 Viewing Logs](#43-viewing-logs)
    - [4.4 Health Checks](#44-health-checks)
    - [4.5 Recovering Stuck Releases](#45-recovering-stuck-releases)
    - [4.6 Operation History](#46-operation-history)
    - [4.7 Inspecting Values](#47-inspecting-values)
    - [4.8 Comparing Values](#48-comparing-values)
    - [4.9 Rolling Back](#49-rolling-back)
    - [4.10 Shelling into Pods](#410-shelling-into-pods)
    - [4.11 Resource Usage](#411-resource-usage)
  - [5. Managing Configuration](#5-managing-configuration)
    - [5.1 Profiles](#51-profiles)
    - [5.2 Profile Management](#52-profile-management)
    - [5.3 Config Get / Set / Edit](#53-config-get--set--edit)
    - [5.4 Connecting to Existing Installations](#54-connecting-to-existing-installations)
  - [6. CI/CD Automation](#6-cicd-automation)
    - [6.1 Non-Interactive Mode](#61-non-interactive-mode)
    - [6.2 Machine-Readable Output](#62-machine-readable-output)
    - [6.3 Scripting Patterns](#63-scripting-patterns)
    - [6.4 GitHub Actions Example Workflow](#64-github-actions-example-workflow)
    - [6.5 GitOps-Style Upgrades](#65-gitops-style-upgrades)
  - [7. Uninstalling](#7-uninstalling)
    - [7.1 Basic Uninstall](#71-basic-uninstall)
    - [7.2 Cleaning Up PVCs and Namespaces](#72-cleaning-up-pvcs-and-namespaces)
    - [7.3 Full Environment Teardown](#73-full-environment-teardown)
    - [7.4 Removing the CLI](#74-removing-the-cli)
  - [8. Recipes](#8-recipes)
    - [8.1 Migrating from Manual Helm to CLI](#81-migrating-from-manual-helm-to-cli)
    - [8.2 Multi-Namespace Production Setup](#82-multi-namespace-production-setup)
    - [8.3 Enabling the Full Data Stack (SQL + Airflow + Superset)](#83-enabling-the-full-data-stack-sql--airflow--superset)
    - [8.4 Setting Up Monitoring (Prometheus + Loki + Grafana)](#84-setting-up-monitoring-prometheus--loki--grafana)
    - [8.5 Dry-Run Workflow for Change Management](#85-dry-run-workflow-for-change-management)

---

## 1. Installation

### 1.1 One-Line Installer

The fastest way to install the Ilum CLI is the bootstrap script. It uses a tiered fallback strategy: it tries `pipx` first, then `uv`, and finally downloads a standalone binary if neither Python package manager is available.

**Linux / macOS:**

```
$ curl -fsSL https://get.ilum.cloud/cli | bash
Detecting package manager...
  Found pipx 1.7.1
Installing ilum via pipx...
  installed package ilum 0.1.0
Done. Run 'ilum --version' to verify.
```

You can pin a specific version by setting the `ILUM_VERSION` environment variable before running the installer:

```
$ ILUM_VERSION=0.1.0 curl -fsSL https://get.ilum.cloud/cli | bash
```

**Windows (PowerShell):**

```
PS> irm https://get.ilum.cloud/cli/windows | iex
```

The Windows installer tries `winget` first, then `pip`, and finally downloads a standalone `.exe` binary. You can pin a version with `$env:ILUM_VERSION = "0.1.0"` before running.

> **Note:** The installers never modify files outside of your user-local package directories. They do not require administrator/root access.

### 1.2 Package Managers

If you prefer explicit control, install directly with your Python package manager of choice. The CLI requires **Python 3.12 or later**.

**pipx** (recommended for CLI tools):

```
$ pipx install ilum
```

**uv**:

```
$ uv tool install ilum
```

**pip** (into a virtual environment):

```
$ pip install ilum
```

**Editable install for development:**

```
$ git clone https://github.com/ilum-cloud/ilum.git
$ cd ilum/ilum-cli
$ pip install -e ".[dev]"
```

### 1.3 Verifying the Installation

After installing, confirm the CLI is on your PATH and print its version:

```
$ ilum --version
ilum 0.1.0
```

Then run a quick health check to verify that your environment has the required tools:

```
$ ilum doctor
```

Doctor checks for `helm` (>= 3.12), `kubectl` (>= 1.28), and `docker` (>= 24.0). If any tool is missing, the output tells you exactly what to install. See [Section 2.9](#29-running-doctor) for full details.

### 1.4 Shell Completion

The ilum CLI supports shell completion for Bash, Zsh, and Fish:

```bash
# Bash
ilum --install-completion bash

# Zsh
ilum --install-completion zsh

# Fish
ilum --install-completion fish
```

After installing completion, restart your shell. Tab completion works for:
- Module names (`ilum module show <TAB>`)
- Profile names (`ilum config use <TAB>`)
- Doctor check names (`ilum doctor --check <TAB>`)

### 1.5 Prerequisites (helm, kubectl, docker)

The Ilum CLI requires three tools to be installed on your system:

| Tool | Minimum Version | Purpose |
|------|----------------|---------|
| `helm` | >= 3.12 | Deploys and manages the Ilum Helm chart |
| `kubectl` | >= 1.28 | Communicates with the Kubernetes cluster |
| `docker` | >= 24.0 | Container runtime for local clusters |

If you plan to create a local Kubernetes cluster, you also need one cluster provider:

| Provider | Description |
|----------|-------------|
| `k3d` | Lightweight k3s clusters in Docker containers (default) |
| `minikube` | Full Kubernetes in a VM or container |
| `kind` | Kubernetes-in-Docker for testing |

**Standalone installation with `ilum deps`.** The fastest way to install all prerequisites is the `ilum deps install` command. It works in both interactive and non-interactive mode (no TTY required), making it ideal for CI pipelines and scripted setups:

```
$ ilum deps install
ℹ Installing helm...
  ✓ helm installed successfully (helm 3.16.3)
  ✓ kubectl already installed (kubectl 1.30.0)
  ✓ docker already installed (docker 27.3.1)
ℹ Installing k3d...
  ✓ k3d installed successfully (k3d 5.7.1)
ℹ Summary: 2 installed, 2 already present.
```

You can also install specific tools, choose a different cluster provider, or preview what would be installed:

```bash
# Install only helm and kubectl
$ ilum deps install helm kubectl

# Install with kind instead of the default k3d
$ ilum deps install --provider kind

# Preview without executing
$ ilum deps install --dry-run
```

To check the status of all dependencies without installing anything, use `ilum deps list`:

```
$ ilum deps list
```

**Automatic installation (interactive mode).** When you run `ilum init` or `ilum quickstart` in an interactive terminal, the CLI checks for each required tool during the preflight step. If a tool is missing or outdated, it offers to install it for you:

```
$ ilum quickstart
╭─ ilum quickstart ────────────────────────────────────────╮
│ Ilum Quickstart                                          │
╰──────────────────────────────────────────────────────────╯

ℹ Step 1/4: Checking prerequisites...
  ✓ helm 3.16.3
  ✗ kubectl not found
? Install kubectl now? Yes
ℹ Installing kubectl via direct download...
  ✓ kubectl installed successfully
  ✓ docker 27.3.1
✓ All prerequisites met.
...
```

The CLI installs tools using platform-appropriate methods:

| Platform | Install Method |
|----------|---------------|
| **Linux** | Official install scripts or direct binary downloads to `/usr/local/bin/` |
| **macOS** | Homebrew (`brew install`) |
| **Windows** | winget (`winget install`) |

After installation, the CLI re-checks the tool version to confirm success. If the automatic install fails, the error message includes a link to the tool's manual installation page.

**Non-interactive mode (CI/scripts).** Auto-installation is only available in interactive terminals. In non-interactive environments (CI pipelines, scripts, cron jobs, or when piping output), the CLI raises an error if a required tool is missing:

```
$ ilum install --yes 2>&1
Error: kubectl is required but not available
  Suggestion: Install manually: https://kubernetes.io/docs/tasks/tools/
```

In CI/CD pipelines, install prerequisites before running the CLI. See [Section 6.4](#64-github-actions-example-workflow) for a complete example.

**Cluster provider auto-install.** When you run `ilum cluster create --provider k3d` or select a provider in the `ilum init` wizard, the CLI checks for the provider binary and offers to install it if missing (interactive mode only). This applies to `k3d`, `minikube`, and `kind`.

**Checking prerequisites without installing.** Use `ilum doctor` to check all tool versions without triggering any installation:

```
$ ilum doctor --check helm --check kubectl --check docker
```

See [Section 2.9](#29-running-doctor) for full details on the doctor command.

> **After `ilum cleanup --deps`:** If you previously removed dependencies with `ilum cleanup --deps` and need to reinstall them, run `ilum deps install` to restore all required tools non-interactively. Alternatively, run `ilum init` interactively or install tools manually using the commands shown below.

**Manual installation (all platforms):**

```bash
# Linux — helm
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Linux — kubectl
curl -fsSLo /tmp/kubectl \
  "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x /tmp/kubectl && sudo mv /tmp/kubectl /usr/local/bin/kubectl

# Linux — docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER && newgrp docker

# Linux — k3d
curl -fsSL https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

# macOS — all tools via Homebrew
brew install helm kubectl k3d
brew install --cask docker
```

---

## 2. Your First Cluster

This section walks you through the end-to-end workflow: initializing the CLI, creating a local Kubernetes cluster, installing the Ilum platform, and verifying the deployment.

### 2.1 Quickstart (One Command)

If you want to get Ilum running as fast as possible with zero decisions, use `ilum quickstart`. It detects your cluster (or creates a local minikube), configures a default profile, and installs Ilum with all default modules — all in a single command.

```
$ ilum quickstart
╭─ ilum quickstart ────────────────────────────────────────╮
│ Ilum Quickstart                                          │
│                                                          │
│ One-command setup: preflight checks, cluster detection,  │
│ and install.                                             │
╰──────────────────────────────────────────────────────────╯

ℹ Step 1/4: Checking prerequisites...
  ✓ helm 3.16.3
  ✓ kubectl 1.31.2
  ✓ docker 27.3.1
✓ All prerequisites met.

ℹ Step 2/4: Detecting Kubernetes cluster...
✓ Cluster reachable (context: minikube)

ℹ Step 3/4: Configuring profile...
✓ Profile 'default' configured with 7 modules.

ℹ Step 4/4: Installing Ilum...
  Command: helm install ilum ilum/ilum --namespace default ...
⠋ Installing Ilum (this may take several minutes)...
✓ Ilum installed successfully!

ℹ Next steps:
  ilum status          Show release and pod status
  ilum module enable   Enable additional modules
  ilum logs core       View Ilum Core logs
  ilum upgrade         Upgrade to a newer version
```

If no cluster is reachable, quickstart automatically creates a local minikube cluster with the `dev` preset (4 CPUs, 8 GB RAM):

```
$ ilum quickstart
...
ℹ Step 2/4: Detecting Kubernetes cluster...
ℹ No reachable cluster found. Creating a local cluster...
ℹ Creating minikube cluster 'ilum-dev'...
✓ Cluster 'ilum-dev' created (context: ilum-dev)
...
```

If a required tool is missing, quickstart offers to install it automatically (interactive mode only -- see [Section 1.5](#15-prerequisites-helm-kubectl-docker)):

```
$ ilum quickstart
...
ℹ Step 1/4: Checking prerequisites...
  ✓ helm 3.16.3
  ✗ kubectl not found
? Install kubectl now? Yes
ℹ Installing kubectl via direct download...
  ✓ kubectl installed successfully
  ✓ docker 27.3.1
✓ All prerequisites met.
...
```

You can override the provider with `--provider`:

```
$ ilum quickstart --provider k3d
```

To enable extra modules beyond the defaults, pass `-m`:

```
$ ilum quickstart -m sql -m airflow
```

> **When to use quickstart vs. init + install:** Use `ilum quickstart` when you want the fastest path to a working Ilum cluster with no customization. Use `ilum init` followed by `ilum install` when you need to pick specific modules, configure a custom namespace, choose a release name, or fine-tune the deployment before installing.

### 2.2 Deployment Presets

Deployment presets are curated module bundles designed for common scenarios. Instead of hand-picking individual modules, you select a preset that includes a tested combination of modules and Helm flags appropriate for your use case.

**The four built-in presets:**

| Preset | Description | Modules |
|--------|-------------|---------|
| `development` | Lightweight local setup for development and testing | core, ui, mongodb, postgresql, minio, jupyter, gitea (7 modules) |
| `production` | Hardened setup with security, monitoring, and replication | core, ui, mongodb, postgresql, minio, jupyter, gitea, monitoring, loki, openldap (10 modules) |
| `data-engineering` | Full data stack with SQL, orchestration, and BI tooling | core, ui, mongodb, postgresql, minio, jupyter, gitea, sql, airflow, superset, trino, hive-metastore (12 modules) |
| `air-gapped` | Offline deployment with a private container registry | core, ui, mongodb, postgresql, minio, jupyter, gitea (7 modules) |

The `production` preset also sets `global.security.enabled=true` and `ilum-core.replicaCount=2`. The `air-gapped` preset requires you to provide `global.imageRegistry` at install time (see [Section 2.3](#23-air-gapped-deployments)).

**List all available presets:**

```
$ ilum preset list
╭─ Deployment Presets ─────────────────────────────────────────────────────────╮
│ Name               Description                                      Modules │
├──────────────────────────────────────────────────────────────────────────────┤
│ development        Lightweight local setup for development and ...   7       │
│ production         Hardened setup with security, monitoring, an...   10      │
│ data-engineering   Full data stack with SQL, orchestration, and...   12      │
│ air-gapped         Offline deployment with a private container ...   7       │
╰──────────────────────────────────────────────────────────────────────────────╯
```

**Use a preset with `ilum install`:**

```
$ ilum install --preset production --yes
╭─ Install Summary ────────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    latest                                        │
│ Preset     production                                    │
│ Modules    core, gitea, jupyter, loki, minio, mongodb,   │
│            monitoring, openldap, postgresql, ui          │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

  Command: helm install ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --atomic \
    --set ilum-core.enabled=true \
    --set ilum-ui.enabled=true \
    --set mongodb.enabled=true \
    --set postgresql.enabled=true \
    --set minio.enabled=true \
    --set ilum-jupyter.enabled=true \
    --set gitea.enabled=true \
    --set kube-prometheus-stack.enabled=true \
    --set global.logAggregation.enabled=true \
    --set global.logAggregation.loki.enabled=true \
    --set global.logAggregation.promtail.enabled=true \
    --set openldap.enabled=true \
    --set global.security.enabled=true \
    --set ilum-core.replicaCount=2

⠋ Installing Ilum...
✓ Ilum installed successfully (release: ilum).
```

**Use a preset with `ilum quickstart`:**

```
$ ilum quickstart --preset data-engineering
```

This deploys Ilum with the full data engineering stack (SQL, Airflow, Superset, Trino, Hive Metastore) in a single command.

**Presets in the `ilum init` wizard.** When you run `ilum init`, the wizard presents a preset selection step before module selection. Choosing a preset pre-populates the module list; choosing "Custom" drops you into the manual module picker:

```
$ ilum init
...
? Choose a deployment preset:
  development — Lightweight local setup for development and testing (7 modules)
  production — Hardened setup with security, monitoring, and replication (10 modules)
  data-engineering — Full data stack with SQL, orchestration, and BI tooling (12 modules)
  air-gapped — Offline deployment with a private container registry (7 modules)
> Custom
```

**Combining presets with extra modules.** Presets define a base set of modules. You can layer additional modules on top with the `-m` flag:

```
$ ilum install --preset production -m airflow -m superset --yes
```

This installs the production preset (10 modules) plus Airflow and Superset, for a total of 12 modules. Dependencies are resolved automatically as usual.

**Air-gapped preset.** The `air-gapped` preset requires `global.imageRegistry` to be set to your private registry URL. You must provide it via `--set` at install time:

```
$ ilum install --preset air-gapped --set global.imageRegistry=registry.local:5000 --yes
```

If you omit the `--set` flag, the CLI reports the placeholder and prompts you to provide the value. See [Section 2.3](#23-air-gapped-deployments) for the full air-gapped workflow.

### 2.3 Air-Gapped Deployments

An air-gapped environment is a network that has no outbound internet access. In these environments, you cannot pull container images from public registries or download Helm charts at install time. The `ilum airgap` command group provides a workflow for exporting everything you need on a connected machine, transferring the bundle to the isolated network, and importing it into a private registry.

The full workflow has five steps:

**Step 1: List required images.**

On a machine with internet access, use `ilum airgap images` to discover every container image needed for your module selection. The command runs `helm template` under the hood and parses all `image:` references from the rendered manifests.

```
$ ilum airgap images --preset air-gapped
╭─ Container Images (23) ──────────────────────────────────────────────╮
│ Image Reference                                                      │
├──────────────────────────────────────────────────────────────────────┤
│ docker.io/ilumcloud/ilum-core:6.7.0                                  │
│ docker.io/ilumcloud/ilum-ui:6.7.0                                    │
│ docker.io/bitnami/mongodb:7.0                                        │
│ docker.io/bitnami/postgresql:16                                      │
│ docker.io/minio/minio:latest                                         │
│ docker.io/ilumcloud/ilum-jupyter:6.7.0                               │
│ docker.io/gitea/gitea:1.21                                           │
│ ...                                                                  │
╰──────────────────────────────────────────────────────────────────────╯
```

You can also use `--module` to discover images for a custom module set, or combine `--preset` with `-m` for additional modules:

```
$ ilum airgap images --preset air-gapped -m sql -m airflow
```

For scripting, use `--format plain` to get one image reference per line with no table formatting:

```
$ ilum airgap images --preset air-gapped --format plain
docker.io/ilumcloud/ilum-core:6.7.0
docker.io/ilumcloud/ilum-ui:6.7.0
docker.io/bitnami/mongodb:7.0
...
```

**Step 2: Export the bundle.**

The `ilum airgap export` command pulls every image, saves them as Docker tar archives, and writes a `manifest.json` describing the bundle contents.

```
$ ilum airgap export /tmp/bundle --preset air-gapped
ℹ Pulling docker.io/ilumcloud/ilum-core:6.7.0...
ℹ Saving docker.io/ilumcloud/ilum-core:6.7.0 → ilumcloud_ilum-core_6.7.0.tar
ℹ Pulling docker.io/ilumcloud/ilum-ui:6.7.0...
ℹ Saving docker.io/ilumcloud/ilum-ui:6.7.0 → ilumcloud_ilum-ui_6.7.0.tar
ℹ Pulling docker.io/bitnami/mongodb:7.0...
ℹ Saving docker.io/bitnami/mongodb:7.0 → bitnami_mongodb_7.0.tar
...
✓ Bundle exported to /tmp/bundle (23 images, 7 modules)
```

The resulting bundle directory looks like this:

```
/tmp/bundle/
├── manifest.json
└── images/
    ├── ilumcloud_ilum-core_6.7.0.tar
    ├── ilumcloud_ilum-ui_6.7.0.tar
    ├── bitnami_mongodb_7.0.tar
    ├── bitnami_postgresql_16.tar
    └── ...
```

**Step 3: Transfer the bundle.**

Copy the entire `/tmp/bundle` directory to the air-gapped environment using whatever transfer mechanism your organization allows (USB drive, secure file transfer, sneakernet). The bundle is fully self-contained.

```
# Example: rsync to a bastion host inside the air-gapped network
$ rsync -avz /tmp/bundle/ bastion.internal:/opt/ilum-bundle/
```

**Step 4: Import into the private registry.**

On a machine inside the air-gapped network that has Docker and access to your private registry, run `ilum airgap import`. This loads each tar archive into Docker, re-tags it for your registry, and pushes it.

```
$ ilum airgap import /tmp/bundle --registry registry.local:5000
ℹ Loading ilumcloud_ilum-core_6.7.0.tar...
ℹ Pushing registry.local:5000/ilumcloud/ilum-core:6.7.0...
✓ docker.io/ilumcloud/ilum-core:6.7.0 → registry.local:5000/ilumcloud/ilum-core:6.7.0
ℹ Loading ilumcloud_ilum-ui_6.7.0.tar...
ℹ Pushing registry.local:5000/ilumcloud/ilum-ui:6.7.0...
✓ docker.io/ilumcloud/ilum-ui:6.7.0 → registry.local:5000/ilumcloud/ilum-ui:6.7.0
ℹ Loading bitnami_mongodb_7.0.tar...
ℹ Pushing registry.local:5000/bitnami/mongodb:7.0...
✓ docker.io/bitnami/mongodb:7.0 → registry.local:5000/bitnami/mongodb:7.0
...
✓ All 23 images imported to registry.local:5000
ℹ
ℹ To install with the private registry, use:
ℹ   ilum install --set global.imageRegistry=registry.local:5000
```

The `--registry` flag is required. It specifies the hostname (and optional port) of your private Docker registry. Each image is re-tagged by replacing the original registry prefix with the target registry while preserving the repository path and tag.

**Step 5: Install Ilum.**

With all images available in the private registry, install Ilum using the `air-gapped` preset and point `global.imageRegistry` at your registry:

```
$ ilum install --preset air-gapped --set global.imageRegistry=registry.local:5000 --yes
╭─ Install Summary ────────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    latest                                        │
│ Preset     air-gapped                                    │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

⠋ Installing Ilum...
✓ Ilum installed successfully (release: ilum).
```

Kubernetes pulls all images from `registry.local:5000` instead of the public internet. No outbound connectivity is required.

> **Tip:** You can also use the air-gapped workflow with any preset or custom module set -- it is not limited to the `air-gapped` preset. For example, to prepare an air-gapped bundle for the production preset, run `ilum airgap export /tmp/bundle --preset production`.

### 2.4 Running ilum init

The `ilum init` command launches an interactive setup wizard that walks you through five steps: preflight checks, cluster selection, profile settings, module selection, and a confirmation summary.

```
$ ilum init
╭─ ilum init ──────────────────────────────────────────────╮
│ Welcome to the Ilum CLI Setup Wizard                     │
│                                                          │
│ This wizard will help you configure your Ilum deployment.│
╰──────────────────────────────────────────────────────────╯

ℹ Checking prerequisites...
  ✓ helm 3.16.3
  ✓ kubectl 1.31.2
  ✓ docker 27.3.1
✓ All prerequisites met.

? Select a Kubernetes context:
  k3d-ilum-dev
  minikube
  arn:aws:eks:eu-central-1:123456:cluster/staging
> >> Create a new local cluster

? Cluster provider: k3d
? Resource preset: dev (CPUs: 4, Memory: 8g)
ℹ Creating k3d cluster 'ilum-dev'...
✓ Cluster 'ilum-dev' created (context: k3d-ilum-dev)

? Profile name: default
? Helm release name: ilum
? Kubernetes namespace: default

? Select modules to enable:
  (↑↓ move, space toggle, a toggle all, d defaults, enter confirm)
  ── CORE ──
  ◉ core - Ilum backend API (Spring Boot on Spark) [required]
  ◉ ui - Ilum web frontend (React + Nginx reverse proxy) [required]
  ○ livy-proxy - Livy-compatible Spark session proxy
  ── NOTEBOOK ──
  ◉ jupyter - Jupyter notebook server with Sparkmagic
  ○ jupyterhub - Multi-user JupyterHub (Kubernetes >= 1.28)
  ○ zeppelin - Apache Zeppelin notebook server
  ── INFRASTRUCTURE ──
  ◉ mongodb - MongoDB document store for metadata [required]
  ◉ postgresql - PostgreSQL relational database
  ◉ gitea - Gitea self-hosted Git service
  ── STORAGE ──
  ◉ minio - MinIO S3-compatible object storage
  ...

? Selected 7 modules. What would you like to do?
> Continue with this selection
  Reset to defaults
  Re-select modules

╭─ Configuration Summary ─────────────────────────────────╮
│ Profile       default                                    │
│ Release Name  ilum                                       │
│ Kubecontext   k3d-ilum-dev                               │
│ Namespace     default                                    │
│ Modules       core, ui, mongodb, postgresql, minio       │
│                 ... +3 more                              │
╰──────────────────────────────────────────────────────────╯

? Save this configuration? Yes
ℹ Creating new profile 'default'
✓ Configuration saved to /home/user/.config/ilum/config.yaml
✓ Setup complete!
ℹ Active profile: default
ℹ Run 'ilum install' to deploy Ilum.
```

**Automatic prerequisite installation.** If any required tool is missing or outdated, the wizard offers to install it before continuing (see [Section 1.5](#15-prerequisites-helm-kubectl-docker)):

```
$ ilum init
╭─ ilum init ──────────────────────────────────────────────╮
│ Welcome to the Ilum CLI Setup Wizard                     │
╰──────────────────────────────────────────────────────────╯

ℹ Checking prerequisites...
  ✓ helm 3.16.3
  ✗ kubectl not found
? Install kubectl now? Yes
ℹ Installing kubectl via direct download...
  ✓ kubectl installed successfully
  ✗ docker not found
? Install docker now? Yes
ℹ Installing Docker via official install script (may require sudo)...
ℹ Adding user 'dev' to the docker group...
  ✓ docker installed successfully
⚠ You may need to log out and back in (or run 'newgrp docker')
  for Docker group permissions to take effect.
✓ All prerequisites met.
...
```

If you decline the installation or it fails, the wizard exits with an error message and a link to the tool's manual installation page.

The wizard enforces that the three required modules -- `core`, `ui`, and `mongodb` -- are always selected. If you pick a module that has dependencies (for example, `sql` requires `postgresql` and `core`), those dependencies are automatically added and reported.

**Fast path with defaults:**

If you want to skip all prompts and accept every default, pass the `--yes` flag:

```
$ ilum init --yes
```

This uses the current kubeconfig context, creates a profile named `default` with release name `ilum` in the `default` namespace, and enables the default module set (core, ui, mongodb, postgresql, minio, jupyter, gitea).

You can also name the profile explicitly:

```
$ ilum init --profile staging --yes
```

### 2.5 Creating a Local Cluster

If you do not already have a Kubernetes cluster, the `ilum cluster create` command provisions one locally. The default provider is **k3d** with the **dev** preset (4 CPUs, 8 GB memory).

```
$ ilum cluster create
ℹ Creating k3d cluster 'ilum-dev'...
✓ Cluster 'ilum-dev' created (context: k3d-ilum-dev)
ℹ Kubecontext: k3d-ilum-dev
ℹ Run 'ilum init' or 'ilum install' to deploy Ilum on this cluster.
```

You can switch the provider and preset:

```
$ ilum cluster create --provider minikube --preset full
ℹ Creating minikube cluster 'ilum'...
✓ Cluster 'ilum' created (context: ilum)
ℹ Kubecontext: ilum
ℹ Run 'ilum init' or 'ilum install' to deploy Ilum on this cluster.
```

The two available presets are:

| Preset | CPUs | Memory | Default Name |
|--------|------|--------|--------------|
| `dev`  | 6    | 12 GB   | `ilum-dev`   |
| `full` | 8    | 18 GB  | `ilum`       |

The three supported providers are `k3d`, `minikube`, and `kind`. Each produces a different kubecontext name format: k3d uses `k3d-<name>`, kind uses `kind-<name>`, and minikube uses the cluster name directly.

To see your tracked clusters:

```
$ ilum cluster list
╭─ Local Clusters ─────────────────────────────────────────────────────╮
│ Name       Provider   Context        Created                        │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-dev   k3d        k3d-ilum-dev   2026-02-14T09:31:42            │
╰──────────────────────────────────────────────────────────────────────╯
```

### 2.6 Installing Ilum

With a cluster running and a profile configured, run `ilum install` to deploy the Ilum platform via Helm. The command reads your active profile for the release name, namespace, and context, then presents a summary for confirmation.

```
$ ilum install --yes
╭─ Install Summary ────────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    latest                                        │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

  Command: helm install ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --atomic \
    --set ilum-core.enabled=true \
    --set ilum-ui.enabled=true \
    --set mongodb.enabled=true \
    --set postgresql.enabled=true \
    --set minio.enabled=true \
    --set ilum-jupyter.enabled=true \
    --set gitea.enabled=true

⠋ Installing Ilum...
✓ Ilum installed successfully (release: ilum).
```

**Dry-run mode** lets you preview the full Helm command without executing it:

```
$ ilum install --dry-run
╭─ Install Summary ────────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    latest                                        │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

  Command: helm install ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --atomic \
    --set ilum-core.enabled=true \
    --set ilum-ui.enabled=true \
    --set mongodb.enabled=true \
    --set postgresql.enabled=true \
    --set minio.enabled=true \
    --set ilum-jupyter.enabled=true \
    --set gitea.enabled=true

ℹ Dry-run mode — no changes applied.
```

**Enable specific modules at install time** with the `-m` flag (repeatable):

```
$ ilum install -m sql -m airflow --yes
╭─ Install Summary ────────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    latest                                        │
│ Modules    sql, airflow                                  │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

  Command: helm install ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --atomic \
    --set ilum-sql.enabled=true \
    --set ilum-core.sql.enabled=true \
    --set airflow.enabled=true \
    --set postgresql.enabled=true \
    --set ilum-core.enabled=true

⠋ Installing Ilum...
✓ Ilum installed successfully (release: ilum).
```

Notice that the `sql` module's dependencies (`postgresql` and `core`) are automatically resolved and included.

Other useful install flags include `--version` to pin a chart version, `--values` / `-f` to supply a custom values file, `--set` for individual Helm overrides, `--namespace` / `-n` to target a specific namespace, `--release` / `-r` to set the Helm release name, and `--timeout` to adjust the Helm timeout (default: `10m`). See [reference.md](reference.md) for the complete flag listing.

### 2.7 Checking Status

The `ilum status` command shows three panels: Release Info, Enabled Modules, and Pod Status.

```
$ ilum status
╭─ Release Info ───────────────────────────────────────────╮
│   Release:   ilum                                        │
│   Namespace: default                                     │
│   Status:    deployed                                    │
│   Chart:     ilum-6.7.0                                  │
│   Revision:  1                                           │
│   Deployed:  2026-02-14 09:35:12                         │
╰──────────────────────────────────────────────────────────╯

╭─ Enabled Modules (Live) ────────────────────────────────╮
│ Module                                                   │
├──────────────────────────────────────────────────────────┤
│ core                                                     │
│ gitea                                                    │
│ jupyter                                                  │
│ minio                                                    │
│ mongodb                                                  │
│ postgresql                                               │
│ ui                                                       │
╰──────────────────────────────────────────────────────────╯

╭─ Pod Status ─────────────────────────────────────────────────────────╮
│ Name                             Phase     Ready   Restarts         │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-core-6f8b4c7d9-xk2pl       Running   ✓       0                │
│ ilum-ui-5c9d3a1b8-mn4qr         Running   ✓       0                │
│ ilum-mongodb-0                   Running   ✓       0                │
│ ilum-postgresql-0                Running   ✓       0                │
│ ilum-minio-7b2e5f9a1-jt8ws      Running   ✓       0                │
│ ilum-jupyter-4d6c8e0f2-rv3hp     Running   ✓       0                │
│ ilum-gitea-0                     Running   ✓       0                │
╰──────────────────────────────────────────────────────────────────────╯
```

The status command also performs **config drift detection**. If the modules tracked in your local CLI config diverge from what is actually enabled on the cluster (for example, someone ran a manual `helm upgrade`), you see warnings like:

```
⚠ Config drift: 'airflow' enabled on cluster but not tracked in config
```

You can suppress sections with flags:

```
$ ilum status --no-pods       # Hide pod readiness table
$ ilum status --no-modules    # Hide the enabled modules panel
```

**Wait for all pods to be ready with `--wait`:**

The `--wait` flag polls pod readiness until all pods reach the `Running` state (or a timeout is hit). This is useful in CI/CD pipelines where you need to gate subsequent steps on a fully-ready cluster:

```
$ ilum status --wait
⠋ Waiting for pods to be ready...
  Pods: 5/7 ready
  Pods: 7/7 ready
✓ All pods are ready.

╭─ Pod Status ─────────────────────────────────────────────────────────╮
│ Name                             Phase     Ready   Restarts         │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-core-6f8b4c7d9-xk2pl       Running   ✓       0                │
│ ilum-ui-5c9d3a1b8-mn4qr         Running   ✓       0                │
│ ...                                                                 │
╰──────────────────────────────────────────────────────────────────────╯
```

The default timeout is 300 seconds (5 minutes). Override it with `--wait-timeout`:

```
$ ilum status --wait --wait-timeout 600
```

If pods do not become ready within the timeout, the command exits with code 1.

**Show recent Kubernetes events with `--events`:**

The `--events` flag appends an Events panel showing recent cluster events for the namespace. This surfaces scheduling failures, image pull errors, and resource issues without requiring a separate `kubectl get events` call:

```
$ ilum status --events
...
╭─ Events ─────────────────────────────────────────────────────────────────────────╮
│ Type      Reason              Object                   Message      Count  Age   │
├──────────────────────────────────────────────────────────────────────────────────┤
│ Normal    Scheduled           pod/ilum-core-6f8b...    Successfully  1     3m    │
│ Normal    Pulled              pod/ilum-core-6f8b...    Container im  1     3m    │
│ Warning   BackOff             pod/ilum-airflow-wo...   Back-off res  4     1m    │
╰──────────────────────────────────────────────────────────────────────────────────╯
```

Filter events by type or age:

```
$ ilum status --events --events-type Warning
$ ilum status --events --events-since 30m
$ ilum status --events --events-type Warning --events-since 1h
```

### 2.8 Accessing the UI

The Ilum web UI is exposed by default on **NodePort 31777**. After installation, open it in your browser:

```
http://localhost:31777
```

If you are running on a remote cluster, replace `localhost` with the node IP address. You can find the node address with:

```
$ kubectl get nodes -o wide
```

> **Note:** If NodePort 31777 is already in use by another service on the cluster, the `ilum install` command detects the conflict automatically. In interactive mode it prompts you to accept a suggested free port or enter a custom one. In non-interactive mode (`--yes`) it auto-assigns the next available port in the 30000-32767 range. You can also override the port explicitly with `--set ilum-ui.service.nodePort=31780`.

### 2.9 Running Doctor

The `ilum doctor` command runs a suite of 13 health checks against your environment and cluster. It validates tool versions, Helm repo configuration, cluster connectivity, namespace existence, pod health, PVC status, RBAC permissions, release state, Kubernetes version compatibility, service endpoints, and health endpoints.

**All checks passing:**

```
$ ilum doctor
╭─ ilum doctor ────────────────────────────────────────────────────────╮
│ Status   Check                Message                               │
├──────────────────────────────────────────────────────────────────────┤
│ ✓        helm                 helm 3.16.3                           │
│ ✓        kubectl              kubectl 1.31.2                        │
│ ✓        docker               docker 27.3.1                        │
│ ✓        helm-repo            ilum Helm repo configured             │
│ ✓        cluster              Connected to Kubernetes v1.31.2       │
│ ✓        namespace            Namespace 'default' exists            │
│ ✓        pods                 All pods are healthy                  │
│ ✓        pvcs                 7 PVCs all bound                      │
│ ✓        rbac                 Required RBAC permissions granted     │
│ ✓        release              Release 'ilum' status: deployed       │
│ ✓        compatibility        Kubernetes v1.31.2 is compatible      │
│ ✓        service-endpoints    All services have endpoints           │
│ ✓        health-endpoints     7 health endpoints responding         │
╰──────────────────────────────────────────────────────────────────────╯
```

**A failure example with suggestion:**

```
$ ilum doctor
╭─ ilum doctor ────────────────────────────────────────────────────────╮
│ Status   Check                Message                               │
├──────────────────────────────────────────────────────────────────────┤
│ ✓        helm                 helm 3.16.3                           │
│ ✗        kubectl              kubectl 1.26.5 < required 1.28        │
│ ✓        docker               docker 27.3.1                        │
│ ✓        helm-repo            ilum Helm repo configured             │
│ ✗        cluster              Cannot connect to Kubernetes cluster  │
│ –        namespace            Skipped (cluster unreachable)         │
│ –        pods                 Skipped (cluster unreachable)         │
│ –        pvcs                 Skipped (cluster unreachable)         │
│ –        rbac                 Skipped (cluster unreachable)         │
│ –        release              Skipped (cluster unreachable)         │
│ –        compatibility        Skipped (cluster unreachable)         │
│ –        service-endpoints    Skipped (cluster unreachable)         │
│ –        health-endpoints     Skipped (cluster unreachable)         │
╰──────────────────────────────────────────────────────────────────────╯
  kubectl: Upgrade kubectl to at least 1.28
  cluster: Check your kubeconfig: kubectl cluster-info
```

When the cluster connectivity check fails, all cluster-dependent checks are automatically skipped to avoid cascading errors.

You can also run a single check by name:

```
$ ilum doctor --check pods
╭─ ilum doctor ────────────────────────────────────────────────────────╮
│ Status   Check   Message                                            │
├──────────────────────────────────────────────────────────────────────┤
│ !        pods    Unhealthy pods: ilum-airflow-worker-0              │
╰──────────────────────────────────────────────────────────────────────╯
  pods: Check pod status: kubectl get pods
```

The available check names are: `helm`, `kubectl`, `docker`, `helm-repo`, `cluster`, `namespace`, `pods`, `pvcs`, `rbac`, `release`, `compatibility`, `service-endpoints`, and `health-endpoints`.

---

## 3. Working with Modules

Ilum is built as a modular platform with 31 independently toggleable modules spanning 10 categories. The `ilum module` sub-commands let you browse, enable, disable, and inspect modules without writing raw Helm flags.

### 3.1 Browsing Modules

**List all available modules:**

```
$ ilum module list
╭─ Ilum Modules ───────────────────────────────────────────────────────────────╮
│ Name             Category         Description                       Default │
├──────────────────────────────────────────────────────────────────────────────┤
│ core             core             Ilum backend API (Spring Boot...  ✓       │
│ ui               core             Ilum web frontend (React + Ng...  ✓       │
│ livy-proxy       core             Livy-compatible Spark session...          │
│ mongodb          infrastructure   MongoDB document store for me...  ✓       │
│ kafka            infrastructure   Apache Kafka event bus                    │
│ postgresql       infrastructure   PostgreSQL relational database    ✓       │
│ clickhouse       infrastructure   ClickHouse columnar analytics...          │
│ gitea            infrastructure   Gitea self-hosted Git service     ✓       │
│ minio            storage          MinIO S3-compatible object st...  ✓       │
│ jupyter          notebook         Jupyter notebook server with ...  ✓       │
│ jupyterhub       notebook         Multi-user JupyterHub (Kuber...           │
│ zeppelin         notebook         Apache Zeppelin notebook server           │
│ sql              sql              Ilum SQL engine (Kyuubi)                  │
│ hive-metastore   sql              Hive Metastore service                    │
│ nessie           sql              Nessie versioned data catalog             │
│ unity-catalog    sql              Unity Catalog for data govern...          │
│ trino            sql              Trino distributed SQL query e...          │
│ airflow          orchestration    Apache Airflow workflow orche...          │
│ kestra           orchestration    Kestra declarative orchestrat...          │
│ n8n              orchestration    n8n workflow automation platf...          │
│ nifi             orchestration    Apache NiFi data integration              │
│ mageai           orchestration    Mage.ai data pipeline tool               │
│ mlflow           ai               MLflow experiment tracking a...          │
│ langfuse         ai               Langfuse LLM observability p...          │
│ superset         analytics        Apache Superset BI dashboards            │
│ streamlit        analytics        Streamlit interactive data ap...          │
│ marquez          analytics        Marquez data lineage tracking             │
│ monitoring       monitoring       Prometheus + Grafana monitori...          │
│ loki             monitoring       Grafana Loki log aggregation              │
│ graphite         monitoring       Graphite metrics exporter                 │
│ openldap         security         OpenLDAP directory service                │
╰──────────────────────────────────────────────────────────────────────────────╯
```

**Filter by category:**

```
$ ilum module list --category sql
╭─ Ilum Modules ───────────────────────────────────────────────────────────────╮
│ Name             Category   Description                             Default │
├──────────────────────────────────────────────────────────────────────────────┤
│ sql              sql        Ilum SQL engine (Kyuubi)                         │
│ hive-metastore   sql        Hive Metastore service                           │
│ nessie           sql        Nessie versioned data catalog                    │
│ unity-catalog    sql        Unity Catalog for data governance                │
│ trino            sql        Trino distributed SQL query engine               │
╰──────────────────────────────────────────────────────────────────────────────╯
```

**Show only default-enabled modules:**

```
$ ilum module list --enabled
╭─ Ilum Modules ───────────────────────────────────────────────────────────────╮
│ Name           Category         Description                         Default │
├──────────────────────────────────────────────────────────────────────────────┤
│ core           core             Ilum backend API (Spring Boot...    ✓       │
│ ui             core             Ilum web frontend (React + Ng...    ✓       │
│ mongodb        infrastructure   MongoDB document store for me...    ✓       │
│ postgresql     infrastructure   PostgreSQL relational database      ✓       │
│ gitea          infrastructure   Gitea self-hosted Git service       ✓       │
│ minio          storage          MinIO S3-compatible object st...    ✓       │
│ jupyter        notebook         Jupyter notebook server with ...    ✓       │
╰──────────────────────────────────────────────────────────────────────────────╯
```

**Inspect a single module in detail:**

```
$ ilum module show sql
╭─ Module: sql ────────────────────────────────────────────────────────╮
│ sql (sql)                                                            │
│   Ilum SQL engine (Kyuubi)                                           │
│                                                                      │
│   Default enabled: no                                                │
│   Enable flags:  ilum-sql.enabled=true, ilum-core.sql.enabled=true   │
│   Disable flags: ilum-sql.enabled=false, ilum-core.sql.enabled=false │
│   Requires: postgresql, core                                         │
│                                                                      │
│   Resolved enable chain (including dependencies):                    │
│     --set postgresql.enabled=true                                    │
│     --set ilum-core.enabled=true                                     │
│     --set ilum-sql.enabled=true                                      │
│     --set ilum-core.sql.enabled=true                                 │
╰──────────────────────────────────────────────────────────────────────╯
```

The "Resolved enable chain" shows every `--set` flag needed, including transitive dependencies. This is the exact set of flags the CLI generates when you run `ilum module enable sql`.

### 3.2 Enabling Modules

The `ilum module enable` command upgrades a running Helm release to turn on one or more modules. It computes a values diff, shows a preview, and asks for confirmation.

```
$ ilum module enable sql --yes
╭─ Enable Summary ─────────────────────────────────────────╮
│ Action     Enable                                        │
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Modules    sql                                           │
╰──────────────────────────────────────────────────────────╯

ℹ Auto-resolved dependencies: postgresql, core

╭─ Values Diff ────────────────────────────────────────────╮
│ Key                          Before    After              │
├──────────────────────────────────────────────────────────┤
│ ilum-sql.enabled             false     true               │
│ ilum-core.sql.enabled        false     true               │
│ postgresql.enabled           true      true               │
│ ilum-core.enabled            true      true               │
╰──────────────────────────────────────────────────────────╯

  Command: helm upgrade ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --reuse-values \
    --set ilum-sql.enabled=true \
    --set ilum-core.sql.enabled=true \
    --set postgresql.enabled=true \
    --set ilum-core.enabled=true

⠋ Enabling modules...
✓ Enabled: sql
```

Compare this to the manual Helm equivalent, which requires you to know all four `--set` flags and their exact keys:

```
$ helm upgrade ilum ilum/ilum --reuse-values \
    --set ilum-sql.enabled=true \
    --set ilum-core.sql.enabled=true \
    --set postgresql.enabled=true \
    --set ilum-core.enabled=true
```

The CLI computes these for you, including transitive dependencies.

You can enable multiple modules at once:

```
$ ilum module enable airflow superset --yes
```

Use `--dry-run` to preview without applying:

```
$ ilum module enable sql --dry-run
```

### 3.3 Understanding Dependency Resolution

Every module declares its dependencies and conflicts in the module registry. When you enable a module, the CLI walks the dependency graph and automatically includes everything that is needed.

**Example: langfuse requires postgresql and clickhouse.**

When you run `ilum module enable langfuse`, the resolver detects two transitive dependencies:

```
langfuse
├── requires: postgresql
└── requires: clickhouse
```

If `postgresql` is already enabled on the cluster, only `clickhouse` is added. If neither is present, both are pulled in automatically. You see this reported as:

```
ℹ Auto-resolved dependencies: clickhouse, postgresql
```

**Conflict detection: hive-metastore vs. nessie.**

Some modules are mutually exclusive. Both `hive-metastore` and `nessie` configure the Ilum Core metastore, but with different backends (Hive vs. Nessie). They are declared as conflicting:

- `hive-metastore` sets `ilum-core.metastore.type=hive`
- `nessie` sets `ilum-core.metastore.type=nessie`

If you attempt to enable both, the CLI raises a conflict warning:

```
⚠ Module 'hive-metastore' conflicts with 'nessie', but both are enabled
```

You must choose one or the other. The `trino` module depends on `hive-metastore`, so enabling Trino implicitly selects the Hive path.

### 3.4 Disabling Modules

The `ilum module disable` command removes modules from a running release:

```
$ ilum module disable sql --yes
╭─ Disable Summary ────────────────────────────────────────╮
│ Action     Disable                                       │
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Modules    sql                                           │
╰──────────────────────────────────────────────────────────╯

╭─ Values Diff ────────────────────────────────────────────╮
│ Key                          Before    After              │
├──────────────────────────────────────────────────────────┤
│ ilum-sql.enabled             true      false              │
│ ilum-core.sql.enabled        true      false              │
╰──────────────────────────────────────────────────────────╯

  Command: helm upgrade ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --reuse-values \
    --set ilum-sql.enabled=false \
    --set ilum-core.sql.enabled=false

⠋ Disabling modules...
✓ Disabled: sql
```

The CLI is **shared-dependency aware**. When you disable `sql`, its dependency `postgresql` is not disabled because other active modules (such as `gitea`) still require it. The resolver walks the remaining active module set and only disables dependencies that are no longer needed by any other enabled module.

You can disable multiple modules at once:

```
$ ilum module disable airflow superset --yes
```

> **Note:** Attempting to disable a required module (`core`, `ui`, or `mongodb`) triggers a warning. These modules are fundamental to the platform and should generally remain enabled.

### 3.5 Module Categories

The 31 modules are organized into 10 categories. The table below summarizes each category with a count and representative examples.

| Category         | Count | Example Modules                       |
|------------------|-------|---------------------------------------|
| core             | 3     | core, ui, livy-proxy                  |
| infrastructure   | 5     | mongodb, kafka, postgresql, clickhouse, gitea |
| storage          | 1     | minio                                 |
| notebook         | 3     | jupyter, jupyterhub, zeppelin         |
| sql              | 5     | sql, hive-metastore, nessie, unity-catalog, trino |
| orchestration    | 5     | airflow, kestra, n8n, nifi, mageai    |
| ai               | 2     | mlflow, langfuse                      |
| analytics        | 3     | superset, streamlit, marquez          |
| monitoring       | 3     | monitoring, loki, graphite            |
| security         | 1     | openldap                              |

You can filter any `ilum module` command by category. For example, to see live status of all orchestration modules on the cluster:

```
$ ilum module status --category orchestration
╭─ Module Status (release: ilum, namespace: default) ──────────────────╮
│ Module     Category        Status     Pods   Health                  │
├──────────────────────────────────────────────────────────────────────┤
│ airflow    orchestration   Enabled    3/3    Healthy                 │
│ kestra     orchestration   Disabled   --     --                      │
│ n8n        orchestration   Disabled   --     --                      │
│ nifi       orchestration   Disabled   --     --                      │
│ mageai     orchestration   Disabled   --     --                      │
╰──────────────────────────────────────────────────────────────────────╯
```

For the full list of module flags, dependency chains, and conflict maps, see [reference.md](reference.md).

---

## 4. Day-2 Operations

Once your Ilum deployment is running, you will eventually need to upgrade it, inspect logs, diagnose problems, and recover from failures. This section covers the operational commands you use after the initial install.

### 4.1 Upgrading Ilum

The `ilum upgrade` command upgrades an existing Helm release to a new chart version, applies value changes, or both. Under the hood it runs an 8-step pipeline:

1. **Resolve profile defaults** -- reads the active profile for release name, namespace, and context.
2. **Ensure Helm repo** -- adds and updates the `ilum` chart repo if not already configured.
3. **Detect stuck release** -- checks for `pending-install`, `pending-upgrade`, or `pending-rollback` status.
4. **Plan upgrade** -- fetches live values, detects drift, resolves module flags, and computes a values diff.
5. **Resolve versions** -- determines the current and target chart versions.
6. **Check breaking changes** -- scans the known breaking changes registry for anything between your current and target versions.
7. **Show summary and confirm** -- presents the upgrade summary, values diff, breaking change warnings, and the exact Helm command.
8. **Execute** -- runs `helm upgrade` with `--reuse-values` and saves a values snapshot for drift detection.

**Basic upgrade to the latest chart version:**

```
$ ilum upgrade --yes
╭─ Upgrade Summary ───────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    6.7.0 → 6.8.0                                │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

ℹ No value changes.
ℹ Upgrade notes: https://ilum.cloud/docs/upgrade-notes/

  Command: helm upgrade ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --atomic \
    --reuse-values

⠋ Upgrading Ilum...
✓ Ilum upgraded successfully (release: ilum).
```

**Pin a specific target version with `--version`:**

```
$ ilum upgrade --version 6.7.1 --yes
```

**Add modules during the upgrade with `-m`:**

```
$ ilum upgrade -m airflow --version 6.8.0 --yes
```

When you pass `-m`, the upgrade resolves that module's dependencies and merges them into the `--set` flags alongside the version bump.

**Supply custom values files or individual overrides:**

```
$ ilum upgrade -f production-values.yaml --set ilum-core.replicaCount=3 --yes
```

**Preview the upgrade without applying it:**

```
$ ilum upgrade --version 6.8.0 --dry-run
╭─ Upgrade Summary ───────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    6.7.0 → 6.8.0                                │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

ℹ No value changes.
ℹ Upgrade notes: https://ilum.cloud/docs/upgrade-notes/

  Command: helm upgrade ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --atomic \
    --reuse-values \
    --version 6.8.0

ℹ Dry-run mode — no changes applied.
```

**Values safety.** Every upgrade follows a fetch-merge-diff-apply pipeline. The CLI fetches the live user-supplied values from the running release, detects any external drift (for example, someone ran a manual `helm upgrade` outside the CLI), computes a diff of the intended changes, and saves a snapshot after execution. If drift is detected, you see a warning:

```
⚠ External changes detected since last CLI operation:
╭─ External Changes (Drift) ──────────────────────────────╮
│ Key                          Before    After              │
├──────────────────────────────────────────────────────────┤
│ ilum-core.replicaCount       1         3                  │
╰──────────────────────────────────────────────────────────╯
ℹ These external changes will be preserved. CLI changes will be applied on top.
```

If nothing has changed -- same chart version, no value changes, and no new modules -- the CLI exits early with a success message instead of running a no-op upgrade:

```
✓ Already on version 6.7.0 with no value changes. Nothing to upgrade.
```

See [reference.md](reference.md) for the complete list of `ilum upgrade` flags.

### 4.2 Breaking Change Warnings

When upgrading across minor versions, the CLI checks a built-in registry of known breaking changes and surfaces warnings before you confirm. This works whether you specify `--version` explicitly or let the CLI resolve the latest version automatically.

**Example: upgrading from 6.5.x to 6.7.x crosses two known breaking changes.**

```
$ ilum upgrade --version 6.7.0
╭─ Upgrade Summary ───────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    6.5.2 → 6.7.0                                │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

⚠ Breaking change (6.5 -> 6.6): MongoDB chart upgraded from bitnami/mongodb 13.x to 14.x — auth key format changed.
  Hint: Back up MongoDB data before upgrading.
⚠ Breaking change (6.6 -> 6.7): Langfuse requires custom Docker image with NEXT_PUBLIC_BASE_PATH baked in.
  Hint: Build custom Langfuse image or disable Langfuse module.
⚠ Upgrading across minor versions (6.5.2 → 6.7.0). Review upgrade notes: https://ilum.cloud/docs/upgrade-notes/
ℹ Upgrade notes: https://ilum.cloud/docs/upgrade-notes/

? Proceed with upgrade? [y/N]
```

Each warning includes:
- The version range where the break was introduced.
- A description of what changed.
- A migration hint with a concrete action to take.

The warnings are informational -- you can still proceed with the upgrade after reviewing them. In non-interactive mode (`--yes`), the warnings are printed but the upgrade proceeds automatically. In CI/CD pipelines, use `--dry-run` first to surface these warnings without applying any changes.

### 4.3 Viewing Logs

The `ilum logs` command streams logs from a module's pod without requiring you to look up pod names or label selectors. You specify the module name, and the CLI resolves the correct pod and container automatically.

**Show the last 100 lines (default) from the core module:**

```
$ ilum logs core
2026-02-14 09:45:12.234  INFO 1 --- [main] i.c.IlumCoreApplication : Started IlumCoreApplication in 12.3 seconds
2026-02-14 09:45:12.456  INFO 1 --- [main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 15 endpoints
2026-02-14 09:45:13.789  INFO 1 --- [scheduling-1] i.c.s.SparkSessionManager : Initializing Spark session pool
```

**Follow logs in real time (`--follow` / `-f`):**

```
$ ilum logs core --follow
```

Press `Ctrl+C` to stop streaming. The CLI exits gracefully without an error.

**Control the number of lines with `--tail`:**

```
$ ilum logs jupyter --tail 50
```

Pass `--tail 0` to retrieve the full log history (all lines):

```
$ ilum logs core --tail 0
```

**Select a specific container with `--container` / `-c`:**

When a pod has multiple containers (for example, sidecars or init containers), the CLI lists them:

```
$ ilum logs airflow
ℹ Pod has multiple containers: airflow-webserver, airflow-scheduler. Use -c to select one.
```

Then target the one you want:

```
$ ilum logs airflow -c airflow-scheduler --follow
```

If you specify a container that does not exist in the pod, the CLI reports the available containers:

```
✗ Container 'wrong-name' not found in pod 'ilum-airflow-0'.
  Available containers: airflow-webserver, airflow-scheduler
```

**View logs from a previously terminated container with `--previous` / `-p`:**

```
$ ilum logs core --previous --tail 200
```

This is useful for inspecting crash logs from a container that has already restarted.

**Pipe logs to external tools:**

Because `ilum logs` writes to stdout (not Rich-formatted), you can pipe it directly:

```
$ ilum logs core --tail 0 | grep ERROR
$ ilum logs core --follow | tee /tmp/core.log
```

When multiple pods match a module's label selector (for example, a module with `replicaCount > 1`), the CLI automatically picks the first pod and tells you:

```
ℹ Multiple pods found (3), showing logs for 'ilum-core-6f8b4c7d9-xk2pl'.
```

> **Note:** The `ilum logs` command uses the module's `pod_label` from the module registry to find pods. Modules without a `pod_label` (such as infrastructure modules that do not own a distinct pod) will report an error. In those cases, use `kubectl logs` directly.

### 4.4 Health Checks

Section [2.9](#29-running-doctor) introduced `ilum doctor` and its full check suite. This section focuses on using doctor for targeted operational checks.

**Run a single check by name with `--check` / `-c`:**

```
$ ilum doctor --check pods
╭─ ilum doctor ────────────────────────────────────────────────────────╮
│ Status   Check   Message                                            │
├──────────────────────────────────────────────────────────────────────┤
│ ✓        pods    All pods are healthy                               │
╰──────────────────────────────────────────────────────────────────────╯
```

The exit code reflects the check result: **0** on pass, **1** on failure. This makes single-check mode useful in scripts:

```
$ ilum doctor --check release && echo "Release is healthy"
```

**Target a specific release and namespace:**

```
$ ilum doctor --release ilum-staging --namespace staging --check pods
```

**Available single checks (13 total):**

| Check                | What it verifies                                       |
|----------------------|--------------------------------------------------------|
| `helm`               | Helm CLI version >= 3.12                               |
| `kubectl`            | kubectl CLI version >= 1.28                            |
| `docker`             | Docker CLI version >= 24.0                             |
| `helm-repo`          | Ilum Helm repo is configured                           |
| `cluster`            | Can connect to the Kubernetes API                      |
| `namespace`          | Target namespace exists                                |
| `pods`               | All pods in the namespace are healthy                  |
| `pvcs`               | All PersistentVolumeClaims are bound                   |
| `rbac`               | CLI has required RBAC permissions                      |
| `release`            | Helm release exists and is in `deployed` state         |
| `compatibility`      | Kubernetes version is compatible with the chart        |
| `service-endpoints`  | Services have backing endpoints (no dangling Services) |
| `health-endpoints`   | Known health endpoints respond (e.g. `/actuator/health`, `/health`) |

The `service-endpoints` check iterates over Services in the namespace and verifies that each has at least one backing pod endpoint. This catches cases where a Deployment has zero ready replicas but the Service still exists. The `health-endpoints` check probes known HTTP health paths for core services (such as `/actuator/health` for Ilum Core, `/health` for Airflow) via pod proxy and reports any that are not responding.

If you pass an unknown check name, the CLI reports the error and exits with code 1:

```
$ ilum doctor --check nonexistent
✗ Unknown check: nonexistent
```

For the complete doctor output format, see [Section 2.9](#29-running-doctor) and [reference.md](reference.md).

### 4.5 Recovering Stuck Releases

A Helm release can get stuck in `pending-install`, `pending-upgrade`, or `pending-rollback` state if a previous operation was interrupted (for example, a network timeout, Ctrl+C during install, or a pod that never became ready). When this happens, subsequent `ilum upgrade` or `ilum module enable` commands fail because Helm refuses to operate on a release in a pending state.

**Detection.** The CLI detects stuck releases automatically at the start of every `ilum upgrade`:

```
$ ilum upgrade --yes
✗ Release 'ilum' is stuck. Use --force-rollback to recover.
```

**Recovery with `--force-rollback`.** Pass the `--force-rollback` flag to roll back the stuck release to the last successful revision before the upgrade proceeds:

```
$ ilum upgrade --force-rollback --yes
⚠ Release 'ilum' is stuck — rolling back.
⠋ Rolling back...
✓ Rollback complete.

╭─ Upgrade Summary ───────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    6.7.0 → 6.7.0                                │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

⠋ Upgrading Ilum...
✓ Ilum upgraded successfully (release: ilum).
```

The recovery flow is:

1. Detect that the release status is one of `pending-install`, `pending-upgrade`, or `pending-rollback`.
2. Run `helm rollback` to restore the last known-good revision.
3. Proceed with the requested upgrade normally.

You can also use `--force-rollback` together with `--dry-run` to test the detection without actually rolling back:

```
$ ilum upgrade --force-rollback --dry-run
```

**Diagnosing the root cause.** After recovering a stuck release, use `ilum doctor --check release` and `ilum logs <module> --previous` to understand what went wrong. The `--previous` flag on `ilum logs` shows output from the terminated container, which typically contains the crash reason.

> **Note:** The `--force-rollback` flag only affects the current upgrade invocation. It does not permanently change any configuration. If the underlying problem persists (for example, insufficient resources causing pods to crash-loop), the next upgrade will also get stuck unless you address the root cause.

### 4.6 Operation History

The CLI maintains an audit log of all operations. Every install, upgrade, enable, disable, and uninstall is recorded with a timestamp, operation type, release name, and outcome.

```
$ ilum history
╭─ Operation History ──────────────────────────────────────────────────────╮
│ Time                  Operation   Release   Status                       │
├──────────────────────────────────────────────────────────────────────────┤
│ 2026-02-14 10:30:12   install     ilum      success                     │
│ 2026-02-14 11:15:44   enable      ilum      success                     │
│ 2026-02-14 14:22:01   upgrade     ilum      success                     │
╰──────────────────────────────────────────────────────────────────────────╯
```

**Show only the last N operations:**

```
$ ilum history --last 5
```

**Filter by operation type:**

```
$ ilum history --operation upgrade
```

**Machine-readable output for CI/CD:**

```
$ ilum history --last 5 --output json
```

### 4.7 Inspecting Values

The `ilum values` command lets you view, filter, and export the live Helm values from a running release. It replaces manual `helm get values` calls with dot-path filtering, revision pinning, diff mode, and YAML export.

**Show all user-supplied values:**

```
$ ilum values
╭─ Values ─────────────────────────────────────────────────────────────╮
│ ilum-core:                                                           │
│   enabled: true                                                      │
│   replicaCount: 1                                                    │
│ ilum-ui:                                                             │
│   enabled: true                                                      │
│ mongodb:                                                             │
│   enabled: true                                                      │
│ ...                                                                  │
╰──────────────────────────────────────────────────────────────────────╯
```

**Filter with a dot-notation path:**

```
$ ilum values --path ilum-core
╭─ Values ─────────────────────────────────────────────────────────────╮
│ enabled: true                                                        │
│ replicaCount: 1                                                      │
│ sql:                                                                 │
│   enabled: false                                                     │
│ ...                                                                  │
╰──────────────────────────────────────────────────────────────────────╯
```

If the path does not exist, the CLI reports the error:

```
$ ilum values --path nonexistent.key
✗ Key path 'nonexistent.key' not found in values.
```

**Include chart defaults (computed values) with `--all`:**

```
$ ilum values --all
```

By default, `ilum values` shows only user-supplied values (the equivalent of `helm get values`). With `--all`, it shows the fully computed values including chart defaults (the equivalent of `helm get values --all`).

**View values at a specific revision with `--revision`:**

```
$ ilum values --revision 3
╭─ Values (revision 3) ───────────────────────────────────────────────╮
│ ilum-core:                                                           │
│   enabled: true                                                      │
│ ...                                                                  │
╰──────────────────────────────────────────────────────────────────────╯
```

This is useful for inspecting what values were active at a previous revision before deciding whether to roll back.

**Diff mode with `--diff`:**

```
$ ilum values --diff
╭─ User-Supplied vs Computed Values ──────────────────────────────────╮
│ Key                          User      Computed                      │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-core.replicaCount       1         1                             │
│ ilum-core.image.tag          (absent)  6.7.0                         │
│ ...                                                                  │
╰──────────────────────────────────────────────────────────────────────╯
```

The diff shows which values you have explicitly set versus what the chart provides by default. This helps identify which values are safe to remove from your overrides.

**Export values to a YAML file with `--export`:**

```
$ ilum values --export my-values.yaml
✓ Values exported to my-values.yaml
```

The exported file is round-trip safe YAML suitable for passing back to `ilum upgrade -f my-values.yaml`. Combined with `--all`, this captures the complete computed state:

```
$ ilum values --all --export full-state.yaml
```

### 4.8 Comparing Values

The `ilum diff` command compares values across different sources. While `ilum values --diff` shows user-supplied vs. computed, `ilum diff` gives you fine-grained control over what to compare.

**Compare user-supplied values against chart defaults (the default):**

```
$ ilum diff
╭─ User-Supplied vs Computed Values ──────────────────────────────────╮
│ Key                          Before    After                         │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-core.replicaCount       1         1                             │
│ ...                                                                  │
╰──────────────────────────────────────────────────────────────────────╯
```

**Compare live values against a local YAML file:**

```
$ ilum diff --source file --values-file production-values.yaml
╭─ Live Values vs production-values.yaml ─────────────────────────────╮
│ Key                          Before    After                         │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-core.replicaCount       1         3                             │
│ airflow.enabled              false     true                          │
╰──────────────────────────────────────────────────────────────────────╯
```

This is useful for previewing what a `helm upgrade -f production-values.yaml` would change.

**Compare a specific revision against current values:**

```
$ ilum diff --source revision --revision 2
╭─ Revision 2 vs Current ────────────────────────────────────────────╮
│ Key                          Before    After                        │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-sql.enabled             false     true                         │
│ ilum-core.sql.enabled        false     true                         │
╰──────────────────────────────────────────────────────────────────────╯
```

**Compare the last CLI snapshot against live values (drift detection):**

```
$ ilum diff --source snapshot
╭─ Last CLI Snapshot vs Live Values ──────────────────────────────────╮
│ Key                          Before    After                         │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-core.replicaCount       1         3                             │
╰──────────────────────────────────────────────────────────────────────╯
```

If no snapshot exists (no previous CLI operation has been recorded), the command prints a message and exits cleanly:

```
⚠ No CLI snapshot found. Run an operation first to create one.
```

**Filter the diff to a specific path:**

```
$ ilum diff --source file --values-file prod.yaml --path ilum-core
```

The four valid sources are: `defaults` (default), `file`, `revision`, and `snapshot`.

### 4.9 Rolling Back

The `ilum rollback` command provides an explicit rollback to a previous Helm revision. Unlike the `--force-rollback` flag on `ilum upgrade` (which is designed for stuck releases), `ilum rollback` is a standalone operation with a values diff preview, confirmation prompt, and dry-run mode.

**Roll back to the previous revision:**

```
$ ilum rollback --yes
╭─ Rollback Plan ──────────────────────────────────────────────────────╮
│ Current revision     3                                               │
│ Current status       deployed                                        │
│ Current chart        ilum-6.8.0                                      │
│ Current deployed     2026-02-15 14:22:01                             │
│                                                                      │
│ Target revision      2                                               │
│ Target status        superseded                                      │
│ Target chart         ilum-6.7.0                                      │
│ Target deployed      2026-02-14 11:15:44                             │
╰──────────────────────────────────────────────────────────────────────╯

╭─ Values Changes (current vs target) ───────────────────────────────╮
│ Key                          Before    After                        │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-sql.enabled             true      false                        │
│ ilum-core.sql.enabled        true      false                        │
╰──────────────────────────────────────────────────────────────────────╯

⠋ Rolling back to revision 2...
✓ Rolled back 'ilum' to revision 2.
```

**Roll back to a specific revision with `--revision`:**

```
$ ilum rollback --revision 1 --yes
```

By default (`--revision 0`), the command targets the immediately previous revision.

**Preview a rollback with `--dry-run`:**

```
$ ilum rollback --dry-run
╭─ Rollback Plan ──────────────────────────────────────────────────────╮
│ Current revision     3                                               │
│ Current status       deployed                                        │
│ ...                                                                  │
│ Target revision      2                                               │
│ Target status        superseded                                      │
│ ...                                                                  │
╰──────────────────────────────────────────────────────────────────────╯

╭─ Values Changes (current vs target) ───────────────────────────────╮
│ Key                          Before    After                        │
├──────────────────────────────────────────────────────────────────────┤
│ ilum-sql.enabled             true      false                        │
╰──────────────────────────────────────────────────────────────────────╯

ℹ Dry run — no changes applied.
```

After a successful rollback, the CLI updates the values snapshot and syncs the module configuration to reflect the rolled-back state. This means subsequent `ilum status` and `ilum upgrade` commands see the correct module list without manual intervention.

If only one revision exists (the initial install), the command reports the error:

```
✗ Only one revision exists. Nothing to roll back to.
```

### 4.10 Shelling into Pods

The `ilum exec` command opens an interactive shell in a module's pod. It resolves the module name to the correct pod automatically, handles multi-pod scenarios with interactive selection, and falls back from bash to sh if bash is not available in the container.

**Open a shell in the core module:**

```
$ ilum exec core
root@ilum-core-6f8b4c7d9-xk2pl:/#
```

The CLI finds the pod matching the module's `pod_label`, connects with `kubectl exec -it`, and defaults to `/bin/bash`. If bash is not available, it automatically falls back to `/bin/sh`:

```
⚠ bash not available, falling back to /bin/sh
$ ilum exec minio
/ #
```

**Specify a different shell with `--shell`:**

```
$ ilum exec core --shell /bin/sh
```

**Run a one-off command instead of an interactive shell:**

```
$ ilum exec core --command "cat /opt/ilum/conf/application.conf"
```

When `--command` is used, the CLI runs the command non-interactively and returns the exit code.

**Select a specific container with `--container`:**

```
$ ilum exec airflow --container airflow-scheduler
```

**Multi-pod selection.** When a module has multiple running pods (for example, `replicaCount > 1`), the CLI presents an interactive selection menu:

```
$ ilum exec core
? Multiple pods for 'core'. Select one:
  ilum-core-6f8b4c7d9-xk2pl
> ilum-core-6f8b4c7d9-ab3cd
  ilum-core-6f8b4c7d9-ef7gh
```

In non-interactive mode, use `--pod` to specify the pod directly:

```
$ ilum exec core --pod ilum-core-6f8b4c7d9-ab3cd
```

**Health warning.** If the target pod has a high restart count (more than 5), the CLI warns you before connecting:

```
⚠ Pod 'ilum-core-6f8b4c7d9-xk2pl' has 12 restarts. It may be unstable.
```

> **Note:** Modules without a `pod_label` (infrastructure modules that do not own a distinct pod) report an error. In those cases, use `kubectl exec` directly.

### 4.11 Resource Usage

The `ilum top` command shows per-module CPU and memory usage. It queries the Kubernetes metrics API (provided by metrics-server) and aggregates resource consumption by module, similar to `kubectl top pods` but organized by Ilum module.

**Show resource usage for all modules:**

```
$ ilum top
╭─ Resource Usage ─────────────────────────────────────────────────────────────────╮
│ Module       Pods   CPU Used   CPU Req   CPU %   Mem Used   Mem Req   Mem %      │
├──────────────────────────────────────────────────────────────────────────────────┤
│ core         1      250m       500m      50%     512Mi      1Gi       50%        │
│ gitea        1      50m        100m      50%     128Mi      256Mi     50%        │
│ jupyter      1      100m       200m      50%     256Mi      512Mi     50%        │
│ minio        1      30m        100m      30%     64Mi       256Mi     25%        │
│ mongodb      1      80m        250m      32%     192Mi      512Mi     38%        │
│ postgresql   1      40m        100m      40%     96Mi       256Mi     38%        │
│ ui           1      20m        50m       40%     48Mi       128Mi     38%        │
│ TOTAL        7      570m       1.3       44%     1.3Gi      2.9Gi    44%         │
╰──────────────────────────────────────────────────────────────────────────────────╯
```

The table shows actual usage versus requested resources and the utilization percentage. The TOTAL row aggregates across all modules.

**Filter to a single module:**

```
$ ilum top core
╭─ Resource Usage ─────────────────────────────────────────────────────────────────╮
│ Module   Pods   CPU Used   CPU Req   CPU %   Mem Used   Mem Req   Mem %          │
├──────────────────────────────────────────────────────────────────────────────────┤
│ core     1      250m       500m      50%     512Mi      1Gi       50%            │
│ TOTAL    1      250m       500m      50%     512Mi      1Gi       50%            │
╰──────────────────────────────────────────────────────────────────────────────────╯
```

**Sort by CPU or memory usage:**

```
$ ilum top --sort-by cpu       # Highest CPU usage first
$ ilum top --sort-by memory    # Highest memory usage first
```

The default sort order is by module name.

**Watch mode with `--watch`:**

```
$ ilum top --watch
```

Watch mode continuously refreshes the table at a configurable interval (default: 5 seconds). Press `Ctrl+C` to stop. Adjust the interval with `--interval`:

```
$ ilum top --watch --interval 10
```

**Metrics-server requirement.** The `ilum top` command requires the Kubernetes metrics-server to be installed on the cluster. If the metrics API is not available, the CLI reports the error with an install command:

```
✗ Metrics API not available. Install metrics-server:
    kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
```

> **Note:** The `ilum top` command maps pods to modules using the `pod_label` from the module registry. Pods that do not match any known module label are grouped under `(other)`.

---

## 5. Managing Configuration

The Ilum CLI stores all of its state in a YAML configuration file. On Linux/macOS this is `~/.config/ilum/config.yaml`; on Windows it is `%APPDATA%\ilum\config.yaml`. Run `ilum config path` to see the exact location on your system. This file holds profile definitions, cluster records, and the active profile pointer. You never need to edit it by hand -- the `ilum config` sub-commands and `ilum init` wizard handle everything -- but the format is straightforward YAML if you want to inspect or version-control it.

### 5.1 Profiles

Profiles let you manage multiple Ilum environments (dev, staging, production) from a single machine. Each profile stores a release name, Kubernetes context, namespace, chart version, enabled modules, and communication settings.

**The profile data structure:**

```yaml
profiles:
  default:
    name: default
    release_name: ilum
    cluster:
      kubecontext: k3d-ilum-dev
      namespace: default
      chart_version: "6.7.0"
      chart_repo: https://charts.ilum.cloud
    enabled_modules:
      - core
      - gitea
      - jupyter
      - minio
      - mongodb
      - postgresql
      - ui
    communication_type: grpc
    auth_type: internal
    custom_values_path: ""
```

**Create a new profile using the init wizard:**

```
$ ilum init --profile staging
```

The wizard walks you through context selection, namespace, module selection, and saves a new profile named `staging`. The newly created profile automatically becomes the active profile.

**Switch between profiles:**

```
$ ilum config set active_profile staging
✓ active_profile = staging
```

After switching, all commands (`ilum status`, `ilum upgrade`, `ilum logs`, etc.) operate against the staging release without needing extra flags.

**View the active profile:**

```
$ ilum config show active_profile
staging
```

**View all profiles in the full configuration:**

```
$ ilum config show
╭─ ilum config ────────────────────────────────────────────╮
│ active_profile: staging                                   │
│ profiles:                                                 │
│   default:                                                │
│     name: default                                         │
│     release_name: ilum                                    │
│     cluster:                                              │
│       kubecontext: k3d-ilum-dev                           │
│       namespace: default                                  │
│       chart_version: '6.7.0'                              │
│       chart_repo: https://charts.ilum.cloud               │
│     enabled_modules:                                      │
│       - core                                              │
│       - gitea                                             │
│       - jupyter                                           │
│       - minio                                             │
│       - mongodb                                           │
│       - postgresql                                        │
│       - ui                                                │
│     communication_type: grpc                              │
│     auth_type: internal                                   │
│     custom_values_path: ''                                │
│   staging:                                                │
│     name: staging                                         │
│     release_name: ilum-staging                            │
│     cluster:                                              │
│       kubecontext: arn:aws:eks:eu-central-1:123456:...    │
│       namespace: staging                                  │
│       chart_version: '6.7.0'                              │
│       chart_repo: https://charts.ilum.cloud               │
│     enabled_modules:                                      │
│       - core                                              │
│       - mongodb                                           │
│       - postgresql                                        │
│       - ui                                                │
│     communication_type: grpc                              │
│     auth_type: internal                                   │
│     custom_values_path: ''                                │
│ clusters: []                                              │
╰──────────────────────────────────────────────────────────╯
```

**Override the profile on any command** by passing `--release`, `--namespace`, and `--context` flags directly. These flags take precedence over the active profile:

```
$ ilum status --release ilum-staging --namespace staging --context my-eks-cluster
```

### 5.2 Profile Management

Beyond creating profiles with `ilum init` and switching with `config set`, the CLI provides dedicated commands for managing profiles:

**List all profiles:**

```
$ ilum config list-profiles
```

**Switch the active profile:**

```
$ ilum config use staging
✓ Switched to profile 'staging'.
```

**Export a profile for team sharing:**

```
$ ilum config export production --out prod-profile.yaml
✓ Profile 'production' exported to prod-profile.yaml
```

**Import a shared profile:**

```
$ ilum config import prod-profile.yaml --name production
✓ Profile 'production' imported.
```

**Validate the configuration file:**

```
$ ilum config validate
✓ Configuration is valid.
```

**Back up the configuration:**

```
$ ilum config backup
✓ Backup saved to /home/user/.config/ilum/config.yaml.bak
```

Export and import are useful for onboarding team members: one person configures the environment, exports the profile, and others import it to get identical settings without running through the wizard.

### 5.3 Config Get / Set / Edit

The `ilum config` sub-commands give you programmatic and interactive access to the configuration file.

**Show the entire config:**

```
$ ilum config show
```

**Read a specific key using dot-notation:**

```
$ ilum config show active_profile
default

$ ilum config show profiles.default.cluster.namespace
default

$ ilum config show profiles.staging.enabled_modules
['core', 'mongodb', 'postgresql', 'ui']
```

If the key does not exist, the CLI exits with code 1:

```
$ ilum config show nonexistent.key
✗ Key not found: nonexistent.key
```

**Set a specific key:**

```
$ ilum config set active_profile production
✓ active_profile = production

$ ilum config set profiles.default.cluster.namespace ilum-system
✓ profiles.default.cluster.namespace = ilum-system
```

The set command validates the new value against the Pydantic configuration model. If the value is invalid for the field type, you get an error with a suggestion.

**Print the config file path:**

```
$ ilum config path
/home/user/.config/ilum/config.yaml
```

This is useful in scripts that need to back up or version-control the config file:

```
$ cp "$(ilum config path)" ~/ilum-config-backup.yaml
```

**Create a default config file:**

```
$ ilum config init
✓ Created config: /home/user/.config/ilum/config.yaml
```

If a config file already exists, the command warns you and does nothing. Pass `--force` to overwrite:

```
$ ilum config init --force
✓ Created config: /home/user/.config/ilum/config.yaml
```

**Open the config in your editor:**

```
$ ilum config edit
```

This opens the config file in `$EDITOR` (falling back to `$VISUAL`, then `vi` on Linux/macOS or `notepad` on Windows). After you save and close the editor, the changes take effect immediately -- there is no separate "apply" step.

> **Note:** The configuration file uses platform-native file locking (`fcntl` on Linux/macOS, `msvcrt` on Windows) to prevent concurrent writes from corrupting the YAML. This means you can safely run CLI commands in parallel (for example, from multiple terminal tabs or CI jobs targeting different profiles) without worrying about write races.

### 5.4 Connecting to Existing Installations

If you already have Ilum running on a cluster (deployed manually with `helm install` or by another team member), the `ilum connect` command imports that release into the CLI's configuration without reinstalling anything.

**Auto-detect mode (recommended).** Run `ilum connect` without arguments to scan all namespaces for Ilum Helm releases:

```
$ ilum connect
⠋ Scanning for Ilum releases...
ℹ Found 1 Ilum release:
  Release:   ilum
  Namespace: default
  Version:   6.7.0
  Status:    deployed

? Connect this release to profile 'default'? [Y/n] y
✓ Profile 'default' updated with release 'ilum' in namespace 'default'.
✓ 7 modules synced to config.
✓ Values snapshot saved (drift detection enabled).

ℹ
ℹ Next steps:
ℹ   ilum status          Show release status
ℹ   ilum upgrade         Upgrade to a newer version
ℹ   ilum module enable   Enable additional modules
```

The connect command:

1. Scans for Helm releases whose chart name is `ilum`.
2. Detects which modules are currently enabled by inspecting the computed values.
3. Saves the release, namespace, context, chart version, and enabled modules to a CLI profile.
4. Takes a values snapshot so that future operations can detect drift.

**Multiple releases.** When the scan finds more than one Ilum release, you get a numbered table to choose from:

```
$ ilum connect
⠋ Scanning for Ilum releases...
╭─ Found Ilum Releases ─────────────────────────────────────────╮
│ #   Release        Namespace   Version   Status               │
├────────────────────────────────────────────────────────────────┤
│ 1   ilum           default     6.7.0     deployed             │
│ 2   ilum-staging   staging     6.6.1     deployed             │
╰────────────────────────────────────────────────────────────────╯
? Select release [1-2]: 2
✓ Profile 'default' updated with release 'ilum-staging' in namespace 'staging'.
✓ 4 modules synced to config.
✓ Values snapshot saved (drift detection enabled).
```

**Direct mode.** If you know the exact release name and namespace, skip the scan entirely:

```
$ ilum connect --release ilum-staging --namespace staging --profile staging
✓ Profile 'staging' updated with release 'ilum-staging' in namespace 'staging'.
✓ 4 modules synced to config.
✓ Values snapshot saved (drift detection enabled).
```

**Connect to a specific Kubernetes context:**

```
$ ilum connect --context arn:aws:eks:eu-central-1:123456:cluster/production --profile production
```

**Non-interactive mode for scripts:**

```
$ ilum connect --yes
```

In `--yes` mode, if multiple releases are found, the CLI auto-selects the first one. If a single release is found, it connects without prompting.

**Stuck release warning.** If the discovered release is in a stuck state (`pending-install`, `pending-upgrade`, or `pending-rollback`), the connect command still imports it but prints a warning:

```
⚠ Release 'ilum' is in 'pending-upgrade' state. Run 'ilum upgrade --force-rollback' to recover.
```

> **Note:** The `ilum connect` command is read-only with respect to the cluster. It does not install, upgrade, or modify the Helm release in any way. It only writes to the local CLI configuration file.

---

## 6. CI/CD Automation

Every mutating CLI command (`install`, `upgrade`, `uninstall`, `module enable`, `module disable`) supports the same pair of automation flags: `--yes` to skip confirmation prompts and `--dry-run` to preview without executing. Combined with deterministic exit codes, this makes the CLI straightforward to integrate into CI/CD pipelines.

> **Prerequisites in CI/CD.** The CLI's automatic tool installation (see [Section 1.5](#15-prerequisites-helm-kubectl-docker)) only works in interactive terminals. In CI/CD pipelines, scripts, and non-interactive environments, you **must** pre-install `helm`, `kubectl`, and `docker` before invoking the CLI. If a required tool is missing, the CLI exits with error code `ILUM-001` instead of offering to install it. See the [GitHub Actions workflow](#64-github-actions-example-workflow) below for a complete example that includes the prerequisite install step.

### 6.1 Non-Interactive Mode

**`--yes` / `-y`** skips all confirmation prompts and accepts defaults. Use it when running from a script, cron job, or CI pipeline where there is no human to type "y":

```
$ ilum install --yes
$ ilum upgrade --version 6.8.0 --yes
$ ilum module enable airflow --yes
$ ilum module disable sql --yes
$ ilum uninstall --yes
```

**`--dry-run`** previews the Helm command and values diff without executing anything. The command prints exactly what it would do and exits with code 0:

```
$ ilum install --dry-run
╭─ Install Summary ────────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    latest                                        │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

  Command: helm install ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --atomic \
    --set ilum-core.enabled=true \
    --set ilum-ui.enabled=true \
    --set mongodb.enabled=true \
    --set postgresql.enabled=true \
    --set minio.enabled=true \
    --set ilum-jupyter.enabled=true \
    --set gitea.enabled=true

ℹ Dry-run mode — no changes applied.
```

The same `--dry-run` flag works on `upgrade`, `uninstall`, `module enable`, and `module disable`:

```
$ ilum upgrade --version 6.8.0 --dry-run
$ ilum uninstall --dry-run
$ ilum module enable sql --dry-run
$ ilum module disable airflow --dry-run
```

> **Note:** `--dry-run` and `--yes` can be combined, but `--dry-run` always takes precedence. When both are present the command previews the change and exits without executing, regardless of `--yes`.

### 6.2 Machine-Readable Output

Use `--output json`, `--output yaml`, or `--output csv` for CI/CD-friendly output:

```bash
# JSON output for status
ilum status --output json

# Module list as YAML
ilum module list --output yaml

# Doctor results as JSON (failures only)
ilum doctor --output json --failures-only

# Resource usage as CSV for spreadsheet import
ilum top --output csv

# Quiet mode (suppress informational messages, JSON still emits)
ilum --quiet --output json module list
```

Available on: `status`, `module list`, `module show`, `module status`, `doctor`, `history`, `values`, `diff`, `rollback`, `top`.

The `--output` flag emits structured data to stdout while suppressing Rich formatting. Combined with `--quiet`, only the data payload is written -- no banners, spinners, or informational messages. This makes the output safe to pipe directly into `jq`, `yq`, or other processing tools:

```bash
# Extract pod names from status JSON
ilum status --output json | jq '.pods[].name'

# Count failing doctor checks
ilum doctor --output json | jq '[.checks[] | select(.status == "FAIL")] | length'
```

**CSV output with `--no-headers` and `--field-selector`.** For shell scripts and CI pipelines that process tabular data with standard Unix tools, the CSV output format is often the most convenient. Combine it with `--no-headers` to omit the header row and `--field-selector` to select specific columns:

```bash
# CSV output with all columns
$ ilum top --output csv
module,pods,cpu_used,cpu_req,cpu_pct,mem_used,mem_req,mem_pct
core,1,250m,500m,50%,512Mi,1Gi,50%
mongodb,1,80m,250m,32%,192Mi,512Mi,38%
...

# No header row — useful for piping to awk/cut
$ ilum top --output csv --no-headers
core,1,250m,500m,50%,512Mi,1Gi,50%
mongodb,1,80m,250m,32%,192Mi,512Mi,38%
...

# Select only specific fields
$ ilum top --output csv --field-selector module,cpu_used,mem_used
module,cpu_used,mem_used
core,250m,512Mi
mongodb,80m,192Mi
...

# Combine for minimal output suitable for parsing
$ ilum status --output csv --no-headers --field-selector name,phase
ilum-core-6f8b4c7d9-xk2pl,Running
ilum-ui-5c9d3a1b8-mn4qr,Running
...
```

The `--no-headers` and `--field-selector` flags work with all output formats (`csv`, `json`, `yaml`) and across all commands that support `--output`.

### 6.3 Scripting Patterns

**Exit codes.** The CLI uses two exit codes:

| Exit Code | Meaning                                    |
|-----------|--------------------------------------------|
| `0`       | Success (or dry-run preview completed)     |
| `1`       | Failure (Helm error, validation error, user abort) |

All error handling follows the same pattern: the CLI catches `IlumError` exceptions, prints the error message and suggestion, then exits with code 1. This makes it safe to use standard shell error handling:

```bash
#!/bin/bash
set -euo pipefail

# Deploy and fail the script if install fails
ilum install --yes

# Check health after install
ilum doctor --check pods
ilum doctor --check release

echo "Deployment verified."
```

**Combining with `ilum doctor` for gating:**

```bash
# Gate a deployment on health checks
if ! ilum doctor --check cluster; then
    echo "Cluster unreachable, aborting deploy" >&2
    exit 1
fi

ilum upgrade --version "$TARGET_VERSION" --yes
```

**Piping logs to files or monitoring tools:**

```bash
# Capture recent logs for a post-deploy report
ilum logs core --tail 200 > /tmp/core-deploy.log

# Stream logs to a monitoring pipeline
ilum logs core --follow | your-log-forwarder --source ilum-core
```

**Reading config values in scripts:**

```bash
# Get the active profile's namespace
NS=$(ilum config show profiles.$(ilum config show active_profile).cluster.namespace)
echo "Deploying to namespace: $NS"

# Back up the config before modifying
cp "$(ilum config path)" "$(ilum config path).bak"
```

### 6.4 GitHub Actions Example Workflow

The following workflow demonstrates a complete CI/CD pipeline: dry-run validation on pull requests, deployment on merge to `main`, and tag-based upgrades for releases.

```yaml
name: Ilum Deploy

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
    tags: ["v*"]

env:
  ILUM_RELEASE: ilum
  ILUM_NAMESPACE: ilum-system

jobs:
  # ── Dry-run on PRs ──────────────────────────────────────
  validate:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install prerequisites (helm, kubectl)
        run: |
          curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
          curl -fsSLo /tmp/kubectl "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          chmod +x /tmp/kubectl && sudo mv /tmp/kubectl /usr/local/bin/kubectl

      - name: Install Ilum CLI
        run: pip install ilum

      - name: Configure kubeconfig
        uses: azure/k8s-set-context@v4
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Dry-run install
        run: |
          ilum install \
            --release $ILUM_RELEASE \
            --namespace $ILUM_NAMESPACE \
            --dry-run

      - name: Run doctor checks
        run: |
          ilum doctor \
            --release $ILUM_RELEASE \
            --namespace $ILUM_NAMESPACE

  # ── Deploy on merge to main ─────────────────────────────
  deploy:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install prerequisites (helm, kubectl)
        run: |
          curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
          curl -fsSLo /tmp/kubectl "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          chmod +x /tmp/kubectl && sudo mv /tmp/kubectl /usr/local/bin/kubectl

      - name: Install Ilum CLI
        run: pip install ilum

      - name: Configure kubeconfig
        uses: azure/k8s-set-context@v4
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Deploy Ilum
        run: |
          ilum install \
            --release $ILUM_RELEASE \
            --namespace $ILUM_NAMESPACE \
            --yes

      - name: Verify deployment
        run: |
          ilum doctor \
            --release $ILUM_RELEASE \
            --namespace $ILUM_NAMESPACE \
            --check release
          ilum doctor \
            --release $ILUM_RELEASE \
            --namespace $ILUM_NAMESPACE \
            --check pods

  # ── Upgrade on tag push ─────────────────────────────────
  upgrade:
    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install prerequisites (helm, kubectl)
        run: |
          curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
          curl -fsSLo /tmp/kubectl "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          chmod +x /tmp/kubectl && sudo mv /tmp/kubectl /usr/local/bin/kubectl

      - name: Install Ilum CLI
        run: pip install ilum

      - name: Configure kubeconfig
        uses: azure/k8s-set-context@v4
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Extract version from tag
        id: version
        run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> "$GITHUB_OUTPUT"

      - name: Upgrade Ilum
        run: |
          ilum upgrade \
            --release $ILUM_RELEASE \
            --namespace $ILUM_NAMESPACE \
            --version ${{ steps.version.outputs.VERSION }} \
            --force-rollback \
            --yes

      - name: Post-upgrade health check
        run: |
          ilum doctor \
            --release $ILUM_RELEASE \
            --namespace $ILUM_NAMESPACE

      - name: Capture deployment logs
        if: always()
        run: |
          ilum logs core --tail 100 > /tmp/core-logs.txt || true
          ilum logs ui --tail 100 > /tmp/ui-logs.txt || true

      - name: Upload logs
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: deployment-logs
          path: /tmp/*-logs.txt
```

Key patterns in this workflow:

- **`--dry-run` on PRs** lets reviewers see the exact Helm command and values diff in the CI output before merging.
- **`--yes` on merge** runs the deployment non-interactively.
- **`--force-rollback` on tag upgrades** ensures recovery from any stuck release state before applying the new version.
- **`ilum doctor` after every deploy** catches problems early. Because each check exits with code 1 on failure, a failing check fails the CI job.
- **`ilum logs` with `if: always()`** captures pod logs as artifacts even when a prior step fails, giving you diagnostic data without SSHing into the cluster.

### 6.5 GitOps-Style Upgrades

For teams that manage infrastructure declaratively, the `ilum upgrade` command supports a tag-based deployment pattern where the desired Ilum version is stored in a Git repository and applied by a CI/CD pipeline on every push.

**Store the desired version in a file:**

```
# ilum-version.env
ILUM_VERSION=6.8.0
ILUM_MODULES=core,ui,mongodb,postgresql,minio,jupyter,gitea,sql,airflow
```

**Apply it in CI:**

```bash
#!/bin/bash
set -euo pipefail

source ilum-version.env

# Convert comma-separated modules to repeated -m flags
MODULE_FLAGS=""
IFS=',' read -ra MODS <<< "$ILUM_MODULES"
for mod in "${MODS[@]}"; do
    MODULE_FLAGS="$MODULE_FLAGS -m $mod"
done

# Dry-run first to surface breaking changes
ilum upgrade \
    --version "$ILUM_VERSION" \
    $MODULE_FLAGS \
    --dry-run

# Apply the upgrade
ilum upgrade \
    --version "$ILUM_VERSION" \
    $MODULE_FLAGS \
    --force-rollback \
    --yes

# Verify
ilum doctor --check release
ilum doctor --check pods
```

**Tag-triggered flow.** The pattern from section 6.3 can be adapted for GitOps: push a Git tag like `v6.8.0` to trigger the upgrade job, which extracts the version from the tag and passes it to `ilum upgrade --version`. This gives you an auditable, version-controlled deployment history where every production change maps to a Git tag.

**Multi-environment promotion.** Combine profiles with CI stages to promote versions across environments:

```bash
# Stage 1: Deploy to dev
ilum config set active_profile dev
ilum upgrade --version "$ILUM_VERSION" --yes
ilum doctor

# Stage 2: Promote to staging (after dev passes)
ilum config set active_profile staging
ilum upgrade --version "$ILUM_VERSION" --yes
ilum doctor

# Stage 3: Promote to production (after staging passes)
ilum config set active_profile production
ilum upgrade --version "$ILUM_VERSION" --force-rollback --yes
ilum doctor
```

Each profile points to a different Kubernetes context and namespace, so the same CLI commands work identically across all environments. The `--force-rollback` on production adds an extra safety net for the most critical environment.

---

## 7. Uninstalling

### 7.1 Basic Uninstall

The `ilum uninstall` command removes the Ilum Helm release from your cluster. It shows a summary, asks for confirmation, and then runs `helm uninstall`.

```
$ ilum uninstall
╭─ Uninstall Summary ─────────────────────────────────────╮
│ Release           ilum                                   │
│ Namespace         default                                │
│ Delete PVCs       No                                     │
│ Delete Namespace  No                                     │
╰──────────────────────────────────────────────────────────╯

  Command: helm uninstall ilum --namespace default

? Proceed with uninstall? Yes
⠋ Uninstalling Ilum...
✓ Release 'ilum' uninstalled.
```

By default, uninstall removes the Helm release (all Kubernetes resources it manages -- Deployments, Services, ConfigMaps, Secrets) but preserves PersistentVolumeClaims and the namespace. This means your data volumes survive, so you can reinstall later without data loss.

The command is **idempotent**: if the release does not exist, it exits cleanly with a message rather than an error:

```
$ ilum uninstall
ℹ Release 'ilum' not found — nothing to uninstall.
```

After uninstalling, the CLI clears the tracked module list from your local config. The profile itself remains, so you can run `ilum install` again without re-running `ilum init`.

To skip the confirmation prompt (useful in CI/CD), pass `--yes`:

```
$ ilum uninstall --yes
```

To preview the uninstall command without executing it, use `--dry-run`:

```
$ ilum uninstall --dry-run
╭─ Uninstall Summary ─────────────────────────────────────╮
│ Release           ilum                                   │
│ Namespace         default                                │
│ Delete PVCs       No                                     │
│ Delete Namespace  No                                     │
╰──────────────────────────────────────────────────────────╯

  Command: helm uninstall ilum --namespace default

ℹ Dry-run mode — no changes applied.
```

### 7.2 Cleaning Up PVCs and Namespaces

When you want a complete teardown -- not just the release, but also the persistent data and the namespace -- use the `--delete-data` and `--delete-namespace` flags.

**Delete PVCs after uninstall:**

```
$ ilum uninstall --delete-data --yes
╭─ Uninstall Summary ─────────────────────────────────────╮
│ Release           ilum                                   │
│ Namespace         default                                │
│ Delete PVCs       Yes                                    │
│ Delete Namespace  No                                     │
╰──────────────────────────────────────────────────────────╯

⚠ PersistentVolumeClaims will be DELETED.

  Command: helm uninstall ilum --namespace default

⠋ Uninstalling Ilum...
✓ Release 'ilum' uninstalled.
⠋ Deleting PersistentVolumeClaims...
✓ Deleted 7 PVC(s): data-ilum-mongodb-0, data-ilum-postgresql-0, ...
```

**Delete the namespace as well:**

```
$ ilum uninstall --delete-data --delete-namespace --yes
╭─ Uninstall Summary ─────────────────────────────────────╮
│ Release           ilum                                   │
│ Namespace         ilum-prod                              │
│ Delete PVCs       Yes                                    │
│ Delete Namespace  Yes                                    │
╰──────────────────────────────────────────────────────────╯

⚠ PersistentVolumeClaims will be DELETED.
⚠ Namespace 'ilum-prod' will be DELETED.

  Command: helm uninstall ilum --namespace ilum-prod

⠋ Uninstalling Ilum...
✓ Release 'ilum' uninstalled.
⠋ Deleting PersistentVolumeClaims...
✓ Deleted 7 PVC(s): data-ilum-mongodb-0, data-ilum-postgresql-0, ...
⠋ Deleting namespace 'ilum-prod'...
✓ Namespace 'ilum-prod' deleted.
```

> **Note:** When `--delete-data` or `--delete-namespace` is used **without** `--yes`, the CLI requires a stricter confirmation: you must type the release name exactly to proceed. This guards against accidental data loss.

```
$ ilum uninstall --delete-data
╭─ Uninstall Summary ─────────────────────────────────────╮
│ Release           ilum                                   │
│ Namespace         default                                │
│ Delete PVCs       Yes                                    │
│ Delete Namespace  No                                     │
╰──────────────────────────────────────────────────────────╯

⚠ PersistentVolumeClaims will be DELETED.

  Command: helm uninstall ilum --namespace default

? Type the release name 'ilum' to confirm: ilum
⠋ Uninstalling Ilum...
✓ Release 'ilum' uninstalled.
⠋ Deleting PersistentVolumeClaims...
✓ Deleted 7 PVC(s): data-ilum-mongodb-0, data-ilum-postgresql-0, ...
```

If you mistype the release name, the operation aborts:

```
? Type the release name 'ilum' to confirm: ilm
✗ Release name does not match. Aborted.
```

You can target a specific release and namespace explicitly with `--release` and `--namespace`:

```
$ ilum uninstall --release ilum-staging --namespace staging --delete-data --yes
```

See [reference.md](reference.md) for the complete flag listing.

### 7.3 Full Environment Teardown

For a complete cleanup -- removing Ilum, the local cluster, CLI config, and optionally all dependencies -- use `ilum cleanup`:

```
$ ilum cleanup --all --dry-run
ℹ Tier 1 — Ilum Release
  Would uninstall release "ilum" from namespace "default"
  Would delete 7 PersistentVolumeClaim(s)
  Would delete namespace "default"

ℹ Tier 2 — Local Cluster
  Would delete cluster "ilum-dev" (minikube)

ℹ Tier 3 — CLI Configuration
  ~/.config/ilum (exists, 3 item(s))
  ~/.local/state/ilum (exists, 2 item(s))
  ~/.cache/ilum (does not exist)
  Would remove the above directories.

ℹ Tier 4 — Dependencies
  Would remove: helm (/usr/local/bin/helm) (v3.14.0)
  Would remove: kubectl (/usr/local/bin/kubectl) (v1.30.0)
  Would remove: minikube (/usr/local/bin/minikube) — will delete all clusters first
  Skipping: k3d (not installed)
  Skipping: kind (not installed)
```

The command runs in **tiers**, each opt-in:

| Tier | Flag | What it removes |
|------|------|----------------|
| 1 | *(always)* | Helm release + PVCs + namespace |
| 2 | `--cluster` | CLI-managed local k8s cluster |
| 3 | `--config` | CLI config, state, cache directories |
| 4 | `--deps` | Dependencies: helm, kubectl, minikube, k3d, kind, docker |
| 5 | `--self` | The CLI itself |

Use `--all` as shorthand for `--cluster --config --deps` (does not include `--self`):

```
$ ilum cleanup --all --yes
```

To also uninstall the CLI itself, add `--self` explicitly:

```
$ ilum cleanup --all --self --yes
```

> **Note:** Docker removal requires extra-strong confirmation (type `REMOVE DOCKER`), because removing Docker affects all containers and images on the machine, not just Ilum.

> **Note:** `--self` is never included in `--all` -- it must always be specified explicitly.

> **Reinstalling after cleanup.** After running `ilum cleanup --deps`, you can reinstall the prerequisites by running `ilum init` in an interactive terminal -- the preflight step detects missing tools and offers to install them automatically. Alternatively, install them manually using the commands listed in [Section 1.5](#15-prerequisites-helm-kubectl-docker). In CI/CD environments, you must install prerequisites manually since auto-install requires an interactive terminal.

### 7.4 Removing the CLI

To remove the Ilum CLI itself, uninstall it with the same package manager you used to install it:

**pipx:**

```
$ pipx uninstall ilum
```

**uv:**

```
$ uv tool uninstall ilum
```

**pip:**

```
$ pip uninstall ilum
```

After uninstalling the package, you can optionally remove the CLI's configuration and state directories. Run `ilum config path` before uninstalling to see the exact location on your system.

**Linux / macOS:**

```
$ rm -rf ~/.config/ilum    # Profiles and config
$ rm -rf ~/.local/share/ilum   # Persistent data
$ rm -rf ~/.local/state/ilum   # State (snapshots, drift data)
$ rm -rf ~/.cache/ilum     # Cache (chart downloads)
```

> **Note:** If you have customized `XDG_CONFIG_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME`, or `XDG_CACHE_HOME`, the `ilum` subdirectory will be under those paths instead of the defaults shown above.

**Windows (PowerShell):**

```
PS> Remove-Item -Recurse "$env:APPDATA\ilum"       # Config and data
PS> Remove-Item -Recurse "$env:LOCALAPPDATA\ilum"   # State and cache
```

---

## 8. Recipes

Each recipe below is self-contained. They assume you have the Ilum CLI installed and a Kubernetes cluster accessible via `kubectl`.

### 8.1 Migrating from Manual Helm to CLI

**Context:** You have an existing Ilum installation that was deployed manually with `helm install` or `helm upgrade`. You manage it through raw Helm commands and custom `--set` flags.

**Goal:** You want to bring that installation under CLI management so you can use `ilum module enable`, `ilum upgrade`, `ilum status`, and other CLI commands going forward -- without redeploying or losing state.

**Solution:**

```
$ ilum connect
⠋ Scanning for Ilum releases...
Found 1 Ilum release:
  Release:   ilum
  Namespace: analytics
  Version:   6.6.0
  Status:    deployed

? Connect this release to profile 'default'? Yes
✓ Profile 'default' updated with release 'ilum' in namespace 'analytics'.
✓ 12 modules synced to config.
✓ Values snapshot saved (drift detection enabled).

Next steps:
  ilum status          Show release status
  ilum upgrade         Upgrade to a newer version
  ilum module enable   Enable additional modules
```

The `connect` command auto-detects the release name, namespace, chart version, and currently enabled modules by reading the live Helm values. It saves everything into a CLI profile and takes a values snapshot for future drift detection.

If you have multiple Ilum releases across namespaces, `connect` presents a selection menu:

```
$ ilum connect
⠋ Scanning for Ilum releases...
╭─ Found Ilum Releases ───────────────────────────────────────╮
│ #   Release       Namespace   Version   Status              │
├─────────────────────────────────────────────────────────────┤
│ 1   ilum-dev      dev         6.7.0     deployed            │
│ 2   ilum-staging  staging     6.6.0     deployed            │
│ 3   ilum-prod     production  6.5.0     deployed            │
╰─────────────────────────────────────────────────────────────╯
? Select release [1-3]: 2
✓ Profile 'default' updated with release 'ilum-staging' in namespace 'staging'.
✓ 9 modules synced to config.
✓ Values snapshot saved (drift detection enabled).
```

To connect each release to its own named profile, use `--profile`:

```
$ ilum connect --release ilum-dev --namespace dev --profile dev
$ ilum connect --release ilum-staging --namespace staging --profile staging
$ ilum connect --release ilum-prod --namespace production --profile prod
```

After connecting, all CLI commands work against the active profile. Switch profiles with `ilum config set active_profile <name>`.

### 8.2 Multi-Namespace Production Setup

**Context:** You have a Kubernetes cluster where you need isolated environments for development, staging, and production.

**Goal:** You want to deploy Ilum into three separate namespaces with different module sets and configurations, each managed independently through CLI profiles.

**Solution:**

Create a profile and install for each environment. Use `ilum init --profile` to create each profile, switch to it with `ilum config set active_profile`, then configure and install:

```
$ ilum init --profile dev --yes
$ ilum config set active_profile dev
$ ilum config set cluster.namespace ilum-dev
$ ilum config set release_name ilum-dev
$ ilum install -m jupyter --yes
✓ Namespace 'ilum-dev' does not exist — creating it.
✓ Namespace 'ilum-dev' created.
✓ Ilum installed successfully (release: ilum-dev).
```

```
$ ilum init --profile staging --yes
$ ilum config set active_profile staging
$ ilum config set cluster.namespace ilum-staging
$ ilum config set release_name ilum-staging
$ ilum install -m sql -m airflow --yes
✓ Namespace 'ilum-staging' does not exist — creating it.
✓ Namespace 'ilum-staging' created.
✓ Ilum installed successfully (release: ilum-staging).
```

```
$ ilum init --profile prod --yes
$ ilum config set active_profile prod
$ ilum config set cluster.namespace ilum-prod
$ ilum config set release_name ilum-prod
$ ilum install \
    -m sql -m airflow -m superset -m monitoring -m loki \
    --values prod-values.yaml \
    --yes
✓ Namespace 'ilum-prod' does not exist — creating it.
✓ Namespace 'ilum-prod' created.
✓ Ilum installed successfully (release: ilum-prod).
```

Switch between environments with:

```
$ ilum config set active_profile staging
$ ilum status    # Shows staging release
```

Or target a specific release directly using `--release` and `--namespace` flags:

```
$ ilum status --release ilum-prod --namespace ilum-prod
```

> **Note:** Each namespace gets its own isolated set of PVCs, Services, and ConfigMaps. There is no cross-namespace interference. The CLI tracks each profile's module list independently.

### 8.3 Enabling the Full Data Stack (SQL + Airflow + Superset)

**Context:** You have a running Ilum installation with the default modules (core, ui, mongodb, postgresql, minio, jupyter, gitea).

**Goal:** You want to add a complete data analytics workflow: SQL queries via Kyuubi, pipeline orchestration via Airflow, and dashboards via Superset.

**Solution:**

First, inspect the dependency chain to understand what will be pulled in:

```
$ ilum module show sql
╭─ Module: sql ────────────────────────────────────────────────────────╮
│ sql (sql)                                                            │
│   Ilum SQL engine (Kyuubi)                                           │
│                                                                      │
│   Default enabled: no                                                │
│   Requires: postgresql, core                                         │
│                                                                      │
│   Resolved enable chain (including dependencies):                    │
│     --set postgresql.enabled=true                                    │
│     --set ilum-core.enabled=true                                     │
│     --set ilum-sql.enabled=true                                      │
│     --set ilum-core.sql.enabled=true                                 │
╰──────────────────────────────────────────────────────────────────────╯

$ ilum module show airflow
╭─ Module: airflow ────────────────────────────────────────────────────╮
│ airflow (orchestration)                                              │
│   Apache Airflow workflow orchestration                              │
│                                                                      │
│   Default enabled: no                                                │
│   Requires: postgresql                                               │
╰──────────────────────────────────────────────────────────────────────╯

$ ilum module show superset
╭─ Module: superset ───────────────────────────────────────────────────╮
│ superset (analytics)                                                 │
│   Apache Superset BI dashboards                                      │
│                                                                      │
│   Default enabled: no                                                │
│   Requires: postgresql                                               │
╰──────────────────────────────────────────────────────────────────────╯
```

All three share `postgresql` as a dependency, which is already enabled by default. Enable them in a single command:

```
$ ilum module enable sql airflow superset --yes
╭─ Enable Summary ─────────────────────────────────────────╮
│ Action     Enable                                        │
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Modules    sql, airflow, superset                        │
╰──────────────────────────────────────────────────────────╯

ℹ Auto-resolved dependencies: core, postgresql

╭─ Values Diff ────────────────────────────────────────────╮
│ Key                          Before    After              │
├──────────────────────────────────────────────────────────┤
│ ilum-sql.enabled             false     true               │
│ ilum-core.sql.enabled        false     true               │
│ airflow.enabled              false     true               │
│ superset.enabled             false     true               │
│ postgresql.enabled           true      true               │
│ ilum-core.enabled            true      true               │
╰──────────────────────────────────────────────────────────╯

⠋ Enabling modules...
✓ Enabled: sql, airflow, superset
```

Verify the modules are running:

```
$ ilum module status --category sql
╭─ Module Status (release: ilum, namespace: default) ──────────────────╮
│ Module           Category   Status     Pods   Health                  │
├──────────────────────────────────────────────────────────────────────┤
│ sql              sql        Enabled    1/1    Healthy                 │
│ hive-metastore   sql        Disabled   --     --                      │
│ nessie           sql        Disabled   --     --                      │
│ unity-catalog    sql        Disabled   --     --                      │
│ trino            sql        Disabled   --     --                      │
╰──────────────────────────────────────────────────────────────────────╯

$ ilum module status --category orchestration
╭─ Module Status (release: ilum, namespace: default) ──────────────────╮
│ Module     Category        Status     Pods   Health                   │
├──────────────────────────────────────────────────────────────────────┤
│ airflow    orchestration   Enabled    3/3    Healthy                  │
│ kestra     orchestration   Disabled   --     --                       │
│ n8n        orchestration   Disabled   --     --                       │
│ nifi       orchestration   Disabled   --     --                       │
│ mageai     orchestration   Disabled   --     --                       │
╰──────────────────────────────────────────────────────────────────────╯
```

To later remove one piece of the stack without affecting the others:

```
$ ilum module disable superset --yes
✓ Disabled: superset
```

PostgreSQL remains running because `sql`, `airflow`, and `gitea` still depend on it.

### 8.4 Setting Up Monitoring (Prometheus + Loki + Grafana)

**Context:** You have a running Ilum installation and you want observability into cluster health, application metrics, and centralized logs.

**Goal:** You want to enable the Prometheus + Grafana monitoring stack for metrics and Grafana Loki for log aggregation.

**Solution:**

Check what the monitoring modules include:

```
$ ilum module show monitoring
╭─ Module: monitoring ─────────────────────────────────────────────────╮
│ monitoring (monitoring)                                              │
│   Prometheus + Grafana monitoring stack                              │
│                                                                      │
│   Default enabled: no                                                │
│   Enable flags:  kube-prometheus-stack.enabled=true                  │
│   Disable flags: kube-prometheus-stack.enabled=false                 │
╰──────────────────────────────────────────────────────────────────────╯

$ ilum module show loki
╭─ Module: loki ───────────────────────────────────────────────────────╮
│ loki (monitoring)                                                    │
│   Grafana Loki log aggregation                                       │
│                                                                      │
│   Default enabled: no                                                │
│   Enable flags:  global.logAggregation.enabled=true,                 │
│                  global.logAggregation.loki.enabled=true,            │
│                  global.logAggregation.promtail.enabled=true         │
╰──────────────────────────────────────────────────────────────────────╯
```

Neither module has external dependencies, so enabling them is straightforward:

```
$ ilum module enable monitoring loki --yes
╭─ Enable Summary ─────────────────────────────────────────╮
│ Action     Enable                                        │
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Modules    monitoring, loki                              │
╰──────────────────────────────────────────────────────────╯

╭─ Values Diff ────────────────────────────────────────────╮
│ Key                                       Before  After  │
├──────────────────────────────────────────────────────────┤
│ kube-prometheus-stack.enabled             false   true   │
│ global.logAggregation.enabled             false   true   │
│ global.logAggregation.loki.enabled        false   true   │
│ global.logAggregation.promtail.enabled    false   true   │
╰──────────────────────────────────────────────────────────╯

⠋ Enabling modules...
✓ Enabled: monitoring, loki
```

Verify the monitoring stack is healthy:

```
$ ilum module status --category monitoring
╭─ Module Status (release: ilum, namespace: default) ──────────────────╮
│ Module       Category     Status     Pods   Health                    │
├──────────────────────────────────────────────────────────────────────┤
│ monitoring   monitoring   Enabled    3/3    Healthy                   │
│ loki         monitoring   Enabled    2/2    Healthy                   │
│ graphite     monitoring   Disabled   --     --                        │
╰──────────────────────────────────────────────────────────────────────╯
```

The `monitoring` module deploys the kube-prometheus-stack, which includes Prometheus for metrics collection, Grafana for dashboards, and Alertmanager for alerts. The `loki` module adds Loki for log storage and Promtail for log collection from all pods in the namespace.

Access Grafana through the Ilum UI at `http://localhost:31777/external/grafana/`, or check the service directly with:

```
$ ilum doctor --check pods
```

If you also want Graphite metrics export (for legacy integrations), add it as a third module:

```
$ ilum module enable graphite --yes
```

### 8.5 Dry-Run Workflow for Change Management

**Context:** You are managing a production Ilum deployment and need to review infrastructure changes before applying them -- for compliance, peer review, or just caution.

**Goal:** You want a preview-then-apply workflow where every change is reviewed as a diff before execution.

**Solution:**

Every mutating CLI command supports `--dry-run`. The flag shows the full operation summary, values diff, and generated Helm command -- then exits without executing anything.

**Preview a version upgrade:**

```
$ ilum upgrade --version 6.8.0 --dry-run
╭─ Upgrade Summary ───────────────────────────────────────╮
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Chart      ilum/ilum                                     │
│ Version    6.7.0 → 6.8.0                                │
│ Modules    core, gitea, jupyter, minio, mongodb,         │
│            postgresql, ui                                │
│ Atomic     True                                          │
╰──────────────────────────────────────────────────────────╯

ℹ Upgrade notes: https://ilum.cloud/docs/upgrade-notes/

  Command: helm upgrade ilum ilum/ilum \
    --namespace default \
    --version 6.8.0 \
    --timeout 10m \
    --atomic \
    --reuse-values

ℹ Dry-run mode — no changes applied.
```

**Preview enabling a module:**

```
$ ilum module enable airflow --dry-run
╭─ Enable Summary ─────────────────────────────────────────╮
│ Action     Enable                                        │
│ Release    ilum                                          │
│ Namespace  default                                       │
│ Modules    airflow                                       │
╰──────────────────────────────────────────────────────────╯

ℹ Auto-resolved dependencies: postgresql

╭─ Values Diff ────────────────────────────────────────────╮
│ Key                    Before    After                    │
├──────────────────────────────────────────────────────────┤
│ airflow.enabled        false     true                    │
│ postgresql.enabled     true      true                    │
╰──────────────────────────────────────────────────────────╯

  Command: helm upgrade ilum ilum/ilum \
    --namespace default \
    --timeout 10m \
    --reuse-values \
    --set airflow.enabled=true \
    --set postgresql.enabled=true

ℹ Dry-run mode — no changes applied.
```

**Preview an uninstall:**

```
$ ilum uninstall --delete-data --dry-run
╭─ Uninstall Summary ─────────────────────────────────────╮
│ Release           ilum                                   │
│ Namespace         default                                │
│ Delete PVCs       Yes                                    │
│ Delete Namespace  No                                     │
╰──────────────────────────────────────────────────────────╯

⚠ PersistentVolumeClaims will be DELETED.

  Command: helm uninstall ilum --namespace default

ℹ Dry-run mode — no changes applied.
```

A typical change management workflow looks like this:

1. Run the command with `--dry-run` and capture the output.
2. Share the diff with your team for review.
3. Once approved, re-run the same command without `--dry-run` (and with `--yes` if running from a script).

```
$ ilum module enable sql --dry-run 2>&1 | tee change-request-42.txt
$ # ... team reviews change-request-42.txt ...
$ ilum module enable sql --yes
```

> **Note:** The `--dry-run` flag is available on `ilum install`, `ilum upgrade`, `ilum uninstall`, `ilum module enable`, and `ilum module disable`. Each command exits with code 0 in dry-run mode regardless of what would happen, so you can safely run dry-runs in CI pipelines. See [reference.md](reference.md) for the full list of supported flags per command.
