# Secrets Management — Citadel

Production-grade secrets management using [external-secrets-operator](https://external-secrets.io/) (ESO).

## Overview

Citadel uses external-secrets-operator to sync secrets from external providers
into native Kubernetes Secrets. This keeps secret material out of Git and YAML manifests
while making it available to pods via standard Secret mounts and environment variables.

Supported providers:

| Provider | Use Case | Auth Method |
|----------|----------|-------------|
| Kubernetes | Dev/test (secrets from another namespace) | ServiceAccount |
| HashiCorp Vault | Production secret management | Kubernetes auth |
| AWS Secrets Manager | Cloud-native AWS workloads | Static credentials or IRSA |
| Azure Key Vault | Cloud-native Azure workloads | Managed Identity |

Key concepts:

- **SecretStore / ClusterSecretStore**: Defines *where* secrets come from (provider + auth)
- **ExternalSecret**: Defines *what* to fetch and *how* to map it into a Kubernetes Secret
- **Refresh interval**: How often ESO re-syncs from the external provider

## Installation

### Automated

```bash
# Installs external-secrets-operator, applies dev SecretStore, verifies CRDs
sudo scripts/12-install-secrets-operator.sh
```

### Manual

```bash
# 1. Add the external-secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

# 2. Install external-secrets-operator with Citadel values
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --values security/secrets/external-secrets-install.yaml

# 3. Wait for the webhook to become ready
kubectl wait --for=condition=Available deployment/external-secrets-webhook \
  -n external-secrets --timeout=120s

# 4. Apply SecretStores
kubectl apply -f security/secrets/secret-stores.yaml
```

## SecretStore Configuration

### Kubernetes Backend (Dev/Test)

No external provider needed. Secrets are stored as native Kubernetes Secrets
and synced across namespaces via ExternalSecret resources:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: kubernetes-backend
  namespace: default
spec:
  provider:
    kubernetes:
      remoteNamespace: default
      server:
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
      auth:
        serviceAccount:
          name: external-secrets-sa
```

### HashiCorp Vault (Production)

Requires Vault Kubernetes auth method configured:

```bash
# On the Vault server:
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc"
vault write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets-sa \
  bound_service_account_namespaces=external-secrets \
  policies=external-secrets-policy
```

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: external-secrets-sa
```

### AWS Secrets Manager

```bash
# Create the credentials secret
kubectl create secret generic aws-credentials \
  -n external-secrets \
  --from-literal=access-key-id=AKIAXXXXXXXXXXXXXXXX \
  --from-literal=secret-access-key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
```

### Azure Key Vault

Uses Managed Identity by default. Grant the cluster identity
"Key Vault Secrets User" role on the Azure Key Vault resource.

## Creating ExternalSecrets

### Simple Key-Value

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-api-key
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: app-api-key
    creationPolicy: Owner
  data:
    - secretKey: api-key
      remoteRef:
        key: secret/data/my-app
        property: api_key
```

### Multiple Keys from One Source

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: default
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: db-host
      remoteRef:
        key: secret/data/database
        property: host
    - secretKey: db-password
      remoteRef:
        key: secret/data/database
        property: password
```

### Template-Based (Connection String Assembly)

```yaml
spec:
  target:
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgresql://{{ .db_username }}:{{ .db_password }}@{{ .db_host }}:{{ .db_port }}/{{ .db_name }}"
  data:
    - secretKey: db_host
      remoteRef:
        key: secret/data/database
        property: host
    # ... additional fields
```

## Python Integration

The `citadel_operator.k3s.secrets` module provides programmatic access:

```python
from citadel_operator.k3s.secrets import (
    ExternalSecretConfig,
    SecretKeyMapping,
    SecretStoreConfig,
    SecretsManager,
)

# Generate a SecretStore manifest
store = SecretStoreConfig(
    name="vault-backend",
    provider="vault",
    namespace="production",
    vault_url="https://vault.example.com",
)
manifest = SecretsManager.create_secret_store(store)

# Generate an ExternalSecret manifest
secret = ExternalSecretConfig(
    name="my-app-secrets",
    namespace="production",
    store_name="vault-backend",
    keys=(
        SecretKeyMapping(
            remote_key="secret/data/my-app",
            secret_key="database-url",
            property="db_url",
        ),
    ),
)
manifest = SecretsManager.create_external_secret(secret)

# Check sync status (requires cluster access)
manager = SecretsManager()
status = manager.check_sync_status("my-app-secrets", "production")
if not status["synced"]:
    print(f"Sync failed: {status['status']}")
```

## Refresh Intervals

| Sensitivity | Interval | Use Case |
|-------------|----------|----------|
| High | `5m` | Frequently-rotated credentials (TLS, tokens) |
| Standard | `1h` | Most application secrets |
| Low | `24h` | Stable configuration values |
| Disabled | `0` | One-time fetch, never refreshed |

## Troubleshooting

### ExternalSecret stuck in "SecretSyncedError"

```bash
# Check ExternalSecret status
kubectl describe externalsecret <name> -n <namespace>

# Check the operator logs
kubectl logs -n external-secrets deployment/external-secrets -f

# Verify the SecretStore is healthy
kubectl get secretstore -n <namespace>
kubectl get clustersecretstore
```

### Provider authentication failing

```bash
# Vault: verify Kubernetes auth
kubectl exec -it vault-0 -- vault read auth/kubernetes/config

# AWS: verify credentials secret exists
kubectl get secret aws-credentials -n external-secrets

# Azure: verify managed identity assignment
az identity show --name <identity-name> --resource-group <rg>
```

### CRDs not installed

```bash
# Verify external-secrets CRDs
kubectl get crd | grep external-secrets

# Expected CRDs:
#   externalsecrets.external-secrets.io
#   secretstores.external-secrets.io
#   clustersecretstores.external-secrets.io
#   clusterexternalsecrets.external-secrets.io
```

## Architecture

```
external-secrets-operator
├── watches ExternalSecret resources
├── authenticates to external provider via SecretStore
├── fetches secret data from provider
├── creates/updates Kubernetes Secret
└── re-syncs at configured refreshInterval

SecretStores
├── kubernetes-backend     →  K8s API (dev/test)
├── vault-backend          →  HashiCorp Vault (production)
├── aws-secrets-manager    →  AWS Secrets Manager
└── azure-key-vault        →  Azure Key Vault
```

## Related

- `security/certificates/` — cert-manager TLS certificate provisioning
- `security/encryption-config.yaml` — etcd data-at-rest encryption
- `security/mutual-auth.yaml` — Cilium SPIFFE-based mTLS
- `scripts/12-install-secrets-operator.sh` — Automated installation script
