# Certificate Management — Citadel

Automated TLS certificate provisioning and rotation using [cert-manager](https://cert-manager.io/).

## Overview

Citadel uses cert-manager to automate the full TLS certificate lifecycle:
key generation, CSR creation, ACME challenge solving, certificate retrieval,
Kubernetes Secret management, and automatic renewal before expiry.

Three ClusterIssuers cover the standard deployment lifecycle:

| Issuer | Purpose | Trust |
|--------|---------|-------|
| `letsencrypt-staging` | Testing and validation | Not browser-trusted |
| `letsencrypt-prod` | Production external services | Browser-trusted |
| `internal-ca` | Service-to-service mTLS | Self-signed (internal only) |

Certificate defaults:
- **Duration**: 90 days
- **Renewal window**: 30 days before expiry
- **Private key algorithm**: ECDSA P-256
- **Rotation policy**: Always (new key on each renewal)

## Installation

### Automated

```bash
# Installs cert-manager, applies ClusterIssuers, verifies the deployment
sudo scripts/09-install-cert-manager.sh
```

### Manual

```bash
# 1. Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# 2. Install cert-manager with Citadel values
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --values security/certificates/cert-manager-install.yaml

# 3. Wait for the webhook to become ready
kubectl wait --for=condition=Available deployment/cert-manager-webhook \
  -n cert-manager --timeout=120s

# 4. Apply ClusterIssuers
kubectl apply -f security/certificates/cluster-issuers.yaml
```

## Issuer Configuration

### Let's Encrypt (External Services)

Before using Let's Encrypt issuers, update the email address in
`security/certificates/cluster-issuers.yaml`:

```yaml
spec:
  acme:
    email: your-actual-email@example.com
```

Always test with `letsencrypt-staging` before switching to `letsencrypt-prod`.
Let's Encrypt production has rate limits: 50 certificates per registered domain
per week.

### Internal CA (Service Mesh)

The internal CA is bootstrapped automatically:

1. A self-signed `ClusterIssuer` creates the root CA certificate
2. The root CA key is stored in `citadel-internal-ca-secret` in the `cert-manager` namespace
3. The `internal-ca` ClusterIssuer uses this CA to sign leaf certificates

No external connectivity required. Suitable for air-gapped environments.

## Requesting Certificates

### Wildcard Certificate (Internal Services)

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-wildcard
  namespace: my-namespace
spec:
  secretName: internal-wildcard-tls
  duration: 2160h     # 90 days
  renewBefore: 720h   # 30 days before expiry
  commonName: "*.internal.citadel.local"
  dnsNames:
    - "*.internal.citadel.local"
    - "internal.citadel.local"
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
```

### Per-Service Certificate

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-service-tls
  namespace: default
spec:
  secretName: my-service-tls-secret
  duration: 2160h
  renewBefore: 720h
  commonName: my-service.default.svc.cluster.local
  dnsNames:
    - my-service.default.svc.cluster.local
    - my-service.default.svc
    - my-service
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
```

### mTLS Client Certificate

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-service-client
  namespace: default
spec:
  secretName: my-service-client-tls-secret
  duration: 2160h
  renewBefore: 720h
  commonName: my-service.default.svc.cluster.local
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  usages:
    - client auth
    - digital signature
```

### External Service (Let's Encrypt)

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: public-api-tls
  namespace: production
spec:
  secretName: public-api-tls-secret
  duration: 2160h
  renewBefore: 720h
  commonName: api.example.com
  dnsNames:
    - api.example.com
    - www.api.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
```

## Python Integration

The `citadel_operator.k3s.certificates` module provides programmatic access:

```python
from citadel_operator.k3s.certificates import CertificateConfig, CertificateManager

# Generate a Certificate YAML manifest
config = CertificateConfig(
    issuer_name="internal-ca",
    common_name="my-service.default.svc.cluster.local",
    dns_names=("my-service.default.svc.cluster.local", "my-service"),
)
manifest = CertificateManager.create_certificate(config)

# Check certificate expiry (requires cluster access)
manager = CertificateManager()
expiring = manager.check_expiry(namespace="default")
for name, expiry_time, needs_renewal in expiring:
    if needs_renewal:
        print(f"Certificate {name} expires at {expiry_time}, renewal pending")
```

## Troubleshooting

### Certificate stuck in "Issuing" state

```bash
# Check Certificate status
kubectl describe certificate <name> -n <namespace>

# Check the CertificateRequest
kubectl get certificaterequest -n <namespace>

# Check cert-manager controller logs
kubectl logs -n cert-manager deployment/cert-manager -f
```

### ACME challenge failing

```bash
# Check the Challenge resource
kubectl get challenge -A
kubectl describe challenge <name> -n <namespace>

# Verify ingress connectivity (HTTP-01)
curl -v http://<domain>/.well-known/acme-challenge/<token>
```

### Webhook not ready

```bash
# cert-manager webhook must be running before any Certificate can be created
kubectl wait --for=condition=Available deployment/cert-manager-webhook \
  -n cert-manager --timeout=120s

# Check webhook logs
kubectl logs -n cert-manager deployment/cert-manager-webhook
```

### Internal CA not signing certificates

```bash
# Verify the root CA certificate exists
kubectl get secret citadel-internal-ca-secret -n cert-manager

# Verify the internal-ca ClusterIssuer is Ready
kubectl get clusterissuer internal-ca -o wide

# Check the CA certificate status
kubectl describe certificate citadel-internal-ca -n cert-manager
```

## Architecture

```
cert-manager controller
├── watches Certificate resources
├── creates CertificateRequest
├── resolves Issuer/ClusterIssuer
├── performs ACME challenge (Let's Encrypt) or CA signing (internal)
├── stores cert + key in Kubernetes Secret
└── renews automatically before expiry

ClusterIssuers
├── letsencrypt-staging  →  ACME staging endpoint
├── letsencrypt-prod     →  ACME production endpoint
├── citadel-selfsigned-bootstrap  →  self-signed (root CA only)
└── internal-ca          →  CA backed by citadel-internal-ca-secret
```

## Related

- `security/mutual-auth.yaml` — Cilium SPIFFE-based mTLS (network layer)
- `security/encryption-config.yaml` — etcd data-at-rest encryption
- `scripts/09-install-cert-manager.sh` — Automated installation script
