# Docker Production Deployment

Run LLMHosts in Docker with optional GPU passthrough, automatic TLS via Traefik, and persistent storage. This guide uses the production Compose file (`docker-compose.prod.yml`) included in the repository.

## Prerequisites

| Component | Version |
|-----------|---------|
| Docker Engine | 24.0+ |
| Docker Compose | v2.20+ (plugin) |
| nvidia-container-toolkit | Latest (GPU only) |

```bash
# Verify Docker
docker --version
docker compose version
```

## Quick Start

```bash
git clone https://github.com/LookNoHandsMom/LLMHosts.com.git
cd LLMHosts.com

# Set your domain (defaults to localhost if not set)
export DOMAIN=llmhosts.example.com
export ACME_EMAIL=you@example.com

# Start LLMHosts + Traefik reverse proxy
docker compose -f docker-compose.prod.yml up -d
```

LLMHosts is now running. Traefik listens on ports 80/443, handles TLS termination via Let's Encrypt, and routes traffic to the llmhosts container internally.

## Deployment Modes

### Mode 1: Standalone (default)

Full self-contained stack with built-in Traefik. Use this on a fresh server where nothing else owns ports 80/443.

```bash
docker compose -f docker-compose.prod.yml up -d
```

### Mode 2: External Proxy

Use this when an existing reverse proxy (Pangolin, Traefik, nginx, Caddy) already runs on the host. The override file disables the built-in Traefik and connects llmhosts to the external proxy's Docker network.

```bash
docker compose -f docker-compose.prod.yml -f docker-compose.external-proxy.yml up -d
```

You must configure your external proxy to forward traffic to `http://llmhosts:4000`. See [External Proxy Configuration](#external-proxy-configuration) below.

## docker-compose.prod.yml Overview

The production Compose file defines three services:

1. **llmhosts** -- The core proxy server (built from the multi-stage `Dockerfile`). Handles API paths: `/v1/*`, `/health`, `/ready`, `/api/keys/*`, `/docs`, `/admin`.
2. **frontend** -- Next.js web dashboard (built from `frontend/Dockerfile.prod`). Serves the landing page, pricing, auth, and dashboard UI.
3. **traefik** -- Automatic HTTPS reverse proxy using the official Traefik v3 Alpine image. Routes API paths (priority 20) to llmhosts:4000 and all other paths (priority 10) to frontend:3000.

Key production defaults:

- `restart: unless-stopped` -- auto-restart on crash or reboot.
- Health check every 30 seconds on `/health`.
- 4 GB memory limit on the llmhosts container.
- Authentication enabled via `LLMHOSTS_SECURITY_REQUIREAUTH=true`.
- 30-minute read timeout for SSE streaming (long-running chat completions).
- Security headers: HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy.
- No ports exposed on the llmhosts container -- only Traefik is publicly reachable.
- Persistent volumes for LLMHosts data and Let's Encrypt certificates.

## Environment Variables

### Domain and TLS

| Variable | Default | Description |
|----------|---------|-------------|
| `DOMAIN` | `localhost` | Hostname for Traefik routing + Let's Encrypt |
| `ACME_EMAIL` | `admin@llmhosts.com` | Email for Let's Encrypt registration |

### Application

Create a `.env` file in the project root (never commit this file):

```bash
cat > .env << 'EOF'
# Domain
DOMAIN=llmhosts.example.com
ACME_EMAIL=you@example.com

# Required for production
LLMHOSTS_SECURITY_REQUIREAUTH=true
LLMHOSTS_SERVER_HOST=0.0.0.0

# Logging
LLMHOSTS_SERVER_LOGLEVEL=info
LLMHOSTS_OBSERVABILITY_LOGFORMAT=json

# Ollama (if running on host or another container)
LLMHOSTS_OLLAMA_HOST=http://host.docker.internal:11434

# Cache
LLMHOSTS_CACHE_ENABLED=true
LLMHOSTS_CACHE_MAXSIZEMB=1024

# Auth (generate with: openssl rand -base64 32)
NEXTAUTH_SECRET=REPLACE_ME_WITH_RANDOM_SECRET

# Sentry error tracking (optional)
# LLMHOSTS_OBSERVABILITY_SENTRYDSN=https://...@sentry.io/...
EOF
```

All LLMHosts config values can be set as environment variables with the format `LLMHOSTS_<SECTION>_<KEY>`. See `llmhosts/config.py` for the full list of sections and fields.

## GPU Passthrough

### 1. Install nvidia-container-toolkit

```bash
# Add the NVIDIA repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```

### 2. Enable GPU in Compose

Edit `docker-compose.prod.yml` and uncomment the GPU lines in the `llmhosts` service:

```yaml
services:
  llmhosts:
    # ... existing config ...
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - LLMHOSTS_SERVER_HOST=0.0.0.0
      - LLMHOSTS_SECURITY_REQUIREAUTH=true
```

Or use a Compose override file to avoid editing the original:

```bash
cat > docker-compose.gpu.yml << 'EOF'
services:
  llmhosts:
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
EOF

docker compose -f docker-compose.prod.yml -f docker-compose.gpu.yml up -d
```

### 3. Verify GPU access

```bash
docker compose -f docker-compose.prod.yml exec llmhosts llmhosts doctor
```

The doctor output should show your GPU under the hardware section.

## Running Ollama Alongside LLMHosts

The most common setup pairs LLMHosts with Ollama for local inference. You have two options:

### Option A: Ollama on the host

If Ollama is installed on the host machine, LLMHosts in Docker can reach it via `host.docker.internal`:

```bash
# In .env
LLMHOSTS_OLLAMA_HOST=http://host.docker.internal:11434
```

On Linux, add this to the llmhosts service in your Compose file:

```yaml
extra_hosts:
  - "host.docker.internal:host-gateway"
```

### Option B: Ollama in Docker

Add Ollama as a service in the Compose file:

```yaml
services:
  ollama:
    image: ollama/ollama:latest
    restart: unless-stopped
    networks:
      - llmhosts-internal
    volumes:
      - ollama_data:/root/.ollama
    # Uncomment for GPU:
    # runtime: nvidia
    # environment:
    #   - NVIDIA_VISIBLE_DEVICES=all

  llmhosts:
    # ... existing config ...
    environment:
      - LLMHOSTS_OLLAMA_HOST=http://ollama:11434
      - LLMHOSTS_SERVER_HOST=0.0.0.0
      - LLMHOSTS_SECURITY_REQUIREAUTH=true
    depends_on:
      ollama:
        condition: service_started

volumes:
  ollama_data:
```

Pull models into the containerized Ollama:

```bash
docker compose -f docker-compose.prod.yml exec ollama ollama pull llama3.2
```

## Persistent Volumes

The production Compose file declares named volumes:

| Volume | Mount Point | Purpose |
|--------|-------------|---------|
| `llmhosts_data` | `/home/llmhost/.llmhosts` | Config, encrypted keys, SQLite audit DB, cache |
| `frontend_data` | `/data` | Frontend SQLite DB (user accounts, sessions, costs) |
| `letsencrypt_data` | `/letsencrypt` | TLS certificates from Let's Encrypt |

To back up LLMHosts data:

```bash
docker run --rm -v llmhosts_data:/data -v $(pwd):/backup alpine \
  tar czf /backup/llmhosts-data-$(date +%Y%m%d).tar.gz /data
```

## External Proxy Configuration

When using `docker-compose.external-proxy.yml`, the built-in Traefik is disabled. Configure your external proxy to route API traffic to `http://llmhosts:4000` and web traffic to `http://llmhosts-frontend:3000`.

### Traefik (file provider / Pangolin)

Add to your Traefik dynamic config (e.g., `dynamic_config.yml`):

```yaml
http:
  routers:
    # API paths (higher priority)
    llmhosts-api-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`llmhosts.com`) && (PathPrefix(`/v1`) || PathPrefix(`/health`) || PathPrefix(`/ready`) || PathPrefix(`/docs`) || PathPrefix(`/openapi.json`) || PathPrefix(`/admin`))
      service: llmhosts-api-service
      priority: 20
      tls:
        certResolver: letsencrypt
    llmhosts-api-redirect:
      entryPoints:
        - web
      middlewares:
        - redirect-to-https
      rule: Host(`llmhosts.com`) && (PathPrefix(`/v1`) || PathPrefix(`/health`) || PathPrefix(`/ready`) || PathPrefix(`/docs`) || PathPrefix(`/openapi.json`) || PathPrefix(`/admin`))
      service: llmhosts-api-service
      priority: 20
    # Frontend catch-all (lower priority)
    llmhosts-frontend-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`llmhosts.com`)
      service: llmhosts-frontend-service
      priority: 10
      tls:
        certResolver: letsencrypt
    llmhosts-frontend-redirect:
      entryPoints:
        - web
      middlewares:
        - redirect-to-https
      rule: Host(`llmhosts.com`)
      service: llmhosts-frontend-service
      priority: 10
  services:
    llmhosts-api-service:
      loadBalancer:
        servers:
          - url: http://llmhosts:4000
    llmhosts-frontend-service:
      loadBalancer:
        servers:
          - url: http://llmhosts-frontend:3000
```

### Caddy

See `contrib/Caddyfile` for a ready-made Caddy reverse proxy config.

### nginx

```nginx
server {
    listen 443 ssl;
    server_name llmhosts.com;

    ssl_certificate     /etc/letsencrypt/live/llmhosts.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llmhosts.com/privkey.pem;

    # API paths → Python proxy
    location ~ ^/(v1|health|ready|docs|openapi\.json|admin) {
        proxy_pass http://llmhosts:4000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # SSE streaming support
        proxy_buffering off;
        proxy_read_timeout 30m;
    }

    # Everything else → Next.js frontend
    location / {
        proxy_pass http://llmhosts-frontend:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

## Health Checks

The `docker-compose.prod.yml` includes a Compose-level health check using Python's httpx (no curl dependency):

```yaml
healthcheck:
  test: ["CMD", "python3", "-c", "import httpx; httpx.get('http://localhost:4000/health').raise_for_status()"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 30s
```

Monitor health status:

```bash
docker compose -f docker-compose.prod.yml ps
# The STATUS column shows "healthy" or "unhealthy"

# Or query directly
docker compose -f docker-compose.prod.yml exec llmhosts \
  python3 -c "import httpx; print(httpx.get('http://localhost:4000/health').text)"
```

## Logging

### View logs

```bash
# All services
docker compose -f docker-compose.prod.yml logs -f

# LLMHosts only
docker compose -f docker-compose.prod.yml logs -f llmhosts

# Traefik access logs
docker compose -f docker-compose.prod.yml logs -f traefik

# Last 100 lines
docker compose -f docker-compose.prod.yml logs --tail 100 llmhosts
```

### Structured JSON logging

Both LLMHosts and Traefik output JSON logs by default in production. Set `LLMHOSTS_OBSERVABILITY_LOGFORMAT=json` in your `.env` for machine-parseable application logs that work well with log aggregation tools (Loki, ELK, etc.).

## Resource Limits

The default memory limit is 4 GB. Adjust based on your workload:

```yaml
deploy:
  resources:
    limits:
      memory: 8G
    reservations:
      memory: 2G
```

For GPU deployments with large models, you may need 8-16 GB depending on model size.

## Scaling Considerations

LLMHosts is designed to run as a single instance per GPU host. For horizontal scaling:

- **Multiple GPU machines**: Run one LLMHosts instance per machine, connect them via the mesh (`llmhosts device add`).
- **Load balancing**: Traefik can load balance across multiple LLMHosts instances if you need to distribute API traffic.
- **Workers**: Increase `LLMHOSTS_SERVER_WORKERS` for more uvicorn workers on a single machine (CPU-bound workloads only; GPU inference is typically the bottleneck).

## Updating

```bash
cd /path/to/LLMHosts.com

# Pull latest code
git pull origin main

# Rebuild and restart
docker compose -f docker-compose.prod.yml build llmhosts
docker compose -f docker-compose.prod.yml up -d llmhosts
```

Named volumes (`llmhosts_data`, `frontend_data`, `letsencrypt_data`) persist across container rebuilds. Your configuration, keys, and audit history are preserved.

### Zero-downtime update (optional)

For uninterrupted service, use a rolling update approach:

```bash
# Build new image
docker compose -f docker-compose.prod.yml build llmhosts

# Replace the container (brief restart, typically < 5 seconds)
docker compose -f docker-compose.prod.yml up -d --no-deps llmhosts
```

## Troubleshooting

### Container exits immediately

```bash
docker compose -f docker-compose.prod.yml logs llmhosts
```

Common causes:
- Missing `.env` file or invalid environment variable values.
- Port conflict if another service uses 80/443 (use external proxy mode instead).

### Cannot reach Ollama from container

```bash
# Test from inside the container
docker compose -f docker-compose.prod.yml exec llmhosts \
  python3 -c "import httpx; print(httpx.get('http://host.docker.internal:11434/api/tags').status_code)"
```

On Linux, `host.docker.internal` requires the `extra_hosts` entry shown above. Alternatively, use the host's LAN IP directly.

### GPU not detected inside container

```bash
# Verify nvidia-container-toolkit is working
docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.0-base nvidia-smi

# If that fails, reinstall nvidia-container-toolkit and restart Docker
sudo systemctl restart docker
```

### Health check failing

```bash
# Check if the server started
docker compose -f docker-compose.prod.yml exec llmhosts llmhosts doctor

# Check the health endpoint directly
docker compose -f docker-compose.prod.yml exec llmhosts \
  python3 -c "import httpx; print(httpx.get('http://localhost:4000/health').text)"
```

### TLS certificate not issuing

```bash
# Check Traefik logs for ACME errors
docker compose -f docker-compose.prod.yml logs traefik | grep -i acme

# Verify DNS points to this server
dig +short llmhosts.example.com

# Verify port 80 is reachable (required for HTTP challenge)
curl -I http://llmhosts.example.com
```

Let's Encrypt requires:
1. DNS A record pointing to the server's public IP.
2. Port 80 open and reachable from the internet (for the HTTP challenge).
3. A valid `DOMAIN` environment variable (not `localhost`).
