# Bare Metal Deployment

Deploy LLMHosts directly on a Linux machine with optional GPU acceleration. This guide covers installation, systemd service setup, reverse proxy configuration, and ongoing maintenance.

## System Requirements

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| OS | Ubuntu 22.04+ / Debian 12+ | Ubuntu 24.04 LTS |
| Python | 3.10+ | 3.12 |
| RAM | 4 GB | 16 GB+ |
| Disk | 2 GB (core only) | 20 GB+ (for local models) |
| GPU | Not required | NVIDIA with 8+ GB VRAM |
| Network | Any | Static IP or DDNS |

LLMHosts works in CPU-only mode. A GPU is only needed if you run local models through Ollama or vLLM.

## Installation

### 1. Install Python 3.12

```bash
# Ubuntu 24.04 ships Python 3.12
sudo apt update && sudo apt install -y python3 python3-pip python3-venv

# Verify
python3 --version
```

### 2. Create a dedicated user (recommended)

```bash
sudo useradd --system --create-home --shell /bin/bash llmhost
sudo -u llmhost -i
```

### 3. Install LLMHosts

Choose the tier that matches your hardware:

```bash
# Core -- proxy + router + cache (~50 MB)
pip install llmhosts

# Smart -- adds ONNX-based kNN router + FAISS semantic cache (~150 MB)
pip install "llmhosts[smart]"

# Full -- adds PyTorch router tiers + sentence-transformers (~2 GB)
pip install "llmhosts[full]"
```

If installing into the system Python, prefer `pipx`:

```bash
pipx install llmhost            # core
pipx install "llmhost[smart]"   # smart tier
```

### 4. Verify installation

```bash
llmhosts --version
llmhosts doctor
```

`llmhosts doctor` checks Python, Ollama connectivity, GPU detection, installed tier, and disk space.

## Configuration

LLMHosts reads configuration from `~/.llmhosts/config.toml`. Generate a default config:

```bash
llmhosts setup
```

The setup wizard walks through spending baseline, hardware detection, Ollama discovery, and API key configuration. Alternatively, create the config manually:

```bash
mkdir -p ~/.llmhosts
cat > ~/.llmhosts/config.toml << 'EOF'
[server]
host = "127.0.0.1"
port = 4000
workers = 1
log_level = "info"

[ollama]
host = "http://127.0.0.1:11434"
auto_discover = true

[router]
prefer_local = true
cost_optimize = true
fallback_to_cloud = true

[cache]
enabled = true
max_size_mb = 512
ttl_seconds = 3600

[security]
require_auth = true
rate_limit_rpm = 60

[dashboard]
enabled = true
tui = false
web = true
EOF
```

Every config value can be overridden with an environment variable using the format `LLMHOSTS_<SECTION>_<KEY>`:

```bash
export LLMHOSTS_SERVER_PORT=8080
export LLMHOSTS_SECURITY_REQUIREAUTH=true
```

### Add cloud API keys

```bash
llmhosts keys add openai sk-proj-abc123...
llmhosts keys add anthropic sk-ant-abc123...
llmhosts keys validate
```

Keys are encrypted at rest with Fernet symmetric encryption and stored in `~/.llmhosts/`.

## Running as a systemd Service

### 1. Install the unit file

Copy the provided unit file (from `contrib/llmhost.service` in the repo):

```bash
sudo cp contrib/llmhost.service /etc/systemd/system/llmhost.service
```

Or create it manually:

```ini
[Unit]
Description=LLMHosts AI Proxy
After=network.target
Wants=ollama.service

[Service]
Type=simple
User=llmhost
ExecStart=/usr/local/bin/llmhosts serve --host 0.0.0.0 --no-tui
Restart=on-failure
RestartSec=5
Environment=LLMHOSTS_SECURITY_REQUIREAUTH=true
WorkingDirectory=/home/llmhost

# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/llmhost/.llmhosts
PrivateTmp=yes

[Install]
WantedBy=multi-user.target
```

### 2. Enable and start

```bash
sudo systemctl daemon-reload
sudo systemctl enable llmhost
sudo systemctl start llmhost
```

### 3. Check status

```bash
sudo systemctl status llmhost
journalctl -u llmhost -f
```

### 4. Adjust the ExecStart path

If you installed with `pipx` or a virtual environment, update `ExecStart` to the correct path:

```bash
# Find the binary
which llmhost
# Example output: /home/llmhost/.local/bin/llmhost

# Update the service file accordingly
sudo systemctl edit llmhost
# Add:
# [Service]
# ExecStart=
# ExecStart=/home/llmhost/.local/bin/llmhosts serve --host 0.0.0.0 --no-tui
```

## Reverse Proxy Setup

LLMHosts binds to `127.0.0.1:4000` by default. Use a reverse proxy to add TLS termination and expose it on ports 80/443.

### Option A: Caddy (recommended)

Caddy handles TLS certificates automatically via Let's Encrypt.

```bash
# Install Caddy
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy
```

Create `/etc/caddy/Caddyfile`:

```
llmhost.example.com {
    reverse_proxy localhost:4000
}
```

Replace `llmhost.example.com` with your actual domain. Caddy will automatically provision a TLS certificate.

```bash
sudo systemctl enable caddy
sudo systemctl restart caddy
```

### Option B: nginx

```bash
sudo apt install -y nginx certbot python3-certbot-nginx
```

Create `/etc/nginx/sites-available/llmhost`:

```nginx
server {
    listen 80;
    server_name llmhost.example.com;

    location / {
        proxy_pass http://127.0.0.1:4000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support (for live dashboard)
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Streaming SSE support (for chat completions)
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
    }
}
```

Enable the site and get a TLS certificate:

```bash
sudo ln -s /etc/nginx/sites-available/llmhost /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

# TLS via Let's Encrypt
sudo certbot --nginx -d llmhost.example.com
```

**Important nginx notes for LLMHosts:**

- `proxy_buffering off` is required for streaming chat completions (SSE).
- `proxy_read_timeout 300s` prevents nginx from killing long-running inference requests.
- WebSocket headers are needed for the live dashboard at `/api/ws/live`.

## Firewall

```bash
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
```

Do **not** expose port 4000 directly -- let the reverse proxy handle external traffic.

## Monitoring and Health Checks

### Health endpoint

LLMHosts exposes `GET /health` which returns a JSON status. Use this for uptime monitoring:

```bash
curl -s http://localhost:4000/health | python3 -m json.tool
```

### systemd watchdog (optional)

Add a health check to the service file to automatically restart on failure:

```ini
[Service]
# ... existing config ...
ExecStartPost=/bin/sleep 5
ExecStartPost=/usr/bin/curl -sf http://localhost:4000/health
```

### External monitoring

Point any HTTP uptime monitor (UptimeRobot, Healthchecks.io, etc.) at:

```
https://llmhost.example.com/health
```

### Dashboard

The web dashboard is available at `http://localhost:4000/dashboard` (or via your reverse proxy). It shows:

- Request throughput and latency
- Backend status (Ollama, cloud providers)
- Cost savings vs direct API calls
- Cache hit rates

### Prometheus metrics

If you have a Prometheus stack, LLMHosts exports metrics at `/api/metrics`.

## Installing Ollama (optional)

If you want to run local models:

```bash
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3.2
ollama pull qwen2.5:7b

# Verify LLMHosts sees it
llmhosts doctor
```

Ollama runs as a systemd service (`ollama.service`) and listens on `127.0.0.1:11434` by default. LLMHosts auto-discovers it.

## Updating

### pip

```bash
pip install --upgrade llmhost
sudo systemctl restart llmhost
```

### pipx

```bash
pipx upgrade llmhost
sudo systemctl restart llmhost
```

### Verify after update

```bash
llmhosts --version
llmhosts doctor
curl -s http://localhost:4000/health
```

## Troubleshooting

### Port already in use

```bash
# Find what's using port 4000
sudo ss -tlnp | grep 4000

# Use a different port
llmhosts serve --port 8080
# Or set in config: LLMHOST_SERVER_PORT=8080
```

### Ollama not detected

```bash
# Check if Ollama is running
systemctl status ollama
curl http://127.0.0.1:11434/api/tags

# If Ollama is on a different host/port, set it in config.toml:
# [ollama]
# host = "http://192.168.1.100:11434"
```

### Permission denied on ~/.llmhosts

```bash
sudo chown -R llmhost:llmhost /home/llmhost/.llmhosts
```

### Streaming responses cut off (nginx)

Ensure `proxy_buffering off` is set in your nginx config. Without it, nginx buffers SSE responses and the client sees delayed or truncated output.
