# Tutorial: Multi-GPU Setup with vLLM

Deploy LLMHosts with multiple GPUs using vLLM for high-throughput inference. vLLM supports tensor parallelism across GPUs for running large models.

**Time**: ~30 minutes | **Prerequisites**: Ubuntu 22.04+, 2+ NVIDIA GPUs (24GB+ VRAM each), Docker, NVIDIA Container Toolkit

## 1. Prerequisites

Complete steps 1-3 from the [Single GPU tutorial](./single-gpu-ollama.md) (drivers, Docker, NVIDIA Container Toolkit).

Verify all GPUs are visible:
```bash
nvidia-smi
# Should show multiple GPUs (GPU 0, GPU 1, etc.)
```

## 2. Run vLLM

vLLM is a high-throughput inference engine with PagedAttention. Run it via Docker:

```bash
docker run -d \
  --name vllm \
  --gpus all \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-2-13b-chat-hf \
  --tensor-parallel-size 2 \
  --max-model-len 4096
```

| Flag | Description |
|------|-------------|
| `--tensor-parallel-size 2` | Split model across 2 GPUs |
| `--max-model-len 4096` | Max context length |
| `--ipc=host` | Required for multi-GPU shared memory |

**For gated models** (Llama 2, etc.), add your HuggingFace token:
```bash
docker run -d \
  --name vllm \
  --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -e HUGGING_FACE_HUB_TOKEN=hf_your_token \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-2-13b-chat-hf \
  --tensor-parallel-size 2
```

Verify vLLM is running:
```bash
curl http://localhost:8000/v1/models
```

## 3. Install and Configure LLMHosts

```bash
pip install llmhosts

cat > ~/.config/llmhost/config.toml << 'EOF'
[server]
host = "0.0.0.0"
port = 4000

[security]
require_auth = true

[cache]
enabled = true
max_size_mb = 2048

# Point to vLLM instead of Ollama
[backends.vllm]
type = "openai"
url = "http://localhost:8000/v1"
models = ["meta-llama/Llama-2-13b-chat-hf"]
EOF
```

## 4. Start and Verify

```bash
llmhosts serve

# In another terminal:
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-13b-chat-hf",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

## 5. Multi-GPU Model Guide

| GPUs | Total VRAM | Recommended Models |
|------|-----------|-------------------|
| 2x RTX 3090 (48GB) | 48GB | Llama 2 13B, CodeLlama 34B (Q4) |
| 2x RTX 4090 (48GB) | 48GB | Llama 2 13B, Mixtral 8x7B (Q4) |
| 4x A100 (320GB) | 320GB | Llama 2 70B, Mixtral 8x22B |

## 6. Running vLLM + Ollama Together

You can run both for different model sizes:

```toml
# config.toml
[backends.vllm]
type = "openai"
url = "http://localhost:8000/v1"
models = ["meta-llama/Llama-2-13b-chat-hf"]

[backends.ollama]
type = "ollama"
url = "http://localhost:11434"
models = ["llama2:7b", "codellama:7b"]
```

LLMHosts routes requests to the appropriate backend based on model name.

## Troubleshooting

| Issue | Fix |
|-------|-----|
| `CUDA out of memory` | Reduce `--max-model-len` or use smaller model |
| vLLM can't see all GPUs | Check `CUDA_VISIBLE_DEVICES` isn't set. Use `--gpus all`. |
| Slow first request | Model loading takes 1-2 minutes. Subsequent requests are fast. |
| `torch.distributed` errors | Ensure `--ipc=host` is set |
| Model download fails | Add `-e HUGGING_FACE_HUB_TOKEN=hf_...` for gated models |

## Next Steps

- [Add remote access with Tailscale](./docker-tailscale.md)
- [Add cloud API fallback for overflow](./cpu-only-byok.md)
- [Production Docker Compose deployment](../deployment/docker-production.md)
