---
title: vLLM
description: Using vLLM for inference on DGX Spark.
---

[vLLM](https://github.com/vllm-project/vllm) is the default runtime in sparkrun. It provides first-class support for solo and multi-node inference with two clustering variants:

- **`vllm-distributed`** — uses vLLM's built-in distributed backend (the default)
- **`vllm-ray`** — uses Ray head/worker orchestration

You can write `runtime: vllm` in a recipe and sparkrun resolves it automatically — see [Runtime alias resolution](#runtime-alias-resolution) below.

## Features

- Solo and multi-node tensor parallelism across DGX Spark nodes
- Broad model support (most HuggingFace models)
- PagedAttention for efficient KV cache management
- Tool calling support
- Two multi-node strategies: vLLM native distributed or Ray clustering

## Container images

Both `vllm-distributed` and `vllm-ray` use the same container images. sparkrun works with ready-built images including:

- [Self-built community images from eugr/spark-vllm-docker](https://github.com/eugr/spark-vllm-docker) (often the first to have the latest fixes)
- [scitrera/dgx-spark-vllm](https://hub.docker.com/r/scitrera/dgx-spark-vllm)
- [Official NVIDIA Images](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm)

## Runtime alias resolution

When a recipe specifies `runtime: vllm` (or omits `runtime` entirely), sparkrun resolves it to a concrete runtime in this order:

1. If the recipe has eugr signals (`build_args`, `mods`, or `recipe_version: "1"`) → **`eugr-vllm`** (see [eugr-vllm](/runtimes/eugr-vllm/))
2. If the recipe hints at Ray usage (`distributed_executor_backend: ray` in defaults, or `--distributed-executor-backend ray` in the command template) → **`vllm-ray`**
3. Otherwise → **`vllm-distributed`** (the default)

You can also set `runtime: vllm-distributed` or `runtime: vllm-ray` explicitly to bypass this resolution.

## Example recipe

This recipe uses `runtime: vllm`, which resolves to `vllm-distributed` by default:

```yaml
model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
runtime: vllm
min_nodes: 1
container: scitrera/dgx-spark-vllm:0.16.0-t5

metadata:
  description: NVIDIA Nemotron 3 Nano 30B (upstream NVFP4)
  maintainer: scitrera.ai <open-source-team@scitrera.com>
  model_params: 30B
  model_dtype: nvfp4

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.8
  max_model_len: 200000
  served_model_name: nemotron3-30b-a3b

command: |
  vllm serve \
      {model} \
      --served-model-name {served_model_name} \
      --max-model-len {max_model_len} \
      --gpu-memory-utilization {gpu_memory_utilization} \
      -tp {tensor_parallel} \
      --host {host} \
      --port {port}
```

## Multi-node with vllm-distributed (default)

`vllm-distributed` uses vLLM's built-in multi-node support. Each node runs the full `vllm serve` command with node-specific arguments:

1. Cleans up existing containers on each node
2. Detects InfiniBand/RDMA interfaces on all hosts
3. Detects the head node IP
4. Launches the head node (rank 0) with `--nnodes`, `--node-rank 0`, and `--master-addr`
5. Waits for the master port to become ready
6. Launches worker nodes in parallel, each with `--node-rank N` and `--headless`

```bash
sparkrun run nemotron3-nano-30b-nvfp4-vllm --tp 2
```

No Ray cluster is involved — vLLM coordinates the nodes directly via its native distributed backend.

## Multi-node with vllm-ray

`vllm-ray` forms a Ray cluster first, then runs the serve command on the head node:

1. Cleans up existing containers on each node
2. Detects InfiniBand/RDMA interfaces on all hosts
3. Starts a Ray head node on the first host
4. Joins worker nodes to the Ray cluster
5. Launches the vLLM serve command on the head container

To use `vllm-ray`, either set `runtime: vllm-ray` explicitly or add the Ray hint to defaults:

```yaml
runtime: vllm
defaults:
  distributed_executor_backend: ray
```

### Ray-specific options

These options apply only to `vllm-ray` (and `eugr-vllm`, which extends it):

| Option | Default | Description |
|---|---|---|
| `--ray-port` | 46379 | Ray GCS port |
| `--dashboard` | off | Enable Ray dashboard |
| `--dashboard-port` | 8265 | Ray dashboard port |

## Choosing between variants

Both variants support the same models and vLLM flags. The difference is in multi-node orchestration:

- **`vllm-distributed`** is simpler — no Ray dependency, each node runs independently with vLLM's built-in coordination. This is the default.
- **`vllm-ray`** provides Ray's cluster management, dashboard, and monitoring. Use it when you need Ray-specific features or when running `eugr-vllm` (which extends `vllm-ray`).

For single-node (solo) workloads, both variants behave identically.
