---
title: llama.cpp
description: Using llama.cpp for GGUF model inference on DGX Spark.
---

[llama.cpp](https://github.com/ggml-org/llama.cpp) support in sparkrun provides a lightweight runtime for GGUF quantized models via `llama-server`.

## Features

- Solo mode with GGUF quantized models
- Loads models directly from HuggingFace (e.g. `Qwen/Qwen3-1.7B-GGUF:Q8_0`)
- Automatic model pre-sync with selective quant file download
- Lightweight alternative to vLLM/SGLang

## Container images

sparkrun works with ready-built images including:

- [scitrera/dgx-spark-llama-cpp](https://hub.docker.com/r/scitrera/dgx-spark-llama-cpp)
- Build your own

## GGUF model syntax

GGUF models use colon syntax to select a quantization variant:

```yaml
model: Qwen/Qwen3-1.7B-GGUF:Q8_0
```

sparkrun pre-downloads only the matching quant files and resolves the local cache path so the container doesn't need to re-download at serve time.

## Example recipe

```yaml
model: Qwen/Qwen3-1.7B-GGUF:Q8_0
runtime: llama-cpp
max_nodes: 1
container: scitrera/dgx-spark-llama-cpp:b8076-cu131

metadata:
  description: Qwen3 1.7B (Q8_0 GGUF)
  model_params: 1.7B
  model_dtype: q8_0

defaults:
  port: 8000
  host: 0.0.0.0
  n_gpu_layers: 99
  ctx_size: 8192

command: |
  llama-server \
      -hf {model} \
      --host {host} \
      --port {port} \
      --n-gpu-layers {n_gpu_layers} \
      --ctx-size {ctx_size} \
      --flash-attn on \
      --jinja \
      --no-webui
```

## Key parameters

| Parameter | Description |
|---|---|
| `n_gpu_layers` | Number of layers offloaded to GPU (99 = all) |
| `ctx_size` | Context window size in tokens |

## Running

```bash
sparkrun run qwen3-1.7b-llama-cpp
sparkrun run qwen3-1.7b-llama-cpp --solo
```

## Experimental: Multi-node via RPC

llama.cpp has an experimental RPC backend for multi-node inference. Worker nodes run `rpc-server` and the head node connects via `--rpc`.

:::caution
This is still evolving upstream. The fastest DGX Spark interconnect uses NCCL and RoCE — llama.cpp's RPC mechanism involves more overhead.
:::
