---
title: GGUF Recipes
description: Using llama.cpp with GGUF quantized models.
---

GGUF recipes use colon syntax in the model field to select a quantization variant. The primary runtime for GGUF is `llama-cpp`, but SGLang also has experimental GGUF support — see [SGLang GGUF](/runtimes/sglang/#experimental-gguf-support) for details.

## Example

```yaml
model: Qwen/Qwen3-1.7B-GGUF:Q8_0
runtime: llama-cpp
max_nodes: 1
container: scitrera/dgx-spark-llama-cpp:b8076-cu131

metadata:
  description: Qwen3 1.7B (Q8_0 GGUF) -- small test model via llama.cpp
  maintainer: scitrera.ai <open-source-team@scitrera.com>
  model_params: 1.7B
  model_dtype: q8_0

defaults:
  port: 8000
  host: 0.0.0.0
  n_gpu_layers: 99
  ctx_size: 8192

command: |
  llama-server \
      -hf {model} \
      --host {host} \
      --port {port} \
      --n-gpu-layers {n_gpu_layers} \
      --ctx-size {ctx_size} \
      --flash-attn on \
      --jinja \
      --no-webui
```

## Key differences from vLLM/SGLang recipes

- **Model field** uses `repo:quant` syntax (e.g. `Qwen/Qwen3-1.7B-GGUF:Q8_0`). sparkrun parses this to download only the matching quantization files.
- **`max_nodes: 1`** — llama-cpp solo mode does not support multi-node distribution.
- **`ctx_size`** replaces `max_model_len` — the context window size in tokens.
- **`n_gpu_layers`** controls how many layers are offloaded to GPU. Set to `99` to offload all layers.

## Model pre-sync

When model pre-sync is enabled (the default), sparkrun:
1. Downloads only the matching GGUF quant files locally
2. Distributes them to target hosts via rsync
3. Rewrites `-hf` to `-m` with the resolved container cache path

The container serves from the local copy without re-downloading.

## Running

```bash
sparkrun run qwen3-1.7b-llama-cpp
sparkrun run qwen3-1.7b-llama-cpp --solo
```

## Experimental: Multi-node via RPC

llama.cpp has an experimental RPC backend for multi-node inference. Worker nodes run `rpc-server` and the head node connects via `--rpc`.

:::caution
This is still evolving both upstream and in sparkrun. The fastest DGX Spark interconnect communication uses NCCL and RoCE — llama.cpp's RPC mechanism involves more overhead.
:::
