---
title: SGLang
description: Using SGLang for inference on DGX Spark.
---

[SGLang](https://github.com/sgl-project/sglang) is a first-class runtime in sparkrun with support for solo and multi-node inference using SGLang's native distributed backend.

## Features

- Solo and multi-node clustering via SGLang's built-in distribution
- Native distributed backend (`--dist-init-addr`, `--nnodes`, `--node-rank`)
- High-throughput inference
- Structured generation support
- Experimental GGUF support (quantized models via SGLang)

## Container images

sparkrun works with ready-built images including:

- [scitrera/dgx-spark-sglang](https://hub.docker.com/r/scitrera/dgx-spark-sglang)
- [Official NVIDIA Images](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/sglang)

## Example recipe

```yaml
model: Qwen/Qwen3-1.7B
runtime: sglang
min_nodes: 1
container: scitrera/dgx-spark-sglang:0.5.8-t5

metadata:
  description: Qwen3 1.7B via SGLang
  model_params: 1.7B
  model_dtype: bf16

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1

command: |
  python3 -m sglang.launch_server \
      --model-path {model} \
      --host {host} \
      --port {port} \
      --tp {tensor_parallel}
```

## Multi-node inference

For multi-node, sparkrun uses SGLang's native distribution:

```bash
sparkrun run my-sglang-recipe --tp 2
```

### SGLang-specific options

| Option | Default | Description |
|---|---|---|
| `--init-port` | 25000 | SGLang distributed init port |

## Experimental: GGUF support

SGLang has experimental support for GGUF quantized models. This lets you run quantized models through SGLang's high-performance serving stack instead of llama.cpp.

:::caution
SGLang GGUF support is experimental. You may need to pin specific attention backends and provide an explicit tokenizer path. Test thoroughly before production use.
:::

### Example: Qwen3.5-397B MoE (Q6_K GGUF, 4-node)

```yaml
model: unsloth/Qwen3.5-397B-A17B-GGUF:Q6_K
runtime: sglang
min_nodes: 4
container: scitrera/dgx-spark-sglang:0.5.8-t5

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 4
  gpu_memory_utilization: 0.8
  max_model_len: 200000
  served_model_name: qwen3.5-397b
  attention_backend: triton
  tool_call_parser: qwen3_coder
  tokenizer_path: Qwen/Qwen3.5-397B-A17B

command: |
  python3 -m sglang.launch_server \
      --model-path {model} \
      --served-model-name {served_model_name} \
      --context-length {max_model_len} \
      --mem-fraction-static {gpu_memory_utilization} \
      --tp-size {tensor_parallel} \
      --host {host} \
      --port {port} \
      --attention-backend {attention_backend} \
      --tokenizer-path {tokenizer_path} \
      --tool-call-parser {tool_call_parser}
```

### Key notes for SGLang GGUF

- **`tokenizer_path`** — GGUF files don't bundle tokenizer configs, so you must point to the original / non-GGUF HuggingFace repo (e.g. `Qwen/Qwen3.5-397B-A17B`)
- Model field uses the same `repo:quant` syntax as llama-cpp GGUF recipes (e.g. `unsloth/Qwen3.5-397B-A17B-GGUF:Q6_K`)
