---
title: Networking Best Practices
description: Best practices for configuring DGX Spark networking for multi-node inference clusters.
---

This guide covers how to configure the high-speed ConnectX-7 network on DGX Spark systems for multi-node clustering. Proper network setup is a prerequisite for performant multi-node inference with NCCL.

## DGX Spark networking overview

Each DGX Spark has two wired network connection types:

| Interface | Speed | Purpose |
|---|---|---|
| Built-in Ethernet (management) | 10 Gbps | SSH, general traffic, management |
| ConnectX-7 (QSFP56 ports) | Up to 200 Gbps | NCCL/RDMA, model sync, container distribution |

The ConnectX-7 NIC has **two QSFP56 ports** and supports up to **200 Gbps** aggregate bandwidth. This can be achieved with a single 200 Gbps cable or two 100 Gbps cables.

### The PCIe split

DGX Spark has a unique hardware constraint: the SoC provides a maximum of **PCIe 5.0 x4 lanes per device**. A single x4 link delivers approximately 100 Gbps of bandwidth — half the NIC's 200 Gbps capacity.

To work around this, each physical QSFP56 port is split into **two PCIe x4 partitions**. Each partition appears as a separate network device pair in Linux — one Ethernet interface and one RoCEv2 interface:

```
# Example output of ibdev2netdev with one cable in the "first" port (p0)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
```

With a single cable plugged into one port, you get **two active Ethernet interfaces** (e.g., `enp1s0f1np0` and `enP2p1s0f1np0`) and **two active RoCEv2 interfaces** (e.g., `rocep1s0f0` and `roceP2p1s0f0`). Each pair represents one PCIe x4 partition capable of ~100 Gbps.

Using two cables (one per physical port) also results in two devices per port — four total — but does not meaningfully increase point-to-point bandwidth between two Sparks. **A single cable is sufficient for full 200 Gbps between two nodes.**

:::note
For clusters of **three or more nodes**, you need a switch (e.g., [MikroTik CRS812-DDQ](https://mikrotik.com/product/crs812_ddq) or [MikroTik CRS804-DDQ](https://mikrotik.com/product/crs804_ddq).
:::

## Network design principles

### Separate management and cluster traffic

The single most important networking decision: **keep cluster (NCCL/RDMA) traffic off the management network, and keep management traffic off the CX-7 interfaces.**

- **Management network (10 Gbps built-in Ethernet):** SSH, system administration, general internet access, package updates.
- **Cluster network (CX-7, up to 200 Gbps):** NCCL communication, model file sync (rsync), container image distribution. Nothing else.

Minimize non-cluster activity on the CX-7 subnets. Stray traffic competes with NCCL and degrades inference performance.

sparkrun follows this principle by default — SSH orchestration and control-plane communication go over the management interface, while model sync, container distribution, and NCCL all use the high-speed CX-7 interfaces. In fact, it would be ideal to NOT do container and model synchronization over CX-7 interfaces; however, it's realistically the most *practical* choice to do so.

### Always use static IPs

Use static IP addresses on every CX-7 interface. Do not use DHCP or auto-assigned addresses.

### Choosing your subnets

You need **two private subnets** for the CX-7 cluster interfaces — one per PCIe partition. These are isolated point-to-point or switched links with no internet routing, so you must use addresses from the RFC 1918 private ranges:

| Range | CIDR | Addresses | Common name |
|---|---|---|---|
| `10.0.0.0` – `10.255.255.255` | `10.0.0.0/8` | ~16.7 million | Class A private |
| `172.16.0.0` – `172.31.255.255` | `172.16.0.0/12` | ~1 million | Class B private |
| `192.168.0.0` – `192.168.255.255` | `192.168.0.0/16` | ~65,000 | Class C private |

Any of these ranges work. Use a **/24 subnet mask** (254 usable addresses) — it is the simplest to reason about and more than enough for any Spark cluster. Even an 8-node cluster only needs 8 addresses per subnet.

**The key rule: your cluster subnets must not overlap with any network your Sparks are already connected to.** Check for conflicts with:

- Your management network / LAN (e.g., if your Sparks get management IPs via DHCP on `192.168.1.0/24`, don't use that range)
- VPN address ranges
- Docker's default bridge networks (`172.17.0.0/16` and similar)
- Any other subnets routed on the machine

A simple approach: pick two adjacent `/24` subnets from a range you know is unused. Throughout this guide we use `192.168.11.0/24` and `192.168.12.0/24` as examples — replace these with whatever is free in your environment.

### Assign IPs to both partitions — on different subnets

Each physical port exposes two Ethernet interfaces (the "twin" pair from the PCIe split). Assign an IP address to **both** interfaces, but they **must be on different subnets**:

| Interface | Subnet | Example IP |
|---|---|---|
| `enp1s0f1np0` (partition 1) | `192.168.11.0/24` | `192.168.11.13` |
| `enP2p1s0f1np0` (partition 2) | `192.168.12.0/24` | `192.168.12.13` |

:::caution
**Never put both partitions on the same subnet.** Same-subnet twin interfaces confuse interface auto-discovery and break routing. This is a common misconfiguration that causes subtle, hard-to-debug failures.
:::

Assigning IPs to both partitions allows tools like sparkrun to potentially be able to use both paths for data transfers (model sync, container distribution), maximizing the available 200 Gbps. NCCL uses RoCEv2 directly and will utilize both RoCE interfaces regardless, but having IPs on both enables addressing both interfaces via TCP/IP interfaces.

### Jumbo frames (MTU 9000)

MTU is maximum transmission unit. It sets the upper bound on packet size. The default & standard MTU is 1500. Jumbo Frames are an "industry-standard" but technically not part of proper standards (which is why your Internet connection won't allow jumbo frames).

Set MTU to **9000** on all CX-7 interfaces that are participating in data transfer activity. Jumbo frames reduce per-packet overhead and measurably improve throughput for large data transfers.

```yaml
# /etc/netplan/40-cx7.yaml — set MTU on both twin interfaces
network:
  version: 2
  ethernets:
    enp1s0f1np0:
      dhcp4: no
      mtu: 9000
      addresses: [192.168.11.13/24]
    enP2p1s0f1np0:
      dhcp4: no
      mtu: 9000
      addresses: [192.168.12.13/24]
```

If your cluster uses a switch, the switch ports should be configured for MTU 9216 to leave room MTU 9000 + overhead.

:::caution
An MTU mismatch *anywhere* in the path may result in packets being silently dropped.
:::

## Setup sequence

The end-to-end setup involves three sparkrun commands. Register the cluster first so subsequent commands can reference it by name:

| Step | Command | Notes |
|---|---|---|
| 1. Register cluster | `sparkrun cluster create mylab --hosts <mgmt-IPs>` | Save hosts by name using management IPs |
| 2. SSH access | `sparkrun setup ssh` | Interactive — prompts for passwords; uses default cluster |
| 3. CX-7 networking | `sparkrun setup cx7` | Requires passwordless SSH from step 2. Will also prompt for passwords as needed |

`sparkrun setup ssh` is interactive — it prompts for passwords and doesn't require pre-existing passwordless SSH. `sparkrun setup cx7` runs remote scripts non-interactively, so it **does** require passwordless SSH to already be in place. That's why SSH setup comes first.

Your cluster stays registered with the **management IPs** — there is no need to update it after CX-7 setup. sparkrun automatically discovers CX-7 interfaces on each host and uses them for high-bandwidth operations (model sync, container distribution, NCCL), while SSH orchestration continues over the management network.

```bash
# Step 1: Register cluster with management IPs
sparkrun cluster create mylab \
  --hosts 10.24.11.13,10.24.11.14 \
  --user dgxuser \
  -d "2-node DGX Spark cluster"
sparkrun cluster set-default mylab

# Step 2: SSH setup (prompts for passwords, uses default cluster)
sparkrun setup ssh

# Step 3: Configure CX-7 interfaces (uses default cluster)
sparkrun setup cx7
```

See [SSH Setup](/getting-started/ssh-setup/) for more details on SSH configuration options.

## Configuring CX-7 interfaces with sparkrun

`sparkrun setup cx7` automates the CX-7 configuration — it SSHs into each host, detects ConnectX-7 interfaces, selects two conflict-free subnets, assigns static IPs with jumbo frames, and applies the netplan configuration. This is the recommended approach.

### Basic usage

```bash
# Specify hosts directly (management IPs)
sparkrun setup cx7 --hosts 10.24.11.13,10.24.11.14

# Or use a saved cluster
sparkrun setup cx7 --cluster mylab

# Preview what would be done without making changes
sparkrun setup cx7 --cluster mylab --dry-run
```

### What it does

1. **Detects** CX-7 interfaces on each host via SSH (runs `ibdev2netdev`, identifies twin pairs)
2. **Selects subnets** — automatically picks two `/24` subnets from RFC 1918 space that don't conflict with any existing networks on the hosts. If hosts already have a valid CX-7 configuration, it preserves those subnets.
3. **Assigns IPs** — derives the last octet of each CX-7 IP from the host's management IP (e.g., management IP `10.24.11.13` → CX-7 IPs `192.168.11.13` and `192.168.12.13`), making the addressing scheme easy to remember.
4. **Applies netplan** — generates and writes `/etc/netplan/40-cx7.yaml` on each host with static IPs, MTU 9000, and runs `netplan apply`. Requires sudo on the target hosts.

Existing valid configurations are preserved — if a host already has correct CX-7 netplan config with the right subnets and MTU, it is skipped. Use `--force` to reconfigure all hosts regardless.

### Specifying subnets

If you prefer specific subnets rather than automatic selection:

```bash
sparkrun setup cx7 --cluster mylab \
  --subnet1 192.168.11.0/24 \
  --subnet2 192.168.12.0/24
```

Both `--subnet1` and `--subnet2` must be provided together. See [Choosing your subnets](#choosing-your-subnets) above for guidance on picking ranges that won't conflict.

### Options reference

| Option | Description |
|---|---|
| `--hosts` / `-H` | Comma-separated host list (management IPs or hostnames) |
| `--hosts-file` | File with hosts (one per line, `#` comments) |
| `--cluster` | Use a saved cluster definition |
| `--user` / `-u` | SSH username (default: from cluster config or current user) |
| `--subnet1` | Override subnet for CX-7 partition 1 (e.g., `192.168.11.0/24`) |
| `--subnet2` | Override subnet for CX-7 partition 2 (e.g., `192.168.12.0/24`) |
| `--mtu` | MTU for CX-7 interfaces (default: `9000`) |
| `--dry-run` / `-n` | Show the plan without making changes |
| `--force` | Reconfigure even if existing config is valid |

### Verify connectivity

After `sparkrun setup cx7` completes, verify the links are working:

```bash
# From spark1 — test both subnets
ping -c 3 192.168.11.14
ping -c 3 192.168.12.14
```

Verify jumbo frames are effective end-to-end:

```bash
ping -M do -s 8972 -c 3 192.168.11.14
```

A successful large-packet ping confirms MTU 9000 is working. If it fails, check MTU on both endpoints and any switch in between.

### Manual configuration

If you prefer to configure netplan by hand (or need to troubleshoot), the process is:

1. Identify your interfaces with `ibdev2netdev` — the "Up" interfaces correspond to your connected port
2. Create `/etc/netplan/40-cx7.yaml` on each node with static IPs on both twin interfaces, different subnets, and MTU 9000:

```yaml
# /etc/netplan/40-cx7.yaml
network:
  version: 2
  ethernets:
    enp1s0f0np0:
      dhcp4: no
      mtu: 9000
      addresses: [192.168.11.13/24]
    enP2p1s0f0np0:
      dhcp4: no
      mtu: 9000
      addresses: [192.168.12.13/24]
```

3. Apply on each node:

```bash
sudo chmod 600 /etc/netplan/40-cx7.yaml
sudo netplan apply
```

## Host preparation checklist

Beyond networking, each DGX Spark in the cluster needs some basic setup.

### Stay on DGX OS

We recommend sticking with the **DGX OS** that ships with the Spark rather than switching to another Linux distribution. DGX OS includes NVIDIA drivers, CUDA, container runtime, and firmware that are tested together. Switching distributions risks driver/firmware compatibility issues.

### Same username across all nodes

Use the **same username** on every Spark in the cluster. sparkrun connects via SSH and expects a consistent user across hosts.

Ideally, the cluster user would be a dedicated service account rather than your personal login. In practice, most Spark deployments have a single user, and that works fine.

### Required packages

These should already be installed on DGX OS, but verify:

```bash
# Confirm these are available
which rsync git docker
```

If anything is missing:

```bash
sudo apt-get update && sudo apt-get install -y rsync git
```

## Verifying RDMA performance

Once networking is configured, use the `perftest` package to verify RDMA bandwidth and latency.

### Bandwidth test

On the receiver node:

```bash
ib_write_bw -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB
```

On the sender node:

```bash
ib_write_bw 192.168.11.14 -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB
```

You should see approximately **100 Gbps** per RoCE interface. With both twins active via NCCL, aggregate bandwidth approaches 200 Gbps.

### Latency test

On the receiver node:

```bash
ib_write_lat -d rocep1s0f1 --report_gbits -R --force-link IB
```

On the sender node:

```bash
ib_write_lat 192.168.11.14 -d rocep1s0f1 --report_gbits -R --force-link IB
```

Typical latency between two directly connected Sparks is approximately **1.5 microseconds**.

## Quick reference

| Item | Recommendation |
|---|---|
| Cable type | QSFP56, 200 Gbps (single cable per node pair is sufficient) |
| IP addressing | Static IPs on both CX-7 partitions, different subnets |
| MTU | 9000 on all CX-7 interfaces (and switch ports) |
| Management traffic | Route over 10 Gbps built-in Ethernet, not CX-7 |
| Inference traffic | Route over 10 Gbps built-in Ethernet, not CX-7 |
| Cluster traffic | NCCL, model sync, container distribution over CX-7 only |
| OS | DGX OS (do not switch distributions) |
| User account | Same username on every node, member of `docker` group |
| Required tools | `rsync`, `git`, `docker` (pre-installed on DGX OS) |
| Switch (3+ nodes) | Required to maximize bandwidth |