# Multi-Node

Run distributed training across multiple nodes.

## CLI

```bash
ml launch task.yaml --num-nodes 2
```

## YAML

```yaml
num_nodes: 2

resources:
  accelerators: B200:8

run: |
  torchrun \
    --nproc_per_node=8 \
    --nnodes=$SKYPILOT_NUM_NODES \
    train.py
```

## SDK

```python
task = sky.Task(
    run='torchrun --nproc_per_node=8 train.py',
    num_nodes=2,
)
```

## Environment variables

SkyPilot automatically sets distributed training variables:

| Variable | Description |
|----------|-------------|
| `SKYPILOT_NUM_NODES` | Total number of nodes |
| `SKYPILOT_NODE_RANK` | Current node's rank (0-indexed) |
| `SKYPILOT_NODE_IPS` | Comma-separated list of node IPs |

## Complete example

```yaml
name: distributed-llm-training

num_nodes: 4

resources:
  infra: mithril
  accelerators: B200:8

workdir: .

setup: |
  pip install -r requirements.txt

run: |
  torchrun \
    --nproc_per_node=8 \
    --nnodes=$SKYPILOT_NUM_NODES \
    --node_rank=$SKYPILOT_NODE_RANK \
    --master_addr=$(echo $SKYPILOT_NODE_IPS | cut -d',' -f1) \
    --master_port=29500 \
    train.py
```

## Notes

- All nodes get same resources specification
- Master node is rank 0
- Network between nodes is automatically configured
