---
title: Tips & Troubleshooting
description: Common issues and practical tips for running sparkrun on DGX Spark systems.
---

## HuggingFace cache owned by root

DGX Spark inference containers typically run as **root** inside rootful Docker. If a container downloads a model from HuggingFace — either intentionally or because a file wasn't pre-cached (e.g., a tokenizer config) — the downloaded files in the HuggingFace cache end up **owned by root**. This causes permission errors on subsequent runs when your non-root user can't read or update the cache.

:::note[Cache directory resolution]
sparkrun uses the same cache directory as `huggingface_hub`, resolved in this order:

1. `--cache-dir` CLI flag or `hf_cache_dir` in `~/.config/sparkrun/config.yaml` (sparkrun-specific override)
2. `HF_HUB_CACHE` environment variable
3. `HUGGINGFACE_HUB_CACHE` environment variable (legacy alias)
4. `HF_HOME` environment variable + `/hub`
5. `~/.cache/huggingface` (default)

If you've set `HF_HOME` or `HF_HUB_CACHE`, sparkrun respects those automatically — no sparkrun-specific configuration needed.
:::

sparkrun avoids this by downloading models **as your user** on the host and syncing them to cluster nodes before launching containers. But if models were previously downloaded inside a container (or you ran a container manually that pulled a model), you may already have root-owned files.

**Symptoms:**

- `Permission denied` or `OSError: [Errno 13]` errors referencing paths under your HuggingFace cache (typically `~/.cache/huggingface`)
- A model that worked before suddenly fails on launch
- sparkrun's model sync step fails with permission errors

**Fix:**

sparkrun has a built-in command that fixes ownership across all cluster hosts in one shot:

```bash
# Fix permissions on your default cluster
sparkrun setup fix-permissions

# Or target specific hosts
sparkrun setup fix-permissions --cluster mylab

# Custom cache location
sparkrun setup fix-permissions --cluster mylab --cache-dir /data/hf-cache

# Preview what would be done
sparkrun setup fix-permissions --cluster mylab --dry-run
```

This runs `chown` on the HuggingFace cache directory across all target hosts. It tries passwordless sudo first; if that fails, it prompts for a password once.

**Automatic fix on every run:**

sparkrun automatically attempts to fix cache ownership on remote hosts before syncing models. If non-interactive sudo is available (either via general NOPASSWD or a scoped sudoers entry), this happens silently. If sudo isn't available, sparkrun logs a warning and continues — the rsync may still succeed if permissions aren't actually broken.

To make this automatic fix work without ever needing a password, run `--save-sudo` once:

```bash
# Install a scoped sudoers entry on all cluster hosts (one-time setup)
sparkrun setup fix-permissions --cluster mylab --save-sudo
```

This installs a minimal sudoers rule on each host that permits **only** `chown -R <user> <cache_dir>` without a password — no broader sudo privileges are granted. After this, every future `sparkrun run` silently fixes cache ownership before model distribution.

```bash
# Preview what the sudoers entry would look like
sparkrun setup fix-permissions --cluster mylab --save-sudo --dry-run
```

Alternatively, you can fix a single machine manually (adjust the path if you've set `HF_HOME`):

```bash
sudo chown -R $USER:$USER ~/.cache/huggingface
```

**Prevention:**

- Let sparkrun handle model downloading and distribution — it downloads as your user and rsyncs to target nodes.
- Avoid running `huggingface-cli download` or model download scripts inside inference containers. Download on the host instead.
- Run `sparkrun setup fix-permissions --save-sudo` once per cluster so that cache ownership is corrected automatically on every launch.

## Linux page cache consuming memory

DGX Spark systems have 128 GB of unified CPU/GPU memory. The Linux kernel's page cache can consume a significant portion of this, reducing the memory available for inference containers. While the kernel will reclaim cached pages under pressure, large cached datasets can cause initial allocation failures or slower model loading.

**Symptoms:**

- Out-of-memory errors when launching large models that should fit in VRAM
- Model loading is slower than expected on a freshly booted system with prior file I/O
- `free -h` shows large amounts of memory in the "buff/cache" column

**Fix:**

sparkrun has a built-in command to drop the page cache across all cluster hosts:

```bash
# Clear page cache on your default cluster
sparkrun setup clear-cache

# Or target specific hosts
sparkrun setup clear-cache --cluster mylab

# Preview what would be done
sparkrun setup clear-cache --cluster mylab --dry-run
```

This runs `sync` followed by writing `3` to `/proc/sys/vm/drop_caches` on each host, which safely drops the page cache, dentries, and inodes.

**Automatic clearing on every run:**

sparkrun automatically attempts to drop the page cache on all target hosts before launching containers. If non-interactive sudo is available (either via general NOPASSWD or a scoped sudoers entry), this happens silently. If sudo isn't available, sparkrun logs a warning and continues.

To make this automatic clearing work without ever needing a password, run `--save-sudo` once:

```bash
# Install a scoped sudoers entry on all cluster hosts (one-time setup)
sparkrun setup clear-cache --cluster mylab --save-sudo
```

This installs a minimal sudoers rule on each host that permits **only** writing to `/proc/sys/vm/drop_caches` without a password — no broader sudo privileges are granted. After this, every future `sparkrun run` silently clears the page cache before launching containers.

```bash
# Preview what the sudoers entry would look like
sparkrun setup clear-cache --cluster mylab --save-sudo --dry-run
```