MIRROR-DEDUPE(1)

NAME
mirror-dedupe - Ubuntu mirror synchronisation with global deduplication

SYNOPSIS
mirror-dedupe [--config DIR] [-h] [--dry-run] [--mirror MIRROR] [--dedupe-only] [--list] [--activate MIRROR] [--deactivate MIRROR] [--test MIRROR] [--delete MIRROR] ]

DESCRIPTION
mirror-dedupe synchronises Ubuntu package repositories and performs global deduplication across multiple mirrors using hardlinks. This significantly reduces disk space usage when hosting multiple Ubuntu mirrors or architectures.

The tool operates in three modes:

1.  Orchestrator mode (default): Spawns subprocesses for each configured mirror, then performs global deduplication across all mirrors.

2.  Single mirror mode (--mirror): Processes only the specified mirror.

3.  Dedupe-only mode (--dedupe-only): Skips mirror synchronisation and only performs deduplication.

OPTIONS

-h, --help
    Show help message and exit.

--dry-run
    Show what would be done without actually making any changes.

--mirror MIRROR
    Process only the specified mirror by name. Used internally by the orchestrator to spawn subprocesses for each mirror.

--dedupe-only
    Only run the deduplication phase, skipping mirror synchronisation.

--config DIR
    Path to configuration directory. Default: /etc/mirror-dedupe.

--list
    List available mirrors, showing which are active (enabled) and which are inactive.

--activate MIRROR
    Activate a mirror by creating a symlink in repos-enabled/.

--deactivate MIRROR
    Deactivate a mirror by removing its symlink from repos-enabled/.

--test MIRROR
    Test a mirror configuration. This performs basic connectivity checks against the upstream URL (and configured GPG key URL, if present) and prints a summary of what the mirror is configured to fetch, without making any changes.

--delete MIRROR
    Deactivate a mirror and delete its data directory. This is a destructive operation and requires an interactive PIN confirmation before proceeding.

CONFIGURATION
Configuration files are located in /etc/mirror-dedupe/:

mirror-dedupe.conf
    Main configuration file containing global settings such as buffer size, parallel downloads, and curl timeout. It also supports a global architectures mask to restrict which architectures are mirrored, and a boolean no_hardlinks option which, when set to true, disables deduplication and causes mirror-dedupe to behave as a plain mirror.

repos-available/*.conf
    Available mirror configuration files. Each file defines a repository mirror with settings including:

    name
        Mirror identifier

    upstream
        Source repository URL

    dest
        Local destination path (relative to repo_root)

    sync_method
        Synchronisation method (rsync or https)

    architectures
        List of architectures to mirror (e.g., amd64, arm64)

    components
        Repository components (e.g., main, universe, multiverse)

    distributions
        Ubuntu releases to mirror (e.g., noble, jammy)

    gpg_key_url
        URL to repository GPG key (optional)

    storage_filters
        Optional per-mirror storage filters that control which files are downloaded and kept on disk without modifying upstream indices. This is a mapping that may contain:

        exclude_packages
            List of shell-style patterns matched against the package name (the Package: field in Packages/Sources). Any entry whose package name matches one of these patterns is skipped during planning: it will not be downloaded and will be removed by the cleanup phase if it already exists in the pool.

        exclude_paths
            List of shell-style patterns matched against the relative pool path (for example, pool/main/g/grafana/grafana-enterprise-nightly_*). Entries whose paths match one of these patterns are also skipped.

           These filters only affect what is stored locally. The upstream
           Release/InRelease and index files are always mirrored
           unchanged and retain their original signatures. Mirrors that use
           storage_filters may advertise packages that are not present on disk.

repos-enabled/*.conf
    Symlinks to configuration files in repos-available/ for mirrors that are currently enabled.

EXAMPLES

Run full synchronisation and deduplication:
    mirror-dedupe

Dry run to see what would be changed:
    mirror-dedupe --dry-run

Process only the 'ubuntu' mirror:
    mirror-dedupe --mirror ubuntu

Only perform deduplication (skip sync):
    mirror-dedupe --dedupe-only

Use custom configuration directory:
    mirror-dedupe --config /path/to/config

SYSTEMD INTEGRATION
The package includes systemd timer and service units:

mirror-dedupe.timer
    Scheduled timer that runs the sync daily at a randomised time between 00:00-06:00 UTC.

mirror-dedupe.service
    Service unit that executes the mirror synchronisation and deduplication.

To check timer status:
    systemctl status mirror-dedupe.timer

To manually trigger a sync:
    systemctl start mirror-dedupe.service

To view logs:
    journalctl -u mirror-dedupe.service

FILES

/etc/mirror-dedupe/mirror-dedupe.conf
    Main configuration file

/etc/mirror-dedupe/repos-available/
    Directory containing available mirror configurations

/etc/mirror-dedupe/repos-enabled/
    Directory containing symlinks to enabled mirror configurations

/srv/mirror/repos/
    Default base directory for mirrored repositories

/var/run/mirror_dedupe.*.pid
    Lock files to prevent concurrent processing of the same mirror

/usr/lib/systemd/system/mirror-dedupe.service
    Systemd service unit

/usr/lib/systemd/system/mirror-dedupe.timer
    Systemd timer unit

DEDUPLICATION
The tool performs global deduplication by:

1.  Collecting all package files across all configured mirrors

2.  Grouping files by SHA256 hash

3.  Downloading the first occurrence of each unique file

4.  Creating hardlinks for duplicate files (same hash, different paths)

This approach significantly reduces disk usage when hosting multiple mirrors or architectures, as identical packages are stored only once.

EXIT STATUS

0   Success

1   Error occurred during synchronisation or deduplication

AUTHOR
Written by Tim Hosking.

COPYRIGHT
Copyright 2025 Tim Hosking.
Licence: MIT

SEE ALSO
curl(1), rsync(1), systemctl(1), journalctl(1)