MIRROR-DEDUPE(1)

NAME
mirror-dedupe - Ubuntu mirror synchronisation with global deduplication

SYNOPSIS
mirror-dedupe [-h] [--dry-run] [--mirror MIRROR] [--dedupe-only] [config_dir]

DESCRIPTION
mirror-dedupe synchronises Ubuntu package repositories and performs global deduplication across multiple mirrors using hardlinks. This significantly reduces disk space usage when hosting multiple Ubuntu mirrors or architectures.

The tool operates in three modes:

1.  Orchestrator mode (default): Spawns subprocesses for each configured mirror, then performs global deduplication across all mirrors.

2.  Single mirror mode (--mirror): Processes only the specified mirror.

3.  Dedupe-only mode (--dedupe-only): Skips mirror synchronisation and only performs deduplication.

OPTIONS

-h, --help
    Show help message and exit.

--dry-run
    Show what would be done without actually making any changes.

--mirror MIRROR
    Process only the specified mirror by name. Used internally by the orchestrator to spawn subprocesses for each mirror.

--dedupe-only
    Only run the deduplication phase, skipping mirror synchronisation.

config_dir
    Path to configuration directory. Default: /etc/mirror-dedupe

CONFIGURATION
Configuration files are located in /etc/mirror-dedupe/:

mirror-dedupe.conf
    Main configuration file containing global settings such as buffer size, parallel downloads, and curl timeout.

repos.d/*.conf
    Individual mirror configuration files. Each file defines a repository mirror with settings including:

    name
        Mirror identifier

    upstream
        Source repository URL

    dest
        Local destination path (relative to repo_root)

    sync_method
        Synchronisation method (rsync or https)

    architectures
        List of architectures to mirror (e.g., amd64, arm64)

    components
        Repository components (e.g., main, universe, multiverse)

    distributions
        Ubuntu releases to mirror (e.g., noble, jammy)

    gpg_key_url
        URL to repository GPG key (optional)

EXAMPLES

Run full synchronisation and deduplication:
    mirror-dedupe

Dry run to see what would be changed:
    mirror-dedupe --dry-run

Process only the 'ubuntu' mirror:
    mirror-dedupe --mirror ubuntu

Only perform deduplication (skip sync):
    mirror-dedupe --dedupe-only

Use custom configuration directory:
    mirror-dedupe /path/to/config

SYSTEMD INTEGRATION
The package includes systemd timer and service units:

mirror-dedupe.timer
    Scheduled timer that runs the sync daily at a randomised time between 00:00-06:00 UTC.

mirror-dedupe.service
    Service unit that executes the mirror synchronisation and deduplication.

To check timer status:
    systemctl status mirror-dedupe.timer

To manually trigger a sync:
    systemctl start mirror-dedupe.service

To view logs:
    journalctl -u mirror-dedupe.service

FILES

/etc/mirror-dedupe/mirror-dedupe.conf
    Main configuration file

/etc/mirror-dedupe/repos.d/
    Directory containing individual mirror configurations

/srv/mirror/repos/
    Default base directory for mirrored repositories

/var/run/mirror_dedupe.*.pid
    Lock files to prevent concurrent processing of the same mirror

/usr/lib/systemd/system/mirror-dedupe.service
    Systemd service unit

/usr/lib/systemd/system/mirror-dedupe.timer
    Systemd timer unit

DEDUPLICATION
The tool performs global deduplication by:

1.  Collecting all package files across all configured mirrors

2.  Grouping files by SHA256 hash

3.  Downloading the first occurrence of each unique file

4.  Creating hardlinks for duplicate files (same hash, different paths)

This approach significantly reduces disk usage when hosting multiple mirrors or architectures, as identical packages are stored only once.

EXIT STATUS

0   Success

1   Error occurred during synchronisation or deduplication

AUTHOR
Written by Tim Hosking.

COPYRIGHT
Copyright 2025 Tim Hosking.
Licence: MIT

SEE ALSO
curl(1), rsync(1), systemctl(1), journalctl(1)