Backups using rsync — PyMedPhys

Toggle navigation sidebar
Toggle in-page Table of Contents

# Backups using rsync

## Background

We want all data to have a “single source of truth”. As such, important data has its reference location at only one site. All day-to-day usage of that data is undergone via network traffic to that reference location. The downside of this is that a network interruption, such as the remote data storage location having a power outage, makes this data unavailable.

As such, just-in-case, important remote datastores are synced locally. This document details an example of one such local sync.

## Pre-requisites

- SSH Tunnels forwarding through the SAMBA port of the remote file share

  - For the purpose of this document it will be assumed that this share is accessible at `rccc-ssh/Physics` at port `44448`. The directory to be backed up is `Physics`

- A local SAMBA share for storing the backup

  - For the purpose of this document it will be assumed that this share is accessible at `rccc-ssh/D`. The directory to back up to is `PhysicsDriveBackup`

- A username and password that is able to access both SAMBA shares

  - For the purpose here, this username will be `pexit` and the remote share will be on the domain `nbccc`, and the local share will be on the domain `rccc`.

- An Ubuntu 20.04 instance with access to both the forwarded SAMBA share and the local SAMBA share

  - For the purpose here, this instance is a VM within Hyper-V with user login name `pexit`.

## Overview

The general approach here will be to:

- Create the permanent SAMBA mount points via fstab

- Set up rsync to run via cron

## Permanently mount the SAMBA shares

To begin, we need to create two mount points. On our machine this was done by running:

    sudo mkdir -p /media/rccc-ssh/D /media/tunnel-nbcc-pdc/Physics

Then, to create the permanent mounts both `fstab` and `cifs-utils` were utilised. This was adapted from the instructions over at https://wiki.ubuntu.com/MountWindowsSharesPermanently.

Firstly `cifs-utils` was installed:

    sudo apt install cifs-utils

Next, a file at `~/.smbcredentials` was created with the contents:

    username=pexit
    password=YOUR_PASSWORD_GOES_HERE

Then the read/write permissions of this file were set as such:

    chmod 600 .smbcredentials

Next, the contents of `/etc/fstab` was updated to include the following:

    //rccc-ssh/Physics  /media/tunnel-nbcc-pdc/Physics  cifs  user,uid=pexit,credentials=/home/pexit/.smbcredentials,domain=nbccc,iocharset=utf8,port=44448  0  0
    //rccc-ssh/D        /media/rccc-ssh/D               cifs  user,uid=pexit,credentials=/home/pexit/.smbcredentials,domain=rccc,iocharset=utf8              0  0

These new mount points can be tested by running `mount /media/tunnel-nbcc-pdc/Physics` and `mount /media/rccc-ssh/D`

## Setup rsync crontab

These instructions for setting up `rsync` are adapted from https://www.howtogeek.com/135533/how-to-use-rsync-to-backup-your-data-on-linux/

To set up the `crontab` run `crontab -e`, then append the following to the bottom of that file:

    0 1 * * * mount /media/rccc-ssh/D ; mount /media/tunnel-nbcc-pdc/Physics ; timeout 4h rsync -av --delete /media/tunnel-nbcc-pdc/Physics/Physics/ /media/rccc-ssh/D/PhysicsDriveBackup/

This will set up cron to make sure the appropriate directories are mounted and then runs `rsync` each night at 1 am. If the task hasn’t completed by 5 am it is stopped ready for it to continue the task on the following night.