# RsyncDirector

Automate and coordinate the running of `rsync` jobs on multiple Linux hosts with a `cron` syntax schedule.

## Overview

`rsyncdirector` enables multiple `rsync` jobs, running locally or remotely, to synchronize on lock files, local or remote, to ensure that data is only mutated by one `rsync` process at a time.  Arbitrary commands can also be interspersed between `sync` `actions`.

For example:  Every 24 hours `rsyncdirector` runs on `laptop01` backing up data to a `backup01`.  Both machines are in the same physical location.  Every 7 days an instance of `rsyncdirector` on `backup01` `rsync`s all of its data to `backup-remote` which is at a remote site.  Before the instance of `rsyncdirector` on `laptop01` starts syncing data to `backup01` it checks to see if a lockfile exists in a predefined location on `backup01`.  If it exists, it indicates that `backup01` is currently syncing data and the `laptop01` `rsyncdirector` waits a configured amount of time and then re-checks, until the lock file no longer exists, or a configured timeout exipres.  Once the `laptop01` `rsyncdirector` sees that the lockfile on `backup01` is gone, it writes a lockfile to `backup01` and starts `rsync`ing data from itself to `backup01`.  In this way, multiple `rsyncdirector` instances can be run across multiple hosts and coordinate so that data is not mutated at the same time.

`rsyncdirector` runs as a `systemd` service via a template unit file to enable running multiple `rsyncdirector` instances on a single host.

`rsyncdirector` communicates to remote hosts over SSH using the Python [Fabric](https://www.fabfile.org/) library which depends on [Paramiko](https://www.paramiko.org/) for the core SSH protocol implementation.  SSH connections are currently authenticated using passphrase-less SSH keys.

`rsync` functionality is implemented by the underlying `rsync` package installed on the host.

Each instance can be configured to run under any arbitrary user and connect to any remote host that it can reach with the supplied credentials.  The user under which it runs must have read permissions for the files that it is configured to `rsync`.  The administrator must distribute public SSH keys to the hosts to which data is to be `rsync`ed and can optionally specify a private key other than the `rsyncdirector` user's default set of private keys.

`rsyncdirector` can also run arbitrary `command` actions interspersed between `sync` actions to enable dumping of databases, or any other commands to be run.

See the annotated, [example config file](rsyncdirector/resources/rsyncdirector.yaml) for details on how to configure an instance of `rsyncdirector`.

There is a companion program, [`rsyncdirector_deploy`](https://github.com/rchapin/rsyncdirector_deploy), to assist in the installation and deployment of instances of `rsyncdirector`.

## Forcing a Run Now

The `rsyncdirector` listens for `SIGHUP` events and when receives one will immediately schedule a `run-once` execution of the configured jobs.

1. Get the PID of the `rsyncdirector` process.  If there are multiple instances of it running you will need to adjust the way you search for the PID.
    ```bash
    pgrep rsyncdirector
    ```

1. Send the `SIGHUP`
    ```bash
    kill -SIGHUP <PID>
    ```

## Key Concepts

Much of the following concepts map to specific configuration parameters.

### cron_schedule
Each `rsyncdirector` instance requires defining a peridocity for which it will run by defining a cron expression in the config which uses the same syntax as a standard Linux cron job.

### job
Each job encapsulates n number of `actions`; either a `sync` or a `command`.  The job defines the specifics for connecting to a remote host for `sync` actions if the job type is `remote`.

 <!-- which either rsyncs data between directories on the localhost or data to a single remote host.  The job defines the specifics for connecting to a remote host for `sync` actions if the job type is `remote`. -->

### action
An action is defined within the scope of a job and is either a `sync` or `command`.  **ALL** `action`s must success for a job to continue running.  If one of the `action`s fail, the job is short-circuited and the next job is executed.

#### sync
A `sync` `action` defines the specific `rsync` command to be run.  Each "inherits" the definitions of the job to enable the concatenation of the `rsync` command to include the specified user, host, and port information for `remote` jobs.  The `source` and `dest` are self-explanatory, and the `opts` list enables the inclusion of any arbitrary options that  the underlying `rsync` implementation on the host allows.

#### command
A `command` `action` defines a specific command to be run _on the host on which `rsyncdirector` is running_.  It includes the command itself and an optional list of arguments.

### lock_files
`lock_files` define an arbitrary number of files that will be created on either the localhost or remote host(s) to signal to other `rsyncdirector` instances that a given instance is running.

### blocks_on
`blocks_on` define an arbitrary number of `lock_files` from other `rsyncdirector` instances that the `job` will wait on before continuing with the `job`.

## Metrics

`rsyncdirector` exposes metrics via the `prometheus_client` via `http://$host:$port/metrics`.  By default, it will listen on port `9090` and can be overridden.

## Installing

There is a companion program, [`rsyncdirector_deploy`](https://github.com/rchapin/rsyncdirector_deploy), that contains all of the configuration templates and automation to assist in the installation and deployment of configurations of `rsyncdirector`.

The easiest way to install and configure it is with the aformentioned program.  If you want to install it by hand, you will need to get the configuration templates from the `rsyncdirector_deploy` repo.

## Building, Developing, and Testing

This requires a compatible version of Python in your path from which the path to the interpreter can be gleaned by running `which python<version>`.  See the `requires-python` entry in `pyproject.toml` at the root of the repo for the currently required version of Python.

Once that is installed, run `./run-tests.sh --dev-setup` to create a development virtual environment.

### Building a Distribution

```bash
python -m build
twine check dist/*
```

#### Uploading to PyPi

1. Generate an API token and add the requisite `[pypi]` entry to the `~/.pypirc` file
    ```ini
    [pypi]
    username = __token__
    password = <API-token>
    ```

1. Then upload the artifacts to PyPi
    ```bash
    python -m twine upload --repository pypi dist/*
    ```

#### Uploading to Nexus

In my case, I have an instance of Nexus running in my network with a `~/.pypirc` file configured for it and publish the artifacts there with the following command:
```bash
twine upload --cert <path-to-nexus-cert>--verbose --repository pypi-[dev|release] dist/rsyncdirector-<version>.tar.gz
```

### Running the Tests

The integration tests require that the user running the tests can ssh to `root@localhost` witout having to enter the password.  To do so add a public ssh key of the user running the tests to the authorized keys of the `root` user on the localhost.

Then, from the root of the repo run
```bash
./run-tests.sh
```

### Setting Up to Run and Debug Integration Tests in VSCode

Most of the requisite configs are already present in the provided `.vscode/settings.json` file.

1. Enable the following in your VSCode settings (Workspace or User settings)
    ```json
    "python.analysis.typeCheckingMode": "basic"
    ```

1. Export the path to the required version of Python that you want to use.  If you already have one that will suffice, skip this step.
    ```bash
    export RSYNCDIRECTORINTTEST_PYTHON=<path-to-python-binary>
    ```

1. Setup the test environment.  The following will create a `.env` file at the root of the repository that VSCode will read while running the tests and build the test containers.
    ```bash
    ./run-tests.sh --dev-setup
    ```

1. Configure VSCode
    1. Press `CTRL+SHIFT+P` and select **Python: Configure Tests**
    1. Select `pytest`.  Even though we are using the `unittest` library for testing this seems to be the only test configuration in VSCode that works for running and debugging the tests for the time being.
    1. Choose **rsyncdirector** as the root directory for the tests
    1. Click on the **Testing** icon in the left-hand side-bar and you should see a list of all of the tests that you can now run or debug.

        > If you do not see any tests listed or see an error in the panel check the **Output** Panel and select **Python** from the dropdown menu for any relevant error messages.

1. Update env vars so that existing test timeouts don't result in Exceptions while you are stepping through the code:  There are a number of env vars written to the `.env` file that determine timeouts for various waits in the test code.  When running the whole test suite, the defaults in the `.env` file are fine.  When stepping through the code in the IDE these timeouts can expire and throw Exceptions when nothing has yet gone wrong and get in your way of debugging the code.  To avoid this, edit the `.env` file and update the following env vars setting to the following suggested values.  If you re-run `run-tests.sh --dev-setup` it will overwrite your changes to the `.env` file.
    1. `RSYNCDIRECTORINTTEST_WAITFOR_TIMEOUT_SECONDS=1000`

### Setting up to Develop in VSCode

1. Install the **Black Formatter** extension from Microsoft.
1. Open a Python file in VS Code.
1. Right-click on the editor to display the context menu.
1. Select **Format Document With....**
1. Select **Configure Default Formatter...** from the drop-down menu.
1. Select **black** as your preferred formatter extension from the list.

### Dependency Management

First-order dependencies for `requirements.txt`, `requirements_test.txt`, are defined in the respective `.in` files.  If you make updates to the first order dependencies you need to "compile" the full dependency list.  First, ensure that `pip-tools` is installed in your dev virtual environment.  `dev` dependendies are just defined in `requirements_dev.txt`.
```bash
pip install .[dev]
pip install .[test]
```

Then run the following
```bash
pip-compile -v --no-emit-trusted-host --no-emit-index-url requirements.in
pip-compile -v --no-emit-trusted-host --no-emit-index-url requirements_test.in
```

## Notes

[This is the solution for terminating a multi process/thread in Python](https://cuyu.github.io/python/2016/08/15/Terminate-multiprocess-in-Python-correctly-and-gracefully).  It was the solution for being able to kill a running rsync job.  Actually run the rsync command in a separate process that I can then kill if we receive a SIGTERM for the parent process.
