<div align="center">

# Mano 
[![CI](https://github.com/onnela-lab/mano/actions/workflows/ci.yml/badge.svg)](https://github.com/onnela-lab/mano/actions/workflows/ci.yml)

</div>

Mano is a Python library and CLI tool that helps you write applications that interact with the
[Beiwe Research Platform](https://www.hsph.harvard.edu/onnela-lab/beiwe-research-platform/).

Mano is developed and maintained by [Onnela Lab](https://www.hsph.harvard.edu/onnela-lab/), at the
Harvard T.H. Chan School of Public Health, and is part of The Beiwe Platform. It is distributed
under the open source BSD-3-Clause license.

### Current List of Features:
- Programmatically request lists of the studies you are authorized on from your Beiwe server.
- Request additional information: the list of participants in your study, and the device settings
  currently enabled on your study.
- *Download data generated by the participants in your study.*

> [!Important]
> Some features of Mano may not be available if the server is running an older version of the Beiwe
> Backend. Please let your System Administrator know if you run into any issues. The Beiwe Platform
> provides extensive documentation for sysadmins to handle any issues they may encounter in the
> upgrade process. Resources can be found [on the Beiwe Backend Wiki](
> https://github.com/onnela-lab/beiwe-backend/wiki), and they can [follow announcement issues on
> backend issues](
> https://github.com/onnela-lab/beiwe-backend/issues?q=is%3Aissue%20state%3Aopen%20label%3AANNOUNCEMENT).
>
> If you are encountering any other issue **[Please Report Them Here](
> https://github.com/onnela-lab/mano/issues/39)**

## Table of contents
1. [Requirements and Compatibility](#requirements-and-compatibility)
2. [Installation](#installation)
3. [Initial Setup](#initial-setup)
4. [API For Keyring Access](#api-for-keyring-access)
5. [API For Accessing Study Information](#api-for-accessing-study-information)
6. [API For Downloading Data](#api-for-downloading-data)

## Requirements and Compatibility
- Mano is compatible with modern versions of Python [at time of writing this means 3.10+]
- Mano's CI runs on macOS (Unix), Ubuntu (Linux), and Windows.
  - We have not historically had many Windows users, so [Please Report Any Issues You
  Encounter](https://github.com/onnela-lab/mano/issues/39) with Windows compatibility.
  - Mano should be fully compatible with the Linux Subsystem for Windows (WSL).

### Old SSL Library Compatibility
Beiwe servers _require_ modern, secure connections, so old versions of SSL/TLS libraries provided by
your operating system may cause issues. The simplest known solution is to install one of the
[Miniconda Python distributions](https://www.anaconda.com/docs/getting-started/miniconda/main),
which bundles a more up to date version of OpenSSL.

## Installation
The simplest way to install `mano` is to just use `pip`

```bash
pip install mano
```

## Initial Setup
To interact with Beiwe and download files you will need your Beiwe Platform `url`, `username`,
`password`, `access key`, and `secret key` in a JSON file. Don't worry, we're going to eventually
encrypt this file. _(Note: we are phasing out the need for the `password` field, but Beiwe servers
running an older version of the backend software may still require it.)_

```json
{
    "beiwe.onnela": {
        "URL": "...",
        "USERNAME": "...",
        "PASSWORD": "...",
        "ACCESS_KEY": "...",
        "SECRET_KEY": "..."
    }
}
```

> [!Tip]
> You can also use the environment variables `BEIWE_URL`, `BEIWE_USERNAME`,
> `BEIWE_PASSWORD`, `BEIWE_ACCESS_KEY`, and `BEIWE_SECRET_KEY` to store these settings. If you do,
> load your keyring using `mano.keyring(None)`. You won't be able to use an environment variable for
> storing study-specific secrets (next).

> [!Note]
> You generate, name and manage your access keys under the Manage Credentials section of your Beiwe
> Platform website.

If you wish to use `mano` to encrypt certain downloaded data stream files at rest, you should
provide a study-specific passphrase (which you must generate) in an extra `SECRETS` section.

```json
{
    "beiwe.onnela": {
        "URL": "...",
        "USERNAME": "...",
        "PASSWORD": "...",
        "ACCESS_KEY": "...",
        "SECRET_KEY": "...",
        "SECRETS": {
            "Beiwe Study Omega": "...",
        }
    }
}
```

You don't want this file sitting around as plain text, so Mano requires you encrypt it. Mano uses
the `crypt.py` utility from the `cryptease` library which was installed along with the `mano`
package.

```bash
$ crypt.py --encrypt ~/.nrg-keyring.json --output-file ~/.nrg-keyring.enc
```

> [!Note]
> "`crypt.py`" _looks_ like a file name, but it is an executable command. You may have a common CLI
> program named simply "`crypt`" installed on your system (or even autocompleted in your CLI), it is
> not the same thing and you should not confuse them.

It is up to you to decide where to store the encrypted version of this file, but we **strongly
recommend** deleting the unencrypted version.


## API For Keyring Access
Before making any API calls Mano must read in your keyring file. The first parameter is the name of
the keyring section as shown above

```python
import mano
Keyring = mano.keyring('beiwe.onnela')
```

Mano still requires that you provide the decryption key for your keyring file. By default it will
prompt you to type it in directly, but there are two mechanisms for providing it programmatically.
- Setting the environment variable `NRG_KEYRING_PASS` where Mano is running.
- As the second argument to the `mano.keyring` function in your code.
  - We recommend against placing the decryption key as text in your code, or in any file that gets
  committed to a source control system like Git. This mechanism is provided so that you can
  programmatically source it from another location.
  - It's tough to know where to store a credential securely. If you are on your own computer we
  recommend using the full drive encryption capability of your operating system to secure it.

> [!Important]
> Non-interactive invocations of your code that do not have access to a decryption key
> will probably cause your code to hang as it waits for user input that cannot happen.

## API For Accessing Study Information
With your `Keyring` loaded you can now access information about your studies, users (a.k.a.
participants, subjects), and device settings using simple functions defined within the `mano` module.

```python
for study in mano.studies(Keyring):
    print(study)

_, study_id = study  # get the last printed study id

for user_id in mano.users(Keyring, study_id):
    print(user_id)

for setting in mano.device_settings(Keyring, study_id):
    print(setting)
```

## API For Downloading Data
With your `Keyring` loaded, you can download collected data from your Beiwe server and extract it to
your filesystem using the `mano.sync` module. While we're at it, we will turn on more verbose
logging so we can see what's happening.

> [!Note]
> The `msync.download` function returns a [Python Standard Library `zipfile.ZipFile`](
> https://docs.python.org/3/library/zipfile.html) object from which you extract files.

```python
import logging
from mano import sync as msync

logging.basicConfig(level=logging.INFO)

output_folder = '/tmp/beiwe-data'  # set this to a real folder location

zf = msync.download(Keyring, study_id, user_id, data_streams=['identifiers'])

zf.extractall(output_folder)
```

> [!WARNING]
> We passed `data_streams=['identifiers']` to `msync.download`. Without that parameter that function
> will request *all* data for *all* data streams, which may amount to many gigabytes of data. Check
> out the [backfill](#backfill) section for more information.


### Encrypt Data Files At Rest
You can pass the `ZipFile` object to `msync.save` if you wish to encrypt data stream files.

```python
lock_streams = ['gps', 'audio_recordings']

zf = msync.download(Keyring, study_id, user_id)

data_encryption_key = Keyring['SECRETS']['Beiwe Study Omega']  # not the keyring decryption key!

msync.save(
    Keyring,
    zf,
    user_id,
    output_folder,
    lock=lock_streams,
    passphrase=data_encryption_key,
)
```

### Backfill
By default `msync.download` attempts to download *all* of the data for the specified `user_id`,
which could end up being prohibitively large. For this reason, the `msync.download` function exposes
parameters for `data_streams`, `time_start`, and `time_end`. By using these parameters you can
limit your download operation to those constraints.

```python
data_streams = ['accel', 'ios_log', 'gps']

time_start = '2015-10-01T00:00:00'
time_end = '2015-12-01T00:00:00'

zf = msync.download(
    Keyring,
    study_id,
    user_id,
    data_streams=data_streams,
    time_start=time_start,
    time_end=time_end,
)

zf.extractall(output_folder)
```

> [!Note]
> The full list of data stream keys is: `accelerometer`, `app_log`, `audio_recordings`,
> `bluetooth`, `calls`, `devicemotion`, `gps`, `gyro`, `identifiers`, `image_survey`, `ios_log`,
> `magnetometer`, `power_state`, `proximity`, `reachability`, `survey_answers`, `survey_timings`,
> `texts`, and `wifi`.

Eventually you may find yourself day-dreaming about a `backfill` function that will slide a window
from some arbitrary starting point to the present time in order to download all of your data in more
digestible chunks. You'll be happy to know that the `mano.sync` module exposes a function for this.

```python
start_date = '2015-01-01T00:00:00'

msync.backfill(
    Keyring,
    study_id,
    user_id,
    output_folder,
    start_date=start_date,
    lock=lock_streams,
    passphrase=passphrase,
)
```

> [!Note]
> If you don't pass anything for the `lock` argument, you will not need `passphrase` either.
