> This folder contains the necessary to generate the atlas data pipeline artifact without using the complex Snakemake logic and dependencies.

# Content of this directory
- `environment.yml` is a Conda environment description. Env setup using `conda env create -n atlasEnv --file environment.yml`
- `forge-config.json` is the configuration to make the Nexus Forge work properly
- `annotation_generate_all.sh` is the shell script to run in order to generate the annotation artifact
- `config.sh` contains all the variables necessary for `annotation_generate_all.sh` to work. It is *sourced* from the above script and is not meant to be executed manually.
- `push_config_maker.sh` is genrated automatically by `config.sh` and is meant

# Before generating data
## Python environment
Prior to generate any data, it is important to check if all the necessary Python packages are installed. Using the Conda environment specification `environment.yml` is a good way to start but feel free to prepare your environment a different way, especially if you want to use modules in dev mode.

## Nexus token
The first thing being done by the script `annotation_generate_all.sh` is to read the Nexus access token (JWT) from a file. By default, the file is configured to be placed in `$HOME/.token_fetch/Token`. This is only true if this file was generated using the CLI `blue-brain-token-fetch`.  

The first time running the CLI `blue-brain-token-fetch`, the followign information will be asked, in addition to your credentials:

```
CLIENT_ID: bbp-atlas-pipeline
REALM_NAME: BBP
SERVER_URL: https://bbpauth.epfl.ch/auth/
```

These informations are then kept in `$HOME/.token_fetch/keycloack_config.yaml` and in the future, only your credentials will be asked again. Note that `blue-brain-token-fetch` does not store your password.

## Define path ans URLs
The paths to the local files the pipeline is going to create are all listed in the file `config.sh`. The default *working directory* is `$HOME/working_dir_atlas` (will be created automatically if non existent), but feel free to choose another directory.

In addition, do not hesitate to explore the other variables in `config.sh`, especially if the pipeline relies, at some point, on different Nexus resources.

# Generating the annotation pipeline artifacts
Once all the configuration is done, you can run all the instructions in one go with the command:
```
./annotation_generate_all.sh
```

This will sequentially execute multiple CLIs.  

In some situation, it can be convenient to run step by step, with manual verifications in between. In that case, you can open `annotation_generate_all.sh` in a text editor and comment out some parts of it. Keep in mind that each CLIs requires as inputs some artifacts generated by the previous steps. In other word, the order matters.

# Pushing data into Nexus
Pushing the data to Nexus is handled by the last part of the script `annotation_generate_all.sh`. The script leverages 4 Python scripts from this very folder:
- `push_atlasrelease.py` to push a new atlas release and its components: annotation volume, template volume and region ontology
- `push_non_mask_volumes.py` to push the placement hints volume (multiple distributions), the direction vector volume and the direction field volume.
- `push_region_masks.py` to push the brain region masks (900+ NRRD)
- `push_region_meshes.py` to push the brain region meshes (900+ OBJ)
- `push_region_summaries.py` to push brain region summaries, that contain region info about volumes, volume ratio, layer and region adjacency

All those Python scripts are called with the proper arguments (file path, directory path, token, etc.) directly configured from the file `config.sh`. 