# Dummy containers for ecosystem development

These containers perform a simple transformation to their input data,
to stand in for the kinds of operations that would be done in a step of entity resolution.
The key reason to use these dummy containers is that they have the property that their
input and their output follow the same specification; their output can be passed back in to another
one of these dummy containers, ad infinitum.
Therefore if we ignore what is going on inside the containers, we can build
pipelines of arbitrary length and complexity out of these containers.

The containers can take N input files (see configuration/options section below) and produce N output files.
N = 1 by default.

Specifically, the input/output specification is: a dataframe
with columns "foo," "bar," "counter," and optionally up to 5 columns of
the form "added_column_{i}" where i is an integer.
This can be in CSV or Parquet format.
Initial input files are generated by the notebook in this directory.

Specifically, the transformation they apply is:

- Concatenating rows from their multiple input files, if necessary, filling with 0 where not all of the input files have the same column
- Incrementing the 'counter' column by 1 (or a configured increment)
- Adding 1 (or a configured increment) new columns of the form added_column_{i}, with all values set to i
- Removing any added_column_{i} columns in excess of 5, starting from the lowest i

There are three containers provided here:

- python_pandas uses Pandas.
- python_pyspark uses Spark from Python. The Spark master URL can be configured with DUMMY_STEP_SPARK_MASTER_URL.
- r uses R and tidyverse packages.

There are instructions to build and run these containers; currently only Singularity is supported.
Go to the individual container directories for these.

`test.py` is a script that demonstrates the output-is-valid-input property of these containers by iterating
30 random containers in a row, randomly choosing between CSV and parquet, each taking the output of the previous.
Instructions for running this test are below.

## Container configuration/options

Inputs to the dummy containers are given as a comma-separated list of paths passed to an environment variable; the default environment variable 
is `DUMMY_CONTAINER_MAIN_INPUT_FILE_PATHS`, but you can also specify *what* the input environment variables should be with the environment variable
`INPUT_ENV_VARS`.

You can (optionally) provide another input file at `/extra_implementation_specific_input_data/input*` (Parquet or CSV) or a different path passed as `DUMMY_CONTAINER_EXTRA_IMPLEMENTATION_SPECIFIC_INPUT_FILE_PATH`.
This is meant to represent an input that is specific to a given implementation.

Output is written to `/results/result.<ext>` or a different comma-separated list of paths passed as `DUMMY_CONTAINER_OUTPUT_PATHS`.
If `DUMMY_CONTAINER_OUTPUT_FILE_TYPE` is `csv` it will be in CSV format, otherwise it will be Parquet.

The environment variable `DUMMY_CONTAINER_BROKEN` makes the container return data that does not meet the specification.

If `/extra_implementation_specific_results/` (or a different path passed as `DUMMY_CONTAINER_LOGGING_DIRECTORY`) is writable,
the container will write an `out.log` file there.

## To run the test with Singularity

### With Root Access
If you have root access on your machine, use `build-containers-local.sh` to build the containers.

```
bash build-containers-local.sh
python test.py
```

### Without Root Access

If you are on a system (e.g. HPC cluster) without root access, you can still build the container remotely. 
This requires a (free, but throttled) account at https://cloud.sylabs.io/ to build containers in the cloud through
Singularity Container Services. Then, use build-containers-remote.sh to build the containers.

```
bash build-containers-remote.sh
python test.py
```

## `test.py` configuration/options

`DUMMY_CONTAINERS_TEST_INPUT_FILE` configures which initial input file to use (defaults to input_file_2.csv).

If `DUMMY_CONTAINERS_TEST_INCLUDE_BROKEN` is true, will randomly introduce containers that do not follow the specification.
As you might imagine, this will make the chain fail.

If `DUMMY_CONTAINERS_TEST_INCLUDE_EXTRA_INPUTS` is true, will sometimes randomly provide multiple previous outputs as inputs
to the next container.

If `DUMMY_CONTAINERS_TEST_INCLUDE_EXTRA_OUTPUTS` is true, will save log files as well as results, **and** sometimes have
a container output multiple files, all of which are passed to the next.