
# Step-by-step instructions to run cross study analysis using Parsl on Lambda

### 1. Create and activate a conda environment to support improvelib and Parsl
```bash
conda create -n parsl parsl numpy pandas scikit-learn pyyaml -y
conda activate parsl
```

### 2. Clone the model repository
```bash
git clone <MODEL_REPO>
cd MODEL_NAME
git checkout <BRANCH>
```

**Requirements**:
1. Model scripts must be organized as:
    - <MODEL_NAME>_preprocess_improve.py
    - <MODEL_NAME>_train_improve.py
    - <MODEL_NAME>_infer_improve.py
3. Your model scripts are executable using the preprocess.sh, train.sh and infer.sh scripts in the model directory.
2. Make sure to follow the IMPROVE lib [documentation](https://jdacs4c-improve.github.io/docs) to ensure the model is compliant with the IMPROVE framework


### 3. Clone IMPROVE repo
Clone the [IMPROVE](https://github.com/JDACS4C-IMPROVE/IMPROVE/tree/develop) repository to a directory of your preference (outside your model directory).

```bash
cd ..
git clone https://github.com/JDACS4C-IMPROVE/IMPROVE
cd IMPROVE
git checkout develop
export CSA_WORKFLOW_DIR=`pwd`/workflows/csa/
```

### 4. Set locations for Model directories, raw data and workflow runs

### 5. Set environemnt and update PYTHONPATH

```bash
source workflows/csa/setup.sh
```

### 6. Download benchmark data for cross study analysis

Download benchmark data to the data destination directory using [this](https://github.com/JDACS4C-IMPROVE/IMPROVE/blob/develop/scripts/get-benchmarks). This will create a `raw_data` direcory in the destination. For example:

```bash
./scripts/get-benchmarks ${CSA_EXPERIMENT_BASE_DIR}
```

The downloaded benchmark data tree is shown below:
```
${CSA_EXPERIMENT_BASE_DIR}/raw_data/
├── splits
│   ├── CCLE_all.txt
│   ├── CCLE_split_0_test.txt
│   ├── CCLE_split_0_train.txt
│   ├── CCLE_split_0_val.txt
│   ├── CCLE_split_1_test.txt
│   ├── CCLE_split_1_train.txt
│   ├── CCLE_split_1_val.txt
│   ├── ...
│   ├── GDSCv2_split_9_test.txt
│   ├── GDSCv2_split_9_train.txt
│   └── GDSCv2_split_9_val.txt
├── x_data
│   ├── cancer_copy_number.tsv
│   ├── cancer_discretized_copy_number.tsv
│   ├── cancer_DNA_methylation.tsv
│   ├── cancer_gene_expression.tsv
│   ├── cancer_miRNA_expression.tsv
│   ├── cancer_mutation_count.tsv
│   ├── cancer_mutation_long_format.tsv
│   ├── cancer_mutation.parquet
│   ├── cancer_RPPA.tsv
│   ├── drug_ecfp4_nbits512.tsv
│   ├── drug_info.tsv
│   ├── drug_mordred_descriptor.tsv
│   └── drug_SMILES.tsv
└── y_data
    └── response.tsv
```




### 5. To run cross study analysis:

- You need a configuration file for how to execute the workflow with parsl, examples are provided in the `IMPROVE/workflows/csa/config/` directory.
- You need a model specific configuration for the CSA

#### Create workflow configuration file**:

The **csa_params.ini** contains parameters necessary for the workflow. All command line options can be set in the config file. The user can change the parameters inside this configuration file.

 - `input_dir` : Path to the benchmark `raw_data` for cross study analysis. 
 - `output_dir` : Path to the output directory. The subdirectories in the `output_dir` will be organized as:
    - `preprocess`: Contains pre-processed data.
    - `train`: Contains trained models.
    - `infer`: Contains inference retults.
 - `source_datasets`: List of source datasets for cross study analysis. With the current benchmark datasets this can be a subset of CCLE, gCSI, GDSCv1, GDSCv2 and CTRPv2.
 - `target_datasets`: List of source datasets for cross study analysis. With the current benchmark datasets this can be a subset of CCLE, gCSI, GDSCv1, GDSCv2 and CTRPv2.
 - `split`: Splits of the source datasets for cross study analysis.
 - `hyperparameters_file`: Path to the json file containing hyperparameters per dataset. In this template two hyperparameter files are given:
    - `hyperparameters_hpo.json`: Contains hyperparameters optimized separately on all source datasets.
    - `hyperparameters_default.json`: Contains default values of the hyperparameters for the model.
 - `model_name`: Name of the model for cross study analysis.
 - `model_dir`: Path to the model directory containing the model scripts.
 - `conda_environment`: Name of your model conda environment.
 - `epochs`: Number of epochs for the model.
 - `available_accelerators`: List of GPU ids to launch the jobs. The required format is: ["id1","id2"]. For example, if you want to choose GPUs 0 and 1 set available_accelerators = ["0","1"]
 - `y_col_name`: Response variable used in the model. eg: `auc`
 - `singularity`: True, if the model files are available in a singularity container.
 - `container`: URI to container image to use for execution, e.g. file:///absolute/path/to/container.sif. Required if the `singularity` option is set to True. 
 - `only_cross_study`: True, if only cross study analysis is needed without within study inferences.

 e.g.

 ```
[CSA]
source_datasets = ["gCSI", "CCLE", "GDSCv1", "GDSCv2", "CTRPv2"]
target_datasets = ["gCSI", "CCLE", "GDSCv1", "GDSCv2", "CTRPv2"]
split = ["0","1","2","3","4","5","6","7","8","9"]
y_col_name = auc
epochs = 150
available_accelerators=["0","1"]
```

**hyperparameters.json** contains a dictionary of optimized hyperparameters for the models. The key to the dictionary is the model name, which contains another dictionary with source dataset names as keys. The two hyperparameters considered for this analysis are: `batch_size` and `learning_rate`. 

e.g.
```json
{
   "MODEL_NAME_1" : {
      "CCLE" : {
         "batch_size" : 32,
         "learning_rate" : 0.001
      },
      "CTRPv2" : {
         "batch_size" : 32,
         "learning_rate" : 0.001
      },
      "GDSCv1" : {
         "batch_size" : 32,
         "learning_rate" : 0.001
      },
      "GDSCv2" : {
         "batch_size" : 32,
         "learning_rate" : 0.001
      },
      "gCSI" : {
         "batch_size" : 32,
         "learning_rate" : 0.001
      }
   },
   "MODEL_NAME_2" : {
      "CCLE" : {
         "batch_size" : 128,
         "learning_rate" : 0.0001
      },
      "CTRPv2" : {
         "batch_size" : 128,
         "learning_rate" : 0.0001
      },
      "GDSCv1" : {
         "batch_size" : 128,
         "learning_rate" : 0.0001
      },
      "GDSCv2" : {
         "batch_size" : 128,
         "learning_rate" : 0.0001
      },
      "gCSI" : {
         "batch_size" : 128,
         "learning_rate" : 0.0001
      }
   }
}
```



#### Execution without singularity container:
  
Make sure to change the `model_name` parameter in `csa_params.ini` to your <MODEL_NAME>.  
Change the `model_scripts_dir` parameter to the path to your model directory.   
Change the `model_environment` parameter to the name of your model conda environment.  
Make changes to `csa_params.ini` as needed for your experimenet.

Preprocesssing:
```bash
csa preprocess \
--model_name $MODEL_NAME \
--conda_env $MODEL_CONDA_ENV  \
--model_dir ${CSA_MODEL_DIR}/$MODEL_DIR
--input_dir ${CSA_RAW_DATA_DIR} \
--output_dir $CSA_EXPERIMENT_BASE_DIR \
--hyperparameters_file ${CSA_CONFIG_DIR}/hyperparameters.json \
--config_file ${CSA_CONFIG_DIR}/csa_params.ini \
--parsl_config_file ${CSA_WORKFLOW_DIR}/config/local.py \
--log_level DEBUG \


```


To run cross study analysis:
```bash
csa workflow  \
--config_file <CONFIG_FILE> \
--hyperparameters_file <HYPERPARAMETERS> \
--parsl_config_file <HOST_CONFIG_FILE> \
--input_dir <CSA_EXPERIMENT_BASE_DIR> \
--output_dir <CSA_EXPERIMENT_DIR> \

```

Specifying model name and environment on the command line:

```bash
 ${CSA_WF}/train_and_infer.py \
--conda_env ${MODEL_CONDA_ENV} \
--model_name ${MODEL_NAME} \
--model_dir ${CSA_MODEL_DIR}/${MODEL_DIR}/ \
--hyperparameters_file ${CSA_CONFIG_DIR}/hyperparameters_default.json \
--config_file ${CSA_CONFIG_DIR}/csa_params.ini \
--parsl_config_file ${CSA_WF}/config/lambda.py \
-i ${CSA_EXPERIMENT_BASE_DIR} \
-o ${CSA_EXPERIMENT_DIR}/1/ 
```



### Reference
1.	Yadu Babuji, Anna Woodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Rohan Kumar, Luksaz Lacinski, Ryan Chard, Justin M. Wozniak, Ian Foster, Michael Wilde and Kyle Chard. "Parsl: Pervasive Parallel Programming in Python." 28th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC). 2019. 10.1145/3307681.3325400
