# `config` Examples Overview

The [`config`](./) directory contains a collection of scripts and config snippets that are used to configure the `mlos_bench` components.

These are meant to be used as examples and starting points for your own configuration, though some can be included as-is in your configuration (e.g. linux kernel configs).

In general the `config` directory layout follows that of the `mlos_bench` module/directory layout (e.g. `remote` and `local` `Environments` making using of `Services`, etc., each with their own `json` configs and shell scripts.).

Full end-to-end examples are provided in the [`cli`](./cli/) directory, and typically and make use of the root [`CompositeEnvironments`](./environments/root/) to combine multiple [`Environments`](./environments/), also referencing [`Services`](./services/), [`Storage`](./storage/), and [`Optimizer`](./optimizers/) configs, into a single [`mlos_bench`](../run.py) run.

## Config parameters

An `Environment` configuration can have two sections, `const_args` and `tunable_params`.
At runtime, at each trial the data from both sections will be merged and passed to the environment as a single dictionary of key/value pairs.
The difference between `const_args` and `tunable_params` is that the `tunable_params` values change from one trial to the next (e.g., when the optimizer proposes a new configuration), whereas values of the `const_args` usually stay constant or get their values from sources other than the optimizer.
Having these two sections allows the user not only define which environment parameters to tune, but also customize the environment without changing the Python code or shell scripts.
Sometimes values for the `const_args` come from outside of the Environment configuration, e.g., from global parameters (see the next section) or storage.
To enforce the presence of such parameters, the user can declare their IDs in the `required_args` section of the configuration.
The system will make sure that each parameter in the `required_args` has a value (either specified in the `const_args` or provided externally) and report an error if the required parameters are missing.
The storage system will only save the values of the `tunable_params` for each trial; we rely on git to preserve the config files along with the `const_args` and `required_args` values, and save the commit hash of the configs for each experiment.

## Globals

As mentioned in the [mlos_bench/README.md](../../README.md), a general rule is that the parameters from the global configs like `global_config_azure.jsonc` and/or `experiment_MyAppBench.jsonc` override the corresponding parameters in other configurations.
That allows us to propagate the values of the parameters that are specific to the experiment into other components of the framework and keep the majority of the config files in our library immutable and reusable.

### Sensitive parameters

We recommend storing sensitive parameters like DB hostname and password or Azure credentials in a separate file that is not checked into the source control.
One can create several such files, e.g., `global_config_storage.json` for DB password and `global_config_azure.jsonc` (usually generated by [`scripts/generate-azure-credentials-config.sh`](../../../scripts/generate-azure-credentials-config.sh)) and include them in the `"globals"` section of the CLI config.

### Experiment-specific parameters

Some global parameters vary from experiment to experiment.
It makes sense to store them in a separate file and include it in the `"globals"` section of the CLI config - or, better yet, specify it in the `--globals` option in the command line.
Just like with other global configs, the values from the experiment configuration will override the corresponding values in the Environment, Service, and Optimizer configs down the configuration tree.
This way we can keep our Environment configs immutable and reusable.

One example of the experiment-specific config is the [`experiment_RedisBench.jsonc`](experiments/experiment_RedisBench.jsonc) file that contains the name of the experiment and parameters like VM size and location that may be different for different experiments but remain constant for all trials of that particular experiment.

## Schemas

The [`schemas`](./schemas/) directory contains the [`jsonschema`](https://json-schema.org/) schemas for the `mlos_bench` config files and may also be helpful when writing your own configs.

For instance including a `"$schema"` attribute at the top of your `json` config file will enable `json` validation and auto-complete in many editors (e.g. [VSCode](https://code.visualstudio.com/)):

```jsonc
{
    "$schema": "https://raw.githubusercontent.com/microsoft/MLOS/main/mlos_bench/mlos_bench/config/schemas/environments/environment-schema.json",


    "class": "mlos_bench.environments.SomeEnviroment",
    "config": {
        // ...
    }
}
```
