# Hypered: Hyperparameter Optimizer Library

This library provides a flexible interface for optimizing hyperparameters of any blackbox system. It implements bayesian optimization and supports the creation of various types of hyperparameter search spaces, such as real, integer, and categorical variables.

## Features

- **Hyperparameter Spaces**: Define real, integer, and categorical variables.
- **Objective Functions**: Easily create minimization and maximization objectives.
- **Experiment Management**: Automatically handles experiment directories and parameter/result files.
- **Parallel Execution**: Supports parallel execution of experiments.

## Installation

To install hypered, simply run:

```bash
pip install hypered
```

## Usage

### Step 1: Model Script

To use hypered you first need to define a model script that takes the hyper-parameters as an input json file and computes the loss/objective value and write it as a json file. Both input and output files should be provided by a command line arguments. The following is an example model script:

**example.py**
```python
import argparse
import json
import numpy as np

def main(params_path: str, results_path: str):
    params = json.loads(open(params_path).read())

    op = params["vars"]["option"]
    x = params["vars"]["x"]

    if op == "first":
        loss = np.square(x - 5)
    elif op == "second":
        loss = np.abs(x - 3) - 2
    else:
        print("Invalid option", op)
        exit(0)

    print(x, loss)
    results = {
        "loss": loss
    }
    with open(results_path, "w") as f:
        f.write(json.dumps(results))

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Simple model.")
    parser.add_argument("params", type=str, help="Params file.")
    parser.add_argument("results", type=str, help="Results file.")
    args = parser.parse_args()
    main(args.params, args.results)
```

### Step 2: Configuration File

Next we need to define a configuration file that specifies the hyper parameters as well as the objective function. Below is an example configuration file:

**example.conf**
```python
optimize(
    name="learning_params",
    objective=minimize("loss"),
    binary="python3 example.py {params_path} {results_path}",
    random_starts=10,
    iterations=30,
    parallelism=8,
    params={
        "output_dir": experiment_dir(),
        "device_id": device_id(4),
        "vars": {
            "option": categorical(["first", "second"]),
            "x": real(-10, 10)
        }
    }
)
```

### Step 3: Running the Hyperparameter Optimizer

To run the hyperparameter optimizer, use the `hypered` script with the path to your configuration file:

```bash
hypered example.conf
```

This will start the optimization process as defined in your configuration file and output the best parameters.

## Reference

### Optimizers

#### `optimize`

This function performs hyperparameter optimization using Gaussian Processes.

**Arguments:**
- `name` (str): The name of the parameter group.
- `objective` (function): The objective function to minimize or maximize.
- `binary` (str): The command line binary to execute the experiment.
- `params` (dict): The dictionary of parameters to optimize.
- `random_starts` (int, optional): The number of random initialization points.
- `iterations` (int, optional): The number of iterations to run the optimization.
- `seed` (int, optional): The random seed for reproducibility.
- `parallelism` (int, optional): The number of parallel jobs to run.
- `cwd` (str, optional): The current working directory for the subprocess.

Note that you can use predefined variables {params_path} and {results_path} in your binary string to specify the path to parameters and results json files accordingly.

### Objectives

#### `minimize`

Minimize a given variable.

#### `maximize`

Maximize a given variable.

### Variables

#### `uniform`

Returns the string identifier for a uniform distribution.

#### `log_uniform`

Returns the string identifier for a log-uniform distribution.

#### `real`

Class for defining a real-valued hyperparameter.

#### `integer`

Class for defining an integer-valued hyperparameter.

#### `categorical`

Class for defining a categorical hyperparameter.

### Utilities

#### `experiment_dir`

Retrieves the experiment directory from the context.

#### `params_path`

Retrieves the parameters path from the context.

#### `results_path`

Retrieves the results path from the context.

#### `device_id`

Returns a device ID in a round-robin fashion.

## License

This library is licensed under the MIT License. See the `LICENSE` file for more details.

## Contributing

Contributions are welcome! Please open an issue or submit a pull request on GitHub.

## Contact

For any questions or issues, please open an issue on the GitHub repository.
