############################
 Optimizing Hyperparameters
############################

Hyperparameter optimization (HPO) is an indispensable step to make it work in real
world.

*****************
 Getting Started
*****************

Create a file that defines the objective function to be optimized:

.. code-block:: python
    :caption: objective.py

    import argparse


    def main() -> None:
        parser = argparse.ArgumentParser()
        parser.add_argument("out_filename", type=str)
        parser.add_argument("--x1", type=float)
        parser.add_argument("--x2", type=float)
        args = parser.parse_args()

        y = (args.x1**2) - (4.0 * args.x1) + (args.x2**2) - args.x2 - (args.x1 * args.x2)

        with open(args.out_filename, "w") as f:
            f.write(f"{y}")


    if __name__ == "__main__":
        main()

Run the following command:

.. code-block:: bash

    python -m aiaccel.hpo.apps.optimize params.x1="[0,2]" params.x2="[0,2]" n_trials=30 -- python ./objective.py --x1={x1} --x2={x2} {out_filename}

The parameters are set as params.x1="[0,2]" and params.x2="[0,2]" and the number of
trials is set to n_trials=30. Specify the command to execute the objective function
after '--'. In the arguments, include the parameters and '{out_filename}'. In
objective.py, output the result of the objective function to '{out_filename}' in JSON
format.

*************
 Basic Usage
*************

Configuration
=============

Basic configuration example:

.. code-block:: yaml

    study:
        _target_: optuna.create_study
        direction: minimize
        storage: # Set this item if results need to be stored in DB
            _target_: optuna.storages.RDBStorage
            url: sqlite:///aiaccel_storage.db
            engine_kwargs:
                connect_args:
                timeout: 30
        study_name: my_study  # Set this item if results need to be stored in DB
        sampler:
            _target_: optuna.samplers.TPESampler
            seed: 0

    command: ["python", "./objective.py", "--x1={x1}", "--x2={x2}", "{out_filename}"]

    params:
        x1: [0, 1]
        x2: [0, 1]

    n_trials: 30
    n_max_jobs: 1

Study Configuration
-------------------

The study configuration controls the overall behavior of the optimization process:

.. code-block:: yaml

    study:
        _target_: optuna.create_study  # default
        direction: minimize     # 'minimize' or 'maximize' depending on your objective
        study_name: my_study    # Name of the study (optional)
        storage:  # This item is not required. This item is not required if there is no need to record it in the file.
            _target_: optuna.storages.RDBStorage
            url: sqlite:///example.db
            engine_kwargs:
                connect_args:
                    timeout: 30
    load_if_exists: true    # Load existing study if it exists
    sampler:
        _target_: optuna.samplers.TPESampler
        seed: 42

Sampler Configuration
---------------------

The sampler determines the algorithm used to search the hyperparameter space:

.. code-block:: yaml

    study:
        _target_: optuna.create_study
        direction: minimize
        sampler:
            _target_: optuna.samplers.TPESampler  # Tree-structured Parzen Estimator (default)
            # TPE-specific parameter
            seed: 42                           # For reproducibility
            n_startup_trials: 10               # Number of random trials before using TPE

Available samplers include:

- TPESampler: Efficient Bayesian optimization approach (recommended for most cases)
- RandomSampler: Simple random search (useful as baseline)
- CmaEsSampler: Covariance Matrix Adaptation Evolution Strategy (good for continuous
  parameters)
- GridSampler: Exhaustive grid search (for small parameter spaces)
- NSGAIISampler: For multi-objective optimization
- NelderMeadSampler: Nelder-Mead optimization

Parameters Configuration
------------------------

The parameters section defines the hyperparameter search space using Optuna's suggestion
methods wrapped by aiaccel:

.. code-block:: yaml

    params:
        _convert_: partial
        _target_: aiaccel.hpo.optuna.hparams_manager.HparamsManager  # default

        # Float parameter example
        x1:
            _target_: aiaccel.hpo.optuna.hparams.Float
            low: 0.0
            high: 1.0
            log: false

        # Another float parameter
        x2:
            _target_: aiaccel.hpo.optuna.hparams.Float
            low: 0.0
            high: 1.0
            log: false

        # Shorthand for float parameters
        x3: [0, 1]

Parameter Types
---------------

aiaccel supports multiple parameter types through different suggestion wrappers:

- Float: For continuous parameters

.. code-block:: yaml

    learning_rate:
        _target_: aiaccel.hpo.optuna.hparams.Float
        name: learning_rate
        low: 0.0001
        high: 0.1
        log: true

- Int: For integer parameters

.. code-block:: yaml

    num_layers:
        _target_: aiaccel.hpo.optuna.hparams.Int
        name: num_layers
        low: 1
        high: 10

- Categorical: For categorical parameters

.. code-block:: yaml

    optimizer:
        _target_: aiaccel.hpo.optuna.hparams.Categorical
        name: optimizer
        choices: ['adam', 'sgd', 'rmsprop']

- DiscreteUniform: For discrete uniform parameters

.. code-block:: yaml

    batch_size:
        _target_: aiaccel.hpo.optuna.hparams.Float
        name: batch_size
        low: 32
        high: 256
        step: 32

- LogUniform: For log-uniform parameters

.. code-block:: yaml

    learning_rate:
        _target_: aiaccel.hpo.optuna.hparams.Float
        name: learning_rate
        low: 0.0001
        high: 0.1
        log: true

- LogInt: For log-int parameters

.. code-block:: yaml

    num_layers:
        _target_: aiaccel.hpo.optuna.hparams.Int
        name: num_layers
        low: 1
        high: 10
        log: true

Command
-------

Command to run the objective function. The objective function is the main function to be
optimized:

.. code-block:: yaml

    command: ["python", "./objective.py", "--x1={x1}", "--x2={x2}", "{out_filename}"]

Other Configuration Options
---------------------------

- n_trials: Number of trials to run
- n_max_jobs: Maximum number of parallel jobs

.. code-block:: yaml

    n_trials: 100
    n_max_jobs: 1  # default : 1

Usage Examples
==============

Here are some common usage patterns:

Start a new study with configuration file:

.. code-block:: bash

    aiaccel-hpo optimize --config=config.yaml

Start a new study with cli:

.. code-block:: bash

    python -m aiaccel.hpo.apps.optimize working_directory=./cli/ params.x1="[0,2]" params.x2="[0,2]" study.sampler._target_=optuna.samplers.TPESampler study.sampler.seed=0 n_trials=30 n_max_jobs=1 -- python ./objective.py --x1={x1} --x2={x2} {out_filename}

Start a new study with configuration file and cli:

.. code-block:: bash

    python -m aiaccel.hpo.apps.optimize --config config.yaml params.x1="[0,2]" params.x2="[0,2]" --

*****************************
 HPO Using NelderMeadSampler
*****************************

Basic Usage
===========

Basic optimization example using NelderMeadSampler:

Search Space
------------

NelderMeadSampler requires a search space as an argument.

.. code-block:: python
    :caption: examples/hpo/samplers/example.py

    search_space = {
        "x": (-10.0, 10.0),
        "y": (-10.0, 10.0),
    }

Objective Function
------------------

Set the Objective Function in the same way as in regular Optuna. The optimization target
is the benchmark function Sphere.

.. code-block:: python
    :caption: examples/hpo/samplers/example.py

    def sphere(trial: optuna.trial.Trial) -> float:
        params = []
        for name, distribution in search_space.items():
            params.append(trial.suggest_float(name, *distribution))

        return float(np.sum(np.asarray(params) ** 2))

Execute Optimization
--------------------

Specify NelderMeadSampler as the sampler and execute the optimization.

.. code-block:: python
    :caption: examples/hpo/samplers/example.py

    study = optuna.create_study(
        sampler=NelderMeadSampler(search_space=search_space, seed=42)
    )
    study.optimize(func=sphere, n_trials=100)

Full code is examples/hpo/samplers/example.py

Pallarel Optimization
=====================

Example pallarel optimization:

.. code-block:: python
    :caption: examples/hpo/samplers/example_parallel.py

    study = optuna.create_study(
        sampler=NelderMeadSampler(search_space=search_space, seed=42, block=True)
    )
    study.optimize(func=sphere, n_trials=100, n_jobs=3)

Parallel execution is enabled by setting the NelderMeadSampler argument block=True and
the study.optimize argument n_jobs>2. By enabling parallel execution, the initial point
calculation and the computation during shrinking can be parallelized, leading to faster
execution compared to serial execution.

Full code is examples/hpo/samplers/example_parallel.py

optuna.study.enqueue_trial
==========================

Example using optuna.study.enqueue_trial:

.. code-block:: python
    :caption: examples/hpo/samplers/example_enqueue.py

    study = optuna.create_study(
        sampler=NelderMeadSampler(search_space=search_space, seed=42)
    )
    study.enqueue_trial({"x": 1.0, "y": 1.0})
    study.enqueue_trial({"x": 1.0, "y": 2.0})
    study.enqueue_trial({"x": 2.0, "y": 1.0})
    study.optimize(func=sphere, n_trials=100)

Utilizing the ask-tell interface, random parameters are explored using enqueue_trial
when NelderMeadSampler fails to output parameters.

Full code is examples/hpo/samplers/example_enqueue.py

Sub Sampler
===========

Example using sub_sampler as optuna.samplers.TPESampler:

.. code-block:: python
    :caption: examples/hpo/samplers/example_sub_sampler.py

    study = optuna.create_study(
        sampler=NelderMeadSampler(
            search_space=search_space,
            seed=42,
            sub_sampler=optuna.samplers.TPESampler(seed=42),
        )
    )
    study.optimize(func=sphere, n_trials=100, n_jobs=3)

When sub_sampler=optuna.samplers.TPESampler is set as an argument for NelderMeadSampler,
TPESampler is used for exploration when NelderMeadSampler fails to output parameters.
When using the sub_sampler function, the argument block=False must be set even if it is
parallel. (Parallel execution is possible even with block=False.)

Full code is examples/hpo/samplers/example_sub_sampler.py
