# Quick Start Guide

## Introduction

In the context of `py-gen-ml`, a protobuf is a structured data format that is used to define the schema for your configuration. It is a language-neutral, platform-neutral, and extensible format that is used to serialize and deserialize data. The protobuf ecosystem allows custom plugins to generate code for other types of serialization formats. `py-gen-ml` uses this to generate Pydantic models and JSON schemas to enable a very flexible and powerful configuration system.

!!! note
    As of yet we do not yet lean into the language-neutral or platform-neutral features of protobuf in `py-gen-ml`. If you are not familiar with protobuf, you can read more about it [here](https://developers.google.com/protocol-buffers).

## Defining

To define a protobuf, you need to create a `.proto` file. This file will contain the definition of the data structure that you want to use in your configuration.

Here is an example of a protobuf definition:

```proto
// proto_intro.proto
syntax = "proto3";

package example;

import "py_gen_ml/extensions.proto";


message MLP {
    int64 num_layers = 1;
    int64 num_units = 2;
    string activation = 3;
}
```

## Generating

Now you can generate configuration utilities using the following command:

```bash
py-gen-ml proto_intro.proto
```

By default, generated code will be written to `src/pgml_out`. For changing this and other options, see the [py-gen-ml command](py-gen-ml-command.md) docs. The following files will be generated:

- `proto_intro_base.py`
- `proto_intro_patch.py`
- `proto_intro_sweep.py`
- `proto_intro_cli_args.py`

Below, we'll cover the details of each file.

## Generated Code
### `proto_intro_base.py`
```python
--8<-- "docs/snippets/src/pgml_out/proto_intro_base.py"
```

This file defines a Pydantic model for you configuration that directly corresponds to the protobuf message. You can use this file to load and validate configuration files written in YAML format.

For example, the following YAML file will be validated according to the schema defined in `proto_intro_base.py`:

```yaml
# example.yaml
num_layers: 2
num_units: 100
activation: relu
```

### `proto_intro_patch.py`

```python
--8<-- "docs/snippets/src/pgml_out/proto_intro_patch.py"
```

This file defines a Pydantic model for you patch configuration. Note that all fields are optional, and you can use this file to load and validate patch files written in YAML format.

For example, the following YAML file will be validated according to the schema defined in `proto_intro_patch.py`:

```yaml
# example_patch.yaml
num_layers: 3
```

This is useful for updating only a subset of the configuration. You can easily load a base configuration and apply patches by using `.from_yaml_files` that is
implemented for every descendant of `pgml.YamlBaseModel`:

```python
# example.py
from pgml_out.proto_intro_base import MLP
from pgml_out.proto_intro_patch import MLPPatch

config_with_patches = MLP.from_yaml_files(["example.yaml", "example_patch.yaml"])
```

### `proto_intro_sweep.py`

```python
--8<-- "docs/snippets/src/pgml_out/proto_intro_sweep.py"
```

This file defines a `pgml.Sweeper` for your configuration. You can use this file to sweep over the values of your configuration. Sweeps should be defined as
additional YAML files that are applied as patches on top of the base configuration.

For example, the following YAML file will be validated according to the schema defined in `proto_intro_sweep.py`:

```yaml
# example_sweep.yaml
num_layers:
    low: 1
    high: 5
```

If you then want to run a hyperparameter sweep, you can do so by using the OptunaSampler:

```python
# example.py
from pgml_out.proto_intro_base import MLP
from pgml_out.proto_intro_sweep import MLPSweep

def train_model(config: MLP) -> float:
    """Train a model and return the accuracy"""

if __name__ == "__main__":
    config = MLP.from_yaml_file("example.yaml")
    sweep = MLPSweep.from_yaml_file("example_sweep.yaml")

    def objective(trial: optuna.Trial) -> float:
        sampler = pgml.OptunaSampler(trial=trial)
        patch = sampler.sample(sweep)
        accuracy = train_model(config.merge(patch))
        return accuracy

    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=100)
```

### `proto_intro_cli_args.py`

```python
# proto_intro_cli_args.py
# Autogenerated code. DO NOT EDIT.
import py_gen_ml as pgml
import typing

import pydantic
import typer

from . import proto_intro_base as base


class MLPArgs(pgml.YamlBaseModel):
    """Multi-layer perceptron configuration"""

    num_layers: typing.Annotated[typing.Optional[int], typer.Option(help="Number of layers"), pydantic.Field(None)]
    """Number of layers"""

    num_units: typing.Annotated[typing.Optional[int], typer.Option(help="Number of units"), pydantic.Field(None)]
    """Number of units"""

    activation: typing.Annotated[typing.Optional[str], typer.Option(help="Activation function"), pydantic.Field(None)]
    """Activation function"""
```

This file defines a Pydantic model for you command line arguments. As you can see, we have chosen to rely on [typer](https://typer.tiangolo.com/)
to handle command line arguments. We have added a convenience function to make the use of this class easier.

```python
# example.py
from pgml_out.proto_intro_base import MLP
from pgml_out.proto_intro_cli_args import MLPArgs
import py_gen_ml as pgml
import typer

app = typer.Typer(pretty_exceptions_enable=False)

@pgml.pgml_cmd(app)
def main(other_arg: str = "other_arg", config_path: str = "example.yaml", cli_args: MLPArgs = typer.Option(...)) -> None:
    print(f"other_arg: {other_arg}")
    print(f"num_layers: {cli_args.num_layers}")
    print(f"num_units: {cli_args.num_units}")
    print(f"activation: {cli_args.activation}")

    config = MLP.from_yaml_file(config_path)
    config = config.apply_cli_args(cli_args)
    print(f"config: {config}")


if __name__ == "__main__":
    app()
```

You can then run your script with command line arguments and configuration files:

```bash
python example.py --help
```

Which prints:
```
 Usage: example.py [OPTIONS]

╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────╮
│ --other-arg                 TEXT                             [default: other_arg]                     │
│ --config-path               TEXT                             [default: example.yaml]                  │
│ --num-layers                INTEGER                          Number of layers [default: None]         │
│ --num-units                 INTEGER                          Number of units [default: None]          │
│ --activation                TEXT                             Activation function [default: None]      │
│ --install-completion        [bash|zsh|fish|powershell|pwsh]  Install completion for the specified     │
│                                                              shell.                                   │
│                                                              [default: None]                          │
│ --show-completion           [bash|zsh|fish|powershell|pwsh]  Show completion for the specified shell, │
│                                                              to copy it or customize the              │
│                                                              installation.                            │
│                                                              [default: None]                          │
│ --help                                                       Show this message and exit.              │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

You can set parameters both via command line arguments and configuration files:

```bash
python example.py --num-layers 3
```

This will set the `num_layers` parameter to `3` and the other parameters will be set to the values in the configuration file.

So the script will output:

```
other_arg: other_arg
num_layers: 3
num_units: None
activation: None
config: num_layers=3 num_units=100 activation='relu'
```

Other args can be configured just as easily:

```bash
python example.py --other-arg "other_arg_2"
```

Which will print:

```
other_arg: other_arg_2
num_layers: None
num_units: None
activation: None
config: num_layers=2 num_units=100 activation='relu'
```
