# Tutorial: Part 2

## What We Will Cover

This tutorial builds upon the [Tutorial: Part 1](tutorial_part1.md) and introduces you to SAYN concepts which will enable you to make your projects more dynamic and efficient.

If you need any help or want to ask a question, please reach out to the team at <sayn@173tech.com>.

## Implementing `parameters` and `presets`

The [Tutorial: Part 1](tutorial_part1.md) got you to implement a first ETL process with SAYN. We will now look into two core SAYN features which will enable you to build efficient and dynamic projects as you scale: `parameters` and `presets`.

For this tutorial, we will use the code generated by `sayn init` and make amends as we go through the new concepts.

A [Github repository](https://github.com/173TECH/sayn_tutorial_part2){target="\_blank"} is available with the final code if you prefer to have all the code written directly. After cloning, you simply need to make sure to rename the `sample_settings.yaml` file to `settings.yaml` for the project to work.

### Step 1: Using `parameters`

#### `parameters` Setup

You can use `parameters` in order to make your SAYN tasks' code dynamic. We will set one project parameter called `user_prefix`. This will enable us to distinguish which user generated tables.

First, add `paramaters` to `project.yaml`. Those `parameters` will contain the default values for the SAYN project:

**`project.yaml`**
``` yaml
required_credentials:
  - warehouse

default_db: warehouse

dags:
  - base

parameters:
  user_prefix: '' #no prefix for prod
```

Then, add the `parameters` to your profiles in `settings.yaml`.

**`settings.yaml`**
``` yaml
default_profile: dev

profiles:
  dev:
    credentials:
      warehouse: test_db
    parameters:
      user_prefix: sg_
  prod:
    credentials:
      warehouse: prod_db
    parameters:
      user_prefix: '' #no prefix for prod

credentials:
  test_db:
    type: sqlite
    database: test.db
  prod_db:
    type: sqlite
    database: prod.db
```

### Step 2: Making Tasks Dynamic With `parameters`

Now that our parameters are setup, we can use those to make our tasks' code dynamic. We will now change our tasks' code to use the `user_prefix` parameter.

#### In `python` tasks

For the Python `load_data` task, we will access the `user_prefix` parameter and then pass it to the functions doing the data processing. You can look into `python/utils/log_creator.py` to see how we use the `user_prefix` parameter to change the table names.

Replace your `pyhton/load_data.py` script with the code below. As you can observe, we create a `user_prefix` variable by reading `self.sayn_config.parameters`. The `parameters` are stored on the Task object and can therefore be accessed in that way.

**`load_data.py`**
```python
import sqlite3
import logging
from .utils.log_creator import prepare_data, generate_load_query
from sayn import PythonTask

class LoadData(PythonTask):
    def setup(self):
       err = False

       # we use this list to control how many battles we want per tournament
       tournament_battles = [
           {"tournament_id": 1, "n_battles": 1000},
           {"tournament_id": 2, "n_battles": 250},
           {"tournament_id": 3, "n_battles": 500},
       ]

       user_prefix = self.sayn_config.parameters['user_prefix']

       try:
           self.data_to_load = prepare_data(tournament_battles, user_prefix=user_prefix)
       except Exception as e:
           err = True
           logging.error(e)

       if err:
           return self.failed()
       else:
           return self.ready()

    def run(self):

        user_prefix = self.sayn_config.parameters['user_prefix']

        # load the logs
        for log_type, log_details in self.data_to_load.items():
            # create table
            logging.info('Creating table: {log_type}.'.format(log_type=log_type))

            self.default_db.execute(log_details['create'])

            # load logs
            logging.info('Loading logs: {log_type}.'.format(log_type=log_type))
            logs = log_details['data']

            for log in logs:
                q_insert = generate_load_query(log_type, log, user_prefix=user_prefix)

                self.default_db.execute(q_insert)

            # done
            logging.info('Done: {log_type}.'.format(log_type=log_type))

        return self.success()
```

#### In `autosql` tasks

You can also access the project `parameters` in `autosql` tasks with the following syntax: `{{parameter_name}}`. SAYN's compilation process uses Jinja. Replace all the SQL queries with the following:

**`dim_arenas.sql`**
```sql
SELECT l.arena_id
     , l.arena_name

FROM {{user_prefix}}logs_arenas l
```

**`dim_fighters.sql`**
```sql
SELECT l.fighter_id
     , l.fighter_name

FROM {{user_prefix}}logs_fighters l
```

**`dim_tournaments.sql`**
```sql
SELECT l.tournament_id
     , l.tournament_name

FROM {{user_prefix}}logs_tournaments l
```

**`f_battles.sql`**
```sql
WITH battles AS (

  SELECT l.tournament_id
       , l.battle_id
       , l.arena_id
       , l.fighter1_id
       , l.fighter2_id
       , l.winner_id

  FROM {{user_prefix}}logs_battles l
)

SELECT t.tournament_name
     , t.tournament_name || '-' || CAST(b.battle_id AS VARCHAR) AS battle_id
     , a.arena_name
     , f1.fighter_name AS fighter1_name
     , f2.fighter_name AS fighter2_name
     , w.fighter_name AS winner_name

FROM battles b

LEFT JOIN {{user_prefix}}dim_tournaments t
  ON b.tournament_id = t.tournament_id

LEFT JOIN {{user_prefix}}dim_arenas a
  ON b.arena_id = a.arena_id

LEFT JOIN {{user_prefix}}dim_fighters f1
  ON b.fighter1_id = f1.fighter_id

LEFT JOIN {{user_prefix}}dim_fighters f2
  ON b.fighter2_id = f2.fighter_id

LEFT JOIN {{user_prefix}}dim_fighters w
  ON b.winner_id = w.fighter_id
```

**`f_fighter_results.sql`**
```sql
WITH fighter1_outcome AS (

  SELECT b.tournament_name
       , b.battle_id
       , b.arena_name
       , b.fighter1_name
       , CASE WHEN b.fighter1_name = b.winner_name THEN 1 ELSE 0 END AS is_winner

  FROM {{user_prefix}}f_battles b
)

, fighter2_outcome AS (

  SELECT b.tournament_name
       , b.battle_id
       , b.arena_name
       , b.fighter2_name
       , CASE WHEN b.fighter2_name = b.winner_name THEN 1 ELSE 0 END AS is_winner

  FROM {{user_prefix}}f_battles b
)

SELECT f1.tournament_name
     , f1.battle_id
     , f1.arena_name
     , f1.fighter1_name AS fighter_name
     , f1.is_winner

FROM fighter1_outcome f1

UNION

SELECT f2.tournament_name
     , f2.battle_id
     , f2.arena_name
     , f2.fighter2_name AS fighter_name
     , f2.is_winner

FROM fighter2_outcome f2
```

**`f_rankings`**
```sql
SELECT fr.fighter_name
     , CAST(SUM(fr.is_winner) AS FLOAT) / COUNT(DISTINCT fr.battle_id) AS win_rate

FROM {{user_prefix}}f_fighter_results fr

GROUP BY 1

ORDER BY 2 DESC
```

Our SQL queries are now all set to read tables which are prefixed with the relevant `user_prefix` parameter depending on the profile used at execution. However, our `autosql` tasks still create tables in a static way as we have the `table` attribute of our `autosql` tasks hardcoded. The next step will fix this.

### Step 3: Using `presets` To Standardise Task Definitions

Because most of our tasks have a similar configuration, we can significantly reduce the YAML task definitions using `presets`. `presets` enable to create standardised tasks which can be used to define other tasks by setting a preset attribute. We define a `modelling` preset and use it in our DAG `base.yaml`. Replace the file `dags/base.yaml` with the following:

**`dags/base.yaml`**
```yaml
presets:
  modelling:
    type: autosql
    file_name: '{{task.name}}.sql'
    materialisation: table
    destination:
      tmp_schema: main
      schema: main
      table: '{{user_prefix}}{{task.name}}'
    parents:
      - load_data

tasks:
  load_data:
    type: python
    class: load_data.LoadData

  #this task sets modelling as its preset attribute
  #therefore it inherits all the attributes from the modelling preset
  dim_tournaments:
    preset: modelling

  dim_arenas:
    preset: modelling

  dim_fighters:
    preset: modelling

  f_battles:
    preset: modelling
    parents:
      - dim_tournaments
      - dim_arenas
      - dim_fighters

  f_fighter_results:
    preset: modelling
    parents:
      - f_battles

  #for that task, we overwrite the modelling preset materialisation attribute as we want this model to be a view
  f_rankings:
    preset: modelling
    materialisation: view
    parents:
      - f_fighter_results
```

Using the `modelling` preset on tasks' definitions will imply:

* those tasks will be `autosql` and materialise as a `table`.
* they will all have the `load_data` task as a parent.
* and they will have all other attributes set in the `modelling` preset.

## Running Our New Project

You can now test running `sayn run` or `sayn -p prod`. The two options will do the following:

* `sayn run`:
    * use our `dev` profile
    * create all tables into a `test.db` database
    * prefix all tables with `sg_` and read from `sg_` prefixed tables
* `sayn run -p prod`:
    * use our `prod` profile
    * create all tables into a `prod.db` database
    * will not use a prefix when creating / reading tables

We have made our SAYN project dynamic with `parameters` and made our tasks' definitions more efficients with `presets`.

## What Next?

This is it, you should now have a good understanding of the core ways of using SAYN. You can play further with this project and easily transfer it to a PostgreSQL database for example by:

* changing the credentials in `settings.yaml`.
* setting the `tmp_schema` and `schema` attributes of your `modelling` preset to `public`.

Otherwise, you can learn more about the specific SAYN features by having a look at the specific sections of the documentation.

Enjoy SAYN :)
