# Redshift

The Redshift driver depends on [psycopg2](https://www.psycopg.org){target="\_blank"} and can be
installed with:

```bash
pip install "sayn[redshift]"
```

The [Redshift](https://aws.amazon.com/redshift/){target="\_blank"} connector looks for the following parameters:

Parameter         | Description                           | Default
----------------- | ------------------------------------- | ---------------------------------------------
host              | Host name or public IP of the cluster | Required on standard user/password connection
port              | Connection port                       | 5439
user              | User name used to connect             | Required
password          | Password for that user                | Required on standard user/password connection
cluster_id        | Cluster id as registered in AWS       |
database          | Database in use upon connection       | Required
profile           | AWS CLI profile                       | Optional on IAM connection (`default` if left empty)
access_key_id     | AWS Access Key ID                     | Required on Access Key IAM connections
secret_access_key | AWS Secret Access Key                 | Required on Access Key IAM connections
session_token     | AWS Sessions Token                    | Required on Access Key IAM connections
bucket            | S3 Bucket                             | Required on S3 Copy Loads
bucket_region     | S3 Bucket Region                      | Required if the Bucket is in a different region


For advanced configurations, SAYN will pass other parameters to `create_engine`, so check the
sqlalchemy [psycopg2](https://docs.sqlalchemy.org/en/13/dialects/postgresql.html#module-sqlalchemy.dialects.postgresql.psycopg2){target="\_blank"}
dialect for extra parameters.

## Connection Types

SAYN supports 3 connection models for Redshift: one standard user/password connection and two IAM based.

### Standard User/Password Connection

If you have a user name and password for redshift use the first model and ensure host and password
are specified.

!!! example "settings.yaml"
    ```yaml
    credentials:
      redshift-conn:
        type: redshift
        host: my-redshift-cluster.adhfjlasdljfd.eu-west-1.redshift.amazonaws.com
        port: 5439
        user: awsuser
        password: 'Pas$w0rd' #use quotes to avoid conflict with special characters
        dbname: models
    ```

### Connecting With IAM

With an IAM based connection SAYN uses the AWS API to obtain a temporary password to stablish the
connection, so only `user`, `dbname` and `cluster_id` are required. AWS CLI `profile` can optionally be provided. It will use the `default` profile if not specified.

!!! example "settings.yaml"
    ```yaml
    credentials:
      redshift-conn:
        type: redshift
        cluster_id: my-redshift-cluster
        user: awsuser
        profile: default
        dbname: models
    ```

For this connection type to work:

* `boto3` needs to be installed in the project virtual environment `pip install boto3`.
* The AWS cli need to be [setup](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration){target="\_blank"}.
* The `user` and `dbname` still need to be specified (use the database user, not the `IAM:user`).
* `host` and `port` can be skipped and these values will be obtained using boto3's `redshift describe-clusters`.

### Connecting With Access Keys

If the AWS CLI is not configured, access_key_id, secret_access_key and session_token can be provided to allow for the IAM connection. As with the method above, `user`, `dbname` and `cluster_id` are required.

!!! example "settings.yaml"
    ```yaml
    credentials:
      redshift-conn:
        type: redshift
        cluster_id: my-redshift-cluster
        user: awsuser
        access_key_id: ###
        secret_access_key: ###
        session_token: ###
        dbname: models
    ```
For this connection type to work:

* `boto3` needs to be installed in the project virtual environment `pip install boto3`.
* The `user` and `dbname` still need to be specified (use the database user, not the `IAM:user`).
* `host` and `port` can be skipped and these values will be obtained using boto3's `redshift describe-clusters`.

## Redshift Specific DDL

### Indexes

Redshift doesn't support index definitions, and so autosql and copy tasks will forbid its definition
in the `ddl` entry in the task definition.

### Sorting

Table sorting can be specified under the `ddl` entry in the task definition

!!! example "tasks/base.yaml"
    ```yaml
    tasks:
      f_battles:
        type: autosql
        file_name: f_battles.sql
        materialisation: table
        destination:
          table: f_battles
        table_properties:
          sorting:
            columns:
              - arena_name
              - fighter1_name
    ```

With the above example, the table `f_battles` will be sorted by `arena_name` and `fighter1_name`
using a compound key (Redshift default). The type of sorting can be changed to interleaved.

!!! example "tasks/base.yaml"
    ```yaml
    tasks:
      f_battles:
        type: autosql
        file_name: f_battles.sql
        materialisation: table
        destination:
          table: f_battles
        table_properties:
          sorting:
            type: interleaved
            columns:
              - arena_name
              - fighter1_name
    ```

For more information, read the latest docs about [SORTKEY](https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html){target="\_blank"}.

### Distribution

We can also specify the type of distribution: even, all or key based. If not specified, the Redshift default is even distribution.

!!! example "tasks/base.yaml"
    ```yaml
    tasks:
      f_battles:
        type: autosql
        file_name: f_battles.sql
        materialisation: table
        destination:
          table: f_battles
        table_properties:
          distribution: all
    ```

If we want to distribute the table by a given column use the following:

!!! example "tasks/base.yaml"
    ```yaml
    tasks:
      f_battles:
        type: autosql
        file_name: f_battles.sql
        materialisation: table
        destination:
          table: f_battles
        table_properties:
          distribution: key(tournament_name)
    ```

For more information, read the latest docs about
[DISTKEY](https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html){target="\_blank"}.
