Metadata-Version: 2.1
Name: ecsdep
Version: 0.1.7
Summary: AWS ECS Deployment Tool With Terraform
Home-page: https://gitlab.com/skitai/ecsdep
Author: Hans Roh
Author-email: hansroh@gmail.com
License: MIT
Download-URL: https://pypi.python.org/pypi/ecsdep
Platform: posix
Classifier: License :: OSI Approved :: MIT License
Classifier: Development Status :: 3 - Alpha
Classifier: Topic :: Utilities
Classifier: Environment :: Console
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: colorama

## Introduce
ECS deploy using `docker compose` and `terraform`.

You need to manage just `yml` file for `docker compose` and:
```shell
ecsdep cluster create
ecsdep service up
```
That's all.


## Running Docker

### Local Testing
Docker contains `terrform`, `awscli` and `ecsdep`.
```shell
docker run -d --privileged \
    --name ecsdep \
    -v /path/to/myproject:/app \
    hansroh/dep:dind
docker exec -it ecsdep bash
```

Within docker,
```shell
pip3 install -U ecsdep
```

### Gitlab CI/CD

Add these lines into `.gitlab-ci.yml`:
```yml
image: hansroh/dep:latest
services:
  - name: docker:dind
    alias: dind-service
before_script:
  - pip3 install -U ecsdep
```



## Creating ECS Cluster and Deploy Services

### Creating Base Dcoker Compose Config File
Create `Dockerfile` and `docker-compose.yml`.

This example launch 1 container named `skitai-app`.

```yaml
version: '3.3'

services:
  skitai-app:
    image: registry.gitlab.com/skitai/ecsdep
    container_name: skitai-app
    build:
      context: .
      dockerfile: ./Dockerfile
    ports:
      - 5000:5000
```

Make sure service name (skitai-app) is same as `container_name`.

And test image build and container.
```shell
docker-compose build
docker-compose up -d
docker-compose down
```



### Adding ECS Related Settings

#### Specify Deploy Containers
Add `services.skitai-app.deploy` key. Otherwise this container will not be included.

```yaml
services:
  skitai-app:
    image: registry.gitlab.com/skitai/ecsdep
    deploy:
```

#### Image Registry Pull Credential
For private registry, you need credential. See [this](https://docs.aws.amazon.com/ko_kr/AmazonECS/latest/developerguide/private-auth.html).
```yaml
services:
  skitai-app:
    image: registry.gitlab.com/skitai/ecsdep
    x-ecs-pull-credentials: arn:aws:secretsmanager:ap-northeast-2:000000000:secret:gitlab/registry/mysecret-PrENMF
```

#### Logging (Optaional)
For integrating Cloud Watch log group.
```yaml
services:
  skitai-app:
    logging:
      x-ecs-driver: awslogs
```

#### Container Health Checking (Optaional)
```yaml
    healthcheck:
      test:
        - "CMD-SHELL"
        - "wget -O/dev/null -q http://localhost:5000 || exit 1"
```

#### Container Level Reosurce Requirements
For minimal memory reservation - called soft memory limit,
```yaml
services:
  skitai-app:
    deploy:
      resources:
        reservations:
          memory: "256M"
```

For hard memory limit,
```yaml
    deploy:
      resources:
        reservations:
          memory: "256M"
        limits:
          memory: "320M"
```
Make sure hard limits must be greater or equal than reservation value.

For miniaml CPU units reservation,
```yaml
    deploy:
      resources:
        reservations:
          cpus: "1024"
```

1024 means 1 vCPU. ECS will deploys container which fulfills these reservation requirements.
Value of `cpus` must be string.

Maybe you are using GPU,
```yaml
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

But `ecsdep` ignores above statement. Just add `x-ecs-gpus`.
```yaml
    deploy:
      resources:
        reservations:
          x-ecs-gpus: 1
          devices: ...
```

Ok, test docker build and container.

```shell
docker-compose build
```


### Terraform Setting
```yaml
services:
  skitai-app:
     ...

x-terraform:
  provider: aws
  region: ap-northeast-2
  template-version: 1.1
  state-backend:
    region: "ap-northeast-2"
    bucket: "states-data"
    key-prefix: "terraform/ecs-cluster"
```

Make sure you create s3 bucket.



### Cluster Settings

**Prerequisitions**
- AWS/Gitlab credentials
  ```shell
  docker login -u <gitlab username> -p <gitlab deploy-token> registry.gitlab.com
  aws configure set aws_access_key_id <AWS_ECS_ACCESS_KEY_ID>
  aws configure set aws_secret_access_key <AWS_ECS_SECRET_ACCESS_KEY>
  ```
- AWS certification for your service domain (ex. mydomain.com and *.mydomain.com)
- SSH key file, for generating key file,
  ```shell
  ssh-keygen -t rsa -b 2048 -C "email@example.com" -f ./mykeypair
  ```
  It makes 2 files. `mykeypair` is private key and `mykeypair.pub` is public key.
  Please keep private key file carefully for accessing your EC2 instances.

  Also `mykeypair.pub` must be placed with current yaml file.


```yaml
services:
  skitai-app:
     ...

x-ecs-cluster:
  name: my-cluster
  public-key-file: "mykeypair.pub"
  instance-type: t3.medium
  ami: amzn2-ami-ecs-hvm-*-x86_64-*
  autoscaling:
    desired: 2
    min: 2
    max: 10
    cpu: 60
    memory: 60
  loadbalancer:
    cert-name: mydoamin.com
  vpc:
    cidr_block: 10.0.0.0/16
```

It creates resources like:
- Load Balancer
- VPC with subnets: 10.0.10.0/24, 10.0.20.0/24 and 10.0.30.0/24
- Cluster Instance Auto Scaling Launch Configuration
- Cluster Auto Scaling Group
- Cluster

#### Using Default VPC
Just remove `vpc` key.


#### Cluster-Level Auto Scaling
`autoscaling.cpu` mean that your cpu reservations of your containers
reaches 60% of all cluster instances's CPU units, scale out ECS instances.
If 0, nothing will be happen.

`autoscaling.memory` is same.

#### Instance Type
If you specified `x-ecs-gpus`, `instance-type` must have GPU.

#### VPC
If you consider VPC peering, choose carefully `cidr_block`.


#### Createing/Destroying ECS Cluster

Finally you can create cluster,

`ecsdep` find default yaml named as `docker-compose.yml` or `compose.ecs.yml`.
```shell
ecsdep cluster create
ecsdep cluster destroy
```

Specifying file is also possible.
```shell
ecsdep -f /path/to/docker-compose.yml cluster create
ecsdep -f /path/to/docker-compose.yml cluster destroy
```



### Deploying Service

```yaml
x-ecs-service:
  name: ecsdep
  stages:
    default:
      env-service-stage: "qa"
      hosts: ["qa.myservice.com"]
      listener-priority: 100
    production:
      env-service-stage: "production"
      hosts: ["myservice.com"]
      listener-priority: 101
      autoscaling:
        min: 3
        max: 7
  loadbalancing:
    pathes:
      - /*
    protocol: http
    healthcheck:
      path: "/ping"
      matcher: "200,301,302,404"
  deploy:
    compatibilities:
      - ec2
    resources:
      limits:
        memory: 256M
        cpus: "1024"
    autoscaling:
      desired: 1
      min: 1
      max: 4
      cpu: 100
      memory: 80
```

#### Staging
`stages` can make deploy stages like `production`, `qa` or `staging`.
It find your current environment variable named `SERVICE_STAGE`.
```shell
export SERVICE_STAGE=qa
```

Current deploy stages is selected by current `SERVICE_STAGE` value matched with `env-service-stage`.

If `SERVICE_STAGE` is `qa`, your container routed `qa.myservice.com` by load balancer.



#### Auto Scaling

Default container-level auto scaling settings are placed in `x-ecs-service.autoscaling`.
But each stages can overwrite this values by `x-ecs-service.[workspace name].autoscaling`.


#### Load Balancing (Optional)

Your container routed to `loadbalancing.pathes` by your load balancer.



##### Using Fargate
If you wnat `fargate` deploy, set `x-ecs-service.deploy.compatibilities` to `- fargate`



#### Resource Limiting

It is required when 'fargate' laubching, but it is optional for 'ec2' launching.

`x-ecs-service.deploy.autoscaling.cpu` and `x-ecs-service.deploy.autoscaling.memory`
are both percentages of reserved cpu units or memory MBytes.

If `x-ecs-service.deploy.resources.limits.cpus` is defined, service cannot use units
over this value.

If `x-ecs-service.deploy.resources.limits.memory` is defined and your container over
this value, container will be terminated.



#### Deploying Service

you need env var `CI_COMMIT_SHA`. It is used as image tag with first 8chars from git commit hash string. it will be provided on gitlab runner, but at local testing, `lateset` is just OK.

```shell
export CI_COMMIT_SHA=latest
export SERVICE_STAGE=qa

ecsdep -f dep/compose.ecs.yml service up
```

Whenever commanding `ecsdep service up`, your containers will be deployed to ECS by rolling update way.

As a results, AWS resources will be created.
- Task Definition
- Update Service and Run

*Note*: Sometimes, service-level auto-scaling settings are not applied at initial deploy.
I don't know why but please deploy twice in this case.



#### Shutdown/Remove Service
```shell
ecsdep service down
```




### Deploying Other Services Into Cluster

It is recommended cluster settings are kept in your main app only.

In other service's yml, keep `x-ecs-cluster.name` only and remove other `x-ecs-cluster` settings.

You just care about `x-ecs-service` and `services` definition.

```yaml
services:
  skitai-app-2:
     ...

x-terrform:
  ...

x-ecs-cluster:
  name: my-cluster
```



### Deploying Service With Multiple Containers

```yaml
services:
  skitai-app:
    deploy:
    ports:
      - 5000
    healthcheck:
      test:
        - "CMD-SHELL"
        - "wget -O/dev/null -q http://localhost:5000 || exit 1"

  skitai-nginx:
    depends_on:
      - skitai-app
    x-ecs-wait-conditions:
      - HEALTHY
    ports:
      - "80:80"
```

Make sure only single loadbalancable container can have host port
mapping like "80:80", other wise just docker internal port like "5000".




### Deploying Non Web Service

Get rid of ports and load balancing settings like:
- `services.your-app.ports`
- `x-ecs-service.loadbalancing`
- `x-ecs-service.stages.default.hosts`
- `x-ecs-service.stages.default.listener-priority`




### Using Secrets

```yaml
version: '3.3'

services:
  skitai-app:
    environment:
      DB_PASSWORD=$DB_PASSWORD

secrets:
  DB_PASSWORD:
    name: "arn:aws:secretsmanager:ap-northeast-1:0000000000:secret:gitlab/registry/hansroh-PrENMF:DBPASSWORD::"
    external: true
```

At deploy time, environment `DB_PASSWORD` is overwritten with
`secrets.DB_PASSWORD.name` value by AWS ECS Service. see AWS Secret Manager document.

