Airflow Run
----------------

Python tool for deploying Airflow Multi-Node Cluster.

Requirements
------------

-  Python >=3.6 (tested)

Goal
----

| To provide a quick way to setup Airflow Multi-Node Cluster (a.k.a. Celery Executor Setup).

Steps
-----
1. Generate config yaml file.
2. Run commands to start webserver, scheduler, worker, (rabbitmq, postgres).
3. Add dag files and run initdb.

Generate config file:
---------------------

.. code:: python

    afr --generate_config

Running the tool in the same directory as config file:
------------------------------------------------------

.. code:: python

    afr --run postgresql
    afr --run initdb
    afr --run rabbitmq
    afr --run webserver
    afr --run scheduler
    afr --run worker --queue {queue name}
    afr --run flower

Or, running the tool specifying config path:
--------------------------------------------

.. code:: python

    afr --run postgresql --config /path/config.yaml

Or, use this environment variable to set the config path:
---------------------------------------------------------

.. code:: python

    export AIRFLOWRUN_CONFIG_PATH="/some_path/config.yaml"

After running webserver, scheduler and worker (postgres and rabbitmq if needed local instances), Add your dag files in the dags subdirectory in the directory you defined in the config file. Then, run the following:

.. code:: python

    afr --run initdb

(* note: make sure you have the correct user permission in the dags, logs subdirectories.)

That is it!!


Default Config yaml file:
-------------------------

.. code-block:: yaml

    private_registry: False
    registry_url: registry.hub.docker.com
    username: ""
    password: ""
    repository: pkuong/airflow-run
    image: airflow-run
    tag: latest
    local_dir: {local directory where you want to mount /dags and /logs folder}
    webserver_port: 8000
    flower_port: 5555
    env:
      AIRFLOW__CORE__EXECUTOR: CeleryExecutor
      AIRFLOW__CORE__LOAD_EXAMPLES: "False"
      AIRFLOW__CORE__DAGS_FOLDER: /usr/local/airflow/airflow/dags
      AIRFLOW__CORE__LOGS_FOLDER: /usr/local/airflow/airflow/logs
      AIRFLOW_HOME: /usr/local/airflow
    rabbitmq:
      name: rabbitmq
      username: {username}
      password: {password}
      host: {IP}
      virtual_host: /
      image: rabbitmq:3-management
      home: /var/lib/rabbitmq
      ui_port: 15672
      port: 5672
      env:
        RABBITMQ_DEFAULT_USER: {username}
        RABBITMQ_DEFAULT_PASS: {password}
    postgresql:
      name: postgresql
      username: {username}
      password: {password}
      host: {host}
      image: postgres
      data: /var/lib/postgresql/data
      port: 5432
      env:
        PGDATA: /var/lib/postgresql/data/pgdata
        POSTGRES_USER: {username}
        POSTGRES_PASSWORD: {password}


Docker image
------------

| This tool is using the following public docker image by default.

.. code:: python

    https://hub.docker.com/repository/docker/pkuong/airflow-run

Building the image:
-------------------

| If you want to build your own image, you can run the following:

.. code:: python

    afd --build --config_path={absolute path to config.yaml} --dockerfile_path={absolute path to directory which contains Dockerfile}

Contributors
------------

-  Paulo Kuong (`@pkuong`_)

.. _@pkuong: https://github.com/paulokuong

.. |Build Status| image:: https://travis-ci.org/paulokuong/airflow-run.svg?branch=master
.. target: https://travis-ci.org/paulokuong/airflow-run
