# -*- coding: utf-8 -*-
from distutils.core import setup

packages = \
['airflow_indexima',
 'airflow_indexima.hooks',
 'airflow_indexima.operators',
 'airflow_indexima.uri']

package_data = \
{'': ['*']}

install_requires = \
['PyHive==0.6.1',
 'apache-airflow==1.10.2',
 'gitpython>=2.1.0,<2.2.0',
 'thrift-sasl==0.3.0',
 'thrift>=0.13.0,<0.14.0']

entry_points = \
{'airflow.plugins': ['indexima = '
                     'airflow_indexima.indexima:IndeximaAirflowPlugin']}

setup_kwargs = {
    'name': 'airflow-indexima',
    'version': '2.2.1',
    'description': 'Indexima Airflow integration',
    'long_description': '# airflow-indexima\n\n\n[![Unix Build Status](https://img.shields.io/travis/geronimo-iia/airflow-indexima/master.svg?label=unix)](https://travis-ci.org/geronimo-iia/airflow-indexima)\n[![PyPI Version](https://img.shields.io/pypi/v/airflow-indexima.svg)](https://pypi.org/project/airflow-indexima)\n[![PyPI License](https://img.shields.io/pypi/l/airflow-indexima.svg)](https://pypi.org/project/airflow-indexima)\n\nVersions following [Semantic Versioning](https://semver.org/)\n\n## Overview\n\n[Indexima](https://indexima.com/) [Airflow](https://airflow.apache.org/) integration based on pyhive.\n\nThis project is used in our prod environment with success.\nAs it a young project, take care of change, any help is welcome :)\n\n\n## Setup\n\n### Requirements\n\n* Python 3.6+\n\n### Installation\n\nInstall this library directly into an activated virtual environment:\n\n```text\n$ pip install airflow-indexima\n```\n\nor add it to your [Poetry](https://poetry.eustace.io/) project:\n\n```text\n$ poetry add airflow-indexima\n```\n\nor you could use it as an [Airflow plugin](https://airflow.apache.org/docs/stable/plugins.html)\n\n## Usage\n\nAfter installation, the package can imported:\n\n```text\n$ python\n>>> import airflow_indexima\n>>> airflow_indexima.__version__\n```\n\nSee [Api documentation](https://geronimo-iia.github.io/airflow-indexima/)\n\n\n### a simple query\n\n```python\nfrom airflow_indexima.operators import IndeximaQueryRunnerOperator\n\n...\n\nwith dag:\n    ...\n    op = IndeximaQueryRunnerOperator(\n        task_id = \'my-task-id\',\n        sql_query= \'DELETE FROM Client WHERE GRPD = 1\',\n        indexima_conn_id=\'my-indexima-connection\'\n    )\n    ...\n```\n\n\n### a load into indexima\n\n```python\nfrom airflow_indexima.operators.indexima import IndeximaLoadDataOperator\n\n...\n\nwith dag:\n    ...\n    op = IndeximaLoadDataOperator(\n        task_id = \'my-task-id\',\n        indexima_conn_id=\'my-indexima-connection\',\n        target_table=\'Client\',\n        source_select_query=\'select * from dsi.client\',\n        truncate=True,\n        load_path_uri=\'jdbc:redshift://my-private-instance.com:5439/db_client?ssl=true&user=airflow-user&password=XXXXXXXX\'\n    )\n    ...\n\n```\n### get load path uri from Connection\n\nIn order to get jdbc uri from an Airflow Connection, you could use:\n\n- ```get_redshift_load_path_uri```\n- ```get_postgresql_load_path_uri```\n\nfrom module ```airflow_indexima.uri```\n\nBoth method have this profile: ```Callable[[str, Optional[ConnectionDecorator]], str]```\n\n\nExample:\n```\n    get_postgresql_load_path_uri(connection_id=\'my_conn\')\n    >> \'jdbc:postgresql://my-db:5432/db_client?ssl=true&user=airflow-user&password=XXXXXXXX\'\n```\n\n## Indexima Connection\n\n\n### Authentication\n\nPyHive supported authentication mode:\n\n- \'NONE\': needs a username without password\n- \'CUSTOM\': needs a username and password (default mode)\n- \'LDAP\': needs a username and password\n- \'KERBEROS\': need a kerberos service name\n- \'NOSASL\': corresponds to hive.server2.authentication=NOSASL in hive-site.xml\n\n\n### Configuration\n\nYou could set those parameters:\n\n- host (str): The host to connect to.\n- port (int): The (TCP) port to connect to.\n- timeout_seconds ([int]): define the socket timeout in second (default None)\n- socket_keepalive ([bool]): enable TCP keepalive, default false.\n- auth (str): authentication mode\n- username ([str]): username to login\n- password ([str]): password to login\n- kerberos_service_name ([str]): kerberos service name\n\n`host`, `port`, `username` and `password` came from airflow Connection configuration.\n\n`timeout_seconds`, `socket_keepalive`, `auth` and `kerberos_service_name` parameters can came from:\n\n1. attribut on Hook/Operator class\n2. Airflow Connection in ```extra``` parameter, like this:\n   ```\n   \'{"auth": "CUSTOM", "timeout_seconds": 90, "socket_keepalive": true}\'\n   ```\n\nSetted attribut override airflow connection configuration.\n\nYou could add a decorator function in order to post process Connection before usage.\nThis decorator will be executed after connection configuration (see next section).\n\n### customize Connection credential access\n\nIf you use another backend to store your password (like AWS SSM), you could define a decorator\nand use it as a function in your dag.\n\n```python\nfrom airflow.models import Connection\nfrom airflow import DAG\n\nfrom airdlow_indexima.uri import define_load_path_factory, get_redshift_load_path_uri\n\n\ndef my_decorator(conn:Connection) -> Connection:\n    # conn instance will be not shared, and use only on connection request\n    conn.password = get_ssm_parameter(param_name=f\'{conn.conn_id}.{con.login}\')\n    return conn\n\n\ndag = DAG(\n    dag_id=\'my_dag\',\n    user_defined_macros={\n        # we define a macro get_load_path_uri\n        \'get_load_path_uri\': define_load_path_factory(\n            conn_id=\'my-redshift-connection\',\n            decorator=my_decorator,\n            factory=get_redshift_load_path_uri)\n        },\n    ...\n)\n\nwith dag:\n    ...\n    op = IndeximaLoadDataOperator(\n        task_id = \'my-task-id\',\n        indexima_conn_id=\'my-indexima-connection\',\n        target_table=\'Client\',\n        source_select_query=\'select * from dsi.client\',\n        truncate=True,\n        load_path_uri=\'{{ get_load_path_uri() }}\'\n    )\n    ...\n\n\n```\n\na Connection decorator must follow this type: ```ConnectionDecorator = Callable[[Connection], Connection]```\n\n```define_load_path_factory``` is a function which take:\n\n- a connnection identifier\n- a decorator ```ConnectionDecorator```\n- an uri factory ```UriGeneratorFactory = Callable[[str, Optional[ConnectionDecorator]], str]```\n\nand return a function with no argument which can be called as a macro in dag\'s operator.\n\n### Optional connection parameters\n\nOn each operator you could set this member:\n\n- auth (Optional[str]): authentication mode (default: {\'CUSTOM\'})\n- kerberos_service_name (Optional[str]): optional kerberos service name\n- timeout_seconds (Optional[Union[int, datetime.timedelta]]): define the socket timeout in second\n                (could be an int or a timedelta)\n- socket_keepalive (Optional[bool]): enable TCP keepalive.\n\nNote:\n\n- if execution_timeout is set, it will be used as default value for timeout_seconds.\n\n## Production Feedback\n\nIn production, you could have few strange behaviour like those that we have meet.\n\n### "TSocket read 0 bytes" \n\nYou could fine this issue https://github.com/dropbox/PyHive/issues/240 on long load query running.\n\nTry this in sequence:\n\n1. check your operator configuration, and set ```timeout_seconds``` member to 3600 second for example.\n   You could have a different behaviour when running a dag with/without airflow context in docker container.\n2. if your facing a broken pipe, after 300s, and you have an AWS NLB V2 :\n   Read again [network-load-balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html), and focus on this:\n   > Elastic Load Balancing sets the idle timeout value for TCP flows to 350 seconds. You cannot modify this value. For TCP listeners, clients or targets can use TCP keepalive packets to reset the idle timeout. TCP keepalive packets are not supported for TLS listeners.\n\n   We have tried for you the "socket_keep_alive", and it did not work at all.\n   Our solution was to remove our NLB and use a simple dns A field on indexima master.\n\n\n### "utf-8" or could not read byte ...\n\nBe very welcome to add ```{ "serialization.encoding": "utf-8"}``` in hive_configuration member of IndeximaHook.\n\nThis setting is set in IndeximaHook.__init__, may you override it ?\n\n\n## Playing Airflow without Airflow Server\n\nWhen I was trying many little things and deals with hive stuff, i wrote a single script that help me a lot.\n\nFeel free to use it (or not) to set your dag by yourself:\n\n```python\nimport os\nimport datetime\nfrom airflow.hooks.base_hook import BaseHook\nfrom airflow import DAG\nfrom airflow_indexima.operators.indexima import IndeximaLoadDataOperator\n\n# here we create our Airflow Connection\nos.environ[\'AIRFLOW_CONN_INDEXIMA_ID\'] = \'hive://my-user:my-password@my-server:10000/default\'\nconn = BaseHook.get_connection(\'indexima_id\')\n\ndag = DAG(\n    dag_id=\'my_dag\',\n    default_args={\n        \'start_date\': datetime.datetime(year=2019, month=12, day=1),\n        \'depends_on_past\': False,\n        \'email_on_failure\': False,\n        \'email\': [],\n    },\n)\n\nwith dag:\n\n    load_operator = IndeximaLoadDataOperator(\n        task_id=\'my_task\',\n        indexima_conn_id=\'indexima_id\',\n        target_table=\'my_table\',\n        source_select_query=(\n            "select * from source_table where "\n            "creation_date_tms between \'2019-11-30T00:00:00+00:00\' and \'2019-11-30T12:59:59.000999+00:00\'"\n        ),\n        truncate=True,\n        truncate_sql=(\n            "DELETE FROM my_table WHERE "\n            "creation_date_tms between \'2019-11-30T00:00:00+00:00\' and \'2019-11-30T12:59:59.000999+00:00\'"\n        ),\n        load_path_uri=\'jdbc:postgresql://myserver:5439/db_common?user=etl_user&password=a_strong_password&ssl=true\',\n        retries=2,\n        execution_timeout=datetime.timedelta(hours=3),\n        sla=datetime.timedelta(hours=1, minutes=30),\n    )\n\n    # here we run the dag\n    load_operator.execute(context={})\n\ndel os.environ[\'AIRFLOW_CONN_INDEXIMA_ID\']\n\n\n```\n\n\n## License\n\n[The MIT License (MIT)](https://geronimo-iia.github.io/airflow-indexima/license)\n\n\n## Contributing\n\nSee [Contributing](https://geronimo-iia.github.io/airflow-indexima/contributing)\n\n### Thanks\n\nThanks to [@bartosz25](https://github.com/bartosz25) for his help with hive connection details... \n',
    'author': 'Jerome Guibert',
    'author_email': 'jguibert@gmail.com',
    'url': 'https://pypi.org/project/airflow_indexima',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'entry_points': entry_points,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)
