Metadata-Version: 2.1
Name: tmg-etl-pipeline
Version: 0.0.2
Summary: TMG ETL pipeline
Home-page: https://github.com/telegraph/tmg-etl-pipeline
Author: TMG Data Platform team
Author-email: data.platform@telegraph.co.uk
License: Apache 2.0
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Requires-Dist: cachetools (==4.1.0)
Requires-Dist: certifi (==2020.4.5.1)
Requires-Dist: chardet (==3.0.4)
Requires-Dist: google-api-core (==1.17.0)
Requires-Dist: google-auth (==1.15.0)
Requires-Dist: google-cloud-core (==1.3.0)
Requires-Dist: google-cloud-logging (==1.15.0)
Requires-Dist: google-cloud-secret-manager (==1.0.0)
Requires-Dist: googleapis-common-protos (==1.51.0)
Requires-Dist: grpc-google-iam-v1 (==0.12.3)
Requires-Dist: grpcio (==1.29.0)
Requires-Dist: idna (==2.9)
Requires-Dist: protobuf (==3.12.1)
Requires-Dist: pyasn1 (==0.4.8)
Requires-Dist: pyasn1-modules (==0.2.8)
Requires-Dist: pytz (==2020.1)
Requires-Dist: PyYAML (==5.3.1)
Requires-Dist: requests (==2.23.0)
Requires-Dist: rsa (==4.0)
Requires-Dist: six (==1.15.0)
Requires-Dist: urllib3 (==1.25.9)

TMG ETL pipeline
==================================

TMG ETL pipeline is the base structure of our ETL pipelines.
Any pipeline should inherit from TMGETLPipeline class and implement the necessary methods.


Quick Start
-----------

Installation
~~~~~~~~~~~~

Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.

.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/

Supported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^
Python >= 3.5

Mac/Linux
^^^^^^^^^

.. code-block:: console

    pip install virtualenv
    virtualenv <your-env>
    source <your-env>/bin/activate
    <your-env>/bin/pip install tmg-etl-pipeline

Example Usage
-------------

.. code:: python

    from tmg_etl_pipeline.etl import TMGETLPipeline


    class MyETL(TMGETLPipeline):

        def run(self):
            # your ETL logic goes here
            # access the configs
            # self.config['some-config-variable']
            # access the secrets
            # self.secrets['some-secret-variable']

        def cleanup(self):
            # your clean up code goes here

    etl = MyETL(app_name='your-app-name', config_path='path/to-the-config-file')
    etl.execute()


