# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['tellius_data_manager',
 'tellius_data_manager.alert_manager',
 'tellius_data_manager.alert_manager.chatops',
 'tellius_data_manager.alert_manager.email',
 'tellius_data_manager.alert_manager.general',
 'tellius_data_manager.config_reader',
 'tellius_data_manager.connectors',
 'tellius_data_manager.connectors.databases',
 'tellius_data_manager.connectors.http',
 'tellius_data_manager.connectors.object_stores',
 'tellius_data_manager.errors',
 'tellius_data_manager.flows',
 'tellius_data_manager.logs',
 'tellius_data_manager.persistence_operators',
 'tellius_data_manager.persistence_operators.dataframe_operators',
 'tellius_data_manager.persistence_operators.dataframe_operators.dataframe_readers',
 'tellius_data_manager.persistence_operators.dataframe_operators.dataframe_writers',
 'tellius_data_manager.pipes',
 'tellius_data_manager.pipes.logics',
 'tellius_data_manager.pipes.pipe_stats',
 'tellius_data_manager.pipes.readers',
 'tellius_data_manager.pipes.transformers',
 'tellius_data_manager.pipes.writers',
 'tellius_data_manager.state_manager']

package_data = \
{'': ['*']}

install_requires = \
['PyYAML>=6.0,<7.0',
 'azure-storage-blob>=12.13.1,<13.0.0',
 'boto3>=1.24.28,<2.0.0',
 'flatten-json>=0.1.13,<0.2.0',
 'logging-json>=0.2.1,<0.3.0',
 'pandas>=1.4.3,<2.0.0',
 'providah>=0.1.15.0,<0.2.0.0',
 'pyOpenSSL>=22.0.0,<23.0.0',
 'requests>=2.28.1,<3.0.0']

setup_kwargs = {
    'name': 'tellius-data-manager',
    'version': '0.2.50',
    'description': '',
    'long_description': '<!--Copyright 2022 Tellius, Inc\n\nLicensed under the Apache License, Version 2.0 (the "License");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an "AS IS" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.-->\n# Tellius Data Manager\n\n[![Build Status](https://drone.app.tellius.com/api/badges/Tellius/tellius-data-manager/status.svg)](https://drone.app.tellius.com/Tellius/tellius-data-manager)\n\n## About\nTellius Data Manager - tdm - takes the approach that a data pipeline is a flow of information from ingest, transform, write, and possible additional transformation. That is, it is impossble to really do ETL or ELT in the modern data stack. Instead, it is ETLTLTLT... etc....\n\nA common mathematical reduction of data pipelines is to represent and even discuss data pipelines using the Abstract Algebraic concept of a Directed Acyclic Graph or DAG. This comes from engineers having at least basic understandings of mathematics and a greater number of data engineers having advanced CS or mathematics degrees (they may have been data scientists at one time). The reason this is so commonly used is it provides a clear explanation representation of how data MOVES through a system. Data in most any situation should not move recursively through a pipeline. In the event that you have created a sequence of steps that is looping - either you have done things incorrectly or you have moved to a grain that is TOO small, and that loop logically belongs embedded within a single step.\n\nThis being said, the logic behind how tdm has been developed is to represent all pipelines as DAGs - although we use a less mathematical language to make using the library more approachable. We currently support two different approaches to defining a pipeline: Declarative through a YAML file and pragmatic through direct coding. Each of these approaches will be described below.\n\n### What is tdm\ntdm, or Tellius Data Manager, is a framework for building ETL, ELT, etc. data pipelines. It is written in python, it is configurable, customizable and built with component reuse in mind. It abstracts away the concept of a secret so that code itself should never contain a secret directly. The  \n\n### What is a data pipeline\nIn essence a pipeline is a sequence of dependent steps, each of which is executed in order and not executed (possibly conditionally) when a predecessor fails.\n\nA data pipeline is a pipeline where each step in the processing pipeline has to do with\n1. Ingestion of data\n2. Transformation of data\n3. Writing of data\n4. Logical operations controlling the processing of the data.\n\nA pipeline is created when each step is implicitly or explicitly linked to a parent process. Each pipeline will require at least one initial pipe and at least one terminal pipe.\n\n### tdm architectural philosophy\n\nThere are many concepts espoused in tdm. And many more that could be added, but are not yet at this time - only when needed would they be.\n\n* Pipelines\n* Logging\n* State Tracking\n* Alerting\n\n#### Pipeline\ntdm - at its core - is based upon the philosophy that each step of a pipeline is an independently executable component. While in this space some would desire that a pipeline can be polyglottic tdm - does not directly espouse this approach - but since it is truly extensible, it is not in any way prevented. *If there is a desire to espouse polyglottism it is on the implementer to build out parallel implementations that operate with the same qualities that tdm does.*_\n\nThis means that the core processing unit of tdm is a Pipe. In implementation a Pipe will have one of 4 basic types\n1. ReaderPipe\n2. TransformerPipe\n3. WriterPipe\n4. LogicsPipe (More on this one later - WIP)\n\n![Basic Pipeline](images/pipe_scenario_1.png)\n\nAbove is a very basic pipeline. This is still useful in many cases. It constitutes the simple act of reading data from one source and writing it to another.\n\n![Transform Pipeline](images/pipe_scenario_2.png)\n\nThis is the next step up in complexity. Here there is an intermediate step of applying a transformation to the data.\n\n![Multi Transform Pipeline](images/pipe_scenario_3.png)\n\nIn some situations transformations have to happen in parallel - or for better execution time they are performed in parallel.\n\n![Complex Pipeline](images/pipe_scenario_4.png)\n\nPipelines can become very complex. In this situation there are multiple data sources, an intermediate write stage. A late stage read and finally a write to ultimately terminate the pipeline.\n\n![Example Pipeline](images/pipe_scenario_5.png)\n\nThis is examplar of what may happen in a pipeline. In this case data is read from a RESTful endpoint. A data column is added to the data and a column represent a State (as in a State in the USA) is modified to a standard format. These two datasets are then joined and then written out to S3.\n\nIn tdm it is standard to define the configuration of your entire pipeline through either a YAML file or in code as a python dict.\n\n#### Logging\nLogging is built into the framework. There is a standardized logging format called TelliusLogger. This should be used where ever you write code. It should replace any tendency to print to stdio and used when capturing and handling exceptions. It extends the standard python logger and uses the same interface. It also captures some additional fields around the runtime.\n\nThe fields captured are open to change as the framework is extended.\n\n#### State Tracking\nThe idea of state for tdm is not meant as a restorable state. It is instead intended to be used primarily for creating an audit trail. This will be used to capture metadata/statistics about the data as well.\n\n#### Alerting\nAlerting is important for production pipelines. An alert, at this time, in tdm nomenclature, corresponds to sending an email when a pipeline error occurs.\n\n## Installing tdm\n\n### System Requirements\n1. Linux Operating System\n2. Python 3.9 or higher\n\n### Gaining Access to Private Repo\ntdm is stored on pypi\n\n```bash\nhttps://pypi.org\n```\n\nand the repo is called\n\n```bash\ntellius-data-manager\n```\n\nIt can be installed as\n```bash\npip install tellius-data-manager\n```\n\nThere is also a prebuilt Docker image at\n\nhttps://hub.docker.com/r/telliusopensource/tellius-data-manager\n\n## Creating a tdm project\n\nAt this time there is no magic button (CLI) for creating a tdm project, so some manual steps will be needed. First, in your user root location you will need to create a hidden directory called .tdm (*only linux systems are supported at this time.*). In this directory create two files, secrets.yml (for storing secret configurations) and config.yml (for storing pipeline configuration - still a WIP)\n\nOnce this is done you need to either us an existing project (repo) and add a sub project for the new group of pipelines. Regardless of the choice, you need to treat it as a standalone project. This means you will have a pyproject.toml, requirements.txt and other python project artifacts local to this pipelines as its dependencies may be different than those of other pipelines.\n\n### Repository Structure\n\nWhen installing tdm there is no CLI at this time, so you have to initialize the project by yourself. tdm operates under an opinionated file structure.\n```bash\nproject_root\n  pipeline_drivers\n  pipelines\n  tests\n```\n\nIf you are planning to deploy with Argo CronWorkflows, then you need an argo directory.\n\nSimilarly, if using Kubernetes CronJobs, then create a manifests directory.\n\n### Drivers vs Pipes\n\nThere will be two different kinds of code written for a pipeline (most often). These are Drivers and Pipes. A driver is a script where a pipeline is configured and orchestrated. Later versions will have the ability to abstract this away completely to a configuration file. Pipes are custom extensions to the various kinds of Pipe described above.\n\n#### Drivers\n\nThe pipeline_drivers directory will have a file called entrypoint.py. This will serve as the main entrypoint to code execution. It will be a CLI that will group all the different pipelines together so that the orchestration engine will have a templatizable execution command.\n\nThe pipeline_drivers directory will also have one or more files defining the various data pipelines. These files will have the following general structure\n\n```python\n\nlogging.setLogger(TelliusLogger)\n_logger = logging.getLogger(__name__)\n\n\ndef some_function(table, state_file, data_file, pipeline_id):\n  # Create a job_id - this uniquely identifies the execution run\n  job_id = uuid.uuid4().hex\n\n  # This will be used to track and log/persist the state (audit) of the piepline\n  state_manager = StateManager(**configuration_dictionary)\n\n  # Errors can happen!\n  try:\n    # Some pipelines may have configuration parameters we have not yet extracted to configuration - put them here.\n    some_parameters...\n\n    # This is where we are going to create a dict to define the configuration. Alternatively you could use the YAMLConfigReader that comes in tdm to read it from a configuration file that is local or stored at the USER_ROOT/.tdm/config.yml (more on this later)\n    flow_model = {\n      configuration of each step here\n    }\n\n    # Define the various pipeline steps\n    read_pipe = QuerySomething(**flow_model[\'read\'])\n    transform_pipe = TransformSomething(**flow_model[\'transform\'])\n    write_pipe = WriteSomething(**flow_model[\'write\'])\n\n    # Sometimes the pipeline here can have control and looping logic. Hopefully this can all be handled through pipe definitions later - but it is not yet there. And for this reason some additional logic may be needed and possibly other parameters. However, we will not do so here.\n\n    start_time = datetime.datetime.now().timestamp()\n    # Build the execution flow\n    read = read_pipe.run()\n    transform = transform_pipe.run(parent=read)\n    write = write_pipe.run(parent=transform)\n  except:\n    stop_time = datetime.datetime.now().timestamp()\n    state_manager.update(\n        stop_time=stop_time,\n        start_time=start_time,\n        flow_model=flow_model,\n        status=status,\n        meta_state={},\n        pull_stats=read_stats,\n        push_stats={},\n        job_id=job_id,\n    )\n```\n\n#### Pipes\nThe pipes directory contains custom Pipe definitions. We find it best to organize the custom pieces in folders with each folder containing all the pipe for a specific pipeline. There should also be a folder called reusable that will contain all pipe that are common across two or more pipelines - this will help promote DRY coding.\n\n### Writing your Entrypoint CLI\nThe entrypoint.py is a CLI code. You can think of it as a driver of drivers. This file will be used by orchestration layers or can be used to create your own customer orchestration layer. It is used to wrap all the various pipelines that you have written into a single templatizable call and makes it friendly to call code from the command line in Dockerized situations.\n\nThe recommendation is to use click. Click is one of the better CLI libraries written for python and simplifies a lot of the argument parsing logic. Using click we would write the CLI something along these lines.\n\n```python\n@click.command()\n@click.option(\'--pipeline\', \'-p\', type=str, required=True, help=\'This is the name of the pipe\')\n@click.option(\'--pipeline_id\', \'-id\', type=str, required=True, help=\'This is a required id to uniquely identify the pipeline\')\n@click.option(\'--state_file\', \'-s\', type=str, required=True, help=\'This is the file where state will be stored\')\n@click.option(\'--data_file\', \'-d\', type=str, required=True, help=\'This is the file bucket where data will be stored\')\ndef tdm(pipeline, pipeline_id, state_file, data_file):\n  if pipeline == "pipe-1":\n    do_something(pipeline_id, state_file, data_file)\n  elif pipeline == "pipe-2":\n    do_something_else(pipeline_id, state_file, data_file)\n  elif pipeline == "pipe-3":\n    do_something_again(pipeline_id, state_file, data_file)\n  else:\n    raise ValueError(f\'Pipeline {pipeline} does not exist.\')\n```\n\n### Argo CronWorkflows vs Kubernets CronJobs\nIf you have an orchestration layer ofr your pipelines that needs additional code, then that code should be stored in another directory. For Argo and Kuberentes, two popular orchestration tools, we recommend placing them in a \nargo_workflows and cronjobs directory, respectively, at the same level as pipeline_drivers and pipes folders. \n\n### Using Secrets in Code\nOne thing, don\'t use secrets in code in any hard coded fashion. The Pipe construct has the ability to read a secrets.yml file - this file will be stored in the users root directory in a .tdm subdirectory\n\nExample secrets.yml below\n\n```bash\nversion: 1\n\nRead Example Secrets:\n  code: SECRET_CODE\n  key: SECRET_KEY\n  \nS3:\n  access_key_id: ADFASDF78789ASD897F\n  secret_access_key_id: AHDH890SDAF897987A890SDF\n```\n\n## Building a Pipeline\n\nThe basics of building a pipeline...\n\nWe start with how you will structure your project. A project repository should have the following structure (Subject to change with tool maturity)\n\n```text\nargo_workflows\npipelines\n  pipeline_drivers\n  pipes\nDockerfile\n```\n\nThis structure is conventional and should always be followed. At current there is no override for directories or directory structure.\n\nThe argo_jobs folder should contain ALL Argo CronWorkflow manifests.\n\nThe pipelines section is where all code lives.\n\nThe pipeline_drivers subdirectory must contain either\n1. Python scripts - one per pragmatic pipeline\n2. A master_driver script (main.py) that reads in the /root/.tdm/config.yml\n3. A combination of the two\n\nE.g.\n```text\nmain.py\nservice_now_mdm.py\nairwatch_mdm.py\n```\n\nThe pipes subdirectory contains all the custom pipelines. This section can be organized however one sees fit. However, it is recommended to define a logical structure that will provide meaning to anyone who is working with the code and for yourself in the future. One such way is\n```text\npipelines\n  pipes\n    service_now\n    airwatch_mdm\n    cloudwatch\n    common\n```\n\n## Pipelines: From Script to Production\n\nMany pipelines start as a script. And there is no issue or incorrectness in this when you are prototyping or working out some unknowns. Sometimes they are put together by amateur or junior developers. Regardless of how they originate, there is a need to move to a production pipelines. And that generally requires a large amount of custom coding, hardening, security awareness and much more. tdm provides both a framework for building custom steps with many of these concerns abstracted away or made easy to use - while also providing out of the box functionality to keep from creating the same code over and over.\n\nWhat\'s covered here, that is different from the technical documentation above the goes into greater detail on how to do very specific extensions to or understanding of functionality of the various components is a step-by-step walk through of moving a script into a tdm production pipeline.\n\nFor simplicity we will assume that the script is written in python. If it is not written in python, then the engineer should add steps to translate to python. We will refer to it as **script.py**\n\n### Original script\n\nWe will be working with the code in the below section.\n\n```python\nimport various_libraries_here\n\nsome_df = pd.DataFrame()\nanother_df = pd.DataFrame()\n\npayload = {}\n\nheaders = {\n  \'aw-tenant-code\': \'SECRET_CODE\',\n  \'Accept\': \'application/json\',\n  \'Content-Type\': \'application/json\',\n  \'Authorization\': \'Basic SECRET_KEY\'}\n\nurl="DYNAMIC_ACCESS_URL"\nresponse = requests.request("GET", url, headers=headers, data = payload)\n\ndata=json.loads(response.text)\nprint(data)\ndata1=data["Devices"]\ndata1=pd.DataFrame(data1, columns=[\'SerialNumber\',\'Uuid\',\'Id_Value\'])\n\n\n\nfor index, row in data1.iterrows():\n    try:\n        info=row[\'info\']\n        id = row[\'Uuid\']\n        payload = {}\n        url = "URL_%s"%id\n        response = requests.request("GET", url, headers=headers, data = payload)\n        data=json.loads(response.text)\n        data=data["some_items"]\n        some_df = pd.DataFrame.from_dict(data)\n        some_df["info"]=row[\'info\']\n        some_df["id"]=row[\'id\']\n        another_df=security_df.append(some_df,sort=False)\n        print(\'Information\',\'info\')\n    except Exception as error:\n        print("Error fetching data %s",id,error)\n\n\nanother_df.rename(columns=COLUMN_MAPPING_DEFINITION)\n\nbucket = \'bucket_name\'\ncsv_buffer = StringIO()\nanother_df.to_csv(csv_buffer,sep =\',\',index=False)\n\nsession = boto3.Session(\naws_access_key_id=\'AWS_KEY\',\naws_secret_access_key=\'AWS_SECRET_KEY\'\n)\n\n# Creating S3 Resource From the Session.\ns3_resource = session.resource(\'s3\')\ns3_resource.Object(bucket, \'FILENAME_AND_PATH.csv\').put(Body=csv_buffer.getvalue())\n```\n\n#### Step 1\nA trained engineer will many issues with the code, and they shouldn\'t worry about most of them as the goal is to rewrite it in a tdm fashion. However, one step will help - and it will help with the eyes. Make sure that you install the pyhton library black\n\n```bash\npip install black\n```\n\nThen execute black against the script\n\n```bash\nblack script.py\n```\n\nand to remove extra non-standard newlines. The result of which would be something like below.\n\n```python\nimport various_libraries_here\n\n\nsome_df = pd.DataFrame()\nsecurity_df = pd.DataFrame()\n\npayload = {}\n\nheaders = {\n    "aw-tenant-code": "SECRET_CODE",\n    "Accept": "application/json",\n    "Content-Type": "application/json",\n    "Authorization": "Basic SECRET_KEY",\n}\n\nurl = "DYNAMIC_ACCESS_URL"\nresponse = requests.request("GET", url, headers=headers, data=payload)\n\ndata = json.loads(response.text)\nprint(data)\ndata1 = data["Devices"]\ndata1 = pd.DataFrame(data1, columns=["info", "id", "value"])\n\nfor index, row in data1.iterrows():\n    try:\n        info = row["info"]\n        id = row["Uuid"]\n        payload = {}\n        url = "URL_%s" % id\n        response = requests.request("GET", url, headers=headers, data=payload)\n        data = json.loads(response.text)\n        data = data["some_items"]\n        some_df = pd.DataFrame.from_dict(data)\n        some_df["info"] = row["info"]\n        some_df["id"] = row["id"]\n        another_df = security_df.append(some_df, sort=False)\n        print("Information", "info")\n    except Exception as error:\n        print("Error fetching data %s", id, error)\n\nanother_df.rename(columns=COLUMN_MAPPING_DEFINITION)\n\nbucket = "bucket_name"\ncsv_buffer = StringIO()\nanother_df.to_csv(csv_buffer, sep=",", index=False)\n\nsession = boto3.Session(\n    aws_access_key_id="AWS_KEY", aws_secret_access_key="AWS_SECRET_KEY"\n)\n\n# Creating S3 Resource From the Session.\ns3_resource = session.resource("s3")\ns3_resource.Object(bucket, "FILENAME_AND_PATH.csv").put(Body=csv_buffer.getvalue())\n```\n\n#### Step 2 - Breaking Down the Processing Steps\n\nThis step can really happen after Step 3 (below), but that should be up to the engineers preference. The importance of the breakdown process is to different steps of the pipeline. Each of which would constitute a different Pipe.\n\nAs we go through the code there will be three general chunks we should look for\n\n1. Reading Data\n2. Transforming Data\n3. Writing Data\n\nIn the above code, which is pretty friendly as it avoids writing to storage locations we will not be using (e.g. local instead of cloud object storage (S3)), we have these as the basic chunks\n\n**Reading Data**\nReading is a two step process in this example. The output of the first read is used as input to the second read.\n\n*Read Step 1*\n```python\nsome_df = pd.DataFrame()\nsecurity_df = pd.DataFrame()\n\npayload = {}\n\nheaders = {\n    "aw-tenant-code": "SECRET_CODE",\n    "Accept": "application/json",\n    "Content-Type": "application/json",\n    "Authorization": "Basic SECRET_KEY",\n}\n\nurl = "DYNAMIC_ACCESS_URL"\nresponse = requests.request("GET", url, headers=headers, data=payload)\n\ndata = json.loads(response.text)\nprint(data)\ndata1 = data["Devices"]\ndata1 = pd.DataFrame(data1, columns=["info", "id", "value"])\n```\n\n*Read 2*\n```python\nfor index, row in data1.iterrows():\n    try:\n        info = row["info"]\n        id = row["id"]\n        payload = {}\n        url = "URL_%s" % id\n        response = requests.request("GET", url, headers=headers, data=payload)\n        data = json.loads(response.text)\n        data = data["some_items"]\n        some_df = pd.DataFrame.from_dict(data)\n        some_df["info"] = row["info"]\n        some_df["id"] = row["id"]\n        another_df = security_df.append(some_df, sort=False)\n        print("Information", "info")\n    except Exception as error:\n        print("Error fetching data %s", id, error)\n```\n\n**Transforming Data**\n\n*Transform*\n\nThe next transformation is just a line of code. It is simple, but also very common. tdm has support for this.\n```python\nanother_df.rename(columns=COLUMN_MAPPING_DEFINITION)\n```\n\n\n\n**Writing Data**\n\nThis final step will usually not require a custom implementation. Here we are dealing with writing a pandas DataFrame out to S3 as a csv file. tdm supports this write operation.\n```python\nbucket = "bucket_name"\ncsv_buffer = StringIO()\nanother_df.to_csv(csv_buffer, sep=",", index=False)\n\nsession = boto3.Session(\n    aws_access_key_id="AWS_KEY", aws_secret_access_key="AWS_SECRET_KEY"\n)\n\n# Creating S3 Resource From the Session.\ns3_resource = session.resource("s3")\ns3_resource.Object(bucket, "FILENAME_AND_PATH.csv").put(Body=csv_buffer.getvalue())\n```\n\nNote that when tdm does not support the needed functionality it is always possible to add support for it.\n\n\n#### Step 3 - Creating Pipes\n\nNow, given chunks of code we need to create Pipe for each of them.\n\n##### Creating the Read Pipe\nLet\'s start with a Pipe to read the data. If you recall from above on how we structure our code, let\'s call this pipeline "Example Pipeline" so that in the pipes directory we create a subdirectory call \'example_pipeline\' and in there a file called \'read.py\'\n```bash\nsome_project\n  pipeline_drivers\n  pipelines\n    example_pipeline\n      read_example_data.py\n```\n\nThe contents of read.py should start as a ReaderPipe type that have the reading part of the script from abnove.\n```python\nclass ReadExampleData(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, **kwargs) -> ReaderPipe:\n    some_df = pd.DataFrame()\n    another_df = pd.DataFrame()\n\n    payload = {}\n\n    headers = {\n        "aw-tenant-code": "SECRET_CODE",\n        "Accept": "application/json",\n        "Content-Type": "application/json",\n        "Authorization": "Basic SECRET_KEY",\n    }\n\n    url = "DYNAMIC_ACCESS_URL"\n    response = requests.request("GET", url, headers=headers, data=payload)\n\n    data = json.loads(response.text)\n    print(data)\n    data1 = data["Devices"]\n    data1 = pd.DataFrame(data1, columns=["info", "id", "value"])\n\n```\n\nThis is not yet executable and is still rather messy\n* remove some_df - it is never used. Also, another_df is not used here either -> remove it.\n* set data={} in requests GET call - and remove payload variable\n* move url to key word argument\n\n```python\nclass ReadExampleData(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, url: str, **kwargs) -> ReaderPipe:\n    headers = {\n        "aw-tenant-code": "SECRET_CODE",\n        "Accept": "application/json",\n        "Content-Type": "application/json",\n        "Authorization": "Basic SECRET_KEY",\n    }\n\n    response = requests.request("GET", url, headers=headers, data={}})\n\n    data = json.loads(response.text)\n    print(data)\n    data1 = data["Devices"]\n    data1 = pd.DataFrame(data1, columns=["info", "id", "value"])\n\n```\n\nRemove the print command. We should never print the data out. Not only can this cause logs (as print commands should rightly be in production) to blow up, but this can result in sensitive information being logged out to less secure systems and/or violate compliance controls. In general, unless you are locally hacking at something - never use print as a \'logger\'.\n\nAlso, you should see that \'Devices\' is all that is needed from the read text - so simplify things further by combining lines of code.\n\nAlso, there is no reason to have data and data1. Make it all one line.\n\n```python\nclass ReadExampleData(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, url: str, **kwargs) -> ReaderPipe:\n    headers = {\n        "aw-tenant-code": "SECRET_CODE",\n        "Accept": "application/json",\n        "Content-Type": "application/json",\n        "Authorization": "Basic SECRET_KEY",\n    }\n\n    response = requests.request("GET", url, headers=headers, data={}})\n\n    data = pd.DataFrame(json.loads(response.text)["Devices"], columns=["info", "id", "value"])\n```\n\nThe next step is to move secrets to secrets.yml. This should be\n\n```bash\nRead Example Secrets:\n  code: SECRET_CODE\n  key: SECRET_KEY\n```\n\nAnd then to use the self._secrets property in the code directly.\n\n```python\nclass ReadExampleData(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, url: str, **kwargs) -> ReaderPipe:\n    headers = {\n        "aw-tenant-code": self._secrets.code,\n        "Accept": "application/json",\n        "Content-Type": "application/json",\n        "Authorization": f"Basic {self._secrets.key}",\n    }\n\n    response = requests.request("GET", url, headers=headers, data={}})\n\n    data = pd.DataFrame(json.loads(response.text)["Devices"], columns=["info", "id", "value"])\n```\n\nWe are almost there, we still need to adjust the metadata and return self.\n\nAlso, change data to df, that is the conventional approach when working with dataframes.\n\n```python\nclass ReadExampleData(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, url: str, **kwargs) -> ReaderPipe:\n    headers = {\n        "aw-tenant-code": self._secrets.code,\n        "Accept": "application/json",\n        "Content-Type": "application/json",\n        "Authorization": f"Basic {self._secrets.key}",\n    }\n\n    response = requests.request("GET", url, headers=headers, data={}})\n\n    df = pd.DataFrame(json.loads(response.text)["Devices"], columns=["info", "id", "value"])\n\n    self._state.update_metadata(key="data", value=df)\n\n    return self\n```\n\nNow, two final things to have a minimally production version of this code. We need to add logging and handle any exceptional situations.\n1. requests call can have a return that has a Status Code that is not 200 - indicating some kind of error.\n2. Other errors can occur - \'Devices\' not present in dataframe, secrets are missing, etc... But such errors will bubble up with clarity. The requests error would end up looking like a data error - which it is not.\n\n```python\nclass ReadExampleData(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, url: str, **kwargs) -> ReaderPipe:\n    headers = {\n        "aw-tenant-code": self._secrets.code,\n        "Accept": "application/json",\n        "Content-Type": "application/json",\n        "Authorization": f"Basic {self._secrets.key}",\n    }\n    response = requests.request("GET", url, headers=headers, data={}})\n\n    if response.status_code != 200:\n      raise ValueError(\n        {\n          "content": response.text,\n          "code": response.status_code,\n          "reason": response.reason,\n        }\n      )\n    else:\n      self._logger.debug(\'Successfully read data from restful endpoint.\')\n\n\n    df = pd.DataFrame(json.loads(response.text)["Devices"], columns=["info", "id", "value"])\n\n    self._state.update_metadata(key="data", value=df)\n\n    return self\n```\n\nWell, we are almost done, I lied. We need to add some docstrings.\n\n```python\nclass ReadExampleData(ReaderPipe):\n  """Example Pipe for reading data from a RESTful endpoint."""\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, url: str, **kwargs) -> ReaderPipe:\n    """Execution of the ReadExampleData data ingestion pipe. Will pull data from a RESTful endpoint and pass the resulting pandas.DataFrame to the metadata layer.\n\n    Args:\n      url: URL endpoint from which data will be retrieved.\n\n    Raises:\n      ValueError: When the API call returns a status code other than 200.\n    """\n    headers = {\n        "aw-tenant-code": self._secrets.code,\n        "Accept": "application/json",\n        "Content-Type": "application/json",\n        "Authorization": f"Basic {self._secrets.key}",\n    }\n    response = requests.request("GET", url, headers=headers, data={}})\n\n    if response.status_code != 200:\n      raise ValueError(\n        {\n          "content": response.text,\n          "code": response.status_code,\n          "reason": response.reason,\n        }\n      )\n    else:\n      self._logger.debug(\'Successfully read data from restful endpoint.\')\n\n\n    df = pd.DataFrame(json.loads(response.text)["Devices"], columns=["info", "id", "value"])\n\n    self._state.update_metadata(key="data", value=df)\n\n    return self\n```\n\n*Note:* it could be argued that we should either pass the filter columns in as parameters or create a new step that filters the data. There are good arguments in both ways,\n\n*Read 2*\nIf you recall, the read process was two steps. Each of these will be a separate ReaderPipe. Yes, one read can be input to another - why not. For reference, we will start with adding the data read to a ReaderPipe\n\n```python\nclass ReadExampleDataIteratively(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, **kwargs) -> ReaderPipe:\n    for index, row in data1.iterrows():\n    try:\n        info = row["info"]\n        id = row["id"]\n        payload = {}\n        url = "URL_%s" % id\n        response = requests.request("GET", url, headers=headers, data=payload)\n        data = json.loads(response.text)\n        data = data["some_items"]\n        some_df = pd.DataFrame.from_dict(data)\n        some_df["info"] = row["info"]\n        some_df["id"] = row["id"]\n        another_df = security_df.append(some_df, sort=False)\n        print("Information", "info")\n    except Exception as error:\n        print("Error fetching data %s", id, error)\n```\n\nThis is just the start, we need to be able to get the data from the previous step. This will - under an assumption - come from a parent Pipe - the first parent Pipe. The way to get the data is then to create a reference to it in the code. Now, since this is done often it is useful to have a pattern to use for this\n\n```python\nclass ReadExampleDataIteratively(ReaderPipe):\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, url: str, **kwargs) -> ReaderPipe:    \n    if len(self._parents) == 1:\n            df: pd.DataFrame = self._parents[0].info["data"]\n        else:\n            raise ValueError("No parent Pipe was provided.")\n\n        if data is None:\n            raise ValueError("\'data\' not found in parent Pipe\'s metadata.")\n\n    for _, row in df.iterrows():\n    try:\n        info = row["info"]\n        id = row["id"]\n        payload = {}\n        url = "URL_%s" % id\n        response = requests.request("GET", url, headers=headers, data=payload)\n        data = json.loads(response.text)\n        data = data["some_items"]\n        some_df = pd.DataFrame.from_dict(data)\n        some_df["info"] = row["info"]\n        some_df["id"] = row["id"]\n        another_df = security_df.append(some_df, sort=False)\n        print("Information", "info")\n    except Exception as error:\n        print("Error fetching data %s", id, error)\n```\n\nNow, without spending the same time breaking things down as before, we should end up with something like this.\n\n```python\nclass ReadExampleDataIteratively(ReaderPipe):\n  """Example Pipe for reading data from a RESTful endpoint iteratively given an input parent with information to iterate over."""\n  def __init__(self, **kwargs):\n    super().__init__(**kwargs)\n\n  def _run(self, **kwargs) -> ReaderPipe:\n    """Execution of the ReadExampleData data ingestion pipe. Will pull data from a RESTful endpoint and pass the resulting pandas.DataFrame to the metadata layer.\n\n    Raises:\n      ValueError: When the API call returns a status code other than 200.\n    """\n    if len(self._parents) == 1:\n            df: pd.DataFrame = self._parents[0].info["data"]\n        else:\n            raise ValueError("No parent Pipe was provided.")\n\n        if data is None:\n            raise ValueError("\'data\' not found in parent Pipe\'s metadata.")\n\n    headers = {\n      "aw-tenant-code": self._secrets.code,\n      "Accept": "application/json",\n      "Content-Type": "application/json",\n      "Authorization": f"Basic {self._secrets.key}",\n    }\n\n    iteratively_constructed_df = pd.DataFrame()\n    for index, row in df.iterrows():\n      try:\n        url = f"URL_{id}"\n        response = requests.request("GET", url, headers=headers, data={})\n        if response.status_code != 200:\n          raise ValueError(\n            {\n              "content": response.text,\n              "code": response.status_code,\n              "reason": response.reason,\n            }\n          )\n\n        local_df = pd.DataFrame.from_dict(\n          json.loads(response.text)[\'some_items\']\n        )\n        local_df["info"] = row["info"]\n        local_df["id"] = row["id"]\n        iteratively_constructed_df = another_df.append(local_df, sort=False)\n        self._logger.info({"Information": "info"})\n      except:\n        raise ValueError(\n          {\n            "Summary": f"Error fetching data for id={id}",\n            \'stack_trace\': trackback.format_exc(),\n          }\n        )\n\n      self._state.update_metadata(key="data", value=iteratively_constructed_df)\n\n      return self\n```\n\nThis last read step we will create in a file called read_example_data_iteratively.py. The file tree should now look like\n\n```bash\nsome_project\n  pipeline_drivers\n  pipelines\n    example_pipeline\n      read_example_data.py\n      read_example_data_iteratively.py\n```\n\n\n##### Creating the Transform Pipe\n\nThe sole transformation is just a line of code. It is simple, but also very common. tdm has support for this.\n```python\nanother_df.rename(columns=COLUMN_MAPPING_DEFINITION)\n```\n\nYou will just use the RenameColumns TransformerPipe.\n\n```python\n# some_config has a parameter for column_map\nremap_columns = RenameColumns(**some_config)  \n\nremapped_columns = remap_columns.run(parent=some_parent)\n```\n\n##### Creating the Write Pipe\n\nIf you recall, we also have methods for reading data. The first pass of this framework had a WriterPipe for each destination. However, the internal writing configuration makes this unnecessary. So you would use a FileWriter\n\n```python\nwriter = FileWriter(**some_config)\n\ndata_wrote = writer.run(parent=remapped_columns)\n```\n\nIn most cases you will not need to worry about creating custom write methods - unless there is a new destination. In this situation (and not covered here) you will need to create a new DataframeWriter (or similar) Writing method that will be constructed within the FileWriter.\n\n#### Building Pipeline Driver\n\nNow that we have created some custom Pipes and decomposed the pipeline, we need to create the orchestration layer to execute the pipeline. This is referred to as a pipeline_driver in the application layer. Each pipeline driver will take the name of the pipeline - this is the convention. Once creating this file, the file tree will be\n\n```bash\nsome_project\n  pipeline_drivers\n    example_pipeline.py\n  pipelines\n    example_pipeline\n      read_example_data.py\n      read_example_data_iteratively\n```\n\nLet\'s walk through creating the pipeline_driver for example_pipeline.\n\n##### Step 1 - Create a skeleton for the drivers\n\nCreate a base driver that has a method call and a set of pipeline instnaces.\n\n```python\ndef example_pipeline(): <- we don\'t know what goes here yet\n\n  workflow_config = {}\n\n  # Create skeleton instances of all the pipes\n  reader = ReadExampleData(**workflow_config[\'read\'])\n  iterative_reader = ReadExampleDataIteratively(**workflow_config[\'read\'])\n  column_renamer = RenameColumns(**workflow_config[\'read\'])\n  writer = FileWriter(**workflow_config[\'read\'])\n\n```\n\n##### Step 2 - create basic orchestration flow\n\nWrite the pipeline orchestration flow\n\n```python\ndef example_pipeline(): <- we don\'t know what goes here yet\n\n  workflow_config = {}\n\n  # Create skeleton instances of all the pipes\n  reader = ReadExampleData(**workflow_config[\'read\'])\n  iterative_reader = ReadExampleDataIteratively(**workflow_config[\'read_iteratively\'])\n  column_renamer = RenameColumns(**workflow_config[\'rename\'])\n  writer = FileWriter(**workflow_config[\'write\'])\n\n  # pipeline orchestration\n  read_data = reader.run()\n  iteratively_read = reader.run(parent=iteratively_read)\n  renamed_columns = column_renamer.run(parent=iteratively_read)\n  written_data = writer.run(parent=renamed_columns)\n```\n\n##### Step 3 - Create Configuration\n\nThe configuration is how we are stating the inputs to the pipeline. You can opt for key word arguments in the constructors, but that would be a user decision - and when declarative pipelines are ready would require additional work.\n\n```python\ndef example_pipeline(\n  pipeline_id: str,\n  job_id: str,\n):\n\n  workflow_config = {\n      {\n      "read": {\n        "name": "Read Data",\n        "secrets": {\n          "type": "yamlconfigreader",\n          "name": "Read RESTful Secrets",\n        },\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "read": {\n        "name": "Read Data",\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "read": {\n        "name": "Read Data Iteratively",\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "write": {\n        "name": "Write Data",\n        "writer": {\n          "name": "S3 Writer",\n          "type": "FileWriter",\n          "config": {\n            "bucket": "name of bucket",\n            "secrets": {\n              "type": "yamlconfigreader",\n              "name": "S3 Secrets",\n            },\n          },\n        }\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n    }\n\n  # Create skeleton instances of all the pipes\n  reader = ReadExampleData(**workflow_config[\'read\'])\n  iterative_reader = ReadExampleDataIteratively(**workflow_config[\'read\'])\n  column_renamer = RenameColumns(**workflow_config[\'read\'])\n  writer = FileWriter(**workflow_config[\'read\'])\n\n  # pipeline orchestration\n  read_data = reader.run()\n  iteratively_read = reader.run(parent=iteratively_read)\n  renamed_columns = column_renamer.run(parent=iteratively_read)\n  written_data = writer.run(parent=renamed_columns)\n```\n\n##### Step 4 - Adding a State Manager\n\nThe StateManager allows you to track the history of a data pipeline. Make sure to use a StateManager that is right for your needs.\n\n```python\ndef example_pipeline(\n  pipeline_id: str,\n  job_id: str,\n  asset: str,\n  state_bucket: str,\n  data_bucket: str\n):\n\n\n  state_manager = S3StateManager(\n    **{\n      "name": f"State Manager for SNOW Data - {asset}",\n      "state_object_name": f"SNOW Data Pull - {asset}",\n      "pipeline_id": pipeline_id,\n      "job_id": job_id,\n      "version": 1,\n      "writer": {\n        "type": "S3CSVWriter",\n        "config": {\n          "container": state_bucket,\n          "secrets": {"name": "S3", "type": "yamlconfigreader"},\n        },\n        "name": "S3 State Writer",\n      },\n      "reader": {\n        "type": "S3CSVReader",\n        "config": {\n          "bucket": state_bucket,\n          "secrets": {"name": "S3", "type": "yamlconfigreader"},\n        },\n        "name": "S3 State Reader",\n      },\n    }\n  )\n\n  workflow_config = {\n      {\n      "read": {\n        "name": "Read Data",\n        "secrets": {\n          "type": "yamlconfigreader",\n          "name": "Read RESTful Secrets",\n        },\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "read": {\n        "name": "Read Data",\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "read": {\n        "name": "Read Data Iteratively",\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "write": {\n        "name": "Write Data",\n        "writer": {\n          "name": "S3 Writer",\n          "type": "FileWriter",\n          "config": {\n            "bucket": data_bucket,\n            "secrets": {\n              "type": "yamlconfigreader",\n              "name": "S3 Secrets",\n            },\n          },\n        }\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n    }\n\n  # Create skeleton instances of all the pipes\n  reader = ReadExampleData(**workflow_config[\'read\'])\n  iterative_reader = ReadExampleDataIteratively(**workflow_config[\'read\'])\n  column_renamer = RenameColumns(**workflow_config[\'read\'])\n  writer = FileWriter(**workflow_config[\'read\'])\n\n  # pipeline orchestration\n  read_data = reader.run()\n  iteratively_read = reader.run(parent=iteratively_read)\n  renamed_columns = column_renamer.run(parent=iteratively_read)\n  written_data = writer.run(parent=renamed_columns)\n```\n\n##### Step 5: Using the StateManager\n\nTo really use the state manager, you need to make sure that you are able to capture data from your pipeline, even if it fails. So we add some exception handling and create an entry(ies) to the StateManager\n```python\nThe StateManager allows you to track the history of a data pipeline. Make sure to use a StateManager that is right for your needs.\n\n```python\ndef example_pipeline(\n  pipeline_id: str,\n  job_id: str,\n  state_container: str,\n  asset: str,\n  state_bucket: str,\n  data_bucket: str,\n):\n\n\n  state_manager = S3StateManager(\n    **{\n      "name": f"State Manager for SNOW Data - {asset}",\n      "state_object_name": f"SNOW Data Pull - {asset}",\n      "pipeline_id": pipeline_id,\n      "job_id": job_id,\n      "version": 1,\n      "writer": {\n        "type": "S3CSVWriter",\n        "config": {\n          "container": state_bucket,\n          "secrets": {"name": "S3", "type": "yamlconfigreader"},\n        },\n        "name": "S3 State Writer",\n      },\n      "reader": {\n        "type": "S3CSVReader",\n        "config": {\n          "bucket": state_bucket,\n          "secrets": {"name": "S3", "type": "yamlconfigreader"},\n        },\n        "name": "S3 State Reader",\n      },\n    }\n  )\n\n  workflow_config = {\n      {\n      "read": {\n        "name": "Read Data",\n        "secrets": {\n          "type": "yamlconfigreader",\n          "name": "Read RESTful Secrets",\n        },\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "read": {\n        "name": "Read Data",\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "read": {\n        "name": "Read Data Iteratively",\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n\n      "write": {\n        "name": "Write Data",\n        "writer": {\n          "name": "S3 Writer",\n          "type": "FileWriter",\n          "config": {\n            "bucket": "name of bucket",\n            "secrets": {\n              "type": "yamlconfigreader",\n              "name": "S3 Secrets",\n            },\n          },\n        }\n        "pipe_id": uuid.uuid4().hex,\n        "job_id": job_id,\n        "pipeline_id": pipeline_id,\n      },\n    }\n\n  start_time = datetime.now().timestamp()\n  try:\n    status = PipeStatus.QUEUED\n\n    # Instances of all pipes\n    reader = ReadExampleData(**workflow_config[\'read\'])\n    iterative_reader = ReadExampleDataIteratively(**workflow_config[\'read\'])\n    column_renamer = RenameColumns(**workflow_config[\'read\'])\n    writer = FileWriter(**workflow_config[\'read\'])\n\n    # pipeline orchestration\n    executed_pipes = []\n    read_data = reader.run()\n    executed_pipes.append(read_data)\n\n    iteratively_read = reader.run(parent=iteratively_read)\n    executed_pipes.append(iteratively_read)\n\n    renamed_columns = column_renamer.run(parent=iteratively_read)\n    executed_pipes.append(renamed_columns)\n\n    written_data = writer.run(parent=renamed_columns)\n    executed_pipes.append(written_data)\n\n    status = PipeStatus.FAILURE if any(\n      [True for result in executed_pipes if result.status == PipeStatus.FAILURE]\n    ) else PipeStatus.SUCCESS\n\n  except:\n    _logger.error(msg=traceback.format_exc())\n    status = PipeStatus.FAILURE\n    raise\n\n  finally:\n    stop_time = datetime.now().timestamp()\n    state_manager.update(\n        stop_time=stop_time,\n        start_time=start_time,\n        flow_model=workflow_config,\n        status=status,\n        meta_state={},\n        pull_stats=read_stats,\n        push_stats={},\n        job_id=job_id,\n    )\n```\n\n##### Step 6: Writing an entrypoint\nAn entrypoint is a CLI that can be used to execute the pipeline in orchestration platform like Argo Workflows, Kubernetes with CronJobs, etc...\n\nThe entrypoint should be something like:\n\n```python\n@click.command()\n@click.option(\n    "--pipeline", "-p", type=str, required=True, help="This is the name of the pipe"\n)\n@click.option(\n    "--pipeline_id",\n    "-id",\n    type=str,\n    required=True,\n    help="This is a required id to uniquely identify the pipeline",\n)\n@click.option(\n    "--asset",\n    "-f",\n    type=str,\n    required=True,\n    help="This is the name of the asset",\n)\n@click.option(\n    "--state_bucket",\n    "-s",\n    type=str,\n    required=True,\n    help="This is the S3 bucket where state will be stored",\n)\n@click.option(\n    "--data_bucket",\n    "-d",\n    type=str,\n    required=True,\n    help="This is the S3 bucket where data will be stored",\n)\ndef tdm(\n    pipeline: str,\n    pipeline_id: str,\n    state_bucket: str,\n    data_bucket: str,\n) -> None:\n    if pipeline == "example_pipeline":\n        example_pipeline(\n            pipeline_id=pipeline_id,\n            job_id=uuid.uuid4().hex,\n            asset=asset,\n            state_bucket=state_bucket,\n            data_bucket=data_bucket,\n        )\n     else:\n        raise ValueError(f\'Pipeline {pipeline} does not exist.\')\n```\n\nThe first part of the code\n\n##### Step 7: Orchestration - Creating the Argo Workflow\nHere we are assuming that you are using an Argo Workflow. This should be similar for each.\n\nMake sure to change the schedule - avoid too many pipelines scheduled at the same time.\n\nEach pipeline needs a unique pipeline_id.\n\nSet the other CLI variables accordingly.\n\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: CronWorkflow\nmetadata:\n  name: example_pipeline\nspec:\n  schedule: "20 * * * *"\n  concurrencyPolicy: "Replace"\n  startingDeadlineSeconds: 0\n  workflowSpec:\n    imagePullSecrets:\n      - name: gcr\n    entrypoint: example_pipeline\n    templates:\n    - name: Example Pipeline\n      container:\n        image: your_image\n        imagePullPolicy: Always\n        command: [ python,\n                   -m,\n                   pipelines.pipeline_drivers.entrypoint,\n                   --pipeline,\n                   example_pipeline,\n                   --pipeline_id,\n                   897289734789123894,\n                   --asset,\n                   some_asset,\n                   --state_bucket,\n                   state_bucket,\n                   --data_bucket,\n                   example_data ]\n```\n\n##### Step 8: Secrets & Configurations\nWhile it is not straightforward at this time to always use a configuration for your pipelines - simple cases you can - it is best to store the alerting configuration in the config.yml in the .tdm directory. When starting development you may not have a configuration for your alerting. Use the one below.\n\n```bash\nversion: 1\n\nalert_managers:\n  - name: Alert Manager\n    type: nullalertmanager\n    config:\n      name: Not Real\n      secrets:\n        type: yamlconfigreader\n        name: Not a secret\n\nalert\n```\n\nSecrets will be stored in the same location as secrets.yml\n\n#### Building & Deploying Pipelines\nBuilding and deploying your pipelines should follow at least a dev/prod flow. That means, you will need an orchestration configuration for each of two different environements, two environments and deployment endpoints for dev and prod, respectively. And this even goes for locations where data is stored - preferably in different cloud accounts.\n\nWe currently using this approach here at Tellius. In our projects we create a pipelines in a monorepo - with customer or domain level separation. This allows us to keep similar code all in one place. And the pipelines are reusable - the same conventional structure will exist in all places. When working on one pipeline we will\n1. Create the new pipeline code\n2. Create a new driver\n3. Create a new entry in the entrypoint file\n4. Create a new test (we have an additional directory for tests)\n5. We will test locally if possible to identify any issues before committing code if possible.\n6. We will commit and push code to development branch\n7. Dev CI flow will execute\n8. Docker Image will be created\n9. Argo CronWorkflow will be pushed to S3\n10. (Manually right now) Argo will manually be updated with the latest version of the manifest.\n11. Argo will be triggered (manually) - maybe a few times - on a dev Argo deployment\n12. We check the logs to see if anything went wrong - eventhough local may have worked - remote may not have.\n13. If successful and ready to push to production go to 14, else iterate and improve (same if failure)\n14. Merge code to main branch\n15. CI/CD flow will execute again\n16. 10-12 will repeat.\n\nThe CI/CD flow is approximately this below at minimum\n\n![Basic Pipeline](images/build_flow.png)\n\n\n\n\n\n\n\n\n\n\n\n\n\n<!--\n### Declarative API\n\nBefore we start, let\'s make it clear, the Declarative API is more advanced in usage than the Pragmatic API, which we describe below. However, it is much cleaner and is what should be used in most production situations. The Pragmatic API is best suited for development and testing of new pipelines and new Pipes.\n\nThe declarative API works by specifying the\n1. A collection of Alert Managers\n2. A Collection of Pipelines\n\n```yaml\nversion: v1\n\nalert_managers:\n  - name: Email Alert Manager\n    type: gmailemailalertmanager\n    config:\n      email_addresses:\n        - alert_email@some_email.something\n      mode: mfa\n      secrets:\n        type: yamlconfigreader\n        name: Gmail Secrets\n\npipelines:\n  - name: Device Inventory\n    pipes:\n      - name: Query Inventory\n        type: QueryDI\n        config:\n          secrets:\n            type: yamlconfigreader\n            name: Inventory Secrets\n\n      - name: Add Date From Current Time\n        type: appenddatecolumn\n        config:\n          parents:\n            - Query Inventory\n\n      - name: Write to Disk\n        type: writelocalfile\n        config:\n          parents:\n            - Add Data From Current Time\n          directory: output_data\n\n  - name: Time Difference\n    pipes:\n      - name: Query Time Difference\n        type: QueryTD\n        config:\n          secrets:\n            type: yamlconfigreader\n            name: Time Difference Secrets\n\n      - name: Add Date From Current Time\n        type: appenddatecolumn\n        config:\n          parents:\n            - Query Time Difference\n```\n\nAll parts of the declarative API require that a field of `name` be present. The name field provides a unique identifier for the\n\nThe first part, that alert_manager section is required when you need to setup or configure alerting for pipelines. This is used in the event that there are issues with a pipeline and an alert is required.\n\nEach AlertManager has a type. This is the name of the class and can be any mix of upper and lowercase.\n\nThere will also be a configuration. The configuration (`config`) section will be the values required to construct the class. When the class requires secrets there will be a special section called _secret_. The secrets section requires specifying a type. The type refers to the class type that will be used to `read` the secrets into memory. The `name` in the secrets section corresponds to the secret as named in the `secrets.yml`.\n\n```yaml\nalert_managers:\n  - name: Email Alert Manager\n    type: gmailemailalertmanager\n    config:\n      email_addresses:\n        - john.aven@tellius.com\n      mode: mfa\n      secrets:\n        type: yamlconfigreader\n        name: Gmail Secrets\n ```\n\nThe pipelines section describes a collection of various different pipelines that will get scheduled and executed. The pipelines have a `name` and `pipes` fields. The pipes field is a collection of different Pipes that will be executed in order of dependency. Each Pipe will have a `name`, `type` and `config` just as the AlertManager. A config within a pipe will have an additional field `parents` that will list, by names, the Pipes in the current pipeline, which must be executed beforehand.\n\n```yaml\n\npipelines:\n  - name: Pipeline 1\n    pipes:\n      - name: Pipe One\n        type: pipelineTypeOne\n        config: SOME_CONFIGURATION_DICTIONARY\n      - name: Pipe Two\n        type: pipelineTypeTow\n        config:\n          some_parameter: some_value\n          parents:\n            - Pipe One\n``` -->\n<!--\n### Pragmatic API\n\nThe Pragmatic API is capable of everything that the Declarative API is. However, it is generally easier for a beginner to write a pipeline pragmatically. An example, and simple, pipeline is seen below\n\n```python\n\n# Set up the pipes (define the pipes)\nread_secrets = {}\nread_configuration = {\n  \'parameter\': \'value\'\n  \'secrets\': pipe_secrets\n}\nread_pipe = SomePipe(**read_configuration)\n\ntransform_configuration = {}\ntransform_pipe = TransformPipe(**transform_configuration)\n\nwrite_configuration = {}\nwrite[\'secrets\'] = {}\nwrite_pipe = WritePipe(**write_configuration)\n\n# Create pragmatic pipe\n\nread = read_pipe.run()\ntransform = transform_pipe.run(parent=read)\nwrite = write_pipe.run(parent=transform)\n\n```\n\nIn the above the notation **some_dictionary is python shorthand for passing a bunch of named keyword arguments.\n\nThis approach is pretty simple. It is allows one to interactively build a pipeline and test it out. This is especially useful locally.\n\n## Orchestration of Pipeline Execution\n\nOrchestrating pipelines is the act of executing the pipeline manually or through the use of an orchestration engine.\n\n### Local Execution\n\nLocally executing a pipeline should only be done during testing. The execution should happen from the root of your project. And assuming the suggested project structure you would called\n```bash\ncd your_project\npython -m pipelines.pipeline_drivers.your_pipeline\n```\n\n### Cron Workflows\n\n## Extending the Library\n\ntdm is designed to be extended both in the library and externally within your own pipeline\'s codebase. We will cover here how to extend the library locally within your pipelines and how to update / modify deployments to support your new code.\n\n### Creating New Transformation, Readers, and Writers\n\nThe library comes with a number of base classes, all of which can be explored in the libraries\' documentation - included in the codebase for ease of access.\n\nTo create a new Pipe that has a type of Read, Write or Transform you will need to create a new class that inherits from these classes and implements the _run(...) method\n\n```python\nfrom tellius_data_manager.pipes.transformers.transformer_pipe import Transform\n\n\nclass NewTransform(Transform):\n    def _run(self, local_arguments: some_type = None, **kwargs):\n# Do Something Here\n```\n\nMake sure to include explicitly any variables that\n\n\n\n### Including New Code in Your Pipeline\n\n### Updating Local Deployments\n\n### Updating Docker Deployments\n\n### Building New Connectors\n\n### Creating Custom Transformations\n\n## Configuring Your Pipeline\n\nThe configuration, in the current version of tdm is describe below\n\n```yaml\nversion: v1\n\npipelines:\n  - name:\n    pipes:\n      - name: Read S3\n\n```\n\nFeatures to Come:\n\n* Auto Argo Creation\n* Kubernetes Secrets\n* AWS Stored Secrets\n\n## Software Design Considerations\n\nSoftware is designed using SOLID OOP design principles, and any extension to it must follow this design. Any extension\nbeyond what is currently present must be discussed - especially if it changes or breaks current design abstractions.\n\nIn terms of design patterns the solution relies on a combination of the\n* Strategy Pattern\n* Polymorphic Factory Pattern\n* Builder Pattern -->\n',
    'author': 'Tellius Open Source',
    'author_email': 'tellius.open.source@tellius.com',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.9,<4.0',
}


setup(**setup_kwargs)
