# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['mojap_metadata',
 'mojap_metadata.converters',
 'mojap_metadata.converters.arrow_converter',
 'mojap_metadata.converters.etl_manager_converter',
 'mojap_metadata.converters.glue_converter',
 'mojap_metadata.converters.glue_converter.specs',
 'mojap_metadata.converters.postgres_converter',
 'mojap_metadata.metadata',
 'mojap_metadata.metadata.specs']

package_data = \
{'': ['*']}

install_requires = \
['PyYAML>=5.4.1',
 'dataengineeringutils3>=1.4.0',
 'jsonschema>=3.0.0',
 'parameterized>=0.7.0,<0.8.0']

extras_require = \
{'arrow': ['pyarrow>=3.0.0'],
 'etl-manager': ['etl-manager>=7.4,<8.0'],
 'glue': ['pydbtools>=5.3.0,<6.0.0'],
 'postgres': ['psycopg2[postgres]>=2.9.2,<3.0.0',
              'testing.postgresql[postgres]>=1.3.0,<2.0.0',
              'SQLAlchemy[postgres]>=1.4.27,<2.0.0']}

setup_kwargs = {
    'name': 'mojap-metadata',
    'version': '1.11.1',
    'description': 'A python package to manage metadata',
    'long_description': '# mojap-metadata\n\nThis python package allows users to read and alter our metadata schemas (using the metadata module) as well as convert our metadata schemas to other schema definitions utilised by other tools (these are defined in the converters module and are defined as Converters).\n\n\n## Installation\n\nMake sure you are using a new version of pip (>=20.0.0)\n\n```bash\npip install git+https://github.com/moj-analytical-services/mojap-metadata\n```\n\nTo install additional dependencies that will be used by the converters (e.g. `etl-manager` and `arrow` extras)\n\n```bash\npip install \'mojap-metadata[etl-manager,arrow] @ git+https://github.com/moj-analytical-services/mojap-metadata\'\n```\n\n<hr>\n\n# Metadata \n\nThis module creates a class called `Metadata` which allows you to interact with our agnostic metadata schemas. The `Metadata` class deals with parsing, manipulating and validating metadata json schemas.\n\n## The Schema\n\nOur metadata schemas are used to define a table. The idea of these schemas are to define the contexts of a table with generic metadata schemas. If you want to use this schema to interact with Oracle, PyArrow or AWS Glue for example, then you can create a Converter class to take the metadata and converter it to a schema that works with that tool (or vice versa).\n\nWhen adding a parameter to the metadata config first thing is to look if it exists in [json-schema](https://json-schema.org/understanding-json-schema/index.html). For example `enum`, `pattern` and `type` are parameters in our column types but come from json schema naming definitions.\n\nAn example of a basic metadata schema:\n\n```json\n{\n    "$schema" : "$schema": "https://moj-analytical-services.github.io/metadata_schema/mojap_metadata/v1.0.0.json",\n    "name": "employees",\n    "description": "table containing employee information",\n    "file_format": "parquet",\n    "columns": [\n        {\n            "name": "employee_id",\n            "type": "int64",\n            "type_desc": "integer",\n            "description": "an ID for each employee",\n            "minimum": 1000,\n            "maximum": 9999\n        },\n        {\n            "name": "employee_name",\n            "type": "string",\n            "type_string": "string",\n            "description": "name of the employee"\n        },\n        {\n            "name": "employee_dob",\n            "type": "date64",\n            "type_desc": "date",\n            "description": "date of birth for the employee in ISO format",\n            "pattern": "^\\\\d{4}-([0]\\\\d|1[0-2])-([0-2]\\\\d|3[01])$"\n        }\n    ]\n}\n```\n\n### Schema Properties\n\n- **name:** String that can be whatever you want to name the table. Best to avoid spaces as most systems do not like that but it will let you do this.\n\n- **file_format:** String denoting the file format.\n\n- **columns:** List of objects where each object descibes a column in your table. Each column object must have at least a `name` and a (`type` or `type_description`).\n\n    - **name:** String denoting the name of the column.\n    - **type:** String specifing the type the data is in. We use data types from the [Apache Arrow project](https://arrow.apache.org/docs/python/api/datatypes.html). We use their type names as it seems to comprehensively cover most of the data types we deal with. _Note: In our naming convention for types we allow `bool` (which is equivalent to `bool_`) and `list` (which is equivalent to `list_`)._\n    - **type_category:** These group different sets of `type` properties into a single superset. These are: `integer`, `float`, `string`, `timestamp`, `bool`, `list`, `struct`. For example we class `int8, int16, int32, int64, uint8, uint16, uint32, uint64` as `integer`. It allows users to give more generic types if your data is not coming from a system or output with strict types (i.e. data exported from Excel or an unknown origin). The Metadata class has default type values for each given `type_category`. See the `default_type_category_lookup` attribute of the `Metadata` class to see said defaults. This field is required if `type` is not set.\n    - **description:** Description of the column.\n    - **enum:** List of what values that column can take. _(Same as the standardised json schema keyword)._\n    - **pattern:** Regex pattern that value has to to match (for string type_categories only). _(Same as the standardised json schema keyword)._\n    - **minLength / maxLength:** The minimum and maximum length of the string (for string type_categories only). _(Same as the standardised json schema keyword)._\n    - **minimum / maximum:** The minumum and maximum value a numerical type can take (for integer and float type_categories only).\n- **partitions:** List of what columns in your dataset are partitions.\n- **table_location:** the location of the table. This is a string that can represent a file path, directory, url, etc.\n- **database_name:**  the name of the database this table belongs to.\n\n#### Additional Schema Parameters\n\nWe allow users to add addition parameters to the table schema object or any of the columns in the schema. If there are specific parameters / tags you want to add to your schema it should still pass validation (as long as the additional parameters are not the same name of ones already used in the schema).\n\n## Usage\n\n```python\nfrom mojap_metadata import Metadata\n\n# Generate basic Metadata Table from dict\nmeta1 = Metadata(name="test", columns=[{"name": "c1", "type": "int64"}, {"name": "c2", "type": "string"}])\n\nprint(meta1.name) # test\nprint(meta1.columns[0]) #\xa0{"name": "c1", "type": "int64"}\nprint(meta1.description) # ""\n\n# Generate meta from dict\nd = {\n    "name": "test",\n    "columns": [\n        {"name": "c1", "type": "int64"},\n        {"name": "c2", "type": "string"}\n    ]\n}\nmeta2 = Metadata.from_dict(d)\n\n# Read / write to json\nmeta3 = Metadata.from_json("path/to/metadata_schema.json")\nmeta3.name = "new_table"\nmeta3.to_json("path/to/new_metadata_schema.json")\n```\n\n## Added Class methods and properties\n\nThe metadata class has some methods and properties that are not part of the schema but helps organise and manage the schema.\n\n### Column Methods\n\nThe class has multiple methods to alter the columns list.\n\n- `column_names`: Get a list of column names\n- `update_column`: If column with name matches replace it otherwise add it to the end\n- `remove_column`: Remove column that matches the given name. Note if a name in the `partitions` property matches that name then it is also removed.\n\n```python\n    meta = Metadata(columns=[\n        {"name": "a", "type": "int8"},\n        {"name": "b", "type": "string"},\n        {"name": "c", "type": "date32"},\n    ])\n    meta.column_names # ["a", "b", "c"]\n \n    meta.update_column({"name": "a", "type": "int64"})\n    meta.columns[0]["type"] # "int64"\n\n    meta.update_column({"name": "d", "type": "string"})\n    meta.column_names # ["a", "b", "c", "d"]\n\n    meta.remove_column("d")\n    assert meta.column_names == ["a", "b", "c"]\n```\n\nThe metadata class is a subclass of MutableMappings, where keys are column names and values are column metadata. \n\n- A metadata column can be accessed using the column name as a key.\n- A new or existing column can be updated using the column name as a key. The key must match the column name. Calls update_column.\n- A column can be deleted. Calls remove_column.\n- Columns of metadata can be iterated over.\n- The length of metadata is defined as the number of columns.\n\n```python\n# Access a specific column\nmeta["c1"] # {"name": "c1", "type": "int64"}\n\n# Add a new column (key must match name)\nmeta["c3"] = {"name": "c3", "type": "bool"}\n\n# Delete a column\ndel meta["c3"]\n\n# Iterate over all columns\nfor col in meta:\n    print(f"column name:{col["name"]}, column type:{col["type"]}")\n\n# Get the number of columns\nlen(meta) # 3\n```\n\n### force_partition_order Property\n\nBy default this is set to None. However can be set to `"start"` or `"end"`. When set to None the Metadata Class will not track column order relative to partitions.\n\n> Note: For Athena we normally set partitions at the end.\n\n```python\nmeta = Metadata(columns=[\n        {"name": "a", "type": "int8"},\n        {"name": "b", "type": "string"},\n        {"name": "c", "type": "date32"},\n    ])\n\nmeta.partitions = ["b"]\nmeta.column_names # ["a", "b", "c"]\n```\n\nIf set to `"start"` or `"end"` then any changes to partitions will affect the column order.\n\n```python\nmeta.force_partition_order = "start"\nmeta.column_names # ["b", "a" ,"c"]\n```\n\n### Generating Metdata objects\n\n<hr>\n\n# Converters\n\nConverters takes a Metadata object and generates something else from it (or can convert something to a Metadata object). Most of the time your converter will convert our schema into another systems schema. \n\n# Usage\n\nFor example the `ArrowConverter` takes our schemas and converts them to a pyarrow schema:\n\n```python\nfrom mojap_metadata import Metadata\nfrom mojap_metadata.converters.arrow_converter import ArrowConverter\n\nd = {\n    "name": "test",\n    "columns": [\n        {"name": "c1", "type": "int64"},\n        {"name": "c2", "type": "string"},\n        {"name": "c3", "type": "struct<k1: string, k2:list<int64>>"}\n    ],\n    "file_format": "jsonl"\n}\nmeta = Metadata.from_dict(d)\n\nac = ArrowConverter()\narrow_schema = ac.generate_from_meta(meta)\n\nprint(arrow_schema) # Could use this schema to read in data as arrow dataframe and cast it to the correct types\n```\n\nAnother use for the arrow converter is to convert it back from an Arrow schema to our metadata. This is especially useful if you have nested data types that would be difficult to write out the full `STRUCT` / `LIST`. Instead you can let Arrow do that for you and then pass the agnostic metadata object into something like the Glue Converter to generate a schema for AWS Glue.\n\n```python\nimport pyarrow as pa\nimport pandas as pd\n\nfrom mojap_metadata.converters.arrow_converter import ArrowConverter\n\ndata = {\n    "a": [0,1],\n    "b": [\n        {"cat": {"meow": True}, "dog": ["bone", "fish"]},\n        {"cat": {"meow": True}, "dog": ["bone", "fish"]},\n    ]\n}\ndf = pd.DataFrame(data)\narrow_df = pa.Table.from_pandas(df)\nac = ArrowConverter()\nmeta = ac.generate_to_meta(arrow_df.schema)\n\nprint(meta.columns) # [{\'name\': \'a\', \'type\': \'int64\'}, {\'name\': \'b\', \'type\': \'struct<cat:struct<meow:bool>, dog:list<string>>\'}]\n```\n\nAnother example is the `GlueConverter` which takes our schemas and converts them to a dictionary that be passed to the glue_client to deploy a schema on AWS Glue.\n\n```python\nimport boto3\nfrom mojap_metadata import Metadata\nfrom mojap_metadata.converters.glue_converter import GlueConverter\n\nd = {\n    "name": "test",\n    "columns": [\n        {"name": "c1", "type": "int64"},\n        {"name": "c2", "type": "string"},\n        {"name": "c3", "type": "struct<k1: string, k2:list<int64>>"}\n    ],\n    "file_format": "jsonl"\n}\nmeta = Metadata.from_dict(d)\n\ngc = GlueConverter()\nboto_dict = gc.generate_from_meta(meta, )\nboto_dict = gc.generate_from_meta(meta, database_name="test_db", table_location="s3://bucket/test_db/test/")\n\nprint(boto_dict) \n\nglue_client = boto3.client("glue")\nglue_client.create_table(**boto_dict) # Would deploy glue schema based on our metadata\n```\n\nAll converter classes are sub classes of the `mojap_metadata.converters.BaseConverter`. This `BaseConverter` has no actual functionality but is a boilerplate class that ensures standardised attributes  for all added `Converters` these are:\n\n- **generate_from_meta:** (function) takes a Metadata object and returns whatever the converter is producing .\n\n- **generate_to_meta:** (function) takes Any object (normally another schema for another system or package) and returns our Metadata object. (i.e. the reverse of generate_from_meta).\n\n- **options:** (Data Class) that are the options for the converter. The base options have a `suppress_warnings` parameter but it doesn\'t mean call converters use this. To get a better understanding of setting options see the `GlueConverter` class or the `tests/test_glue_converter.py` to see how they are set.\n\nincluded alongside `GlueConverter` is `GlueTable` that can overlay a metadata object, dictionary, or path to metadata file. it has one method:\n- **generate_from_meta:** generates a glue table from the metadata object, dict, or string path, takes the following arguments:\n    - _metadata:_ the metadata object, dict, or string path that is to be overlaid\n    - _table\\_location:_ a kwarg, the location of the table data. This can also be a property of the metadata object, dict, or file\n    - _database\\_name:_ a kwarg, the name of the glue database to put the table. This can also be a property of the metadata object, dict, or file\n\n\n##\xa0Further Usage\n\nSee the [mojap-aws-tools repo](https://github.com/moj-analytical-services/mojap-aws-tools-demo) which utilises the converters a lot in different tutorials.\n\n## Contributing and Design Considerations\n\nEach new converter (if not expanding on existing converters) should be added as a new submodule within the parent `converters` module. This is especially true if the new converter has additional package dependencies. By design the standard install of this package is fairly lightweight. However if you needed the `ArrowConverter` you would need to install the additional package dependencies for the arrow converter:\n\n```bash\npip install \'mojap-metadata[arrow] @ git+https://github.com/moj-analytical-services/mojap-metadata\'\n```\n\nThis means we can continuely add converters (as submodules) and add optional package dependencies ([see pyproject.toml](./pyproject.toml) ) without making the default install any less lightweight. `mojap_metadata` would only error if someone tries to import a converter subclass that with having the additional dependencies dependencies installed).\n\n## Postgres Converter\n\nPostgres Converter provides the following functionality\n\n1. Conenction to postgres database\n2. Extract the metadata from the tables\n3. Convert the extracted ouptut into Metadata object \n\n- **get_object_meta** (function) takes the table name, schema name then the extracts the metadata from postgres database and \nconverts into Metadata object \n\n- **generate_to_meta:** (function) takes the database connection and returns a list of Metadata object for all the (non-system schemas) schemas and tables from the connection.\n',
    'author': 'MoJ Data Engineering',
    'author_email': 'dataengineering@digital.justice.gov.uk',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'python_requires': '>=3.8,<3.11',
}


setup(**setup_kwargs)
