# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['pydeequ']

package_data = \
{'': ['*']}

install_requires = \
['numpy>=1.14.1', 'pandas>=0.23.0']

extras_require = \
{'pyspark': ['pyspark>=2.4.7,<3.1.1']}

setup_kwargs = {
    'name': 'pydeequ',
    'version': '0.1.8',
    'description': 'PyDeequ - Unit Tests for Data',
    'long_description': '# PyDeequ\n\nPyDeequ is a Python API for [Deequ](https://github.com/awslabs/deequ), a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets. PyDeequ is written to support usage of Deequ in Python .\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) ![Coverage](https://img.shields.io/badge/coverage-90%25-green)\n\nThere are 4 main components of Deequ, and they are:\n- Metrics Computation:\n    - `Profiles` leverages Analyzers to analyze each column of a dataset.\n    - `Analyzers` serve here as a foundational module that computes metrics for data profiling and validation at scale.\n- Constraint Suggestion:\n    - Specify rules for various groups of Analyzers to be run over a dataset to return back a collection of constraints suggested to run in a Verification Suite.\n- Constraint Verification:\n    - Perform data validation on a dataset with respect to various constraints set by you.   \n- Metrics Repository\n    - Allows for persistence and tracking of Deequ runs over time.\n\n![](imgs/pydeequ_architecture.jpg)\n\n## 🎉 Announcements 🎉\n- We\'ve release a blogpost on integrating PyDeequ onto AWS leveraging services such as AWS Glue, Athena, and SageMaker! Check it out: [Monitor data quality in your data lake using PyDeequ and AWS Glue](https://aws.amazon.com/blogs/big-data/monitor-data-quality-in-your-data-lake-using-pydeequ-and-aws-glue/).\n- Check out the [PyDeequ Release Announcement Blogpost](https://aws.amazon.com/blogs/big-data/testing-data-quality-at-scale-with-pydeequ/) with a tutorial walkthrough the Amazon Reviews dataset!\n- Join the PyDeequ community on [PyDeequ Slack](https://join.slack.com/t/pydeequ/shared_invite/zt-qopmmfgm-ajKRyxx0HgCiK50b9JhAFg) to chat with the devs!\n\n## Quickstart\n\nThe following will quickstart you with some basic usage. For more in-depth examples, take a look in the [`tutorials/`](tutorials/) directory for executable Jupyter notebooks of each module. For documentation on supported interfaces, view the [`documentation`](https://pydeequ.readthedocs.io/).\n\n### Installation\n\nYou can install [PyDeequ via pip](https://pypi.org/project/pydeequ/).\n\n```\npip install pydeequ\n```\n\n### Set up a PySpark session\n```python\nfrom pyspark.sql import SparkSession, Row\nimport pydeequ\n\nspark = (SparkSession\n    .builder\n    .config("spark.jars.packages", pydeequ.deequ_maven_coord)\n    .config("spark.jars.excludes", pydeequ.f2j_maven_coord)\n    .getOrCreate())\n\ndf = spark.sparkContext.parallelize([\n            Row(a="foo", b=1, c=5),\n            Row(a="bar", b=2, c=6),\n            Row(a="baz", b=3, c=None)]).toDF()\n```\n\n### Analyzers\n\n```python\nfrom pydeequ.analyzers import *\n\nanalysisResult = AnalysisRunner(spark) \\\n                    .onData(df) \\\n                    .addAnalyzer(Size()) \\\n                    .addAnalyzer(Completeness("b")) \\\n                    .run()\n\nanalysisResult_df = AnalyzerContext.successMetricsAsDataFrame(spark, analysisResult)\nanalysisResult_df.show()\n```\n\n### Profile\n\n```python\nfrom pydeequ.profiles import *\n\nresult = ColumnProfilerRunner(spark) \\\n    .onData(df) \\\n    .run()\n\nfor col, profile in result.profiles.items():\n    print(profile)\n```\n\n### Constraint Suggestions\n\n```python\nfrom pydeequ.suggestions import *\n\nsuggestionResult = ConstraintSuggestionRunner(spark) \\\n             .onData(df) \\\n             .addConstraintRule(DEFAULT()) \\\n             .run()\n\n# Constraint Suggestions in JSON format\nprint(suggestionResult)\n```\n\n### Constraint Verification\n\n```python\nfrom pydeequ.checks import *\nfrom pydeequ.verification import *\n\ncheck = Check(spark, CheckLevel.Warning, "Review Check")\n\ncheckResult = VerificationSuite(spark) \\\n    .onData(df) \\\n    .addCheck(\n        check.hasSize(lambda x: x >= 3) \\\n        .hasMin("b", lambda x: x == 0) \\\n        .isComplete("c")  \\\n        .isUnique("a")  \\\n        .isContainedIn("a", ["foo", "bar", "baz"]) \\\n        .isNonNegative("b")) \\\n    .run()\n\ncheckResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult)\ncheckResult_df.show()\n```\n\n### Repository\n\nSave to a Metrics Repository by adding the `useRepository()` and `saveOrAppendResult()` calls to your Analysis Runner.\n```python\nfrom pydeequ.repository import *\nfrom pydeequ.analyzers import *\n\nmetrics_file = FileSystemMetricsRepository.helper_metrics_file(spark, \'metrics.json\')\nrepository = FileSystemMetricsRepository(spark, metrics_file)\nkey_tags = {\'tag\': \'pydeequ hello world\'}\nresultKey = ResultKey(spark, ResultKey.current_milli_time(), key_tags)\n\nanalysisResult = AnalysisRunner(spark) \\\n    .onData(df) \\\n    .addAnalyzer(ApproxCountDistinct(\'b\')) \\\n    .useRepository(repository) \\\n    .saveOrAppendResult(resultKey) \\\n    .run()\n```\n\nTo load previous runs, use the `repository` object to load previous results back in.\n\n```python\nresult_metrep_df = repository.load() \\\n    .before(ResultKey.current_milli_time()) \\\n    .forAnalyzers([ApproxCountDistinct(\'b\')]) \\\n    .getSuccessMetricsAsDataFrame()\n```\n\n## [Contributing](https://github.com/awslabs/python-deequ/blob/master/CONTRIBUTING.md)\nPlease refer to the [contributing doc](https://github.com/awslabs/python-deequ/blob/master/CONTRIBUTING.md) for how to contribute to PyDeequ.\n\n## [License](https://github.com/awslabs/python-deequ/blob/master/LICENSE)\n\nThis library is licensed under the Apache 2.0 License.\n\n******\n\n## Contributing Developer Setup\n\n1. Setup [SDKMAN](#setup-sdkman)\n1. Setup [Java](#setup-java)\n1. Setup [Apache Spark](#setup-apache-spark)\n1. Install [Poetry](#poetry)\n1. Run [tests locally](#running-tests-locally)\n\n### Setup SDKMAN\n\nSDKMAN is a tool for managing parallel Versions of multiple Software Development Kits on any Unix based\nsystem. It provides a convenient command line interface for installing, switching, removing and listing\nCandidates. SDKMAN! installs smoothly on Mac OSX, Linux, WSL, Cygwin, etc... Support Bash and ZSH shells. See\ndocumentation on the [SDKMAN! website](https://sdkman.io).\n\nOpen your favourite terminal and enter the following:\n\n```bash\n$ curl -s https://get.sdkman.io | bash\nIf the environment needs tweaking for SDKMAN to be installed,\nthe installer will prompt you accordingly and ask you to restart.\n\nNext, open a new terminal or enter:\n\n$ source "$HOME/.sdkman/bin/sdkman-init.sh"\n\nLastly, run the following code snippet to ensure that installation succeeded:\n\n$ sdk version\n```\n\n### Setup Java\n\nInstall Java Now open favourite terminal and enter the following:\n\n```bash\nList the AdoptOpenJDK OpenJDK versions\n$ sdk list java\n\nTo install For Java 11\n$ sdk install java 11.0.10.hs-adpt\n\nTo install For Java 11\n$ sdk install java 8.0.292.hs-adpt\n```\n\n### Setup Apache Spark\n\nInstall Java Now open favourite terminal and enter the following:\n\n```bash\nList the Apache Spark versions:\n$ sdk list spark\n\nTo install For Spark 3\n$ sdk install spark 3.0.2\n```\n\n### Poetry\n\nPoetry [Commands](https://python-poetry.org/docs/cli/#search)\n\n```bash\npoetry install\n\npoetry update\n\n# --tree: List the dependencies as a tree.\n# --latest (-l): Show the latest version.\n# --outdated (-o): Show the latest version but only for packages that are outdated.\npoetry show -o\n```\n\n## Running Tests Locally\n\nTake a look at tests in `tests/dataquality` and `tests/jobs`\n\n```bash\n$ poetry run pytest\n```',
    'author': 'Calvin Wang',
    'author_email': 'calviwan@amazon.com',
    'maintainer': 'Calvin Wang',
    'maintainer_email': 'calviwan@amazon.com',
    'url': 'https://pydeequ.readthedocs.io',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'python_requires': '>=3.6.2,<4',
}


setup(**setup_kwargs)
