{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrate Data Validation Into Your Pipeline\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prep environment and logging\n",
    "\n",
    "import json\n",
    "import great_expectations as ge\n",
    "import great_expectations.jupyter_ux\n",
    "from great_expectations.datasource.types import BatchKwargs\n",
    "from datetime import datetime\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Watch** a [short tutorial video](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#video) or **read** [the written tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation)\n",
    "\n",
    "We'd love it if you **reach out for help on** the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get a DataContext\n",
    "This represents your project that you set up using `great_expectations init`. [Read more in the tutorial](https://great-expectations.readthedocs.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-datacontext-object)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Get a pipeline run id\n",
    "\n",
    "Generate a run id, a timestamp, or any other string that is meaningful to you that will help you refer to validation results. We recommend they be chronologically sortable.\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#set-a-run-id)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's make a simple sortable timestamp. Note this could come from your pipeline runner.\n",
    "run_id = datetime.utcnow().isoformat().replace(\":\", \"\") + \"Z\"\n",
    "run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Pick a data asset & expectation suite name\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#choose-data-asset-and-expectation-suite)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "great_expectations.jupyter_ux.list_available_data_asset_names(context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_asset_name = \"REPLACE ME!\" # TODO: replace with your value!\n",
    "expectation_suite_name = \"warning\" # TODO: replace with your value!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Load a batch of data from the data asset you want to validate\n",
    "\n",
    "Learn about `get_batch` in [this tutorial]](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#obtain-a-batch-to-validate)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### If your pipeline processes Pandas Dataframes:\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(file_path_to_validate)\n",
    "batch = context.get_batch(data_asset_name, expectation_suite_name, BatchKwargs(df=df))\n",
    "batch.head()\n",
    "```\n",
    "\n",
    "##### If your pipeline processes Spark Dataframes:\n",
    "```python\n",
    "from pyspark.sql import SparkSession\n",
    "from great_expectations.dataset import PandasDataset, SqlAlchemyDataset, SparkDFDataset\n",
    "\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "df = SparkDFDataset(spark.read.csv(file_path_to_validate))\n",
    "batch = context.get_batch(data_asset_name, expectation_suite_name, BatchKwargs(df=df))\n",
    "batch.head()\n",
    "```\n",
    "\n",
    "##### If your pipeline processes SQL queries:\n",
    "\n",
    "* A. To validate an existing table:\n",
    "\n",
    "```python\n",
    "data_asset_name = 'USE THE TABLE NAME'\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name=expectation_suite_name,\n",
    "                        BatchKwargs(table=data_asset_name)) \n",
    "batch.head()\n",
    "```\n",
    "\n",
    "* B. To validate a query result set:\n",
    "\n",
    "```python\n",
    "data_asset_name = 'USE THE NAME YOU SPECIFIED WHEN YOU CREATED THE EXPECTATION SUITE FOR THIS QUERY'\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name=expectation_suite_name,\n",
    "                        BatchKwargs(query='SQL FOR YOUR QUERY'))\n",
    "batch.head()\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = context.get_batch(COPY THE APPROPRIATE CODE SNIPPET FROM THE CELL ABOVE)\n",
    "batch.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Validate the batch\n",
    "\n",
    "This is the \"workhorse\" of Great Expectations. Call it in your pipeline code after loading the file and just before passing it to your computation.\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#validate)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_result = batch.validate(run_id=run_id)\n",
    "\n",
    "if validation_result[\"success\"]:\n",
    "    print(\"This file meets all expectations from a valid batch of {0:s}\".format(data_asset_name))\n",
    "else:\n",
    "    print(\"This file is not a valid batch of {0:s}\".format(data_asset_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Review the validation results\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#review-validation-results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(validation_result, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Validation Operators\n",
    "\n",
    "The `validate` method evaluates one batch of data against one expectation suite and returns a dictionary of validation results. This is sufficient when you explore your data and get to know Great Expectations.\n",
    "When deploying Great Expectations in a **real data pipeline, you will typically discover additional needs**:\n",
    "\n",
    "* validating a group of batches that are logically related\n",
    "* validating a batch against several expectation suites such as using a tiered pattern like `warning` and `failure`\n",
    "* doing something with the validation results (e.g., saving them for a later review, sending notifications in case of failures, etc.).\n",
    "\n",
    "`Validation Operators` provide a convenient abstraction for both bundling the validation of multiple expectation suites and the actions that should be taken after the validation.\n",
    "\n",
    "[Read more about Validation Operators](https://docs.greatexpectations.io/en/latest/features/validation_operators_and_actions.html?utm_source=notebook&utm_medium=integrate_validation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is an example of invoking a validation operator that is configured by default in the great_expectations.yml file\n",
    "\n",
    "results = context.run_validation_operator(\n",
    "    assets_to_validate=[batch],\n",
    "    run_id=run_id,\n",
    "    validation_operator_name=\"action_list_operator\",\n",
    ")\n",
    "\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations! You used your Expectations to validate one of your data assets!\n",
    "\n",
    "## Next steps:\n",
    "\n",
    "### 1. Data Docs\n",
    "Jump back to the command line and run `great_expectations build-docs` to see your Data Docs. These will now include a **data quality report** built from the validations that helps you understand and communicate about your data.\n",
    "### 2. Explore the documentation & community\n",
    "You are now among the elite data professionals who know how to build robust protection for pipelines and machine learning models. Join the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack) to see how others are wielding these superpowers."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
