{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrate Data Validation Into Your Pipeline\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prep environment and logging\n",
    "\n",
    "import json\n",
    "import os\n",
    "import logging\n",
    "import great_expectations as ge\n",
    "import great_expectations.jupyter_ux\n",
    "import pandas as pd\n",
    "import uuid # used to generate run_id\n",
    "from datetime import datetime\n",
    "\n",
    "import tzlocal\n",
    "\n",
    "great_expectations.jupyter_ux.setup_notebook_logging()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integrate data validation into your pipeline\n",
    "\n",
    "[**Watch a short tutorial video**](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#video)\n",
    "\n",
    "\n",
    "[**Read more in the tutorial**](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation)\n",
    "\n",
    "**Reach out for help on** [**Great Expectations Slack**](https://greatexpectations.io/slack)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a DataContext object\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a pipeline run id\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#set-a-run-id)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2019-07-12T191931.543981Z'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a run-id that GE will use to key shared parameters\n",
    "run_id = datetime.utcnow().isoformat().replace(\":\", \"\") + \"Z\"\n",
    "run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose data asset name and expectation suite name\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#choose-data-asset-and-expectation-suite)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context.list_expectation_suite_keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_asset_name = \"REPLACE ME!\" # TODO: replace with your value!\n",
    "expectation_suite_name = \"my_suite\" # TODO: replace with your value!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtain the batch to validate\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#obtain-a-batch-to-validate)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### If your pipeline processes Pandas Dataframes:\n",
    "\n",
    "```\n",
    "df = pd.read_csv(file_path_to_validate)\n",
    "df.head()\n",
    "batch = context.get_batch(data_asset_name, expectation_suite_name, df)\n",
    "```\n",
    "\n",
    "##### If your pipeline processes Spark Dataframes:\n",
    "```\n",
    "from pyspark.sql import SparkSession\n",
    "from great_expectations.dataset import PandasDataset, SqlAlchemyDataset, SparkDFDataset\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "df = SparkDFDataset(spark.read.csv(file_path_to_validate))\n",
    "df.spark_df.show()\n",
    "batch = context.get_batch(data_asset_name, expectation_suite_name, df)\n",
    "```\n",
    "\n",
    "##### If your pipeline processes SQL querues:\n",
    "```\n",
    "batch = context.get_batch(data_asset_name, expectation_suite_name, query=\"SELECT * from ....\") # the query whose result set you want to validate\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the batch\n",
    "\n",
    "This is the \"workhorse\" method of Great Expectations. Call it in your pipeline code after loading the file and just before passing it to your computation.\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#validate)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_result = batch.validate(run_id=run_id)\n",
    "\n",
    "if validation_result[\"success\"]:\n",
    "    print(\"This file meets all expectations from a valid batch of {0:s}\".format(data_asset_name))\n",
    "else:\n",
    "    print(\"This file is not a valid batch of {0:s}\".format(data_asset_name))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the validation results\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#review-validation-results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(validation_result, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finishing touches - notifications and saving validation results and validated batches\n",
    "\n",
    "#### Notifications\n",
    "You want to be notified when the pipeline validated a batch, especially when the validation failed.\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#send-notifications)\n",
    "\n",
    "#### Saving validation results\n",
    "\n",
    "To enable the storing of validation results, uncomment the `result_store` section in your great_expectations.yml file. \n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#save-validation-results)\n",
    "\n",
    "#### Saving failed batches\n",
    "\n",
    "When a batch fails validation (it does not pass all the expectations of the data asset), it is useful to save the batch along with the validation results for future review. You can enable this option in your project's great_expectations.yml file. \n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#save-failed-batches)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
