{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrate Data Validation Into Your Pipeline\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import great_expectations as ge\n",
    "import great_expectations.jupyter_ux\n",
    "from great_expectations.datasource.types import BatchKwargs\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integrate data validation into your pipeline\n",
    "\n",
    "[**Watch a short tutorial video**](https://greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#video)\n",
    "\n",
    "\n",
    "[**Read more in the tutorial**](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation)\n",
    "\n",
    "**Reach out for help on** [**Great Expectations Slack**](https://greatexpectations.io/slack)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a DataContext object\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a pipeline run id\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#set-a-run-id)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a run id - a pipeline run id, a timestamp or any other string that is meaningful to you \n",
    "# and will help you refer to validation results. We recommend they be chronologically sortable.\n",
    "run_id = datetime.utcnow().isoformat().replace(\":\", \"\") + \"Z\"\n",
    "run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose data asset name and expectation suite name\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#choose-data-asset-and-expectation-suite)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "great_expectations.jupyter_ux.list_available_data_asset_names(context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_asset_name = \"REPLACE ME!\" # TODO: replace with your value!\n",
    "expectation_suite_name = \"warning\" # TODO: replace with your value!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtain the batch to validate\n",
    "\n",
    "Learn about `get_batch` in [this tutorial]](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#obtain-a-batch-to-validate)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### If your pipeline processes Pandas Dataframes:\n",
    "\n",
    "```\n",
    "import pandas as pd\n",
    "df = pd.read_csv(file_path_to_validate)\n",
    "batch = context.get_batch(data_asset_name, expectation_suite_name, BatchKwargs(df=df))\n",
    "batch.head()\n",
    "```\n",
    "\n",
    "##### If your pipeline processes Spark Dataframes:\n",
    "```\n",
    "from pyspark.sql import SparkSession\n",
    "from great_expectations.dataset import PandasDataset, SqlAlchemyDataset, SparkDFDataset\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "df = SparkDFDataset(spark.read.csv(file_path_to_validate))\n",
    "batch = context.get_batch(data_asset_name, expectation_suite_name, BatchKwargs(df=df))\n",
    "batch.head()\n",
    "```\n",
    "\n",
    "##### If your pipeline processes SQL querues:\n",
    "\n",
    "* A. To validate an existing table:\n",
    "\n",
    "```\n",
    "data_asset_name = 'USE THE TABLE NAME'\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name=expectation_suite_name,\n",
    "                        BatchKwargs(table=data_asset_name)) \n",
    "batch.head()\n",
    "```\n",
    "\n",
    "* B. To validate a query result set:\n",
    "\n",
    "```\n",
    "data_asset_name = 'USE THE NAME YOU SPECIFIED WHEN YOU CREATED THE EXPECTATION SUITE FOR THIS QUERY'\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name=expectation_suite_name,\n",
    "                        BatchKwargs(query='SQL FOR YOUR QUERY'))\n",
    "batch.head()\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = context.get_batch(COPY THE APPROPRIATE CODE SNIPPET FROM THE CELL ABOVE)\n",
    "batch.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the batch\n",
    "\n",
    "This is the \"workhorse\" method of Great Expectations. Call it in your pipeline code after loading the file and just before passing it to your computation.\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#validate)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_result = batch.validate(run_id=run_id)\n",
    "\n",
    "if validation_result[\"success\"]:\n",
    "    print(\"This file meets all expectations from a valid batch of {0:s}\".format(data_asset_name))\n",
    "else:\n",
    "    print(\"This file is not a valid batch of {0:s}\".format(data_asset_name))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the validation results\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#review-validation-results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(validation_result, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation Operators\n",
    "\n",
    "The `validate` method evaluates one batch of data against one expectation suite and returns a dictionary of validation results. This is sufficient when you explore your data and get to know Great Expectations.\n",
    "When deploying Great Expectations in a real data pipeline, you will typically discover additional needs:\n",
    "\n",
    "* validating a group of batches that are logically related\n",
    "* validating a batch against several expectation suites\n",
    "* doing something with the validation results (e.g., saving them for a later review, sending notifications in case of failures, etc.).\n",
    "\n",
    "Validation Operators provide a convenient abstraction for both bundling the validation of multiple expectation suites and the actions that should be taken after the validation.\n",
    "\n",
    "[Read more about Validation Operators](https://docs.greatexpectations.io/en/latest/features/validation_operators_and_actions.html?utm_source=notebook&utm_medium=integrate_validation)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is an example of invoking a validation operator that is configured by default in the great_expectations.yml file\n",
    "\n",
    "results = context.run_validation_operator(\n",
    "    assets_to_validate=[batch],\n",
    "    run_id=run_id,\n",
    "    validation_operator_name=\"action_list_operator\",\n",
    ")\n",
    "\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
