{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validation Playground\n",
    "\n",
    "**Watch** a [short tutorial video](https://greatexpectations.io/videos/getting_started/integrate_expectations) or **read** [the written tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data)\n",
    "\n",
    "We'd love it if you **reach out for help on** the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import great_expectations as ge\n",
    "from great_expectations.profile import ColumnsExistProfiler\n",
    "import great_expectations.jupyter_ux\n",
    "from great_expectations.datasource.types import BatchKwargs\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get a DataContext\n",
    "This represents your **project** that you just created using `great_expectations init`. [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#get-a-datacontext-object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. List the CSVs in your folder\n",
    "\n",
    "The `DataContext` will now introspect your pyspark `Datasource` and list the CSVs it finds. [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#list-data-assets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ge.jupyter_ux.list_available_data_asset_names(context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Pick a csv and the expectation suite\n",
    "\n",
    "Internally, Great Expectations represents csvs and dataframes as `DataAsset`s and uses this notion to link them to `Expectation Suites`. [Read more about the validate method in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#pick-a-data-asset-and-expectation-suite)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_asset_name = \"ONE_OF_THE_CSV_DATA_ASSET_NAMES_FROM_ABOVE\" # TODO: replace with your value!\n",
    "normalized_data_asset_name = context.normalize_data_asset_name(data_asset_name)\n",
    "normalized_data_asset_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We recommend naming your first expectation suite for a table `warning`. Later, as you identify some of the expectations that you add to this suite as critical, you can move these expectations into another suite and call it `failure`. [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#choose-data-asset-and-expectation-suite)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expectation_suite_name = \"warning\" # TODO: replace with your value!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.a. If you don't have an expectation suite, let's create a simple one\n",
    "\n",
    "You need expectations to validate your data. Expectations are grouped into Expectation Suites. \n",
    "\n",
    "If you don't have an expectation suite for this data asset, the notebook's next cell will create a suite of very basic expectations, so that you have some expectations to play with. The expectation suite will have `expect_column_to_exist` expectations for each column.\n",
    "\n",
    "If you created an expectation suite for this data asset, you can skip executing the next cell (if you execute it, it will do nothing).\n",
    "\n",
    "To create a more interesting suite, open the [create_expectations.ipynb](create_expectations.ipynb) notebook.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    context.get_expectation_suite(normalized_data_asset_name, expectation_suite_name)\n",
    "except great_expectations.exceptions.DataContextError:\n",
    "    context.create_expectation_suite(data_asset_name=normalized_data_asset_name, expectation_suite_name=expectation_suite_name, overwrite_existing=True);\n",
    "    batch_kwargs = context.yield_batch_kwargs(data_asset_name)\n",
    "    batch = context.get_batch(normalized_data_asset_name, expectation_suite_name, batch_kwargs)\n",
    "    ColumnsExistProfiler().profile(batch)\n",
    "    batch.save_expectation_suite()\n",
    "    expectation_suite = context.get_expectation_suite(normalized_data_asset_name, expectation_suite_name)\n",
    "    context.build_data_docs()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Load a batch of data you want to validate\n",
    "\n",
    "To learn more about `get_batch` with other data types (such as existing pandas dataframes, SQL tables or Spark), see [this tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#load-a-batch-of-data-to-validate)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_kwargs = context.yield_batch_kwargs(data_asset_name)\n",
    "batch = context.get_batch(normalized_data_asset_name, expectation_suite_name, batch_kwargs)\n",
    "batch.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Get a pipeline run id\n",
    "\n",
    "Generate a run id, a timestamp, or a meaningful string that will help you refer to validation results. We recommend they be chronologically sortable.\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=validate_data#set-a-run-id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's make a simple sortable timestamp. Note this could come from your pipeline runner.\n",
    "run_id = datetime.utcnow().isoformat().replace(\":\", \"\") + \"Z\"\n",
    "run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Validate the batch\n",
    "\n",
    "This is the \"workhorse\" of Great Expectations. Call it in your pipeline code after loading data and just before passing it to your computation.\n",
    "\n",
    "[Read more about the validate method in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#validate-the-batch)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_result = batch.validate(run_id=run_id)\n",
    "\n",
    "if validation_result[\"success\"]:\n",
    "    print(\"This data meets all expectations for {}\".format(str(data_asset_name)))\n",
    "else:\n",
    "    print(\"This data is not a valid batch of {}\".format(str(data_asset_name)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.a. OPTIONAL: Review the JSON validation results\n",
    "\n",
    "Don't worry - this blob of JSON is meant for machines. Continue on or skip this to see this in Data Docs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(json.dumps(validation_result, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Validation Operators\n",
    "\n",
    "The `validate` method evaluates one batch of data against one expectation suite and returns a dictionary of validation results. This is sufficient when you explore your data and get to know Great Expectations.\n",
    "When deploying Great Expectations in a **real data pipeline, you will typically discover additional needs**:\n",
    "\n",
    "* validating a group of batches that are logically related\n",
    "* validating a batch against several expectation suites such as using a tiered pattern like `warning` and `failure`\n",
    "* doing something with the validation results (e.g., saving them for a later review, sending notifications in case of failures, etc.).\n",
    "\n",
    "`Validation Operators` provide a convenient abstraction for both bundling the validation of multiple expectation suites and the actions that should be taken after the validation.\n",
    "\n",
    "[Read more about Validation Operators in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#save-validation-results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is an example of invoking a validation operator that is configured by default in the great_expectations.yml file\n",
    "\n",
    "results = context.run_validation_operator(\n",
    "    assets_to_validate=[batch],\n",
    "    run_id=run_id,\n",
    "    validation_operator_name=\"action_list_operator\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. View the Validation Results in Data Docs\n",
    "\n",
    "Let's now build and look at your Data Docs. These will now include an **data quality report** built from the `ValidationResults` you just created that helps you communicate about your data with both machines and humans.\n",
    "\n",
    "[Read more about Data Docs in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#view-the-validation-results-in-data-docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context.open_data_docs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations! You ran Validations!\n",
    "\n",
    "## Next steps:\n",
    "\n",
    "### 1. Author more interesting Expectations\n",
    "\n",
    "Here we used some **extremely basic** `Expectations`. To really harness the power of Great Expectations you can author much more interesting and specific `Expectations` to protect your data pipelines and defeat pipeline debt. Go to [create_expectations.ipynb](create_expectations.ipynb) to see how!\n",
    "\n",
    "### 2. Explore the documentation & community\n",
    "\n",
    "You are now among the elite data professionals who know how to build robust descriptions of your data and protections for pipelines and machine learning models. Join the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack) to see how others are wielding these superpowers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
