{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import great_expectations as ge\n",
    "import great_expectations.jupyter_ux\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Author Expectations\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[**Watch a short tutorial video**](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#video)\n",
    "\n",
    "[**Read more in the tutorial**](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations)\n",
    "\n",
    "**Reach out for help on** [**Great Expectations Slack**](https://tinyurl.com/great-expectations-slack)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a DataContext object\n",
    "[Read more in the tutorial](https://great-expectations.readthedocs.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-datacontext-object)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List data assets in your project\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#data-assets)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "great_expectations.jupyter_ux.list_available_data_asset_names(context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load a batch of data from the data asset you want to validate\n",
    "\n",
    "Learn about `get_batch` in [this tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-batch)\n",
    "\n",
    "__Quick Guide:__\n",
    "\n",
    "##### If you want to validate data in Pandas Dataframes or in Spark Dataframes:\n",
    "\n",
    "* A. If GE listed and profiled your files correctly:\n",
    "\n",
    "```\n",
    "data_asset_name = CHOOSE FROM THE LIST ABOVE\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                          expectation_suite_name)\n",
    "```\n",
    "* B. Otherwise (you want to control the logic of reading the data):\n",
    "\n",
    "```\n",
    "df = load the data into a dataframe, e.g., df = SparkDFDataset(spark.read.csv... or pd.read_csv(...\n",
    "data_asset_name = COME UP WITH A NAME - THIS WILL CREATE A NEW DATA ASSET.\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                          expectation_suite_name, \n",
    "                          df)\n",
    "```\n",
    "\n",
    "\n",
    "##### If you want to validate data in a database:\n",
    "\n",
    "* A. To validate an existing table:\n",
    "\n",
    "```\n",
    "data_asset_name = 'CHOOSE THE NAME OF YOUR TABLE FROM THE LIST OF DATA ASSETS ABOVE'\n",
    "df = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name='my_suite') \n",
    "```\n",
    "\n",
    "* B. To validate a query result set:\n",
    "\n",
    "```\n",
    "data_asset_name = 'NAME YOUR QUERY (E.G., daily_users_query) - THIS WILL CREATE A NEW DATA ASSET'\n",
    "df = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name='my_suite',\n",
    "                        query='SQL FOR YOUR QUERY')\n",
    "```\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = context.get_batch(COPY THE APPROPPRIATE CODE SNIPPET FROM THE CELL ABOVE\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optionally, customize options used to read your data (e.g., separators, header, etc) by setting reader options in `get_batch`\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#reader-options)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is how you can see which data batch was loaded\n",
    "df._batch_kwargs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Author Expectations\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#create-expectations)\n",
    "\n",
    "See available expectations in the [expectation glossary](https://docs.greatexpectations.io/en/latest/glossary.html?utm_source=notebook&utm_medium=create_expectations)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#example:\n",
    "\n",
    "column_name = df.get_table_columns()[0]\n",
    "df.expect_column_values_to_not_be_null(column_name)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add more expectations here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add more expectations here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add more expectations here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the expectations\n",
    "\n",
    "Expectations that were true on this data batch were added. To view all the expectations you added so far about this data asset, do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.get_expectation_suite()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    \n",
    "    \n",
    "If you decide not to save some expectations that you created, use [remove_expectaton method](https://docs.greatexpectations.io/en/latest/module_docs/data_asset_module.html?highlight=remove_expectation&utm_source=notebook&utm_medium=create_expectations#great_expectations.data_asset.data_asset.DataAsset.remove_expectation)\n",
    "\n",
    "\n",
    "The following call will save the expectation suite as a JSON file in great_expectations/expectations directory of your project:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.save_expectation_suite() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You created and saved expectations for at least one of the data assets.\n",
    "\n",
    "### We will show you how to set up validation - the process of checking if new files of this type conform to your expectations before they are processed by your pipeline's code. \n",
    "\n",
    "### Go to [integrate_validation_into_pipeline.ipynb](integrate_validation_into_pipeline.ipynb) to proceed.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
