{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import great_expectations as ge\n",
    "import great_expectations.jupyter_ux\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Author Expectations\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
     "Watch a [short tutorial video](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#video) or read [the written tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations)\n",
    "\n",
    "We'd love it if you **reach out for help on** the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get a DataContext.\n",
    "This represents your project that you set up using `great_expectations init`. [Read more in the tutorial](https://great-expectations.readthedocs.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-datacontext-object)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. List data assets in your project\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#data-assets)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "great_expectations.jupyter_ux.list_available_data_asset_names(context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Pick a data asset & set the expectation suite name\n",
    "We recommend you name your first expectation suite for a given data asset `warning`. Later, as you identify some of the expectations that you add to this suite as critical, you can move these expectations into another suite and call it `failure`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_asset_name = \"REPLACE ME!\" # TODO: replace with your value!\n",
    "expectation_suite_name = \"warning\" # TODO: replace with your value!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create the new expectation suite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context.create_expectation_suite(data_asset_name=data_asset_name, expectation_suite_name=expectation_suite_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Load a batch of data from the data asset you want to validate\n",
    "\n",
    "Learn about `get_batch` in [this tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-batch)\n",
    "\n",
    "__Quick Guide:__\n",
    "\n",
    "##### If you want to validate data in Pandas Dataframes or in Spark Dataframes:\n",
    "\n",
    "* A. If GE listed and profiled your files correctly:\n",
    "\n",
    "```python\n",
    "data_asset_name = CHOOSE FROM THE LIST ABOVE\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                          expectation_suite_name,\n",
    "                          context.yield_batch_kwargs(data_asset_name))\n",
    "```\n",
    "* B. Otherwise (you want to control the logic of reading the data):\n",
    "\n",
    "```python\n",
    "df = load the data into a dataframe, e.g., df = SparkDFDataset(spark.read.csv... or pd.read_csv(...\n",
    "data_asset_name = COME UP WITH A NAME - THIS WILL CREATE A NEW DATA ASSET.\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                          expectation_suite_name, \n",
    "                          df)\n",
    "```\n",
    "\n",
    "\n",
    "##### If you want to validate data in a database:\n",
    "\n",
    "* A. To validate an existing table:\n",
    "\n",
    "```python\n",
    "data_asset_name = 'CHOOSE THE NAME OF YOUR TABLE FROM THE LIST OF DATA ASSETS ABOVE'\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name='my_suite'\n",
    "                        BatchKwargs(table=data_asset_name)) \n",
    "```\n",
    "\n",
    "* B. To validate a query result set:\n",
    "\n",
    "```python\n",
    "data_asset_name = 'NAME YOUR QUERY (E.G., daily_users_query) - THIS WILL CREATE A NEW DATA ASSET'\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                        expectation_suite_name='my_suite',\n",
    "                        BatchKwargs(query='SQL FOR YOUR QUERY'))\n",
    "```\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# COPY THE APPROPRIATE CODE SNIPPET FROM THE CELL ABOVE\n",
    "batch = context.get_batch(data_asset_name, \n",
    "                          expectation_suite_name,\n",
    "                          context.yield_batch_kwargs(data_asset_name))\n",
    "batch.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optionally, customize batch options (for example for csvs: delimiters, header, etc) in `get_batch`\n",
    "This is how you can see which data batch was loaded\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#reader-options)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch._batch_kwargs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Author Expectations\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#create-expectations)\n",
    "\n",
    "See available expectations in the [expectation glossary](https://docs.greatexpectations.io/en/latest/glossary.html?utm_source=notebook&utm_medium=create_expectations).\n",
    "You can also see available expectations by hovering over data elements in the HTML page generated by profiling your dataset.\n\n",
    "Here is an example expectation that checks if the values in the batch's first column are null."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_name = batch.get_table_columns()[0]\n",
    "batch.expect_column_values_to_not_be_null(column_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add more expectations here. **Hint** start with `batch.expect_` and hit tab for Jupyter's autocomplete to see all the expectations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.expect_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Review and save your Expectations\n",
    "\n",
    "Expectations that are `True` on this data batch are added automatically. To view all the expectations you added so far about this data asset, run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.get_expectation_suite()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    \n",
    "    \n",
    "If you decide not to save some expectations that you created, use [remove_expectaton method](https://docs.greatexpectations.io/en/latest/module_docs/data_asset_module.html?highlight=remove_expectation&utm_source=notebook&utm_medium=create_expectations#great_expectations.data_asset.data_asset.DataAsset.remove_expectation)\n",
    "\n",
    "\n",
    "The following method will save the expectation suite as a JSON file in the `great_expectations/expectations` directory of your project:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.save_expectation_suite()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations! You created and saved expectations for at least one of your data assets.\n",
    "\n",
    "## Next steps:\n",
    "\n",
    "### 1. Data Docs\n",
    "Jump back to the command line and run `great_expectations build-docs` to see your Data Docs. These are created from the expectations you just made and help you understand and communicate about your data.\n",
    "### 2. Validation\n",
    "Validation is the process of checking if new batches of this data meet to your expectations before they are processed by your pipeline.\n",
    "### Go to [integrate_validation_into_pipeline.ipynb](integrate_validation_into_pipeline.ipynb) to see how!\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
