{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import great_expectations as ge\n",
    "import great_expectations.jupyter_ux\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Author Expectations\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Watch a [short tutorial video](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#video) or read [the written tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations)\n",
    "\n",
    "We'd love it if you **reach out for help on** the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get a DataContext.\n",
    "This represents your project that you set up using `great_expectations init`. [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-datacontext-object)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. List data assets in your project\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#data-assets)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "great_expectations.jupyter_ux.list_available_data_asset_names(context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Pick a data asset and set the expectation suite name\n",
    "\n",
    "Your data_asset_name will consist of three components: a datasource, generator, and generator_asset, but you can usually provide only some of them as long as the name is not ambiguous. See [more in the reference](https://docs.great_expectations.io/en/latest/reference/data_context_reference.html#data-asset-names). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_asset_name = \"REPLACE_ME\"  # TODO: replace with your value!\n",
    "data_asset_name = context.normalize_data_asset_name(data_asset_name)\n",
    "data_asset_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We recommend you name your first expectation suite for a given data asset `warning`. Later, as you identify some of the expectations that you add to this suite as critical, you can move these expectations into another suite and call it `failure`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expectation_suite_name = \"warning\"  # TODO: replace with your value!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create a new expectation suite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We are creating an empty expectation suite. All the expectations we will create in this notebook\n",
    "# will be added to it.\n",
    "\n",
    "context.create_expectation_suite(data_asset_name=data_asset_name, expectation_suite_name=expectation_suite_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Load a batch of data from the data asset you want to validate\n",
    "\n",
    "Learn about `get_batch` in [this tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-batch)\n",
    "\n",
    "How you get a batch of data will depend on your datasource (pandas, SQL, Spark, etc). `BatchKwargs` tell your specific datasource how to fetch a batch of data and might include a filepath, table name, SQL query, or even an existing DataFrame loaded outside of Great Expectations. Choose the best batch_kwargs for your situation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you're working with data loaded from files, and GE listed and profiled your files correctly:\n",
    "batch_kwargs = context.yield_batch_kwargs(data_asset_name)\n",
    "\n",
    "# If you would like to validate data in a database, using an entire table or view:\n",
    "# batch_kwargs = {'table': 'name_of_table_to_validate'}  # Add a 'schema' key if you need to specify that explicitly\n",
    "\n",
    "# If you would like to validate data in a database, using a query to construct a temporary table:\n",
    "# batch_kwargs = {'query': 'SELECT YOUR_ROWS FROM YOUR_TABLE'}\n",
    "\n",
    "# If you would like to control reading of data outside of Great Expectations, and provide a pre-built Dataframe:\n",
    "# df = spark.read.csv(...)\n",
    "# df = pd.readcsv(...)\n",
    "# batch_kwargs = {'dataset': df}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = context.get_batch(data_asset_name, \n",
    "                          expectation_suite_name,\n",
    "                          batch_kwargs)\n",
    "batch.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optionally, customize and review batch options\n",
    "BatchKwargs are extremely flexible and allow you to specify additional information to use when building the batch, such as filetypes, delimiters, headers, or other parameters. You can add additional batch_kwargs when you build the kwargs or when you call `get_batch`.\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#reader-options)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See the batch kwargs used to load your batch\n",
    "batch.batch_kwargs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The datasource can add and store additional identifying information to ensure you can track a batch through\n",
    "# your pipeline\n",
    "batch.batch_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Author Expectations\n",
    "\n",
    "With a batch, you can add expectations by calling its 'expect' methods.\n",
    "\n",
    "See available expectations in the [expectation glossary](https://docs.greatexpectations.io/en/latest/glossary.html?utm_source=notebook&utm_medium=create_expectations).\n",
    "You can also see available expectations by hovering over data elements in the HTML page generated by profiling your dataset.\n",
    "\n",
    "Below is an example expectation that checks if the values in the batch's first column are null.\n",
    "\n",
    "[Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#create-expectations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_name = batch.get_table_columns()[0]\n",
    "batch.expect_column_values_to_not_be_null(column_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add more expectations here. **Hint** start with `batch.expect_` and hit tab for Jupyter's autocomplete to see all the expectations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.expect_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Review and save your Expectations\n",
    "\n",
    "Expectations that are `True` on this data batch are added automatically. To view all the expectations you added so far about this data asset, run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.get_expectation_suite()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    \n",
    "    \n",
    "If you decide not to save some expectations that you created, use [remove_expectaton method](https://docs.greatexpectations.io/en/latest/module_docs/data_asset_module.html?highlight=remove_expectation&utm_source=notebook&utm_medium=create_expectations#great_expectations.data_asset.data_asset.DataAsset.remove_expectation). You can also choose not to filter expectations that were `False` on this batch.\n",
    "\n",
    "\n",
    "The following method will save the expectation suite as a JSON file in the `great_expectations/expectations` directory of your project:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.save_expectation_suite()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations! You created and saved expectations for at least one of your data assets.\n",
    "\n",
    "## Next steps:\n",
    "\n",
    "### 1. Data Docs\n",
    "Jump back to the command line and run `great_expectations build-docs` to see your Data Docs. These are created from the expectations you just made and help you understand and communicate about your data.\n",
    "### 2. Validation\n",
    "Validation is the process of checking if new batches of this data meet to your expectations before they are processed by your pipeline.\n",
    "### Go to [integrate_validation_into_pipeline.ipynb](integrate_validation_into_pipeline.ipynb) to see how!\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}