Data documentation compiled by Great Expectations

###### Great Expectations Walkthrough

#### How to work with Great Expectations

Welcome! Now that you have initialized your project, the best way to work with Great Expectations is in this iterative dev loop:

1. Let Great Expectations create a (terrible) first draft suite, by running `great_expectations suite new`.
2. View the suite here in Data Docs.
3. Edit the suite in a Jupyter notebook by running `great_expectations suite edit`
4. Repeat Steps 2-3 until you are happy with your suite.
5. Commit this suite to your source control repository.

#### What are Expectations?

- expect_column_to_exist
- expect_table_row_count_to_be_between
- expect_column_values_to_be_unique
- expect_column_values_to_not_be_null
- expect_column_values_to_be_between
- expect_column_values_to_match_regex
- expect_column_mean_to_be_between
- expect_column_kl_divergence_to_be_less_than
- ... and many more

An expectation is a falsifiable, verifiable statement about data.

Expectations provide a language to talk about data characteristics and data quality - humans to humans, humans to machines and machines to machines.

Expectations are both data tests and docs!

#### Expectations can be presented in a machine-friendly JSON

{
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "user_id"
}
}

A machine can test if a dataset conforms to the expectation.

#### Validation produces a validation result object

{
"success": false,
"result": {
"element_count": 253405,
"unexpected_count": 7602,
"unexpected_percent": 2.999
},
"expectation_config": {
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "user_id"
}
}

Here's an example Validation Result (not from your data) in JSON format. This object has rich context about the test failure.

#### Validation results save you time.

This is an example of what a single failed Expectation looks like in Data Docs. Note the failure includes unexpected values from your data. This helps you debug pipelines faster.

#### Great Expectations provides a large library of expectations.

Nearly 50 built in expectations allow you to express how you understand your data, and you can add custom expectations if you need a new one.

#### Now explore and edit the sample suite!

This sample suite shows you a few examples of expectations.

Note this is not a production suite and was generated using only a small sample of your data.

When you are ready, press the How to Edit button to kick off the iterative dev loop.

##### How to Edit This Expectation Suite

Expectations are best edited interactively in Jupyter notebooks.

To automatically generate a notebook that does this run:

Once you have made your changes and run the entire notebook you can kill the notebook by pressing Ctr-C in your terminal.

Because these notebooks are generated from an Expectation Suite, these notebooks are entirely disposable.

#### Expectation Suite

A collection of Expectations defined for batches of data.

Actions
Table of Contents

##### Overview

###### Info

Expectation Suite Name | default
Great Expectations Version | 0.7.0-beta

###### Table-Level Expectations

- Must have greater than or equal to 0 rows.

- Must have a list of columns in a specific order, but that order is not specified.

###### Notes

This Expectation suite currently contains 43 total Expectations across 7 columns.

##### Age

- value types must belong to this set: DOUBLE_PRECISION DoubleType FLOAT FloatType NUMERIC float.

- may have any number of unique values.

- may have any fraction of unique values.

- values must not be null, at least 50 % of the time.

- minimum value may have any numerical value.

- maximum value may have any numerical value.

- mean may have any numerical value.

- median may have any numerical value.

- quantiles must be within the following value ranges.

- Quantile | Min Value | Max Value
  --- | --- | ---
  0.05 | Any | Any
  Q1 | Any | Any
  Median | Any | Any
  Q3 | Any | Any
  0.95 | Any | Any

- can match any distribution.

##### Name

- value types must belong to this set: CHAR StringType TEXT VARCHAR str string.

- may have any number of unique values.

- may have any fraction of unique values.

- values must not be null, at least 50 % of the time.

##### PClass

- value types must belong to this set: CHAR StringType TEXT VARCHAR str string.

- may have any number of unique values.

- may have any fraction of unique values.

- values must not be null, at least 50 % of the time.

- values must not match this regular expression: ^\s+|\s+$.

- distinct values must belong to a set, but that set is not specified.

##### Sex

- value types must belong to this set: CHAR StringType TEXT VARCHAR str string.

- may have any number of unique values.

- may have any fraction of unique values.

- values must not be null, at least 50 % of the time.

- values must not match this regular expression: ^\s+|\s+$.

- distinct values must belong to a set, but that set is not specified.

##### SexCode

- value types must belong to this set: BIGINT INTEGER IntegerType LongType SMALLINT int.

- may have any number of unique values.

- may have any fraction of unique values.

- values must not be null, at least 50 % of the time.

- distinct values must belong to a set, but that set is not specified.

##### Survived

- value types must belong to this set: BIGINT INTEGER IntegerType LongType SMALLINT int.

- may have any number of unique values.

- may have any fraction of unique values.

- values must not be null, at least 50 % of the time.

- distinct values must belong to a set, but that set is not specified.

##### Unnamed: 0

- value types must belong to this set: BIGINT INTEGER IntegerType LongType SMALLINT int.

  ###### Notes:

  #### These are expectation notes

  - you can use markdown
  - or just strings

- may have any number of unique values.

  ###### Notes:

  This is a

  string list,

  assigned to the 'content' key of a notes dict.

  Cool

  huh?

- may have any fraction of unique values.

  ###### Notes:

  This is just a single string, assigned to the 'content' key of a notes dict.

- values must not be null, at least 50 % of the time.

  ###### Notes:

  This is a list

  of strings

  assigned to the notes

  key.

- values must be unique.

  ###### Notes:

  This is a single string assigned to the 'notes' key.