Data documentation compiled by Great Expectations

###### Great Expectations Walkthrough

#### How to work with Great Expectations

Welcome! Now that you have initialized your project, the best way to work with Great Expectations is in this iterative dev loop:

1. Let Great Expectations create a (terrible) first draft suite, by running `great_expectations suite new`.
2. View the suite here in Data Docs.
3. Edit the suite in a Jupyter notebook by running `great_expectations suite edit`
4. Repeat Steps 2-3 until you are happy with your suite.
5. Commit this suite to your source control repository.

#### What are Expectations?

- expect_column_to_exist
- expect_table_row_count_to_be_between
- expect_column_values_to_be_unique
- expect_column_values_to_not_be_null
- expect_column_values_to_be_between
- expect_column_values_to_match_regex
- expect_column_mean_to_be_between
- expect_column_kl_divergence_to_be_less_than
- ... and many more

An expectation is a falsifiable, verifiable statement about data.

Expectations provide a language to talk about data characteristics and data quality - humans to humans, humans to machines and machines to machines.

Expectations are both data tests and docs!

#### Expectations can be presented in a machine-friendly JSON

{
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "user_id"
}
}

A machine can test if a dataset conforms to the expectation.

#### Validation produces a validation result object

{
"success": false,
"result": {
"element_count": 253405,
"unexpected_count": 7602,
"unexpected_percent": 2.999
},
"expectation_config": {
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "user_id"
}
}

Here's an example Validation Result (not from your data) in JSON format. This object has rich context about the test failure.

#### Validation results save you time.

This is an example of what a single failed Expectation looks like in Data Docs. Note the failure includes unexpected values from your data. This helps you debug pipelines faster.

#### Great Expectations provides a large library of expectations.

Nearly 50 built in expectations allow you to express how you understand your data, and you can add custom expectations if you need a new one.

#### Now explore and edit the sample suite!

This sample suite shows you a few examples of expectations.

Note this is not a production suite and was generated using only a small sample of your data.

When you are ready, press the How to Edit button to kick off the iterative dev loop.

#### Data Asset Profile

Overview of data values, statistics, and distributions.

Actions
Table of Contents

##### Overview

###### Dataset info

Number of variables | 7
Number of observations expect_table_row_count_to_be_between | 1313
Missing cells | 6.06%

###### Variable types

int | 3
float | 1
string | 3
datetime | 0
bool | 0
unknown | 0

Show Expectation Types...

- expect_column_values_to_be_in_type_list 7
- expect_column_unique_value_count_to_be_between 7
- expect_column_proportion_of_unique_values_to_be_between 7
- expect_column_values_to_not_be_null 7
- expect_column_values_to_be_in_set 7
- expect_column_distinct_values_to_be_in_set 4
- expect_column_values_to_not_match_regex 3
- expect_table_row_count_to_be_between 1
- expect_table_columns_to_match_ordered_list 1
- expect_column_values_to_be_unique 1
- expect_column_min_to_be_between 1
- expect_column_max_to_be_between 1
- expect_column_mean_to_be_between 1
- expect_column_median_to_be_between 1
- expect_column_quantile_values_to_be_between 1
- expect_column_kl_divergence_to_be_less_than 1

##### Unnamed: 0 expect_column_to_exist

###### Type: int expect_column_values_to_be_of_type
expect_column_values_to_be_in_type_list

###### Properties

Distinct (n) expect_column_unique_value_count_to_be_between | 1313
Distinct (%) expect_column_proportion_of_unique_values_to_be_between | 100.0%
Missing (n) expect_column_values_to_not_be_null | 0
Missing (%) expect_column_values_to_not_be_null | 0.0%

###### Example Values expect_column_values_to_be_in_set

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

##### Name expect_column_to_exist

###### Type: string expect_column_values_to_be_of_type
expect_column_values_to_be_in_type_list

###### Properties

Distinct (n) expect_column_unique_value_count_to_be_between | 1310
Distinct (%) expect_column_proportion_of_unique_values_to_be_between | 99.8%
Missing (n) expect_column_values_to_not_be_null | 0
Missing (%) expect_column_values_to_not_be_null | 0.0%

###### Example Values expect_column_values_to_be_in_set

Carlsson, Mr Frans Olof Connolly, Miss Kate Kelly, Mr James Allen, Miss Elisabeth Walton Allison, Master Hudson Trevor Allison, Miss Helen Loraine Allison, Mr Hudson Joshua Creighton Allison, Mrs Hudson JC (Bessie Waldo Daniels) Anderson, Mr Harry Andrews, Miss Kornelia Theodosia Andrews, Mr Thomas, jr Appleton, Mrs Edward Dale (Charlotte Lamson) Artagaveytia, Mr Ramon Astor, Colonel John Jacob Astor, Mrs John Jacob (Madeleine Talmadge Force) Aubert, Mrs Leontine Pauline Barkworth, Mr Algernon H Baumann, Mr John D Baxter, Mr Quigg Edmond Baxter, Mrs James (Helene DeLaudeniere Chaput)

##### PClass expect_column_to_exist

###### Type: string expect_column_values_to_be_of_type
expect_column_values_to_be_in_type_list

###### Properties

Distinct (n) expect_column_unique_value_count_to_be_between | 4
Distinct (%) expect_column_proportion_of_unique_values_to_be_between | 0.3%
Missing (n) expect_column_values_to_not_be_null | 0
Missing (%) expect_column_values_to_not_be_null | 0.0%

###### Example Values expect_column_values_to_be_in_set

3rd 1st 2nd *

###### Value Counts expect_column_distinct_values_to_be_in_set

##### Age expect_column_to_exist

###### Type: float expect_column_values_to_be_of_type
expect_column_values_to_be_in_type_list

###### Properties

Distinct (n) expect_column_unique_value_count_to_be_between | 75
Distinct (%) expect_column_proportion_of_unique_values_to_be_between | 9.9%
Missing (n) expect_column_values_to_not_be_null | 557
Missing (%) expect_column_values_to_not_be_null | 42.4%

###### Quantiles

0.05 expect_column_quantile_values_to_be_between | 6.0
Q1 expect_column_quantile_values_to_be_between | 21.0
Median expect_column_quantile_values_to_be_between expect_column_median_to_be_between | 28.0
Q3 expect_column_quantile_values_to_be_between | 39.0
0.95 expect_column_quantile_values_to_be_between | 57.0

###### Statistics

Mean expect_column_mean_to_be_between | 30.40
Minimum expect_column_min_to_be_between | 0.17
Maximum expect_column_max_to_be_between | 71.00

###### Example Values expect_column_values_to_be_in_set

22.0 21.0 30.0 18.0 36.0 24.0 26.0 27.0 28.0 19.0 20.0 23.0 25.0 32.0 45.0 29.0 31.0 33.0 35.0 39.0

###### Histogram expect_column_kl_divergence_to_be_less_than

##### Sex expect_column_to_exist

###### Type: string expect_column_values_to_be_of_type
expect_column_values_to_be_in_type_list

###### Properties

Distinct (n) expect_column_unique_value_count_to_be_between | 2
Distinct (%) expect_column_proportion_of_unique_values_to_be_between | 0.2%
Missing (n) expect_column_values_to_not_be_null | 0
Missing (%) expect_column_values_to_not_be_null | 0.0%

###### Example Values expect_column_values_to_be_in_set

male female

###### Value Counts expect_column_distinct_values_to_be_in_set

##### Survived expect_column_to_exist

###### Type: int expect_column_values_to_be_of_type
expect_column_values_to_be_in_type_list

###### Properties

Distinct (n) expect_column_unique_value_count_to_be_between | 2
Distinct (%) expect_column_proportion_of_unique_values_to_be_between | 0.2%
Missing (n) expect_column_values_to_not_be_null | 0
Missing (%) expect_column_values_to_not_be_null | 0.0%

###### Example Values expect_column_values_to_be_in_set

0 1

###### Value Counts expect_column_distinct_values_to_be_in_set

##### SexCode expect_column_to_exist

###### Type: int expect_column_values_to_be_of_type
expect_column_values_to_be_in_type_list

###### Properties

Distinct (n) expect_column_unique_value_count_to_be_between | 2
Distinct (%) expect_column_proportion_of_unique_values_to_be_between | 0.2%
Missing (n) expect_column_values_to_not_be_null | 0
Missing (%) expect_column_values_to_not_be_null | 0.0%

###### Example Values expect_column_values_to_be_in_set

0 1

###### Value Counts expect_column_distinct_values_to_be_in_set