Data documentation compiled by Great Expectations

###### Great Expectations Walkthrough

#### How to work with Great Expectations

Welcome! Now that you have initialized your project, the best way to work with Great Expectations is in this iterative dev loop:

1. Let Great Expectations create a (terrible) first draft suite, by running `great_expectations suite new`.
2. View the suite here in Data Docs.
3. Edit the suite in a Jupyter notebook by running `great_expectations suite edit`
4. Repeat Steps 2-3 until you are happy with your suite.
5. Commit this suite to your source control repository.

#### What are Expectations?

- expect_column_to_exist
- expect_table_row_count_to_be_between
- expect_column_values_to_be_unique
- expect_column_values_to_not_be_null
- expect_column_values_to_be_between
- expect_column_values_to_match_regex
- expect_column_mean_to_be_between
- expect_column_kl_divergence_to_be_less_than
- ... and many more

An expectation is a falsifiable, verifiable statement about data.

Expectations provide a language to talk about data characteristics and data quality - humans to humans, humans to machines and machines to machines.

Expectations are both data tests and docs!

#### Expectations can be presented in a machine-friendly JSON

{
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "user_id"
}
}

A machine can test if a dataset conforms to the expectation.

#### Validation produces a validation result object

{
"success": false,
"result": {
"element_count": 253405,
"unexpected_count": 7602,
"unexpected_percent": 2.999
},
"expectation_config": {
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "user_id"
}
}

Here's an example Validation Result (not from your data) in JSON format. This object has rich context about the test failure.

#### Validation results save you time.

This is an example of what a single failed Expectation looks like in Data Docs. Note the failure includes unexpected values from your data. This helps you debug pipelines faster.

#### Great Expectations provides a large library of expectations.

Nearly 50 built in expectations allow you to express how you understand your data, and you can add custom expectations if you need a new one.

#### Now explore and edit the sample suite!

This sample suite shows you a few examples of expectations.

Note this is not a production suite and was generated using only a small sample of your data.

When you are ready, press the How to Edit button to kick off the iterative dev loop.

##### How to Edit This Expectation Suite

Expectations are best edited interactively in Jupyter notebooks.

To automatically generate a notebook that does this run:

great_expectations suite edit default

Once you have made your changes and run the entire notebook you can kill the notebook by pressing Ctr-C in your terminal.

Because these notebooks are generated from an Expectation Suite, these notebooks are entirely disposable.

#### Expectation Validation Result

Evaluates whether a batch of data matches expectations.

Actions

Validation Filter:

Show All Failed Only
Table of Contents

##### Overview

Expectation Suite: default
Data asset: None
Status: Failed

###### Statistics

Evaluated Expectations | 51
Successful Expectations | 43
Unsuccessful Expectations | 8
Success Percent | ≈84.31%

Show more info...

###### Info

Great Expectations Version | 0.9.7+17.g02805059.dirty
Run Name | 20200322T170247.671855Z
Run Time | 2020-03-22T17:02:47.671855Z

###### Batch Kwargs

ge_batch_id | f499679e-6c5e-11ea-8d20-acde48001122

##### Table-Level Expectations

Status | Expectation | Observed Value
--- | --- | ---
| Must have greater than or equal to 0 rows. | 1,313
| Must have a list of columns in a specific order, but that order is not specified. | ['Unnamed: 0', 'Name', 'PClass', 'Age', 'Sex', 'Survived', 'SexCode']

##### Unnamed: 0

Status | Expectation | Observed Value
--- | --- | ---
| value types must belong to this set: BIGINT BYTEINT DECIMAL INT INTEGER IntegerType LongType SMALLINT TINYINT int. | int64
| may have any number of unique values. | 1,313
| may have any fraction of unique values. | 1
| values must not be null, at least 50 % of the time. | 100% not null
| values must belong to this set: [ ]. 1,313 unexpected values found. 100% of 1,313 total rows.
Sampled Unexpected Values
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| 100% unexpected
| values must be unique. | 0% unexpected

##### Name

Status | Expectation | Observed Value
--- | --- | ---
| value types must belong to this set: CHAR STRING StringType TEXT VARCHAR str string. | 0% unexpected
| may have any number of unique values. | 1,310
| may have any fraction of unique values. | ≈0.9977151561
| values must not be null, at least 50 % of the time. | 100% not null
| values must belong to this set: [ ]. 1,313 unexpected values found. 100% of 1,313 total rows.
Sampled Unexpected Values
Carlsson, Mr Frans Olof
Connolly, Miss Kate
Kelly, Mr James
Allen, Miss Elisabeth Walton
Allison, Master Hudson Trevor
Allison, Miss Helen Loraine
Allison, Mr Hudson Joshua Creighton
Allison, Mrs Hudson JC (Bessie Waldo Daniels)
Anderson, Mr Harry
Andrews, Miss Kornelia Theodosia
Andrews, Mr Thomas, jr
Appleton, Mrs Edward Dale (Charlotte Lamson)
Artagaveytia, Mr Ramon
Astor, Colonel John Jacob
Astor, Mrs John Jacob (Madeleine Talmadge Force)
Aubert, Mrs Leontine Pauline
Barkworth, Mr Algernon H
Baumann, Mr John D
Baxter, Mr Quigg Edmond
Baxter, Mrs James (Helene DeLaudeniere Chaput)
| 100% unexpected
| values must not match this regular expression: ^\s+|\s+$. 3 unexpected values found. ≈0.2285% of 1,313 total rows.
Unexpected Value | Count
--- | ---
Barber, Ms | 1
Daly, Mr Peter Denis | 1
Geiger, Miss Emily | 1
| ≈0.22848% unexpected

##### PClass

Status | Expectation | Observed Value
--- | --- | ---
| value types must belong to this set: CHAR STRING StringType TEXT VARCHAR str string. | 0% unexpected
| may have any number of unique values. | 4
| may have any fraction of unique values. | ≈0.003046458492
| values must not be null, at least 50 % of the time. | 100% not null
| values must belong to this set: [ ]. 1,313 unexpected values found. 100% of 1,313 total rows.
Unexpected Value | Count
--- | ---
3rd | 711
1st | 322
2nd | 279
* | 1
| 100% unexpected
| values must not match this regular expression: ^\s+|\s+$. | 0% unexpected
| distinct values must belong to a set, but that set is not specified. | ['*', '1st', '2nd', '3rd']

##### Age

Status | Expectation | Observed Value
--- | --- | ---
| value types must belong to this set: DOUBLE_PRECISION DoubleType FLOAT FLOAT4 FLOAT8 FloatType NUMERIC float. | float64
| may have any number of unique values. | 75
| may have any fraction of unique values. | ≈0.09920634921
| values must not be null, at least 50 % of the time. | ≈57.578% not null
| values must belong to this set: [ ]. 756 unexpected values found. ≈57.58% of 1,313 total rows.
Sampled Unexpected Values
22.0
21.0
30.0
18.0
36.0
24.0
26.0
27.0
28.0
19.0
20.0
23.0
25.0
32.0
45.0
29.0
31.0
33.0
35.0
39.0
| ≈57.578% unexpected
| minimum value may have any numerical value. | 0.17
| maximum value may have any numerical value. | 71
| mean may have any numerical value. | ≈30.39798942
| median may have any numerical value. | 28
| quantiles must be within the following value ranges.
Quantile | Min Value | Max Value
--- | --- | ---
0.05 | Any | Any
Q1 | Any | Any
Median | Any | Any
Q3 | Any | Any
0.95 | Any | Any
Quantile | Value
--- | ---
0.05 | 6.0
Q1 | 21.0
Median | 28.0
Q3 | 39.0
0.95 | 57.0
| can match any distribution. | KL Divergence: None (-infinity, infinity, or NaN)

##### Sex

Status | Expectation | Observed Value
--- | --- | ---
| value types must belong to this set: CHAR STRING StringType TEXT VARCHAR str string. | 0% unexpected
| may have any number of unique values. | 2
| may have any fraction of unique values. | ≈0.001523229246
| values must not be null, at least 50 % of the time. | 100% not null
| values must belong to this set: [ ]. 1,313 unexpected values found. 100% of 1,313 total rows.
Unexpected Value | Count
--- | ---
male | 851
female | 462
| 100% unexpected
| values must not match this regular expression: ^\s+|\s+$. | 0% unexpected
| distinct values must belong to a set, but that set is not specified. | ['female', 'male']

##### Survived

Status | Expectation | Observed Value
--- | --- | ---
| value types must belong to this set: BIGINT BYTEINT DECIMAL INT INTEGER IntegerType LongType SMALLINT TINYINT int. | int64
| may have any number of unique values. | 2
| may have any fraction of unique values. | ≈0.001523229246
| values must not be null, at least 50 % of the time. | 100% not null
| values must belong to this set: [ ]. 1,313 unexpected values found. 100% of 1,313 total rows.
Unexpected Value | Count
--- | ---
0 | 863
1 | 450
| 100% unexpected
| distinct values must belong to a set, but that set is not specified. | [0, 1]

##### SexCode

Status | Expectation | Observed Value
--- | --- | ---
| value types must belong to this set: BIGINT BYTEINT DECIMAL INT INTEGER IntegerType LongType SMALLINT TINYINT int. | int64
| may have any number of unique values. | 2
| may have any fraction of unique values. | ≈0.001523229246
| values must not be null, at least 50 % of the time. | 100% not null
| values must belong to this set: [ ]. 1,313 unexpected values found. 100% of 1,313 total rows.
Unexpected Value | Count
--- | ---
0 | 851
1 | 462
| 100% unexpected
| distinct values must belong to a set, but that set is not specified. | [0, 1]