# Overview
This repository contains a collection of scripts and data related to the public_bi dataset. The data is stored in the tables folder, and the scripts for processing this data are located in the scripts directory.  
# Selectivity Tables
One of the key scripts in this repository is selectivity_table.py. This script generates two CSV files: selectivity.csv and selectivity_mini.csv. These files contain information about the selectivity of various queries on the public_bi dataset. **selectivity.csv**  is not present in repository due to its large size, but can be generated by running the selectivity_table.py script.
## Selectivity.csv
The selectivity.csv file contains a query for each predicate for each unique value a column has in the public_bi dataset. In addition, for each column it has queries on the values not present in the column to achieve 0 and 1 selectivity (for example, with ≠ predicate, for each  column that has enough unique values, we have a query that would  filter based on a value that is not present in the column and larger than min and  smaller than max to prevent block skipping).  Each row in the selectivity.csv file represents a different query, and the selectivity that this query achieves when run against the public_bi dataset. The selectivity of a query is a measure of the proportion of the total data that the query returns.  
## Selectivity_mini.csv
The selectivity_mini.csv file is a smaller version of the selectivity.csv file. For each predicate, there exists a query that achieves the nearest possible selectivity to each value in the range from 0.000 to 1.000, incrementing by 0.001. This means that for every predicate, there is a query tailored to closely match each selectivity level within this range when executed against the public_bi dataset. This file is sufficient for most applications, as it contains a representative sample of the selectivity of each predicate.
# Misc
- [ ] why Euro2016 is different?
- [ ] USCensus does not fit