{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "70472390-e9ca-4ae6-bf44-ea173ecd6ea3",
   "metadata": {},
   "source": [
    "# Clean\n",
    "\n",
    "This notebook loads your assembled data and prepares it for modeling.\n",
    "\n",
    "The final output is a `sup` that is fully ready for modeling and analysis.\n",
    "\n",
    "Here's what happens in this notebook:\n",
    "- We fill gaps in the data using reasonable assumptions\n",
    "- We prepare the data for further analysis by marking clusters\n",
    "- We process sales validity information\n",
    "- We run our own sales scrutiny heuristic to make sure we only use trustworthy sales that reflect market value\n",
    "\n",
    "These operations are necessary for modeling, but nevertheless inject a certain amount of subjectivity into the model, which is why we keep the results of the **Clean** notebook separate from those produced in the **Assemble** notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f52848e147c86a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change these as desired\n",
    "\n",
    "# The slug of the locality you are currently working on\n",
    "locality = \"us-ky-louisville\"\n",
    "\n",
    "# Whether to print out a lot of stuff (can help with debugging) or stay mostly quiet\n",
    "verbose = True\n",
    "\n",
    "# Clear previous state for this notebook and start fresh\n",
    "clear_checkpoints = True\n",
    "\n",
    "# Set to true to have sales scrutiny drop sales rather than just flagging them\n",
    "sales_scrutiny_drop_outliers = False    # Drop outlier sales in sales clusters\n",
    "sales_scrutiny_drop_heuristics = True  # Drop sales that match suspicious metadata patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94e2b17504a799e4",
   "metadata": {},
   "source": [
    "# 1. Basic setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5091e3d7-3524-4d8a-b74f-e300e63b146f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Environment setup completed.\n"
     ]
    }
   ],
   "source": [
    "import init_notebooks\n",
    "init_notebooks.setup_environment()\n",
    "locality = init_notebooks.check_for_different_locality(locality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "initial_id",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import a bunch of stuff\n",
    "from openavmkit.pipeline import (\n",
    "    init_notebook,\n",
    "    from_checkpoint,\n",
    "    delete_checkpoints,\n",
    "    write_checkpoint,\n",
    "    read_pickle,\n",
    "    load_settings,\n",
    "    examine_sup,\n",
    "    fill_unknown_values_sup,\n",
    "    process_sales,\n",
    "    mark_ss_ids_per_model_group_sup,\n",
    "    mark_horizontal_equity_clusters_per_model_group_sup,\n",
    "    run_sales_scrutiny,\n",
    "    write_notebook_output_sup\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bac878e081b1c36f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "locality = us-ky-louisville\n",
      "base path = C:\\Users\\jacks\\Documents\\Non-Game Stuff\\Programming\\openavmkit\\notebooks\\pipeline\n",
      "current path = C:\\Users\\jacks\\Documents\\Non-Game Stuff\\Programming\\openavmkit\\notebooks\\pipeline\\data\\us-ky-louisville\n"
     ]
    }
   ],
   "source": [
    "init_notebook(locality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6c9e992b-bc8e-4571-b31c-10e787af2e06",
   "metadata": {},
   "outputs": [],
   "source": [
    "if clear_checkpoints:\n",
    "    delete_checkpoints(\"2-clean\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be1bd860-b07c-46cf-a362-e0e74eb0eb2a",
   "metadata": {},
   "source": [
    "# 2. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "17f058f7-20b3-4a09-a5ff-519559ae6fe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "settings = load_settings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9ac4ae929de30695",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the data\n",
    "sales_univ_pair = read_pickle(\"out/1-assemble-sup\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5ac7f077-104c-43ba-99b6-f29e60179f94",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "EXAMINING UNIVERSE...\n",
      "\n",
      "            FIELD                 TYPE     NON-ZERO    %    NON-NULL    %                    UNIQUE                 \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "             LAND             \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           NUMERIC            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "assr_land_value                                                              Float64      277,198   98%    282,400  100%                                         \n",
      "depth_ft_1                                                                   float64      268,482   95%    282,400  100%                                         \n",
      "depth_ft_2                                                                   float64       97,037   34%    282,400  100%                                         \n",
      "depth_ft_3                                                                   float64       18,962    7%    282,400  100%                                         \n",
      "depth_ft_4                                                                   float64        7,204    3%    282,400  100%                                         \n",
      "dist_to_osm_aiport                                                           float64       18,160    6%     18,644    7%                                         \n",
      "dist_to_osm_educational                                                      float64       59,058   21%     59,468   21%                                         \n",
      "dist_to_osm_golf_courses                                                     float64       96,440   34%     97,917   35%                                         \n",
      "dist_to_osm_golf_courses_audubon_country_club                                float64        5,490    2%      5,496    2%                                         \n",
      "dist_to_osm_golf_courses_big_spring_country_club                             float64        6,337    2%      6,350    2%                                         \n",
      "dist_to_osm_golf_courses_bobby_nichols_golf_course                           float64        3,621    1%      3,628    1%                                         \n",
      "dist_to_osm_golf_courses_charlie_vettner_golf_course                         float64        5,967    2%      6,028    2%                                         \n",
      "dist_to_osm_golf_courses_cherokee_golf_course                                float64        7,768    3%      7,790    3%                                         \n",
      "dist_to_osm_golf_courses_cherry_valley_golf_course                           float64            0    0%          0    0%                                         \n",
      "dist_to_osm_golf_courses_crescent_hill_golf_course                           float64        6,171    2%      6,180    2%                                         \n",
      "dist_to_osm_golf_courses_elk_run_golf_club                                   float64            0    0%          0    0%                                         \n",
      "dist_to_osm_golf_courses_farmsley_golf_center                                float64        5,367    2%      5,388    2%                                         \n",
      "dist_to_osm_golf_courses_glenmary_country_club                               float64        3,475    1%      3,637    1%                                         \n",
      "dist_to_osm_golf_courses_heritage_hill_golf_course                           float64            0    0%          0    0%                                         \n",
      "dist_to_osm_golf_courses_hidden_creek_golf_club                              float64            0    0%          0    0%                                         \n",
      "dist_to_osm_golf_courses_hunting_creek_golf_course                           float64        1,807    1%      2,256    1%                                         \n",
      "dist_to_osm_golf_courses_hurstbourne_country_club                            float64        4,527    2%      4,652    2%                                         \n",
      "dist_to_osm_golf_courses_iroquois_golf_course                                float64        7,703    3%      7,739    3%                                         \n",
      "dist_to_osm_golf_courses_long_run_golf_course                                float64        1,860    1%      1,862    1%                                         \n",
      "dist_to_osm_golf_courses_louisville_country_club                             float64        3,174    1%      3,201    1%                                         \n",
      "dist_to_osm_golf_courses_midland_trail_golf_club                             float64        2,189    1%      2,202    1%                                         \n",
      "dist_to_osm_golf_courses_new_albany_golf_course                              float64            0    0%          0    0%                                         \n",
      "dist_to_osm_golf_courses_owl_creek_country_club                              float64        1,271    0%      1,279    0%                                         \n",
      "dist_to_osm_golf_courses_oxmoor_country_club                                 float64        4,772    2%      4,816    2%                                         \n",
      "dist_to_osm_golf_courses_persimmon_ridge_golf_club                           float64          695    0%        695    0%                                         \n",
      "dist_to_osm_golf_courses_polo_fields_golf_and_country_club                   float64        3,085    1%      3,293    1%                                         \n",
      "dist_to_osm_golf_courses_quail_chase_golf_course                             float64        5,283    2%      5,302    2%                                         \n",
      "dist_to_osm_golf_courses_seneca_golf_course                                  float64        7,676    3%      7,680    3%                                         \n",
      "dist_to_osm_golf_courses_shawnee_golf_course                                 float64        5,396    2%      5,445    2%                                         \n",
      "dist_to_osm_golf_courses_standard_country_club                               float64        5,084    2%      5,156    2%                                         \n",
      "dist_to_osm_golf_courses_sun_valley_golf_course                              float64        3,656    1%      3,676    1%                                         \n",
      "dist_to_osm_golf_courses_the_crossings_golf_course                           float64          241    0%        243    0%                                         \n",
      "dist_to_osm_golf_courses_unnamed_golf_courses                                float64          179    0%        179    0%                                         \n",
      "dist_to_osm_golf_courses_valhalla_golf_club                                  float64        3,669    1%      3,676    1%                                         \n",
      "dist_to_osm_golf_courses_valley_view_golf_course                             float64            0    0%          0    0%                                         \n",
      "dist_to_osm_golf_courses_wildwood_country_club                               float64        4,248    2%      4,306    2%                                         \n",
      "dist_to_osm_golf_courses_wooded_view_golf_course                             float64            0    0%          0    0%                                         \n",
      "dist_to_osm_golf_courses_woodhaven_country_club                              float64        3,063    1%      3,096    1%                                         \n",
      "dist_to_osm_parks                                                            float64      218,737   77%    220,185   78%                                         \n",
      "dist_to_osm_water_bodies                                                     float64      151,865   54%    152,554   54%                                         \n",
      "dist_to_road_ft_1                                                            float64      260,972   92%    282,400  100%                                         \n",
      "dist_to_road_ft_2                                                            float64       80,585   29%    282,400  100%                                         \n",
      "dist_to_road_ft_3                                                            float64        9,627    3%    282,400  100%                                         \n",
      "dist_to_road_ft_4                                                            float64        1,648    1%    282,400  100%                                         \n",
      "frontage_ft_1                                                                float64      268,482   95%    282,400  100%                                         \n",
      "frontage_ft_2                                                                float64       97,037   34%    282,400  100%                                         \n",
      "frontage_ft_3                                                                float64       18,962    7%    282,400  100%                                         \n",
      "frontage_ft_4                                                                float64        7,204    3%    282,400  100%                                         \n",
      "geom_aspect_ratio                                                            float64      282,400  100%    282,400  100%                                         \n",
      "geom_rectangularity_num                                                      Float64      282,400  100%    282,400  100%                                         \n",
      "geom_vertices                                                                 int64       282,391  100%    282,400  100%                                         \n",
      "land_area_gis_delta_percent                                                  Float64      282,383  100%    282,400  100%                                         \n",
      "land_area_gis_delta_sqft                                                     float64      282,383  100%    282,400  100%                                         \n",
      "land_area_gis_sqft                                                           float64      282,400  100%    282,400  100%                                         \n",
      "land_area_given_sqft                                                         Float64      282,383  100%    282,400  100%                                         \n",
      "land_area_somers_ft                                                          float64      268,482   95%    282,400  100%                                         \n",
      "land_area_sqft                                                               float64      282,400  100%    282,400  100%                                         \n",
      "land_area_sqft_log                                                           float64      282,400  100%    282,400  100%                                         \n",
      "latitude                                                                     float64      282,400  100%    282,400  100%                                         \n",
      "latitude_norm                                                                float64      282,399  100%    282,400  100%                                         \n",
      "longitude                                                                    float64      282,400  100%    282,400  100%                                         \n",
      "longitude_norm                                                               float64      282,399  100%    282,400  100%                                         \n",
      "median_c_rent                                                                float64      202,346   72%    202,346   72%                                         \n",
      "median_g_rent                                                                float64      203,104   72%    203,104   72%                                         \n",
      "median_income                                                                float64      265,412   94%    265,412   94%                                         \n",
      "osm_frontage_motorway_ft                                                     float64        1,036    0%    282,400  100%                                         \n",
      "osm_frontage_primary_ft                                                      float64       10,243    4%    282,400  100%                                         \n",
      "osm_frontage_residential_ft                                                  float64      231,975   82%    282,400  100%                                         \n",
      "osm_frontage_secondary_ft                                                    float64       13,884    5%    282,400  100%                                         \n",
      "osm_frontage_service_ft                                                      float64       60,482   21%    282,400  100%                                         \n",
      "osm_frontage_tertiary_ft                                                     float64       17,173    6%    282,400  100%                                         \n",
      "osm_frontage_trunk_ft                                                        float64          643    0%    282,400  100%                                         \n",
      "osm_frontage_unclassified_ft                                                 float64        2,406    1%    282,400  100%                                         \n",
      "osm_road_angle_1                                                             float64      268,482   95%    268,482   95%                                         \n",
      "osm_road_angle_2                                                             float64       97,037   34%     97,037   34%                                         \n",
      "osm_road_angle_3                                                             float64       18,962    7%     18,962    7%                                         \n",
      "osm_road_angle_4                                                             float64        7,204    3%      7,204    3%                                         \n",
      "osm_total_frontage_ft                                                        float64      268,482   95%    282,400  100%                                         \n",
      "polar_angle                                                                  float64      282,400  100%    282,400  100%                                         \n",
      "polar_radius                                                                 float64      282,400  100%    282,400  100%                                         \n",
      "proximity_to_osm_aiport                                                      float64       18,643    7%    282,400  100%                                         \n",
      "proximity_to_osm_educational                                                 float64       59,467   21%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses                                                float64       97,916   35%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_audubon_country_club                           float64        5,495    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_big_spring_country_club                        float64        6,349    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_bobby_nichols_golf_course                      float64        3,627    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_charlie_vettner_golf_course                    float64        6,027    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_cherokee_golf_course                           float64        7,789    3%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_cherry_valley_golf_course                      float64            0    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_crescent_hill_golf_course                      float64        6,179    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_elk_run_golf_club                              float64            0    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_farmsley_golf_center                           float64        5,387    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_glenmary_country_club                          float64        3,636    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_heritage_hill_golf_course                      float64            0    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_hidden_creek_golf_club                         float64            0    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_hunting_creek_golf_course                      float64        2,255    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_hurstbourne_country_club                       float64        4,651    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_iroquois_golf_course                           float64        7,738    3%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_long_run_golf_course                           float64        1,861    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_louisville_country_club                        float64        3,200    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_midland_trail_golf_club                        float64        2,201    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_new_albany_golf_course                         float64            0    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_owl_creek_country_club                         float64        1,278    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_oxmoor_country_club                            float64        4,815    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_persimmon_ridge_golf_club                      float64          694    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_polo_fields_golf_and_country_club              float64        3,292    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_quail_chase_golf_course                        float64        5,301    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_seneca_golf_course                             float64        7,679    3%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_shawnee_golf_course                            float64        5,444    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_standard_country_club                          float64        5,155    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_sun_valley_golf_course                         float64        3,675    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_the_crossings_golf_course                      float64          242    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_unnamed_golf_courses                           float64          178    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_valhalla_golf_club                             float64        3,675    1%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_valley_view_golf_course                        float64            0    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_wildwood_country_club                          float64        4,305    2%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_wooded_view_golf_course                        float64            0    0%    282,400  100%                                         \n",
      "proximity_to_osm_golf_courses_woodhaven_country_club                         float64        3,095    1%    282,400  100%                                         \n",
      "proximity_to_osm_parks                                                       float64      220,184   78%    282,400  100%                                         \n",
      "proximity_to_osm_water_bodies                                                float64      152,553   54%    282,400  100%                                         \n",
      "within_osm_aiport                                                              bool        18,644    7%    282,400  100%                                         \n",
      "within_osm_educational                                                         bool        59,468   21%    282,400  100%                                         \n",
      "within_osm_golf_courses                                                        bool        97,917   35%    282,400  100%                                         \n",
      "within_osm_golf_courses_audubon_country_club                                   bool         5,496    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_big_spring_country_club                                bool         6,350    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_bobby_nichols_golf_course                              bool         3,628    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_charlie_vettner_golf_course                            bool         6,028    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_cherokee_golf_course                                   bool         7,790    3%    282,400  100%                                         \n",
      "within_osm_golf_courses_cherry_valley_golf_course                              bool             0    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_crescent_hill_golf_course                              bool         6,180    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_elk_run_golf_club                                      bool             0    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_farmsley_golf_center                                   bool         5,388    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_glenmary_country_club                                  bool         3,637    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_heritage_hill_golf_course                              bool             0    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_hidden_creek_golf_club                                 bool             0    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_hunting_creek_golf_course                              bool         2,256    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_hurstbourne_country_club                               bool         4,652    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_iroquois_golf_course                                   bool         7,739    3%    282,400  100%                                         \n",
      "within_osm_golf_courses_long_run_golf_course                                   bool         1,862    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_louisville_country_club                                bool         3,201    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_midland_trail_golf_club                                bool         2,202    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_new_albany_golf_course                                 bool             0    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_owl_creek_country_club                                 bool         1,279    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_oxmoor_country_club                                    bool         4,816    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_persimmon_ridge_golf_club                              bool           695    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_polo_fields_golf_and_country_club                      bool         3,293    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_quail_chase_golf_course                                bool         5,302    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_seneca_golf_course                                     bool         7,680    3%    282,400  100%                                         \n",
      "within_osm_golf_courses_shawnee_golf_course                                    bool         5,445    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_standard_country_club                                  bool         5,156    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_sun_valley_golf_course                                 bool         3,676    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_the_crossings_golf_course                              bool           243    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_unnamed_golf_courses                                   bool           179    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_valhalla_golf_club                                     bool         3,676    1%    282,400  100%                                         \n",
      "within_osm_golf_courses_valley_view_golf_course                                bool             0    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_wildwood_country_club                                  bool         4,306    2%    282,400  100%                                         \n",
      "within_osm_golf_courses_wooded_view_golf_course                                bool             0    0%    282,400  100%                                         \n",
      "within_osm_golf_courses_woodhaven_country_club                                 bool         3,096    1%    282,400  100%                                         \n",
      "within_osm_parks                                                               bool       220,185   78%    282,400  100%                                         \n",
      "within_osm_water_bodies                                                        bool       152,554   54%    282,400  100%                                         \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           BOOLEAN            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "is_geom_complex                                                                bool         1,819    1%    282,400  100%                            [False, True]\n",
      "is_geom_elongated                                                              bool        24,152    9%    282,400  100%                            [False, True]\n",
      "is_geom_irregular                                                              bool        30,840   11%    282,400  100%                            [False, True]\n",
      "is_geom_triangular                                                             bool         7,811    3%    282,400  100%                            [False, True]\n",
      "is_vacant                                                                      bool        45,917   16%    282,400  100%                            [False, True]\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "         CATEGORICAL          \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "land_class                                                                    string      282,400  100%    282,400  100%                    ['Structure', 'Land']\n",
      "neighborhood                                                                  string      199,752   71%    199,752   71%                                      162\n",
      "osm_road_face_1                                                               object      268,482   95%    268,482   95%                                        9\n",
      "osm_road_face_2                                                               object       97,037   34%     97,037   34%                                        9\n",
      "osm_road_face_3                                                               object       18,962    7%     18,962    7%                                        9\n",
      "osm_road_face_4                                                               object        7,204    3%      7,204    3%                                        9\n",
      "osm_road_name_1                                                               object      268,482   95%    268,482   95%                                   12,862\n",
      "osm_road_name_2                                                               object       97,037   34%     97,037   34%                                   23,192\n",
      "osm_road_name_3                                                               object       18,962    7%     18,962    7%                                   12,543\n",
      "osm_road_name_4                                                               object        7,204    3%      7,204    3%                                    6,596\n",
      "osm_road_type_1                                                               object      268,482   95%    268,482   95%                                       12\n",
      "osm_road_type_2                                                               object       97,037   34%     97,037   34%                                       14\n",
      "osm_road_type_3                                                               object       18,962    7%     18,962    7%                                       13\n",
      "osm_road_type_4                                                               object        7,204    3%      7,204    3%                                       12\n",
      "zoning                                                                        string      238,164   84%    238,164   84%                                       63\n",
      "\n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "         IMPROVEMENT          \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           NUMERIC            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "assr_impr_value                                                              Float64      234,617   83%    282,400  100%                                         \n",
      "bldg_area_finished_sqft                                                      Float64      236,483   84%    282,400  100%                                         \n",
      "bldg_condition_num                                                           Float64        6,308    2%    282,400  100%                                         \n",
      "bldg_effective_age_years                                                     Float64      224,641   80%    282,400  100%                                         \n",
      "bldg_effective_year_built                                                    Float64      224,999   80%    270,916   96%                                         \n",
      "bldg_rooms_bath_full                                                         Float64      236,188   84%    282,400  100%                                         \n",
      "bldg_stories                                                                 Float64            4    0%     45,936   16%                                         \n",
      "bldg_units                                                                   Float64        4,995    2%     50,912   18%                                         \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "         CATEGORICAL          \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "bldg_ac                                                                      category     282,400  100%    282,400  100%                 ['No', 'UNKNOWN', 'Yes']\n",
      "bldg_condition_txt                                                           category      52,225   18%     52,225   18%                                        7\n",
      "bldg_type                                                                    category     282,400  100%    282,400  100%                                       38\n",
      "\n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "            OTHER             \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           NUMERIC            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "assr_market_value                                                            Float64      279,514   99%    282,400  100%                                         \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           BOOLEAN            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "is_vacant                                                                      bool        45,917   16%    282,400  100%                            [False, True]\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "         CATEGORICAL          \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "address                                                                       string      282,400  100%    282,400  100%                                  276,019\n",
      "key                                                                           string      282,400  100%    282,400  100%                                  282,400\n",
      "model_group                                                                   object      282,400  100%    282,400  100%             ['UNKNOWN', 'single_family']\n",
      "property_owner                                                                string      282,288  100%    282,288  100%                                  222,222\n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "UNCLASSIFIED:                 \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "bldg__year_built                                                             float64      226,359   80%    226,359   80%                                      192\n",
      "geometry                                                                     geometry     282,400  100%    282,400  100%                                  282,400\n",
      "property_owner_address                                                        object      282,271  100%    282,271  100%                                  204,366\n",
      "total_population                                                             float64      282,196  100%    282,196  100%                                      529\n",
      "\n",
      "EXAMINING SALES...\n",
      "\n",
      "            FIELD                 TYPE     NON-ZERO    %    NON-NULL    %                    UNIQUE                 \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "         IMPROVEMENT          \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           NUMERIC            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "bldg_age_years                                                               Float64      183,983   77%    190,431   80%                                         \n",
      "bldg_year_built                                                              Float64      190,431   80%    190,431   80%                                         \n",
      "\n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "            OTHER             \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           NUMERIC            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "sale_age_days                                                                Float64      237,834  100%    237,834  100%                                         \n",
      "sale_day                                                                     Float64      237,834  100%    237,834  100%                                         \n",
      "sale_month                                                                   Float64      237,834  100%    237,834  100%                                         \n",
      "sale_price                                                                   Float64      202,340   85%    237,834  100%                                         \n",
      "sale_quarter                                                                 Float64      237,834  100%    237,834  100%                                         \n",
      "sale_year                                                                    Float64      237,834  100%    237,834  100%                                         \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "           BOOLEAN            \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "vacant_sale                                                                    bool        14,616    6%    237,834  100%                            [True, False]\n",
      "valid_sale                                                                     bool       237,834  100%    237,834  100%                                   [True]\n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "         CATEGORICAL          \n",
      "------------------------------ ---------- ---------- ----- ---------- ----- ----------------------------------------\n",
      "key                                                                           string      237,834  100%    237,834  100%                                  237,834\n",
      "key_sale                                                                      string      237,834  100%    237,834  100%                                  237,834\n",
      "sale_date                                                                   datetime64[ns]    237,834  100%    237,834  100%                                    6,803\n",
      "sale_year_month                                                               string      237,834  100%    237,834  100%                                      309\n",
      "sale_year_quarter                                                             string      237,834  100%    237,834  100%                                      103\n",
      "\n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "UNCLASSIFIED:                 \n",
      "============================== ========== ========== ===== ========== ===== ========================================\n",
      "geometry                                                                     geometry     231,356   97%    231,356   97%                                  231,356\n",
      "sale_type                                                                     string      105,515   44%    105,515   44%                                       15\n"
     ]
    }
   ],
   "source": [
    "examine_sup(sales_univ_pair, settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cebf2f4c-16ec-4be4-a5d4-f50b36d5fde6",
   "metadata": {},
   "source": [
    "# 3. Fill unknowns\n",
    "\n",
    "Modeling functions are unable to process null data, so you need to fill them in somehow.   \n",
    "The goal is to **eliminate all gaps in your data,** at least for fields you intend to turn into modeling variables.\n",
    "\n",
    "Consult the documentation for more details and best practices on filling unknown values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b3f6c117-2685-4cb6-970e-9dd21fa3c37c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill holes in the data with sensible defaults\n",
    "sales_univ_pair = fill_unknown_values_sup(sales_univ_pair, settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89c2d9c6-bfea-4bbd-ae74-078e54732f5b",
   "metadata": {},
   "source": [
    "# 4. Clustering\n",
    "\n",
    "We cluster all similar properties and give each cluster a unique ID.  \n",
    "Later, we'll use these ID's whenever we want to run a horizontal equity study.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a9e16813-3e0c-4177-b9b9-1f65e059f6af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Marking horizontal equity clusters...\n",
      "\n",
      "Marking LAND horizontal equity clusters...\n",
      "\n",
      "Marking IMPROVEMENT horizontal equity clusters...\n"
     ]
    }
   ],
   "source": [
    "settings = load_settings()\n",
    "sales_univ_pair = from_checkpoint(\"2-clean-00-horizontal-equity\", mark_horizontal_equity_clusters_per_model_group_sup,\n",
    "    {\n",
    "        \"sup\": sales_univ_pair,\n",
    "        \"settings\": settings,\n",
    "        \"verbose\": verbose,\n",
    "        \"do_land_clusters\": True,\n",
    "        \"do_impr_clusters\": True\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67036e5e-468f-4ccd-a784-940d86c0c13a",
   "metadata": {},
   "source": [
    "## 5. Process sales\n",
    "\n",
    "We process sales validity information to set all the right codes for later use.  \n",
    "We calculate time trends for sales over time to generate time-adjusted sale prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "139bd8e6-38bd-499f-b68b-412febf914dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before univ merge len = 237834\n",
      "After univ merge len = 237834\n",
      "Using 78664 sales...\n",
      "--> 4039 vacant sales\n",
      "--> 74625 improved sales\n",
      "--> 72269 valid for ratio study\n",
      "--> 4039 valid for land ratio study\n",
      "len before validate = 78664\n",
      "Invalid sales validation filter disabled, skipping...\n",
      "len after validate = 78664\n",
      "len after hydrate = 78664\n",
      "Applying time adjustment...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jacks\\Documents\\Non-Game Stuff\\Programming\\openavmkit\\openavmkit\\time_adjustment.py:431: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_impr[\"sale_price_per_impr_sqft\"] = div_df_z_safe(\n",
      "C:\\Users\\jacks\\Documents\\Non-Game Stuff\\Programming\\openavmkit\\openavmkit\\time_adjustment.py:434: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_land[\"sale_price_per_land_sqft\"] = div_df_z_safe(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Using period: Q\n",
      "--> Crunching time adjustment...\n",
      "--> Flattening time adjustment...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jacks\\Documents\\Non-Game Stuff\\Programming\\openavmkit\\openavmkit\\time_adjustment.py:230: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df[\"sale_date\"] = pd.to_datetime(df[\"sale_date\"])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---->interpolating missing periods...\n",
      "--> Time adjustment calculated for 2083 days.\n",
      "len after enrich = 78664\n",
      "len after clip = 78664\n"
     ]
    }
   ],
   "source": [
    "sales_univ_pair = from_checkpoint(\"2-clean-01-process_sales\", process_sales,\n",
    "    {\n",
    "        \"sup\": sales_univ_pair,\n",
    "        \"settings\": load_settings(),\n",
    "        \"verbose\": verbose\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c331f62-f35b-47b4-8a2e-19149f95a2c5",
   "metadata": {},
   "source": [
    "## 6. Scrutinize sales\n",
    "\n",
    "We cluster all sales of similar properties in similar locations.  \n",
    "We flag individual sales that are anomalously high or low for their local cluster.  \n",
    "This helps us catch potentially invalid sales that slipped by the assessor's notice.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d1bedf35-32b7-4a9a-bf30-46950c270483",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UserWarning: You didn't provide a `deed_id` in `analysis.sales_scrutiny.deed_id`, so no deed-based sales validation heuristic can be run\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validating sales by heuristic, 78664 total sales\n",
      "--> 22132 bad keys for heuristic: flag_dupe_date_price\n",
      "--> 36 bad keys for heuristic: flag_false_vacant\n",
      "Dropped 22162 invalid sales keys identified by heuristic\n",
      "\n",
      "No manual exclusions file specified in settings, skipping manual exclusions\n",
      "Processing model group: single_family\n",
      "--> Flagged 7 vacant sales\n",
      "--> Flagged 4205 improved sales\n",
      "--> Unmarked sales before: 50877\n",
      "--> Unmarked sales after: 46665\n",
      "--> Marked 4212 new potentially invalid sales\n"
     ]
    }
   ],
   "source": [
    "sales_univ_pair = from_checkpoint(\"2-clean-02-sales-scrutiny\", run_sales_scrutiny,\n",
    "    {\n",
    "        \"sup\": sales_univ_pair, \n",
    "        \"settings\": load_settings(), \n",
    "        \"drop_cluster_outliers\": sales_scrutiny_drop_outliers, \n",
    "        \"drop_heuristic_outliers\": sales_scrutiny_drop_heuristics, \n",
    "        \"verbose\": verbose\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563e780e-8475-41c7-955e-2688c0837da0",
   "metadata": {},
   "source": [
    "# 7. Write out results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b0e5e1eb-b1bd-4b77-9e49-2a5e8e75c3fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...out/2-clean-sup.pickle\n",
      "...out/look/2-clean-universe.parquet\n",
      "...out/look/2-clean-sales.parquet\n",
      "...out/look/2-clean-sales-hydrated.parquet\n"
     ]
    }
   ],
   "source": [
    "write_notebook_output_sup(sales_univ_pair, \"2-clean\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7977159-a81f-4de2-b27e-1605f871ae6d",
   "metadata": {},
   "source": [
    "# 8. Look at it on a map!\n",
    "- Take the files output in the previous step and put them in a map viewer like QGIS, ArcGIS, or Felt\n",
    "- Look at them with your eyeballs\n",
    "- Make sure the data looks correct\n",
    "- If not, go back and fix it!\n",
    "- Don't proceed to the next step until everything looks right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8576dff4-3dd0-401b-b880-fcaffbc9cb46",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
