{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "04e3eceb-1900-4416-afcb-5e54b164d590",
   "metadata": {},
   "source": [
    "# Assessment Quality\n",
    "This notebook runs a few quick studies on already-cleaned and prepared data.\n",
    "\n",
    "You will need to have previously run the pipeline through the end of either 1-assemble or 2-clean.  \n",
    "Horizontal equity studies are only available if you have run through 2-clean, but the rest will work with just 1-assemble."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1add99ee-77ad-46d5-ae3e-72751ab29379",
   "metadata": {},
   "source": [
    "**User settings**\n",
    "\n",
    "Change the below to whatever you need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d207bf5-ef3e-4d5f-b32c-84753c4590a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The slug of the locality you are currently working on\n",
    "locality = \"us-nc-guilford\"\n",
    "\n",
    "# The model group we're operating on\n",
    "model_group = \"single_family\"\n",
    "\n",
    "# General settings:\n",
    "prediction_field = \"assr_market_value\"\n",
    "land_prediction_field = \"assr_land_value\"\n",
    "sales_field = \"sale_price\"\n",
    "\n",
    "# Study period:\n",
    "end_date = \"2025-01-01\"\n",
    "start_date = \"2024-01-01\"\n",
    "\n",
    "# Ratio study settings:\n",
    "max_trim = 0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "982a9304-7b86-4491-80a5-249716dc80a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import init_notebooks\n",
    "init_notebooks.setup_environment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78e7fa4a-26df-4c04-b83d-6cdc1d79bf91",
   "metadata": {},
   "source": [
    "# Calculations\n",
    "Don't change anything below this line. This does the actual calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4a9df73-baef-4db5-9d23-1c474b9805c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load functions from OpenAVMKit\n",
    "from openavmkit.pipeline import (\n",
    "    init_notebook,\n",
    "    read_sales_univ, \n",
    "    run_ratio_study, \n",
    "    run_horizontal_equity_study, \n",
    "    run_vertical_equity_study, \n",
    "    plot_prediction_vs_sales\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3175bcc5-28dd-4092-b7c2-619e03f48444",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_notebook(locality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd4a32cb-94c4-46a1-945a-93e623e17986",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openavmkit.pipeline import load_settings\n",
    "from openavmkit.data import read_predictions\n",
    "from openavmkit.filters import select_filter\n",
    "\n",
    "# Read our data\n",
    "try:\n",
    "    sales_univ_pair = read_sales_univ(\"out/look/3-spatial-lag-\")\n",
    "    print(\"Read files from notebook 3\")\n",
    "except ValueError as e:\n",
    "    try:\n",
    "        sales_univ_pair = read_sales_univ(\"out/look/2-clean-\")\n",
    "        print(\"Read files from notebook 2\")\n",
    "    except ValueError as e:\n",
    "        sales_univ_pair = read_sales_univ(\"out/look/1-assemble-\")\n",
    "        print(\"Error reading files from notebook 2.\\nBacking up to notebook 1, basic tests will work but horizontal equity won't be available.\")\n",
    "\n",
    "if prediction_field == \"prediction\":\n",
    "    df_sales = sales_univ_pair.sales\n",
    "    df_pred_sales = read_predictions(\"ensemble\", model_group, \"sales\")\n",
    "    df_sales = df_sales.merge(df_pred_sales, on=\"key_sale\", how=\"left\")\n",
    "    df_sales = df_sales[~df_sales[\"prediction\"].isna()]\n",
    "    if len(sales_filters) > 0:\n",
    "        df_sales = select_filter(df_sales, sales_filters)\n",
    "    sales_univ_pair.sales = df_sales\n",
    "    \n",
    "    df_univ = sales_univ_pair.universe\n",
    "    df_pred_univ = read_predictions(\"ensemble\", model_group, \"universe\")\n",
    "    df_univ = df_univ.merge(df_pred_univ, on=\"key\", how=\"left\")\n",
    "    df_univ = df_univ[~df_univ[\"prediction\"].isna()]\n",
    "    if len(univ_filters) > 0:\n",
    "        df_sales = select_filter(df_univ, univ_filters)\n",
    "    sales_univ_pair.universe = df_univ"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2224867-4ae3-4444-8d70-6a26bcdf0b59",
   "metadata": {},
   "source": [
    "## Accuracy -- Ratio Study"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56da9715-b51e-4aa3-809d-c7ed351d1c65",
   "metadata": {},
   "outputs": [],
   "source": [
    "ratio_study = run_ratio_study(\n",
    "    sales_univ_pair,\n",
    "    model_group,\n",
    "    prediction_field,\n",
    "    sales_field,\n",
    "    start_date,\n",
    "    end_date,\n",
    "    land_only = False,\n",
    "    max_trim = max_trim\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc7ffd6d-d754-4ba3-ae9a-f41afab6da07",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(ratio_study.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3af77c4-80f0-45bb-a196-60ddef7655b5",
   "metadata": {},
   "source": [
    "# Is that good?\n",
    "\n",
    "You want to see these values:\n",
    " - Median ratio within bounds\n",
    " - COD below target\n",
    " - If the main stat is out of range, you still pass if the target is within your 95% CI\n",
    " - IAAO standard grades you on the *trimmed* score\n",
    "\n",
    "**Example:**  \n",
    "Large to mid-sized single family residential improved should have a COD below `15`and a median ratio between `0.90` and `1.10`, according to IAAO guidelines.  \n",
    "Local standards vary widely; many insist on a median ratio between `0.95` to `1.05`, others deliberately target a low-ball figure of e.g. `0.9`, `.0.8`, `0.7`, etc.\n",
    "\n",
    "Things to consider:\n",
    "- Low COD (< 5) *can* indicate sales chasing, COD very close to 0 is particularly suspicious\n",
    "  - Horizontal equity study (below) can tell you more about that\n",
    "- Local standards often copy IAAO guidelines, but not always\n",
    "- Always look at untrimmed scores to keep yourself honest\n",
    "- **Always check local standards**\n",
    "\n",
    "\n",
    "Below you will find a summary of [IAAO Standard on ratio studies](https://www.iaao.org/wp-content/uploads/Standard_on_Ratio_Studies.pdf), Table 2-3, page 34.\n",
    "\n",
    "------\n",
    "\n",
    "<table style=\"margin-left:0; margin-right:auto;\">\n",
    "  <tr>\n",
    "      <th style=\"text-align:left;\">Jurisdiction Size/Profile/Market Activity</th>\n",
    "      <th style=\"text-align:left;\">Abbreviation</th>\n",
    "  </tr>\n",
    "  <tr>\n",
    "      <td>Very large jurisdictions/densely populated/newer properties/active markets</td>\n",
    "      <td>3</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "      <td>Large to mid-sized jurisdictions/older & newer properties/less active markets</td>\n",
    "      <td>2</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "      <td>Rural or small jurisdictions/little development/depressed markets</td>\n",
    "      <td>1</td>\n",
    "  </tr>\n",
    "</table>\n",
    "\n",
    "<table style=\"margin-left:0; margin-right:auto;\">\n",
    "    <tr>\n",
    "        <th style=\"text-align:left;\">General Property Class</th>\n",
    "        <th style=\"text-align:left;\">Examples</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>Residential Improved</td>\n",
    "        <td>single family dwellings, condominiums, manuf. housing, 2-4 family units</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>Income-producing properties</td>\n",
    "        <td>commercial, industrial, apartments</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "*\"The COD performance recommendations are based upon representative and adequate sample sizes, with outliers trimmed and a 95%\n",
    "level of confidence.\"*\n",
    "\n",
    "<table style=\"margin-left:0; margin-right:auto;\">\n",
    "  <tr><th style=\"text-align:left;\">Class + Size/Profile/Activity</th><th style=\"text-align:left;\">COD Range</th></tr>\n",
    "  <tr><td>Residential improved</td><td></td></tr>\n",
    "  <tr><td>3</td><td>5.0 - 10.0</td></tr>\n",
    "  <tr><td>2</td><td>5.0 - 15.0</td></tr>\n",
    "  <tr><td>1</td><td>5.0 - 20.0</td></tr>\n",
    "  <tr><td>Income-producing properties</td><td></td></tr>\n",
    "  <tr><td>3</td><td>5.0 - 15.0</td></tr>\n",
    "  <tr><td>2</td><td>5.0 - 20.0</td></tr>\n",
    "  <tr><td>1</td><td>5.0 - 25.0</td></tr>\n",
    "  <tr><td>Residential vacant land</td><td></td></tr>\n",
    "  <tr><td>3</td><td>5.0 - 15.0</td></tr>\n",
    "  <tr><td>2</td><td>5.0 - 20.0</td></tr>\n",
    "  <tr><td>1</td><td>5.0 - 25.0</td></tr>\n",
    "  <tr><td>Other (non-agricultural) vacant land</td><td></tr>\n",
    "  <tr><td>3</td><td>5.0 - 20.0</td></tr>\n",
    "  <tr><td>2</td><td>5.0 - 25.0</td></tr>\n",
    "  <tr><td>1</td><td>5.0 - 30.0</td></tr>\n",
    "</table>\n",
    "\n",
    "*\"These types of property are provided for general guidance only and may not represent jurisdictional requirements\"*  \n",
    "*\"CODs lower than 5.0 may indicate sales chasing or non-representative samples\"*  \n",
    "*\"Appraisal level recommendation for each type of property shown should be between `0.90` and `1.10`\"*  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a473ebd-3dcf-4f76-9b38-eb6034aad9d8",
   "metadata": {},
   "source": [
    "## Consistency -- Horizontal Equity\n",
    "\n",
    "If you have run notebook 2, this part will tell you about the *consistency* of your valuations, regardless of how close they are to sale prices.  \n",
    "If you have not run notebook 2, this part will be skipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39a15762-edc8-4d9b-ae5a-f34ffd546a46",
   "metadata": {},
   "outputs": [],
   "source": [
    "he_study = run_horizontal_equity_study(\n",
    "    sales_univ_pair, \n",
    "    model_group, \n",
    "    prediction_field\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a64842d6-4b36-46d5-984b-5704d68072d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "if he_study is not None:\n",
    "    display(he_study.summary.print())\n",
    "else:\n",
    "    print(\"No horizontal equity id found, skipping...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba0704a-b7bd-470f-bb26-44a25a466261",
   "metadata": {},
   "source": [
    "## Detect sales chasing\n",
    "With a horizontal equity study in hand, we can check for sales chasing by seeing if any clusters that include sales have very different valuations from physically similar properties"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0169e300-2b48-4537-9a85-b90c8a2db8d7",
   "metadata": {},
   "source": [
    "# Is that good?\n",
    "\n",
    "OpenAVMKit's horizontal equity study is not an IAAO standard and is not written into in any local standards that we know of.\n",
    "\n",
    "Here are our own recommendations:\n",
    "\n",
    "- Median CHD should be about the same or less than the IAAO COD guidelines for the property type (e.g. `15` for typical single family)\n",
    "- 75th %ile CHD should not be crazy high\n",
    "- A very low COD (< 5) *and* a high Median CHD, is a strong indicator of sales chasing\n",
    "  - This is because properties that sold are basically copying the sale price, whereas unsold properties have wildly different valuations, even when their characteristics and locations are similar\n",
    "\n",
    "The test works by dividing your model group into clusters of similarly-located, physically similar properties (e.g. same land use, same location, same size, same type, same age, etc). You would naturally expect similar valuations for such properties.\n",
    "\n",
    "- The CHD measures variation in valuation within a cluster\n",
    "- It has nothing to do with sale prices\n",
    "- It has nothing to do with accuracy\n",
    "- It only measures horizontal uniformity (consistency)\n",
    "\n",
    "The test is not completely objective because drawing clusters is always subjective. This test is only as good as the quality of the clusters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eea4c8f-16ac-4555-a8d1-e461001c63b7",
   "metadata": {},
   "source": [
    "## Fairness -- Vertical Equity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afa9ae5a-bff2-4b2b-b095-a06f2d161f0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ve_study = run_vertical_equity_study(\n",
    "    sales_univ_pair,\n",
    "    model_group,\n",
    "    prediction_field,\n",
    "    sales_field,\n",
    "    \"census_tract\",\n",
    "    start_date,\n",
    "    end_date\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1376d91a-6be0-4a8b-acb9-a734f44b133d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ve_study.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "506d85c2-d23f-4faf-b07d-d9896a877fdb",
   "metadata": {},
   "source": [
    "# Is that good?\n",
    "\n",
    "IAAO guidelines:\n",
    "- PRD:\n",
    "  - fails if outside `0.98` and `1.03` \n",
    "  - passes if 95% CI is within tolerance range\n",
    "  - if 95% CI overlaps 1.0, not statistically significant evidence of vertical inequity bias\n",
    "- PRB:\n",
    "  - fails if outside `-0.10` to `0.10`\n",
    "  - passes if 95% CI is within tolerance range\n",
    "  - should ideally fall between `-0.05` and `0.05`\n",
    "  - if 95% CI overlaps 0.0, not statistically significant evidence of vertical inequity bias\n",
    "- PRB is generally preferred to PRD\n",
    "- **always consult your local standards**\n",
    "\n",
    "Quotes from the IAAO standard on ratio studies:\n",
    "\n",
    "*\"As a general matter, the PRB coefficient should fall between –0.05 and 0.05. PRBs for which 95% confidence intervals fall outside of this range indicate that one can reasonably conclude that assessment levels change by more than 5% when values are halved or doubled.\"*\n",
    "\n",
    "*\"PRBs for which 95% confidence intervals fall outside the range of –0.10 to 0.10 indicate unacceptable vertical inequities\"*\n",
    "\n",
    "*\"PRD standards are not absolute and may be less meaningful when samples are small or when wide variation in prices exist. In such cases, statistical tests of vertical equity hypotheses should be substituted.\"*\n",
    "\n",
    "*\"Alternatively, assessing officials can rely on the PRB, which is less sensitive to atypical prices and ratios\"*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba482ba8-1109-4c9a-bcd0-c01699d775b0",
   "metadata": {},
   "source": [
    "# Visualizations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e739ec8-524a-4bda-93d4-be4dedf6e519",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_prediction_vs_sales(\n",
    "    sales_univ_pair,\n",
    "    model_group,\n",
    "    prediction_field,\n",
    "    sales_field,\n",
    "    start_date,\n",
    "    end_date\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d2749e-487e-4898-be02-2fa56a469675",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plot_prediction_vs_sales(\n",
    "    sales_univ_pair,\n",
    "    model_group,\n",
    "    land_prediction_field,\n",
    "    sales_field,\n",
    "    start_date,\n",
    "    end_date,\n",
    "    land_only=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9204e6f-7a37-4f0b-b1a1-3c1858e9293b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ve_study.plot_quantiles(ylim=None)\n",
    "ve_study.plot_quantiles(ylim=[0.6,0.8], ci_bounds=True)\n",
    "ve_study.plot_quantiles(ylim=None, grouped=True)\n",
    "ve_study.plot_quantiles(ylim=[0.6,0.8], ci_bounds=True, grouped=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9754a16f-573a-444d-95fb-f29e585702f4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (my_library_env)",
   "language": "python",
   "name": "my_library_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
