{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce053d77-e779-410c-b7ee-0abf0d457344",
   "metadata": {},
   "source": [
    "# Assemble\n",
    "\n",
    "This notebook loads and assembles all your basic data sources, including tabulor and geospatial data.\n",
    "\n",
    "The final output is two dataframes:\n",
    "\n",
    "- UNIVERSE\n",
    "- SALES\n",
    "\n",
    "The SALES dataframe represents transactions or parcels; these are ownership transfers with prices, dates, and metadata.  \n",
    "The UNIVERSE dataframe represents the parcels themselves (land and buildings, and their associated characteristics).\n",
    "\n",
    "These will be packaged together in a handy data structure called a `SalesUniversePair`, or `sup` for short. `openavmkit` provides many handy functions that carefully perform operations on `sup`s without mixing up their fields.\n",
    "\n",
    "The key thing to understand is that the **Assemble** notebook outputs a `sup` that represents *factual assertions* about the world. In later notebooks, we will have to add assumptions, opinions, and educated guesses, but we first will establish the firmest facts we can in this notebook.\n",
    "\n",
    "You can think of the two dataframes in the `sup` as answering the following questions:\n",
    "\n",
    "- UNIVERSE:\n",
    "  - Where is each parcel located in space, and what is its shape?\n",
    "  - What are the *current* characteristics of each parcel?\n",
    "    - Which parcels have buildings and which are vacant lots?\n",
    "    - How big is each parcel?\n",
    "    - What is the age/size/quality/condition/etc of each building?\n",
    "- SALES:\n",
    "  - Which parcels have sold?\n",
    "  - What prices did they sell for?\n",
    "  - What dates did they sell on?\n",
    "  - Which sales were valid?\n",
    "  - What characteristics were different *at the time of sale* from how the parcel is now?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "initial_id",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change these as desired\n",
    "\n",
    "# The slug of the locality you are currently working on\n",
    "locality = \"us-ky-louisville\"\n",
    "\n",
    "# Whether to print out a lot of stuff (can help with debugging) or stay mostly quiet\n",
    "verbose = True\n",
    "\n",
    "# Clear previous state for this notebook and start fresh\n",
    "clear_checkpoints = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b11d468-1d7d-43eb-81ed-a983c5fb78b7",
   "metadata": {},
   "source": [
    "# 1. Basic setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1414e0d-5738-4713-8b1d-e6d66930b286",
   "metadata": {},
   "outputs": [],
   "source": [
    "import init_notebooks\n",
    "init_notebooks.setup_environment()\n",
    "locality = init_notebooks.check_for_different_locality(locality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e670e4a6-4e2e-4d78-95bd-978a2bd12d98",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import OpenAVMkit:\n",
    "from openavmkit.pipeline import ( \n",
    "    init_notebook,\n",
    "    from_checkpoint,\n",
    "    delete_checkpoints,\n",
    "    examine_df,\n",
    "    examine_df_in_ridiculous_detail,\n",
    "    examine_sup,\n",
    "    examine_sup_in_ridiculous_detail,\n",
    "    cloud_sync,\n",
    "    load_settings,\n",
    "    load_dataframes,\n",
    "    process_dataframes,\n",
    "    process_sales,\n",
    "    enrich_sup_streets,\n",
    "    tag_model_groups_sup,\n",
    "    write_notebook_output_sup\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c38b672-1f29-4e9f-b32c-671810b1c721",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_notebook(locality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c05828b-a5bf-420f-8dfc-e1fe86f3afcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "if clear_checkpoints:\n",
    "    delete_checkpoints(\"1-assemble\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9b2b771-08a9-4512-9fa8-4dd2119c213c",
   "metadata": {},
   "source": [
    "# 2. Sync with Cloud\n",
    "- If you have configured cloud storage, syncs with your remote storage\n",
    "- Reconciles your local input files with the versions on the remote server\n",
    "- Pulls down whatever is newer from the remote server\n",
    "- Uploads whatever is newer from your local machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f4d61d-52f3-4c1c-9bef-d06a412f091e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cloud_sync(locality, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b73f042-a66d-46f8-b879-83ca5b9c21b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "settings = load_settings()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38e301d1-0416-4512-8da9-130569a2b1e0",
   "metadata": {},
   "source": [
    "# 3. Load & process data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0427dfa-883f-49a5-8344-353264eec4e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load all of our initial dataframes, but don't do anything with them just yet\n",
    "dataframes = from_checkpoint(\"1-assemble-01-load_dataframes\", load_dataframes,\n",
    "    {\n",
    "        \"settings\": load_settings(),\n",
    "        \"verbose\": verbose\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f5ce66d-8269-42c7-b60d-29c2363c1df8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# load all of our initial dataframes and assemble our data\n",
    "sales_univ_pair = from_checkpoint(\"1-assemble-02-process_data\", process_dataframes,\n",
    "    {\n",
    "        \"dataframes\": dataframes,\n",
    "        \"settings\": load_settings(), \n",
    "        \"verbose\": verbose\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c5e8e0d-d29f-4bcd-aadc-f502a3e21dea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate street frontages\n",
    "sales_univ_pair = from_checkpoint(\"1-assemble-03-enrich_streets\", enrich_sup_streets,\n",
    "    {\n",
    "        \"sup\": sales_univ_pair,\n",
    "        \"settings\":load_settings(), \n",
    "        \"verbose\":verbose\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4878531b-cbe9-401f-a6c1-ed10b9710599",
   "metadata": {},
   "source": [
    "# 4. Inspect results\n",
    "\n",
    "## 4.1 Examine\n",
    "\n",
    "- Run the next cell and look at the printed out results.\n",
    "- Note the \"Non-zero\" and \"Non-null\" columns in particular and make sure they're what you expect\n",
    "- This view is for a quick glance to get a good idea of what all your data is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e862e4b7-db7d-4662-8a1c-be900f0a12b2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "examine_sup(sales_univ_pair, load_settings())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15f9fd42-90eb-41c0-baa1-e417f3753890",
   "metadata": {},
   "source": [
    "## 4.2 Examine in ridiculous detail\n",
    "\n",
    "- You've looked, now LOOK AGAIN. This cell will run `describe()` for each numeric field and `value_counts()` for each categorical field.\n",
    "- Use this info to decide which variables are useful/useless\n",
    "- Consult this readout when you build your modeling group filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a9639c-48bf-4086-a114-78651516527f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "examine_sup_in_ridiculous_detail(sales_univ_pair, load_settings())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c35091f-a668-4661-af16-58ef31e1c1f0",
   "metadata": {},
   "source": [
    "## 4.3 Look at it on a map\n",
    "\n",
    "- Go to your `out/look/` folder\n",
    "- There should be parquets there\n",
    "- Drop them into ArcGIS, QGIS, or Felt\n",
    "- Look at your location fields and make sure they make sense"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2014e0b6-fe72-4d90-b294-fd141d608de1",
   "metadata": {},
   "source": [
    "# 5. Tag modeling groups\n",
    "- Separates rows into groups like \"single family\", \"townhomes\" and \"commercial\" as specified by the user\n",
    "- These groups will guide all further processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6b9d80b-4d56-4c87-ba14-eac97be442fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_univ_pair = from_checkpoint(\"1-assemble-04-tag_modeling_groups\", tag_model_groups_sup,\n",
    "    {\n",
    "        \"sup\": sales_univ_pair, \n",
    "        \"settings\": load_settings(), \n",
    "        \"verbose\": verbose\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc25049d-3eee-486b-996e-142f0701eb74",
   "metadata": {},
   "source": [
    "# 6. Write out results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "752ae7c9-b1a3-4694-8d4e-e8e0ebbedd56",
   "metadata": {},
   "outputs": [],
   "source": [
    "write_notebook_output_sup(\n",
    "    sales_univ_pair, \n",
    "    \"1-assemble\", \n",
    "    parquet=True, \n",
    "    gpkg=False, \n",
    "    shp=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f170f00-38bd-4f8c-a734-28fb0cd9a92f",
   "metadata": {},
   "source": [
    "# 7. Look at it on a map!\n",
    "- Take the files output in the previous step and put them in a map viewer like QGIS, ArcGIS, or Felt\n",
    "- Look at them with your eyeballs\n",
    "- Make sure the data looks correct\n",
    "- If not, go back and fix it!\n",
    "- Don't proceed to the next step until everything looks right"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
