{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Tutorial 2: ACS 5-Year Aggregate Data\n",
                "\n",
                "This tutorial covers the most common use case: fetching aggregate statistics from the ACS 5-Year estimates.\n",
                "\n",
                "**Goal:** Get poverty statistics for all places (cities, towns, CDPs) in a state."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Setup"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import os\n",
                "from cendat import CenDatHelper\n",
                "from dotenv import load_dotenv\n",
                "\n",
                "load_dotenv()\n",
                "cdh = CenDatHelper(key=os.getenv(\"CENSUS_API_KEY\"))"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 1: Find the ACS 5-Year Product"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# The \\) at the end matches products ending with a closing paren,\n",
                "# which filters out sub-products like /profile, /subject, etc.\n",
                "cdh.list_products(years=[2023], patterns=r\"acs/acs5\\)\")\n",
                "cdh.set_products()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 2: Explore Variable Groups\n",
                "\n",
                "For products like ACS with thousands of variables, groups are essential:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Search for poverty-related groups\n",
                "cdh.list_groups(patterns=\"poverty\")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Let's use B17001 (Poverty Status by Sex by Age)\n",
                "cdh.set_groups(\"B17001\")\n",
                "\n",
                "# See what variables are in this group\n",
                "cdh.describe_groups()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 3: Select Variables and Geography"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# B17001_001E = Total population for poverty calculation\n",
                "# B17001_002E = Population below poverty level\n",
                "cdh.set_variables([\"B17001_001E\", \"B17001_002E\"])\n",
                "\n",
                "# 160 = Places (cities, towns, CDPs)\n",
                "cdh.set_geos([\"160\"])"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 4: Get Data with Names"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "response = cdh.get_data(\n",
                "    include_names=True,      # Include place names\n",
                "    include_attributes=True  # Include margins of error\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 5: Analyze"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Convert to DataFrame\n",
                "df = response.to_polars(concat=True, destring=True)\n",
                "df.glimpse()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Quick tabulation: how many places have >10,000 population?\n",
                "response.tabulate(\"state\", where=\"B17001_001E > 10_000\")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Weighted by population\n",
                "response.tabulate(\n",
                "    \"state\",\n",
                "    weight_var=\"B17001_001E\",\n",
                "    where=\"B17001_001E > 10_000\"\n",
                ")"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "name": "python",
            "version": "3.11.0"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 4
}