{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting votes\n",
    "> Let's see how how well votes of politicians in polls can be predicted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The strategy**:\n",
    "- first: only include a politician id and a poll id as features \n",
    "- second: include text features based on the poll title and or description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TL;DR**\n",
    "- using only politician id and poll id we find an 88% accuracy (over validation given random split) => individual outcome is highly associated with votes of others in the same poll"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**:\n",
    "- combine poll title and description for feature generation\n",
    "- try transformer based features\n",
    "- visualise most incorrect predicted polls and politicians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from fastai.tabular.all import (\n",
    "    Categorify,\n",
    "    CategoryBlock,\n",
    "    Normalize,\n",
    "    TabularPandas,\n",
    "    tabular_learner,\n",
    ")\n",
    "import polars as pl\n",
    "from bundestag.fine_logging import setup_logging\n",
    "import logging\n",
    "from bundestag.paths import get_paths\n",
    "from bundestag.data.transform.abgeordnetenwatch.transform import (\n",
    "    get_polls_parquet_path,\n",
    "    get_votes_parquet_path,\n",
    "    get_mandates_parquet_path,\n",
    ")\n",
    "from bundestag.ml.poll_clustering import SpacyTransformer, clean_text\n",
    "from bundestag.ml.vote_prediction import (\n",
    "    poll_splitter,\n",
    "    plot_predictions,\n",
    "    get_embeddings,\n",
    "    get_poll_proponents,\n",
    "    plot_poll_embeddings,\n",
    "    plot_politician_embeddings,\n",
    ")\n",
    "from plotnine import scale_color_manual\n",
    "from functools import partial\n",
    "\n",
    "logger = logging.getLogger(__name__)\n",
    "setup_logging(logging.DEBUG)\n",
    "\n",
    "paths = get_paths(\"../data\")\n",
    "paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading preprocessed dataframes (see `03_abgeordnetenwatch.ipynb`). First let's the votes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "legislature_id = 132\n",
    "file = get_votes_parquet_path(legislature_id, paths.preprocessed_abgeordnetenwatch)\n",
    "file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_votes = pl.read_parquet(file)\n",
    "df_all_votes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading further info on politicians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = get_mandates_parquet_path(legislature_id, paths.preprocessed_abgeordnetenwatch)\n",
    "file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_mandates = pl.read_parquet(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_mandates = df_mandates.with_columns(\n",
    "    **{\n",
    "        \"party\": pl.when(\n",
    "            pl.col(\"party\").is_in(pl.lit([\"DIE LINKE\", \"Die Linke. (Gruppe)\"]))\n",
    "        )\n",
    "        .then(pl.lit(\"Die Linke\"))\n",
    "        .otherwise(pl.col(\"party\"))\n",
    "    }\n",
    ").with_columns(\n",
    "    **{\n",
    "        \"party\": pl.when(pl.col(\"party\").is_in(pl.lit([\"DIE GRÜNEN\"])))\n",
    "        .then(pl.lit(\"BÜNDNIS 90/DIE GRÜNEN\"))\n",
    "        .otherwise(pl.col(\"party\"))\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_mandates[\"party\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading data on polls (description, title and so on)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = get_polls_parquet_path(legislature_id, paths.preprocessed_abgeordnetenwatch)\n",
    "file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_polls = pl.read_parquet(file)\n",
    "df_polls.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modelling using only poll and politician ids as features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split into train and validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating train / valid split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# splits = RandomSplitter(valid_pct=0.2)(df_all_votes)\n",
    "splits = poll_splitter(df_all_votes, valid_pct=0.2)\n",
    "splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting target variable and count frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_col = \"vote\"\n",
    "print(f\"target values: {df_all_votes[y_col].value_counts()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Final data preprocessing for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to = TabularPandas(\n",
    "    df_all_votes.to_pandas(),\n",
    "    cat_names=[\"politician name\", \"poll_id\"],\n",
    "    y_names=[y_col],\n",
    "    procs=[Categorify],\n",
    "    y_block=CategoryBlock,\n",
    "    splits=splits,\n",
    ")\n",
    "\n",
    "dls = to.dataloaders(bs=512)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding the learning rate for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = tabular_learner(dls)\n",
    "lrs = learn.lr_find()\n",
    "lrs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the artificial neural net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(5, lrs.valley)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_predictions(learn, df_all_votes, df_mandates, df_polls, splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "accuracy:\n",
    "- random split: 88% \n",
    "- poll based split: ~50%, politician embedding itself insufficient to reasonably predict vote"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting resulting embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.model.cpu()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = get_embeddings(learn)\n",
    "embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proponents = get_poll_proponents(df_all_votes, df_mandates)\n",
    "proponents.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings[\"politician name\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_pl = {\n",
    "    \"politician name\": pl.DataFrame(\n",
    "        {\n",
    "            \"politician name__emb_component_0\": embeddings[\"politician name\"][\n",
    "                \"politician name__emb_component_0\"\n",
    "            ],\n",
    "            \"politician name__emb_component_1\": embeddings[\"politician name\"][\n",
    "                \"politician name__emb_component_1\"\n",
    "            ],\n",
    "            \"politician name\": embeddings[\"politician name\"][\"politician name\"],\n",
    "        }\n",
    "    ),\n",
    "    \"poll_id\": pl.DataFrame(\n",
    "        {\n",
    "            \"poll_id__emb_component_0\": embeddings[\"poll_id\"][\n",
    "                \"poll_id__emb_component_0\"\n",
    "            ].values[1:],\n",
    "            \"poll_id__emb_component_1\": embeddings[\"poll_id\"][\n",
    "                \"poll_id__emb_component_1\"\n",
    "            ].values[1:],\n",
    "            \"poll_id\": [int(v) for v in embeddings[\"poll_id\"][\"poll_id\"].values[1:]],\n",
    "        }\n",
    "    ),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party_colors = scale_color_manual(\n",
    "    breaks=[\n",
    "        \"AfD\",\n",
    "        \"BSW (Gruppe)\",\n",
    "        \"BÜNDNIS 90/DIE GRÜNEN\",\n",
    "        \"CDU/CSU\",\n",
    "        \"Die Linke\",\n",
    "        \"FDP\",\n",
    "        \"fraktionslos\",\n",
    "        \"SPD\",\n",
    "    ],\n",
    "    values=[\"blue\", \"purple\", \"green\", \"black\", \"red\", \"yellow\", \"grey\", \"salmon\"],\n",
    ")\n",
    "\n",
    "plot_poll_embeddings(df_all_votes, df_polls, embeddings_pl, df_mandates, party_colors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_politician_embeddings(df_all_votes, df_mandates, embeddings_pl, party_colors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "embed scatters after pca:\n",
    "- poll based split => mandates form two groups\n",
    "- random split => polls and mandates each form 2-3 groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modelling using `poll_title`-based features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LDA topic weights as features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_col = \"poll_title\"\n",
    "nlp_col = f\"{source_col}_nlp_processed\"\n",
    "num_topics = 8\n",
    "\n",
    "st = SpacyTransformer()\n",
    "\n",
    "# load data and prepare text for modelling\n",
    "df_polls_lda = df_polls.with_columns(\n",
    "    **{\n",
    "        nlp_col: pl.col(source_col).map_elements(\n",
    "            partial(clean_text, nlp=st.nlp), return_dtype=pl.List(pl.String)\n",
    "        )\n",
    "    }\n",
    ")\n",
    "\n",
    "# modelling\n",
    "st.fit_lda(df_polls_lda[nlp_col].to_list(), num_topics=num_topics)\n",
    "\n",
    "# creating text features using fitted model\n",
    "df_polls_lda = st.transform(df_polls_lda, col=nlp_col)\n",
    "\n",
    "# inspecting\n",
    "display(df_polls_lda.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_votes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_input = df_all_votes.join(\n",
    "    df_polls_lda[[\"poll_id\"] + st.nlp_cols],\n",
    "    on=\"poll_id\",\n",
    ")\n",
    "df_input.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = poll_splitter(df_input, valid_pct=0.2)\n",
    "splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to = TabularPandas(\n",
    "    df_input.to_pandas(),\n",
    "    cat_names=[\n",
    "        \"politician name\",\n",
    "    ],  # 'poll_id'\n",
    "    cont_names=st.nlp_cols,  # using the new features\n",
    "    y_names=[y_col],\n",
    "    procs=[Categorify, Normalize],\n",
    "    y_block=CategoryBlock,\n",
    "    splits=splits,\n",
    ")\n",
    "\n",
    "dls = to.dataloaders(bs=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = tabular_learner(dls)\n",
    "lrs = learn.lr_find()\n",
    "lrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(\n",
    "    5,\n",
    "    #                     2e-2)\n",
    "    lrs.valley,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_predictions(learn, df_all_votes, df_mandates, df_polls, splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "poll_id split:\n",
    "- politician name + poll_id + 10 lda topics based on poll title do not improve the accuracy\n",
    "- politician name + <s>poll_id</s> + 5 lda topics based on poll title: ~49%\n",
    "- politician name + <s>poll_id</s> + 10 lda topics based on poll title: ~57%\n",
    "- politician name + <s>poll_id</s> + 25 lda topics based on poll title: ~45%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modelling using `poll_description`-based features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LDA topic weights as features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_col = \"poll_description\"\n",
    "nlp_col = f\"{source_col}_nlp_processed\"\n",
    "num_topics = 25\n",
    "\n",
    "st = SpacyTransformer()\n",
    "\n",
    "# load data and prepare text for modelling\n",
    "df_polls_lda = df_polls.with_columns(\n",
    "    **{\n",
    "        nlp_col: pl.col(source_col).map_elements(\n",
    "            partial(clean_text, nlp=st.nlp), return_dtype=pl.List(pl.String)\n",
    "        )\n",
    "    }\n",
    ")\n",
    "\n",
    "# modelling\n",
    "st.fit_lda(df_polls_lda[nlp_col].to_list(), num_topics=num_topics)\n",
    "\n",
    "# creating text features using fitted model\n",
    "df_polls_lda = df_polls_lda.pipe(\n",
    "    st.transform,\n",
    "    col=nlp_col,\n",
    ")\n",
    "\n",
    "# inspecting\n",
    "display(df_polls_lda.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_input = df_all_votes.join(\n",
    "    df_polls_lda[[\"poll_id\"] + st.nlp_cols],\n",
    "    on=\"poll_id\",\n",
    ")\n",
    "df_input.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = poll_splitter(df_input, valid_pct=0.2)\n",
    "splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to = TabularPandas(\n",
    "    df_input.to_pandas(),\n",
    "    cat_names=[\n",
    "        \"politician name\",\n",
    "    ],  # 'poll_id'\n",
    "    cont_names=st.nlp_cols,  # using the new features\n",
    "    y_names=[y_col],\n",
    "    procs=[Categorify, Normalize],\n",
    "    y_block=CategoryBlock,\n",
    "    splits=splits,\n",
    ")\n",
    "\n",
    "dls = to.dataloaders(bs=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = tabular_learner(dls)\n",
    "lrs = learn.lr_find()\n",
    "lrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(\n",
    "    5,\n",
    "    #                     2e-2)\n",
    "    lrs.valley,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_predictions(learn, df_all_votes, df_mandates, df_polls, splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "poll_id split:\n",
    "- politician name + <s>poll_id</s> + 5 lda topics based on poll description: ~51%\n",
    "- politician name + <s>poll_id</s> + 10 lda topics based on poll description: ~53%\n",
    "- politician name + <s>poll_id</s> + 20 lda topics based on poll description: ~56%\n",
    "- politician name + <s>poll_id</s> + 25 lda topics based on poll description: ~59%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "264.4px",
    "width": "476.4px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
