{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysing \"Namentliche Abstimmungen\" in the Bundestag\n",
    "\n",
    "> How close, in terms of roll call votes, are members of the parliament to the present parties? How predictable are individual votes for given polls? Answers to these questions can be found here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data sources\n",
    "\n",
    "Bundestag page `https://www.bundestag.de/parlament/plenum/abstimmung/liste`. Contains roll call votes with information on presence / absence and vote (yes/no/abstain) for each member of the Bundestag over a longer time period.\n",
    "\n",
    "`abgeordnetenwatch.de` API `https://www.abgeordnetenwatch.de/api` (they also have a GUI [here](https://www.abgeordnetenwatch.de)). Contains information on politicians, parliaments, legislative periods and mandates including and beyond the Bundestag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import polars as pl\n",
    "from fastai.tabular.all import (\n",
    "    Categorify,\n",
    "    CategoryBlock,\n",
    "    RandomSplitter,\n",
    "    TabularPandas,\n",
    "    tabular_learner,\n",
    ")\n",
    "from functools import partial\n",
    "from rich import print as pprint\n",
    "\n",
    "from bundestag.fine_logging import setup_logging\n",
    "from bundestag.paths import get_paths\n",
    "from bundestag.ml.similarity import (\n",
    "    get_votes_by_party,\n",
    "    pivot_party_votes_df,\n",
    "    prepare_votes_of_mdb,\n",
    "    align_mdb_with_parties,\n",
    "    compute_similarity,\n",
    "    plot,\n",
    "    align_party_with_all_parties,\n",
    ")\n",
    "from bundestag.ml.poll_clustering import SpacyTransformer, clean_text\n",
    "from bundestag.ml.vote_prediction import (\n",
    "    plot_predictions,\n",
    "    get_embeddings,\n",
    "    plot_politician_embeddings,\n",
    "    plot_poll_embeddings,\n",
    ")\n",
    "from bundestag.gui import MdBGUI, PartyGUI\n",
    "import logging\n",
    "from plotnine import (\n",
    "    scale_color_manual,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logger = logging.getLogger(__name__)\n",
    "setup_logging()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if this notebook is run via `make docs` then the environment variable is set\n",
    "makedocs = os.getenv(\"MAKEDOCS\") is not None\n",
    "logger.info(f\"Running nb with {makedocs=}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comment-in the below cell to download prepared data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import bundestag.data.download.huggingface as download_hf\n",
    "# download_hf.run(Path(\"../data\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1 - Who is similar to whom?\n",
    "> Which party does a politicians voting behavior align the most with? Which parties are closest in terms of cast votes?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading the data**\n",
    "\n",
    "If you have cloned the repo you should already have a `bundestag.de_votes.parquet` file in the root directory of the repo. If not feel free to download that file directly.\n",
    "\n",
    "If you want to have a closer look at the preprocessing please check out `nbs/00_html_parsing.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_paths = get_paths(Path(\"../data\"))\n",
    "_fig_path = Path(\"./images\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = _paths.preprocessed_bundestag / \"bundestag.de_votes.parquet\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pl.read_parquet(file)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Votes by party"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party_votes = get_votes_by_party(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party_votes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-arranging `party_votes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party_votes_pivoted = pivot_party_votes_df(party_votes)\n",
    "party_votes_pivoted.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Which party does a politicians voting behavior align the most with?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collecting the politicians votes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdb = \"Peter Altmaier\"\n",
    "mdb_votes = prepare_votes_of_mdb(df, mdb)\n",
    "mdb_votes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing the politician against the parties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdb_and_parties = align_mdb_with_parties(mdb_votes, party_votes_pivoted)\n",
    "mdb_and_parties.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdb_vs_parties = compute_similarity(mdb_and_parties, \"_party\")\n",
    "mdb_vs_parties.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdb_vs_parties[\"Fraktion/Gruppe_party\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(\n",
    "    mdb_vs_parties,\n",
    "    title_overall=f\"Overall similarity of {mdb} with all parties\",\n",
    "    title_over_time=f\"{mdb} vs time\",\n",
    ")\n",
    "plt.tight_layout()\n",
    "if makedocs:\n",
    "    plt.savefig(_fig_path / \"mdb_similarity_vs_time.png\")\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/mdb_similarity_vs_time.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Which parties are closest in terms of cast votes?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collecting party votes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party_votes_pivoted.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party = \"SPD\"\n",
    "partyA_vs_rest = align_party_with_all_parties(party_votes_pivoted, party)\n",
    "partyA_vs_rest.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "partyA_vs_rest = compute_similarity(partyA_vs_rest, suffix=\"_b\")\n",
    "\n",
    "partyA_vs_rest.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(\n",
    "    partyA_vs_rest,\n",
    "    title_overall=f\"Overall similarity of {party} with all parties\",\n",
    "    title_over_time=f\"{party} vs time\",\n",
    "    party_col=\"Fraktion/Gruppe_b\",\n",
    ")\n",
    "plt.tight_layout()\n",
    "if makedocs:\n",
    "    plt.savefig(_fig_path / \"party_similarity_vs_time.png\")\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/party_similarity_vs_time.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**GUI to inspect similarities**\n",
    "\n",
    "To make the above exploration more interactive, the class `MdBGUI` and `PartyGUI` was implemented to quickly go through the different parties and politicians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdb = MdBGUI(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not makedocs:\n",
    "    display(mdb.render())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party = PartyGUI(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not makedocs:\n",
    "    display(party.render())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2 - How will a politician cast his or her vote?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data used below was processed using `nbs/03_abgeordnetenwatch.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = _paths.preprocessed_abgeordnetenwatch\n",
    "legislature_id = 111\n",
    "file = path / f\"polls_{legislature_id}.parquet\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_polls = pl.read_parquet(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clustering polls using Latent Dirichlet Allocation (LDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_col = \"poll_title\"\n",
    "nlp_col = f\"{source_col}_nlp_processed\"\n",
    "num_topics = 5  # number of topics / clusters to identify\n",
    "\n",
    "st = SpacyTransformer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_polls.head()[source_col].map_elements(\n",
    "    partial(clean_text, nlp=st.nlp), return_dtype=pl.List(pl.String)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data and prepare text for modelling\n",
    "df_polls_lda = df_polls.with_columns(\n",
    "    **{\n",
    "        nlp_col: pl.col(source_col).map_elements(\n",
    "            partial(clean_text, nlp=st.nlp), return_dtype=pl.List(pl.String)\n",
    "        )\n",
    "    }\n",
    ")\n",
    "df_polls_lda.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# modelling clusters\n",
    "st.fit_lda(df_polls_lda[nlp_col].to_list(), num_topics=num_topics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating text features using fitted model\n",
    "df_polls_lda, nlp_feature_cols = st.transform(\n",
    "    df_polls_lda, col=nlp_col, return_new_cols=True\n",
    ")\n",
    "\n",
    "# inspecting clusters\n",
    "display(df_polls_lda.head(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st.lda_topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predicting votes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_votes = pl.read_parquet(path / f\"votes_{legislature_id}.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_mandates = pl.read_parquet(path / f\"mandates_{legislature_id}.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting data set into training and validation set. Splitting randomly here because it leads to an interesting result, albeit not very realistic for production."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = RandomSplitter(valid_pct=0.2)(df_all_votes)\n",
    "splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_col = \"vote\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training a neural net to predict `vote` based on embeddings for `poll_id` and `politician name`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_votes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to = TabularPandas(\n",
    "    df_all_votes.to_pandas(),\n",
    "    cat_names=[\n",
    "        \"politician name\",\n",
    "        \"poll_id\",\n",
    "    ],  # columns in `df_all_votes` to treat as categorical\n",
    "    y_names=[y_col],  # column to use as a target for the model in `learn`\n",
    "    procs=[Categorify],  # processing of features\n",
    "    y_block=CategoryBlock,  # how to treat `y_names`, here as categories\n",
    "    splits=splits,\n",
    ")  # how to split the data\n",
    "\n",
    "dls = to.dataloaders(bs=512)\n",
    "learn = tabular_learner(dls)  # fastai function to set up a neural net for tabular data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lrs = learn.lr_find()  # searches the learning rate\n",
    "pprint(lrs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(\n",
    "    5, lrs.valley\n",
    ")  # performs training using one-cycle hyperparameter schedule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Predictions over unseen data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspecting the predictions of the neural net over the validation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_mandates.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_votes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_predictions(\n",
    "    learn, df_all_votes, df_mandates, df_polls, splits, n_worst_politicians=5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting our dataset randomly leads to a surprisingly good accuracy of ~88% over the validation set. The most reasonable explanation is that the model encountered polls and how most politicians voted for them already during training. \n",
    "\n",
    "This can be interpreted as, if it is known how most politicians will vote during a poll, then the vote of the remaining politicians is highly predictable. Splitting the data set by `poll_id`, as can be done using `vp.poll_splitter` leads to random chance predictions. Anything else would be surprising as well since the only available information provided to the model is who is voting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Visualising learned embeddings**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides the actual prediction it also is interesting to inspect what the model actually learned. This can sometimes lead to [surprises](https://github.com/entron/entity-embedding-rossmann).\n",
    "\n",
    "So let's look at the learned embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.model.cpu()\n",
    "embeddings = get_embeddings(learn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make sense of the embeddings for `poll_id` as well as `politician name` we apply Principal Component Analysis (so one still kind of understands what distances mean) and project down to 2d. \n",
    "\n",
    "Using the information which party was most strongly (% of their votes being \"yes\"), so its strongest proponent, we color code the individual polls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_pl = {\n",
    "    \"politician name\": pl.DataFrame(\n",
    "        {\n",
    "            \"politician name__emb_component_0\": embeddings[\"politician name\"][\n",
    "                \"politician name__emb_component_0\"\n",
    "            ],\n",
    "            \"politician name__emb_component_1\": embeddings[\"politician name\"][\n",
    "                \"politician name__emb_component_1\"\n",
    "            ],\n",
    "            \"politician name\": embeddings[\"politician name\"][\"politician name\"],\n",
    "        }\n",
    "    ),\n",
    "    \"poll_id\": pl.DataFrame(\n",
    "        {\n",
    "            \"poll_id__emb_component_0\": embeddings[\"poll_id\"][\n",
    "                \"poll_id__emb_component_0\"\n",
    "            ].values[1:],\n",
    "            \"poll_id__emb_component_1\": embeddings[\"poll_id\"][\n",
    "                \"poll_id__emb_component_1\"\n",
    "            ].values[1:],\n",
    "            \"poll_id\": [int(v) for v in embeddings[\"poll_id\"][\"poll_id\"].values[1:]],\n",
    "        }\n",
    "    ),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_mandates[\"party\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "party_colors = scale_color_manual(\n",
    "    breaks=[\n",
    "        \"AfD\",\n",
    "        \"BSW\",\n",
    "        \"DIE GRÜNEN\",\n",
    "        \"CDU/CSU\",\n",
    "        \"DIE LINKE\",\n",
    "        \"FDP\",\n",
    "        \"fraktionslos\",\n",
    "        \"SPD\",\n",
    "    ],\n",
    "    values=[\"blue\", \"purple\", \"green\", \"black\", \"red\", \"yellow\", \"grey\", \"salmon\"],\n",
    ")\n",
    "\n",
    "fig = plot_poll_embeddings(\n",
    "    df_all_votes, df_polls, embeddings_pl, df_mandates=df_mandates, colors=party_colors\n",
    ")\n",
    "fig.show()\n",
    "if makedocs:\n",
    "    fig.save(_fig_path / \"poll_embeddings.png\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/poll_embeddings.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The politician embeddings are color coded using the politician's party membership"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plot_politician_embeddings(df_all_votes, df_mandates, embeddings_pl, party_colors)\n",
    "fig.show()\n",
    "if makedocs:\n",
    "    fig.save(_fig_path / \"mandate_embeddings.png\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/mandate_embeddings.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The politician embeddings may be the most surprising finding in its clarity. It seems we find for polls and politicians 2-3 clusters, but for politicians with a significant grouping of mandates associated with the government coalition. It seems we find one cluster for the government parties and one for the government opposition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
