{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qEniIUh2yiGb"
      },
      "source": [
        "# Multi-class Classification Support\n",
        "\n",
        "[![Slides](https://img.shields.io/badge/🦌-ReHLine-blueviolet)](https://rehline-python.readthedocs.io/en/latest/)\n",
        "\n",
        "\n",
        "`PLQ_Ridge_Classifier` extends binary PLQ-ERM to multi-class problems via the `multi_class`\n",
        "parameter, supporting two standard decomposition strategies.\n",
        "\n",
        "**One-vs-Rest (OvR)** fits $K$ binary classifiers, one per class, each trained on the\n",
        "full dataset with relabelled targets ($+1$ for the class, $-1$ for all others).\n",
        "Prediction selects the class with the highest decision score:\n",
        "\n",
        "$$\\widehat{y} = \\arg\\max_{k \\in \\{1, \\dots, K\\}} f_k(\\mathbf{x})$$\n",
        "\n",
        "**One-vs-One (OvO)** fits $\\binom{K}{2}$ binary classifiers, one per class pair $(i, j)$,\n",
        "each trained only on samples belonging to those two classes.\n",
        "Prediction uses majority voting across all pairwise classifiers:\n",
        "\n",
        "$$\\widehat{y} = \\arg\\max_{k \\in \\{1, \\dots, K\\}} \\sum_{j \\neq k} \\mathbf{1}\\bigl[f_{kj}(\\mathbf{x}) > 0\\bigr]$$\n",
        "\n",
        "In both cases, each binary sub-problem is solved by a standard `PLQ_Ridge_Classifier`\n",
        "with the same loss, regularizer, and solver settings passed to the parent estimator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "W5JaV_p2Ztua"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from rehline import plq_Ridge_Classifier\n",
        "\n",
        "\n",
        "# generate data\n",
        "X_mc, y_mc = make_classification(\n",
        "    n_samples=10000, n_features=20, n_informative=10,\n",
        "    n_classes=4, n_clusters_per_class=1, random_state=42\n",
        ")\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_mc, y_mc, test_size=0.2, random_state=42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P_kBHXWG_qxa"
      },
      "source": [
        "### One-vs-Rest (OvR)\n",
        "\n",
        "\n",
        "In OvR, we train $K$ binary classifiers, one per class. Classifier $k$ learns to distinguish\n",
        "class $k$ from all other classes. The final prediction selects the class whose classifier\n",
        "reports the highest decision score:\n",
        "\n",
        "$$\\widehat{y} \\;=\\; \\arg\\max_{k \\in \\{1, \\dots, K\\}} f_k(\\mathbf{x})$$\n",
        "\n",
        "where $f_k(\\mathbf{x})$ is the signed distance from $\\mathbf{x}$ to the decision boundary of classifier $k$."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eNgPMJdctWyi",
        "outputId": "aef4af08-862f-457e-feed-aa6338f57d58"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "plq OvR accuracy: 0.7770\n"
          ]
        }
      ],
      "source": [
        "# predict using ovr method\n",
        "plq_ovr = plq_Ridge_Classifier(\n",
        "    loss={'name': 'svm'}, C=1.0,\n",
        "    fit_intercept=True, max_iter=50000, multi_class='ovr',\n",
        ")\n",
        "plq_ovr.fit(X_train, y_train)\n",
        "\n",
        "y_pred = plq_ovr.predict(X_test)\n",
        "print(f\"plq OvR accuracy: {accuracy_score(y_test, y_pred):.4f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 532
        },
        "id": "P1Fd-Jg5I3fv",
        "outputId": "61dac411-d294-4a23-e6ef-59f6e3dede82"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 219912 characters]",
            "text/plain": [
              "<Figure size 1400x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# results visualization\n",
        "pca = PCA(n_components=2, random_state=42)\n",
        "X_test_2d = pca.fit_transform(X_test)\n",
        "\n",
        "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
        "\n",
        "colors = [\"#4C72B0\", \"#DD8452\", \"#55A868\", \"#C44E52\"]\n",
        "ax = axes[0]\n",
        "for k, cls in enumerate(plq_ovr.classes_):\n",
        "    mask_correct = (y_test == cls) & (y_pred == cls)\n",
        "    mask_wrong   = (y_test == cls) & (y_pred != cls)\n",
        "    ax.scatter(X_test_2d[mask_correct, 0], X_test_2d[mask_correct, 1],\n",
        "               c=colors[k], label=f\"Class {cls}\", s=15, alpha=0.6)\n",
        "    ax.scatter(X_test_2d[mask_wrong, 0], X_test_2d[mask_wrong, 1],\n",
        "               c=colors[k], marker=\"x\", s=40, alpha=0.9)\n",
        "ax.set_title(\"OvR — PCA Projection\\n(× = misclassified)\", fontsize=12)\n",
        "ax.set_xlabel(\"PC 1\"); ax.set_ylabel(\"PC 2\")\n",
        "ax.legend(fontsize=9)\n",
        "\n",
        "cm = confusion_matrix(y_test, y_pred)\n",
        "ConfusionMatrixDisplay(cm, display_labels=plq_ovr.classes_).plot(\n",
        "    ax=axes[1], colorbar=False, cmap=\"Blues\"\n",
        ")\n",
        "axes[1].set_title(\"OvR — Confusion Matrix\", fontsize=12)\n",
        "\n",
        "plt.suptitle(f\"OvR Results\",\n",
        "             fontsize=14, y=1.02)\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bnl41NChAlUr"
      },
      "source": [
        "### One-vs-One (OvO)\n",
        "\n",
        "In OvO, we train $\\binom{K}{2}$ binary classifiers, one for each pair of classes $(i, j)$.\n",
        "Each classifier $f_{ij}$ votes for either class $i$ or class $j$. The final prediction\n",
        "is the class that receives the most votes:\n",
        "\n",
        "$$\\widehat{y} = \\arg\\max_{k \\in \\{1, \\dots, K\\}} \\sum_{j \\neq k} \\mathbf{1}\\bigl[f_{kj}(\\mathbf{x}) > 0\\bigr]$$\n",
        "\n",
        "where $\\mathbf{1}[\\cdot]$ is the indicator function, and $f_{kj}(\\mathbf{x}) > 0$ means\n",
        "classifier $(k, j)$ votes for class $k$."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s9nalNTit9Jl",
        "outputId": "c7c22b0b-50e3-4374-c7a0-344f88776d49"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "plq OvO accuracy: 0.8025\n"
          ]
        }
      ],
      "source": [
        "# predict using ovo method\n",
        "plq_ovo = plq_Ridge_Classifier(\n",
        "    loss={'name': 'svm'}, C=0.7,\n",
        "    fit_intercept=True, max_iter=50000, multi_class='ovo',\n",
        ")\n",
        "plq_ovo.fit(X_train, y_train)\n",
        "y_pred = plq_ovo.predict(X_test)\n",
        "\n",
        "print(f\"plq OvO accuracy: {accuracy_score(y_test, y_pred):.4f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 532
        },
        "id": "3Q4Y19nTI-w_",
        "outputId": "0de1d6b4-64ff-4f10-b14b-767ed12ae519"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 216908 characters]",
            "text/plain": [
              "<Figure size 1400x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# results visualization\n",
        "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
        "\n",
        "ax = axes[0]\n",
        "for k, cls in enumerate(plq_ovo.classes_):\n",
        "    mask_correct = (y_test == cls) & (y_pred == cls)\n",
        "    mask_wrong   = (y_test == cls) & (y_pred != cls)\n",
        "    ax.scatter(X_test_2d[mask_correct, 0], X_test_2d[mask_correct, 1],\n",
        "               c=colors[k], label=f\"Class {cls}\", s=15, alpha=0.6)\n",
        "    ax.scatter(X_test_2d[mask_wrong, 0], X_test_2d[mask_wrong, 1],\n",
        "               c=colors[k], marker=\"x\", s=40, alpha=0.9)\n",
        "ax.set_title(\"OvO — PCA Projection\\n(× = misclassified)\", fontsize=12)\n",
        "ax.set_xlabel(\"PC 1\"); ax.set_ylabel(\"PC 2\")\n",
        "ax.legend(fontsize=9)\n",
        "\n",
        "cm = confusion_matrix(y_test, y_pred)\n",
        "ConfusionMatrixDisplay(cm, display_labels=plq_ovo.classes_).plot(\n",
        "    ax=axes[1], colorbar=False, cmap=\"Blues\"\n",
        ")\n",
        "axes[1].set_title(\"OvO — Confusion Matrix\", fontsize=12)\n",
        "\n",
        "plt.suptitle(f\"OvO Results\",\n",
        "             fontsize=14, y=1.02)\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
