{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp models.fm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FM\n",
    "> A pytorch implementation of Factorization Machine.\n",
    "\n",
    "Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions. Factorization Machine type algorithms are a combination of linear regression and matrix factorization, the cool idea behind this type of algorithm is it aims model interactions between features (a.k.a attributes, explanatory variables) using factorized parameters. By doing so it has the ability to estimate all interactions between features even with extremely sparse data.\n",
    "\n",
    "Factorization machines (FM) [Rendle, 2010], proposed by Steffen Rendle in 2010, is a supervised algorithm that can be used for classification, regression, and ranking tasks. It quickly took notice and became a popular and impactful method for making predictions and recommendations. Particularly, it is a generalization of the linear regression model and the matrix factorization model. Moreover, it is reminiscent of support vector machines with a polynomial kernel. The strengths of factorization machines over the linear regression and matrix factorization are: (1) it can model χ -way variable interactions, where χ is the number of polynomial order and is usually set to two. (2) A fast optimization algorithm associated with factorization machines can reduce the polynomial computation time to linear complexity, making it extremely efficient especially for high dimensional sparse inputs. For these reasons, factorization machines are widely employed in modern advertisement and products recommendations.\n",
    "\n",
    "Most recommendation problems assume that we have a consumption/rating dataset formed by a collection of *(user, item, rating*) tuples. This is the starting point for most variations of Collaborative Filtering algorithms and they have proven to yield nice results; however, in many applications, we have plenty of item metadata (tags, categories, genres) that can be used to make better predictions. This is one of the benefits of using Factorization Machines with feature-rich datasets, for which there is a natural way in which extra features can be included in the model and higher-order interactions can be modeled using the dimensionality parameter d. For sparse datasets, a second-order FM model suffices, since there is not enough information to estimate more complex interactions.\n",
    "\n",
    "![https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img4.png](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img4.png)\n",
    "\n",
    "$$f(x) = w_0 + \\sum_{p=1}^Pw_px_p + \\sum_{p=1}^{P-1}\\sum_{q=p+1}^Pw_{p,q}x_px_q$$\n",
    "\n",
    "This model formulation may look familiar — it's simply a quadratic linear regression. However, unlike polynomial linear models which estimate each interaction term separately, FMs instead use factorized interaction parameters: feature interaction weights are represented as the inner product of the two features' latent factor space embeddings:\n",
    "\n",
    "$$f(x) = w_0 + \\sum_{p=1}^Pw_px_p + \\sum_{p=1}^{P-1}\\sum_{q=p+1}^P\\langle v_p,v_q \\rangle x_px_q$$\n",
    "\n",
    "This greatly decreases the number of parameters to estimate while at the same time facilitating more accurate estimation by breaking the strict independence criteria between interaction terms. Consider a realistic recommendation data set with 1,000,000 users and 10,000 items. A quadratic linear model would need to estimate U + I + UI ~ 10 billion parameters. A FM model of dimension F=10 would need only U + I + F(U + I) ~ 11 million parameters. Additionally, many common MF algorithms (including SVD++, ALS) can be re-formulated as special cases of the more general/flexible FM model class.\n",
    "\n",
    "The above equation can be rewritten as:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\hat{y}(\\textbf{x}) = w_{0} + \\sum_{i=1}^{n} w_{i} x_{i} +  \\sum_{i=1}^n \\sum_{j=i+1}^n \\hat{w}_{ij} x_{i} x_{j}\n",
    "\\end{align*}$$\n",
    "\n",
    "where,\n",
    "\n",
    "- $w_0$ is the global bias\n",
    "- $w_i$ denotes the weight of the i-th feature,\n",
    "- $\\hat{w}_{ij} = v_i^Tv_j$ denotes the weight of the cross feature $x_ix_j$\n",
    "- $v_i \\in \\mathcal{R}^k$ denotes the embedding vector for feature $i$\n",
    "- $k$ denotes the size of embedding vector\n",
    "\n",
    "<aside>\n",
    "💡 For large, sparse datasets...FM and FFM is good. But for small, dense datasets...try to avoid.\n",
    "\n",
    "</aside>\n",
    "\n",
    "Factorization machines appeared to be the method which answered the challenge!\n",
    "\n",
    "|  | Accuracy | Speed | Sparsity |\n",
    "| --- | --- | --- | --- |\n",
    "| Collaborative Filtering | Too Accurate | Suitable | Suitable |\n",
    "| SVM | Too Accurate | Suitable | Unsuitable |\n",
    "| Random Forest/CART | General Accuracy | Unsuitable | Unsuitable |\n",
    "| Factorization Machines (FM) | General Accuracy | Quick | Designed for it |\n",
    "\n",
    "To learn the FM model, we can use the MSE loss for regression task, the cross entropy loss for classification tasks, and the BPR loss for ranking task. Standard optimizers such as SGD and Adam are viable for optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *\n",
    "from fastcore.nb_imports import *\n",
    "from fastcore.test import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import torch\n",
    "\n",
    "from recohut.models.layers.common import FeaturesEmbedding, FeaturesLinear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class FactorizationMachine(torch.nn.Module):\n",
    "\n",
    "    def __init__(self, reduce_sum=True):\n",
    "        super().__init__()\n",
    "        self.reduce_sum = reduce_sum\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        :param x: Float tensor of size ``(batch_size, num_fields, embed_dim)``\n",
    "        \"\"\"\n",
    "        square_of_sum = torch.sum(x, dim=1) ** 2\n",
    "        sum_of_square = torch.sum(x ** 2, dim=1)\n",
    "        ix = square_of_sum - sum_of_square\n",
    "        if self.reduce_sum:\n",
    "            ix = torch.sum(ix, dim=1, keepdim=True)\n",
    "        return 0.5 * ix\n",
    "\n",
    "class FM(torch.nn.Module):\n",
    "    \"\"\"\n",
    "    A pytorch implementation of Factorization Machine.\n",
    "    Reference:\n",
    "        S Rendle, Factorization Machines, 2010.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, field_dims, embed_dim):\n",
    "        super().__init__()\n",
    "        self.embedding = FeaturesEmbedding(field_dims, embed_dim)\n",
    "        self.linear = FeaturesLinear(field_dims)\n",
    "        self.fm = FactorizationMachine(reduce_sum=True)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        :param x: Long tensor of size ``(batch_size, num_fields)``\n",
    "        \"\"\"\n",
    "        x = self.linear(x) + self.fm(self.embedding(x))\n",
    "        return torch.sigmoid(x.squeeze(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **References:-**\n",
    "- S Rendle, Factorization Machines, 2010.\n",
    "- https://github.com/rixwew/pytorch-fm/blob/master/torchfm/model/fm.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "%reload_ext watermark\n",
    "%watermark -a \"Sparsh A.\" -m -iv -u -t -d -p recohut"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
