{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning word embeddings - word2vec\n",
    "\n",
    "\\- [Saurabh Mathur](https://saurabhmathur96.github.io/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aim of this experiment is to use the algorithm developed by [*Tomas Mikolov et al.*](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) to learn high quality vector representations of text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The skip-gram model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given,\n",
    "\n",
    "a sequence of words $ w_1, w_2, .., w_T $, predict the next word.\n",
    "\n",
    "The objective is to maximize average log probability.\n",
    "\n",
    "\n",
    "$$ AverageLogProbability = \\frac{1}{T} \\sum_{t=1}^{T} \\sum_{-c \\leqslant j\\leqslant c, j \\neq 0} log\\ p (w_{t+j} | w_t) $$\n",
    "\n",
    "where $ c $ is the length of context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic skip-gram model\n",
    "\n",
    "The basic skip-gram formulation defines $ p (w_{t+j} | w_t) $ in terms of softmax as -\n",
    "\n",
    "$$ p (wo | wi) = \\frac{ exp(v'^{T} _{wo} \\cdot v_{wi})  }{ \\sum^{W}_{w=1}  exp(v'^{T} _{w} \\cdot v_{wi} ) } $$\n",
    "\n",
    "where $vi$ and $vo$ are input and output word vectors.\n",
    "\n",
    "This is extremely costly and this impractical as, W is huge ( ~ $10^5-10^7$ terms ).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three proposed methods to get around this limitation.\n",
    "- Heirarchial softmax\n",
    "- Negative sampling\n",
    "- Subsample frequent words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm using Google's Tensorflow library for the implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the data, I'm using the [text8 dataset](http://mattmahoney.net/dc/textdata) which is a 100MB sample of cleaned English Wikipedia dump on Mar. 3, 2006"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os, urllib\n",
    "\n",
    "    \n",
    "def fetch_data(url):\n",
    "    \n",
    "    filename = url.split(\"/\")[-1]\n",
    "    datadir = os.path.join(os.getcwd(), \"data\")\n",
    "    filepath = os.path.join(datadir, filename)\n",
    "    \n",
    "    if not os.path.exists(datadir):\n",
    "        os.makedirs(datadir)\n",
    "    if not os.path.exists(filepath):\n",
    "        urllib.urlretrieve(url, filepath)\n",
    "    \n",
    "    return filepath\n",
    "\n",
    "url = \"http://mattmahoney.net/dc/text8.zip\"\n",
    "filepath = fetch_data(url)\n",
    "print (\"Data at {0}.\".format(filepath))\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os, zipfile\n",
    "\n",
    "def read_data(filename):\n",
    "    with zipfile.ZipFile(filename) as f:\n",
    "        data = tf.compat.as_str(f.read(f.namelist()[0])).split()\n",
    "    return data\n",
    "\n",
    "\n",
    "words = read_data(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take only the top $c$ words, mark rest as UNK (unknown)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_dataset(words, vocabulary_size):\n",
    "    count = [[ \"UNK\", -1 ]].extend(\n",
    "        collections.Counter(words).most_common(vocabulary_size))\n",
    "    word_to_index = { word: i for i, (word, _) in enumerate(count) }\n",
    "    data = [word_to_index.get(word, 0) for word in words] # map unknown words to 0\n",
    "    unk_count = data.count(0) # Number of unknown words\n",
    "    count[0][1] = unk_count\n",
    "    index_to_word= dict(zip(word_to_index.values(), word_to_index.keys()))\n",
    "    \n",
    "    return data, count, word_to_index, index_to_word\n",
    "\n",
    "vocabulary_size = 50000\n",
    "data, count, word_to_index, index_to_word = build_dataset(words, vocabulary_size)\n",
    "print (\"data: {0}\".format(data[:5]))\n",
    "print (\"count: {0}\".format(count[:5]))\n",
    "print (\"word_to_index: {0}\".format(word_to_index.items()[:5]))\n",
    "print (\"index_to_word: {0}\".format(index_to_word.items()[:5]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
