{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent neural network\n",
    "\n",
    "Malaya use [Long-Short-Term-Memory](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) for all RNN gates.\n",
    "\n",
    "LSTM References:\n",
    "\n",
    "1. Hochreiter, Sepp; Schmidhuber, Jürgen (1997-11-01). \"Long Short-Term Memory\". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Malaya use recurrent neural network architecture on some models.\n",
    "\n",
    "#### Emotion Analysis\n",
    "\n",
    "1. `malaya.emotion.deep_model('bahdanau')`\n",
    "2. `malaya.emotion.deep_model('luong')`\n",
    "\n",
    "#### Sentiment Analysis\n",
    "\n",
    "1. `malaya.sentiment.deep_model('bahdanau')`\n",
    "2. `malaya.sentiment.deep_model('luong')`\n",
    "\n",
    "#### Subjectivity Analysis\n",
    "\n",
    "1. `malaya.subjective.deep_model('bahdanau')`\n",
    "2. `malaya.subjective.deep_model('luong')`\n",
    "\n",
    "#### Toxicity Analysis\n",
    "\n",
    "1. `malaya.toxic.deep_model('bahdanau')`\n",
    "2. `malaya.toxic.deep_model('luong')`\n",
    "\n",
    "You can read more about [Recurrent Neural Network here](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).\n",
    "\n",
    "#### References\n",
    "\n",
    "1. Li, Xiangang; Wu, Xihong (2014-10-15). \"Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition\". arXiv:1410.4281 [cs.CL].\n",
    "\n",
    "2. Hochreiter, Sepp; Schmidhuber, Jürgen (1997-11-01). \"Long Short-Term Memory\". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735.\n",
    "\n",
    "3. Schmidhuber, Jürgen (January 2015). \"Deep Learning in Neural Networks: An Overview\". Neural Networks. 61: 85–117. arXiv:1404.7828. doi:10.1016/j.neunet.2014.09.003. PMID 25462637."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bidirectional recurrent neural network\n",
    "\n",
    "Malaya use [Long-Short-Term-Memory](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) for all BiRNN gates.\n",
    "\n",
    "LSTM References:\n",
    "\n",
    "1. Hochreiter, Sepp; Schmidhuber, Jürgen (1997-11-01). \"Long Short-Term Memory\". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Malaya use bidirectional recurrent neural network in some models.\n",
    "\n",
    "#### Entities Recognition\n",
    "\n",
    "1. `malaya.entity.deep_model()`\n",
    "\n",
    "#### POS Recognition\n",
    "\n",
    "1. `malaya.pos.deep_model()`\n",
    "\n",
    "You can read more about [Bidirectional Recurrent Neural Network here](https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks).\n",
    "\n",
    "#### References\n",
    "\n",
    "1. M. Schuster, K.K. Paliwal. Bidirectional recurrent neural networks (November 1997). https://ieeexplore.ieee.org/document/650093"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seq2Seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Malaya use seq2seq in some models.\n",
    "\n",
    "#### Stemmer\n",
    "\n",
    "1. `malaya.stem.deep_model()`\n",
    "\n",
    "You can read more about [Seq2Seq here](https://google.github.io/seq2seq/).\n",
    "\n",
    "#### References\n",
    "\n",
    "1. Ilya Sutskever, Oriol Vinyals: “Sequence to Sequence Learning with Neural Networks”, 2014; [http://arxiv.org/abs/1409.3215 arXiv:1409.3215]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT (Deep Bidirectional Transformers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Malaya use BERT in some models.\n",
    "\n",
    "#### Emotion Analysis\n",
    "\n",
    "1. `malaya.emotion.bert()`\n",
    "\n",
    "#### Sentiment Analysis\n",
    "\n",
    "1. `malaya.sentiment.bert()`\n",
    "\n",
    "#### Subjectivity Analysis\n",
    "\n",
    "1. `malaya.subjective.bert()`\n",
    "\n",
    "#### Toxicity Analysis\n",
    "\n",
    "1. `malaya.toxic.bert()`\n",
    "\n",
    "#### Relevancy Analysis\n",
    "\n",
    "1. `malaya.relevancy.bert()`\n",
    "\n",
    "#### Entities Recognition\n",
    "\n",
    "1. `malaya.entities.bert()`\n",
    "\n",
    "#### POS Recognition\n",
    "\n",
    "1. `malaya.pos.bert()`\n",
    "\n",
    "#### Text similarity\n",
    "\n",
    "1. `malaya.similarity.bert()`\n",
    "\n",
    "#### References\n",
    "\n",
    "1. Jacob Devlin, Ming-Wei Chang, Kenton Lee: “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2018; [http://arxiv.org/abs/1810.04805 arXiv:1810.04805]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Skip-thought Vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Malaya use skip-thought in some models.\n",
    "\n",
    "#### Summarization\n",
    "\n",
    "1. `malaya.summarize.deep_skipthought()`\n",
    "\n",
    "#### References\n",
    "\n",
    "1. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun: “Skip-Thought Vectors”, 2015; [http://arxiv.org/abs/1506.06726 arXiv:1506.06726]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### References\n",
    "\n",
    "1. N. Samsudin, Mazidah Puteh, Abdul Razak Hamdan, Mohd Zakree Ahmad Nazri, Normalization of noisy texts in Malaysian online reviews; https://www.researchgate.net/publication/287050449_Normalization_of_noisy_texts_in_Malaysian_online_reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spelling Correction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### References\n",
    "\n",
    "1. N. Samsudin, Mazidah Puteh, Abdul Razak Hamdan, Mohd Zakree Ahmad Nazri, Normalization of noisy texts in Malaysian online reviews; https://www.researchgate.net/publication/287050449_Normalization_of_noisy_texts_in_Malaysian_online_reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### References\n",
    "\n",
    "1. A Multiscale Visualization of Attention in the Transformer Model, https://github.com/jessevig/bertviz\n",
    "2. Topic Modeling visualization, https://github.com/bmabey/pyLDAvis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
