{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.36 s, sys: 840 ms, total: 5.2 s\n",
      "Wall time: 4.77 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import malaya"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why augmentation\n",
    "\n",
    "Let say you have a very limited labelled corpus, and you want to add more, but labelling is very costly.\n",
    "\n",
    "So, text augmentation! We provided few augmentation interfaces in Malaya."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Synonym\n",
    "\n",
    "Use dictionary of synonym to replace words with it synonyms. Synonym data from [Malaya-Dataset/90k-synonym](https://github.com/huseinzol05/Malaya-Dataset#90k-synonym).\n",
    "\n",
    "```python\n",
    "def synonym(\n",
    "    string: str,\n",
    "    threshold: float = 0.5,\n",
    "    top_n = 5,\n",
    "    cleaning_function: Callable = augmentation_textcleaning,\n",
    "    **kwargs\n",
    "):\n",
    "    \"\"\"\n",
    "    augmenting a string using synonym, https://github.com/huseinzol05/Malaya-Dataset#90k-synonym\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    string: str\n",
    "    threshold: float, optional (default=0.5)\n",
    "        random selection for a word.\n",
    "    top_n: int, (default=5)\n",
    "        number of nearest neighbors returned. Length of returned result should as top_n.\n",
    "    cleaning_function: function, (default=malaya.text.function.augmentation_textcleaning)\n",
    "        function to clean text.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "string = 'saya suka makan ayam dan ikan'\n",
    "text = 'Perdana Menteri berkata, beliau perlu memperoleh maklumat terperinci berhubung isu berkenaan sebelum kerajaan dapat mengambil sebarang tindakan lanjut. Bagaimanapun, beliau yakin masalah itu dapat diselesaikan dan pentadbiran kerajaan boleh berfungsi dengan baik.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['saya suka makan ayam dan ikan',\n",
       " 'saya mencinta makan ayam jantan dan ikan',\n",
       " 'saya mencinta makan makan ayam jantan dan ikan',\n",
       " 'saya suka makan makan ayam dan ikan',\n",
       " 'saya suka makan ayam dan ikan']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.augmentation.synonym(string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Perdana menteri menunjukkan beliau perlu memperoleh maklumat Terperinci menghayati isu berkenaan sebelum kerajaan dapat mengawali sebarang nombor lanjut bagaimanapun beliau beramanah sedih itu dapat diselesaikan dan pengurusannya kerajaan berupaya berfungsi dengan baik',\n",
       " 'Perdana menteri menunjukkan beliau wajib mengusahakan data Terperinci menghayati penerbitan berkenaan di hadapan jajahan menggunakannya mendapatkan sebarang digit tertua bagaimanapun beliau beramanah suram itu dapatkan diselesaikan dan pengurusannya kabinet boleh mengangkut dengan baik',\n",
       " 'Ulung uskup merupakan beliau wajib mengusahakan data Terperinci mempunyai penerbitan berkenaan di hadapan kerajaan menggunakannya berkumpul sebarang nombor gelap bagaimanapun beliau beramanah daif itu memperoleh diselesaikan dan pengurusannya kerajaan boleh mencari dengan baik',\n",
       " 'Ulung menteri merupakan beliau wajib memupuk dokumen Terperinci mempunyai pengeluaran berkenaan sebelum kerajaan menangani berkumpul sebarang nombor gelap masih beliau yakin daif itu tiba diselesaikan dan pengurusannya pemerintah boleh mengesani dengan baik',\n",
       " 'Perdana uskup menunjukkan beliau wajib pelihara dokumen Terperinci mempunyai keluaran berkenaan sebelumnya kerajaan menangani berkumpul sebarang nombor jahat Bagaimana pun beliau yakin daif itu maju diselesaikan dan pengurusannya komandan boleh mengesani dengan baik']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.augmentation.synonym(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Wordvector\n",
    "\n",
    "dictionary of synonym is quite hard to populate, required some domain experts to help us. So we can use wordvector to find nearest words.\n",
    "\n",
    "```python\n",
    "def wordvector(\n",
    "    string: str,\n",
    "    wordvector,\n",
    "    threshold: float = 0.5,\n",
    "    top_n: int = 5,\n",
    "    soft: bool = False,\n",
    "    cleaning_function: Callable = augmentation_textcleaning,\n",
    "):\n",
    "    \"\"\"\n",
    "    augmenting a string using wordvector.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    string: str\n",
    "    wordvector: object\n",
    "        wordvector interface object.\n",
    "    threshold: float, optional (default=0.5)\n",
    "        random selection for a word.\n",
    "    soft: bool, optional (default=False)\n",
    "        if True, a word not in the dictionary will be replaced with nearest jarowrinkler ratio.\n",
    "        if False, it will throw an exception if a word not in the dictionary.\n",
    "    top_n: int, (default=5)\n",
    "        number of nearest neighbors returned. Length of returned result should as top_n.\n",
    "    cleaning_function: function, (default=malaya.text.function.augmentation_textcleaning)\n",
    "        function to clean text.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/wordvector.py:114: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/wordvector.py:125: The name tf.InteractiveSession is deprecated. Please use tf.compat.v1.InteractiveSession instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "vocab_wiki, embedded_wiki = malaya.wordvector.load_wiki()\n",
    "word_vector_wiki = malaya.wordvector.load(embedded_wiki, vocab_wiki)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['saya suka makan ayam dan ikan',\n",
       " 'kamu gemar minum ayam serta ayam',\n",
       " 'anda pandai tidur ayam atau ular',\n",
       " 'kami senang mandi ayam mahupun keju',\n",
       " 'aku ingin berehat ayam tetapi lembu']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.augmentation.wordvector(\n",
    "    string, word_vector_wiki, soft = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['perdana menteri berkata beliau perlu memperoleh maklumat terperinci berhubung isu berkenaan sebelum kerajaan dapat mengambil sebarang tindakan lanjut bagaimanapun beliau yakin masalah itu dapat diselesaikan dan pentadbiran kerajaan boleh berfungsi dengan baik',\n",
       " 'perdana kementerian menyatakan beliau perlu memperoleh maklumat terperinci berkaitan persoalan berkaitan selepas kerajaan dapat mendapat sebarang tindakan terperinci walaupun dia sedar gangguan itu boleh dibuktikan serta pentadbiran kerajaan dapat dikelaskan dengan baik',\n",
       " 'perdana setiausaha mengatakan beliau perlu memperoleh maklumat terperinci berhadapan prosedur tertentu setelah kerajaan dapat menghabiskan sebarang tindakan lanjutan namun baginda bimbang kelemahan itu harus dilaksanakan atau pentadbiran kerajaan harus bertindak dengan baik',\n",
       " 'perdana jabatan mendapati beliau perlu memperoleh maklumat terperinci sejajar artikel tersebut ketika kerajaan dapat mengubah sebarang tindakan ringkas maka mereka menyangka gejala itu perlu dikesan mahupun pentadbiran kerajaan perlu dirujuk dengan baik',\n",
       " 'perdana duta mencadangkan beliau perlu memperoleh maklumat terperinci bertentangan kontroversi berlainan sejak kerajaan dapat memakan sebarang tindakan positif tetapi saya takut risiko itu mampu diperhatikan tetapi pentadbiran kerajaan akan dikira dengan baik']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.augmentation.wordvector(\n",
    "    text, word_vector_wiki, soft = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Transformer\n",
    "\n",
    "Problem with wordvector, it just replaced a word for near synonym without understood the whole sentence context, so, Transformer comes to the rescue!\n",
    "\n",
    "```python\n",
    "def transformer(\n",
    "    string: str,\n",
    "    model,\n",
    "    threshold: float = 0.5,\n",
    "    top_p: float = 0.9,\n",
    "    top_k: int = 100,\n",
    "    temperature: float = 1.0,\n",
    "    top_n: int = 5,\n",
    "    cleaning_function: Callable = None,\n",
    "):\n",
    "\n",
    "    \"\"\"\n",
    "    augmenting a string using transformer + nucleus sampling / top-k sampling.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    string: str\n",
    "    model: object\n",
    "        transformer interface object. Right now only supported BERT, ALBERT and ELECTRA.\n",
    "    threshold: float, optional (default=0.5)\n",
    "        random selection for a word.\n",
    "    top_p: float, optional (default=0.8)\n",
    "        cumulative sum of probabilities to sample a word. \n",
    "        If top_n bigger than 0, the model will use nucleus sampling, else top-k sampling.\n",
    "    top_k: int, optional (default=100)\n",
    "        k for top-k sampling.\n",
    "    temperature: float, optional (default=0.8)\n",
    "        logits * temperature.\n",
    "    top_n: int, (default=5)\n",
    "        number of nearest neighbors returned. Length of returned result should as top_n.\n",
    "    cleaning_function: function, (default=None)\n",
    "        function to clean text.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/modeling.py:240: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow_core/python/layers/core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:79: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
      "\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:93: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
      "\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/sampling.py:26: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:114: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.random.categorical` instead.\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:118: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.\n",
      "\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:120: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
      "\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:121: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.\n",
      "\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:127: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.\n",
      "\n",
      "WARNING:tensorflow:From /Users/huseinzolkepli/Documents/Malaya/malaya/transformers/electra/__init__.py:129: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n",
      "INFO:tensorflow:Restoring parameters from /Users/huseinzolkepli/Malaya/electra-model/base/electra-base/model.ckpt\n"
     ]
    }
   ],
   "source": [
    "electra = malaya.transformer.load(model = 'electra')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Perdana Menteri berkata , kerajaan sudah memperoleh maklumat terperinci berhubung masalah berkenaan supaya kerajaan dapat mengambil pelbagai tindakan sewajarnya . Bagaimanapun , beliau yakin masalah itu berjaya diselesaikan dan akhirnya terdahulu boleh diselesaikan dengan baik .',\n",
       " 'Perdana Menteri berkata , kerajaan perlu memperoleh maklumat terperinci berhubung isu berkenaan supaya kerajaan dapat mengambil serius tindakan segera . Bagaimanapun , beliau berharap masalah itu boleh diselesaikan dan akhirnya kementerian boleh diselesaikan dengan baik .',\n",
       " 'Perdana Menteri berkata , kerajaan telah memperoleh maklumat terperinci berhubung isu berkenaan supaya kerajaan dapat mengambil beberapa tindakan sewajarnya . Bagaimanapun , beliau berharap masalah itu perlu diselesaikan dan siasatan BN boleh diselesaikan dengan baik .',\n",
       " 'Perdana Menteri berkata , kerajaan akan memperoleh maklumat terperinci berhubung isu berkenaan supaya kerajaan dapat mengambil sebarang tindakan susulan . Bagaimanapun , beliau mengharapkan masalah itu dapat diselesaikan dan membolehkan tidak boleh ditangani dengan baik .',\n",
       " 'Perdana Menteri berkata , kerajaan sudah memperoleh maklumat terperinci berhubung isu berkenaan supaya kerajaan dapat mengambil sebarang tindakan lanjut . Bagaimanapun , beliau berharap masalah itu dapat diselesaikan dan hanya masih boleh diselesaikan dengan baik .']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.augmentation.transformer(text, electra)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
