{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How Malaya gathered corpus? \n",
    "\n",
    "We use a translator to translate from a validated English dataset to a Bahasa dataset.\n",
    "\n",
    "Everyone agree that Google Translate is the best online translator in this world, but the problem here, to subscribe the API from Google Cloud is really insane expensive.\n",
    "\n",
    "Good thing about [https://translate.google.com/](https://translate.google.com/), it open for public internet! So we just code a headless browser using Selenium with PhantomJS as the backbone, that's all!\n",
    "\n",
    "You can check the source code here, [translator/](https://github.com/huseinzol05/Malaya/tree/master/translator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from translate_selenium import Translate, Translate_Concurrent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate a sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('sample-joy') as fopen:\n",
    "    dataset = list(filter(None, fopen.read().split('\\n')))\n",
    "len(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "translator = Translate(from_lang = 'en', to_lang = 'ms')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get list of supported language in here, [https://cloud.google.com/translate/docs/languages](https://cloud.google.com/translate/docs/languages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4 ms, sys: 0 ns, total: 4 ms\n",
      "Wall time: 1.23 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'seorang lelaki yang saya mengagumi begitu banyak meminta saya untuk pergi bersamanya'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "translator.translate(dataset[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.23 seconds, it took a very long time to translate a single sentence. What if you have 100k of sentences? It will cost you around 123000 seconds! insane to wait!\n",
    "\n",
    "So, we provide multihreading translator, concurrently translate multi sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate batch of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "translators = Translate_Concurrent(batch_size = 3, from_lang = 'en', to_lang = 'ms')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████| 1/1 [00:01<00:00,  1.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8 ms, sys: 12 ms, total: 20 ms\n",
      "Wall time: 1.44 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['kawan yang sudah berkahwin rapat hanya mempunyai anak pertamanya',\n",
       " 'pengenalan rapat menangis untuk saya saya merasa gembira kerana ada yang peduli',\n",
       " 'seorang lelaki yang saya mengagumi begitu banyak meminta saya untuk pergi bersamanya']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "translators.translate_batch(dataset[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See, we predicted 3 sentences at almost wall time. You can increase the `batch_size` to any size you want, limit is your spec now, this method will never make Google blocked your IP. Malaya already tested it more than 300k of sentences.\n",
    "\n",
    "Remember, 1 translator took a quite toll, here I spawned 10 translators, look from my `top`,\n",
    "```text\n",
    "PID   USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\n",
    "14628 husein    20   0 3175700 398980  43036 S  33.6  2.4   5:38.05 phantomjs                                                                                                  \n",
    "14652 husein    20   0 3188824 408880  43084 S  29.9  2.5   5:34.62 phantomjs                                                                                                  \n",
    "14489 husein    20   0 3204708 411520  43064 S  28.6  2.5   5:35.29 phantomjs                                                                                                  \n",
    "14466 husein    20   0 3171668 400304  43008 S  24.6  2.5   5:26.74 phantomjs                                                                                                  \n",
    "14443 husein    20   0 3181056 403228  42916 S  21.9  2.5   5:26.24 phantomjs                                                                                                  \n",
    "14512 husein    20   0 3187592 416036  42956 S  20.3  2.6   5:30.03 phantomjs                                                                                                  \n",
    "14558 husein    20   0 3206104 419800  43640 S  19.9  2.6   5:30.76 phantomjs                                                                                                  \n",
    "14535 husein    20   0 3179416 405508  43196 S  18.3  2.5   5:27.54 phantomjs                                                                                                  \n",
    "14420 husein    20   0 3202472 422448  43064 S  17.6  2.6   5:26.78 phantomjs                                                                                                  \n",
    "14581 husein    20   0 3181132 401892  43056 S  16.3  2.5   5:33.48 phantomjs\n",
    "```\n",
    "\n",
    "1 translator cost me around,\n",
    "```text\n",
    "PID   USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\n",
    "14628 husein    20   0 3175700 398980  43036 S  33.6  2.4   5:38.05 phantomjs  \n",
    "```\n",
    "\n",
    "My machine specifications,\n",
    "```text\n",
    "H/W path       Device       Class          Description\n",
    "======================================================\n",
    "                            system         G1.Sniper H6 (To be filled by O.E.M.)\n",
    "/0                          bus            G1.Sniper H6\n",
    "/0/3d                       processor      Intel(R) Core(TM) i5-4690 CPU @ 3.50GHz\n",
    "/0/42                       memory         16GiB System Memory\n",
    "/0/42/0                     memory         DIMM [empty]\n",
    "/0/42/1                     memory         8GiB DIMM DDR3 Synchronous 1600 MHz (0.6 ns)\n",
    "/0/42/2                     memory         DIMM [empty]\n",
    "/0/42/3                     memory         8GiB DIMM DDR3 Synchronous 1600 MHz (0.6 ns)\n",
    "/0/100                      bridge         4th Gen Core Processor DRAM Controller\n",
    "/0/100/1                    bridge         Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller\n",
    "/0/100/1/0                  display        GM206 [GeForce GTX 960]\n",
    "/0/100/1/0.1                multimedia     NVIDIA Corporation\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**So, beware of your machine!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
