{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.83 s, sys: 755 ms, total: 5.58 s\n",
      "Wall time: 5.38 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import malaya"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "We know that social media texts from Twitter, Facebook and Instagram are very noisy and we want to clean as much as possible to make our machines understand the structure of sentence much better. In Malaya, we standardize our text preprocessing,\n",
    "\n",
    "1. Malaya can replace special words into tokens to reduce dimension curse. `rm10k` become `<money>`.\n",
    "2. Malaya can expand hashtags, `#drmahathir` become `dr mahathir`.\n",
    "3. Malaya can put tags for special words, `#drmahathir` become `<hashtag> drmahathir </hashtag>`.\n",
    "4. Malaya can expand english contractions.\n",
    "5. Malaya can translate english words to become bahasa malaysia words. Again, this translation is using dictionary, it will not understand semantically. Purpose of this translation just to standardize to become bahasa Malaysia.\n",
    "6. Remove postfix for a word, `kerjakan` become `kerja`.\n",
    "7. Normalize elongated words, but this required speller object.\n",
    "\n",
    "These are defaults setting for `preprocessing()`,\n",
    "\n",
    "```\n",
    "def preprocessing(\n",
    "    normalize = [\n",
    "        'url',\n",
    "        'email',\n",
    "        'percent',\n",
    "        'money',\n",
    "        'phone',\n",
    "        'user',\n",
    "        'time',\n",
    "        'date',\n",
    "        'number',\n",
    "    ],\n",
    "    annotate = [\n",
    "        'allcaps',\n",
    "        'elongated',\n",
    "        'repeated',\n",
    "        'emphasis',\n",
    "        'censored',\n",
    "        'hashtag',\n",
    "    ],\n",
    "    lowercase = True,\n",
    "    fix_unidecode = True,\n",
    "    expand_hashtags = True,\n",
    "    expand_english_contractions = True,\n",
    "    translate_english_to_bm = True,\n",
    "    remove_prefix_postfix = True,\n",
    "    maxlen_segmenter = 20,\n",
    "    validate = True,\n",
    "    speller = None,\n",
    "):\n",
    "```\n",
    "\n",
    "#### normalize\n",
    "\n",
    "Supported `normalize`,\n",
    "\n",
    "1. hashtag\n",
    "2. cashtag\n",
    "3. tag\n",
    "4. user\n",
    "5. emphasis\n",
    "6. censored\n",
    "7. acronym\n",
    "8. eastern_emoticons\n",
    "9. rest_emoticons\n",
    "10. emoji\n",
    "11. quotes\n",
    "12. percent\n",
    "13. repeat_puncts\n",
    "14. money\n",
    "15. email\n",
    "16. phone\n",
    "17. number\n",
    "18. allcaps\n",
    "19. url\n",
    "20. date\n",
    "21. time\n",
    "\n",
    "You can check all supported list at `malaya.preprocessing.get_normalize()`.\n",
    "\n",
    "Example, if you set `money` and `number`, and input string is `RM10k`, the output is `<money>`.\n",
    "\n",
    "#### annotate\n",
    "\n",
    "Supported `annotate`,\n",
    "\n",
    "1. hashtag\n",
    "2. allcaps\n",
    "3. elongated\n",
    "4. repeated\n",
    "5. emphasis\n",
    "6. censored\n",
    "\n",
    "Example, if you set `hashtag`, and input string is `#drmahathir`, the output is `<hashtag> drmahathir </hashtag>`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load default paramaters\n",
    "\n",
    "default parameters able to expand hashtag, `#mahathirmohamad` into `mahathir mohamad`, but initial load is quite slow and translate english to bahasa malaysia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "string_1 = 'CANT WAIT for the new season of #mahathirmohamad ＼(^o^)／!!! #davidlynch #tvseries :))), TAAAK SAAABAAR!!!'\n",
    "string_2 = 'kecewa #johndoe movie and it suuuuucks!!! WASTED RM10... rm10 #badmovies :/'\n",
    "string_3 = \"@husein:  can't wait for the Nov 9 #Sentiment talks!  YAAAAAAY !!! :-D http://sentimentsymposium.com/.\"\n",
    "string_4 = 'aahhh, malasnye nak pegi keje harini #mondayblues'\n",
    "string_5 = '#drmahathir #najibrazak #1malaysia #mahathirnajib'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 15.5 s, sys: 3.12 s, total: 18.6 s\n",
      "Wall time: 21.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "preprocessing = malaya.preprocessing.preprocessing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 78.1 ms, sys: 1.63 ms, total: 79.7 ms\n",
      "Wall time: 81.6 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<allcaps> tak boleh tunggu </allcaps> untuk yang baru musim daripada <hashtag> mahathir mohamad </hashtag> \\\\(^o^)/ ! <repeated> <hashtag> david lynch </hashtag> <hashtag> tv series </hashtag> <happy> , <allcaps> taak <elongated> saabaar <elongated> </allcaps> ! <repeated>'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.03 ms, sys: 43 µs, total: 2.07 ms\n",
      "Wall time: 2.11 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'kecewa <hashtag> john doe </hashtag> filem dan ia suucks <elongated> ! <repeated> <allcaps> dibazir </allcaps> <money> . <repeated> <money> <hashtag> bad movies </hashtag> <annoyed>'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.58 ms, sys: 24 µs, total: 1.61 ms\n",
      "Wall time: 1.72 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<user> : boleh tidak tunggu untuk yang <date> <hashtag> sentimen </hashtag> talks ! <allcaps> yaay <elongated> </allcaps> ! <repeated> :-d <url>'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.12 ms, sys: 759 µs, total: 4.88 ms\n",
      "Wall time: 4.26 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'aahh <elongated> , malas nak pergi kerja hari ini <hashtag> isnin blues </hashtag>'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.58 ms, sys: 855 µs, total: 8.43 ms\n",
      "Wall time: 7.82 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<hashtag> dr mahathir </hashtag> <hashtag> najib razak </hashtag> <hashtag> 1 malaysia </hashtag> <hashtag> mahathir najib </hashtag>'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load default paramaters with spelling correction to normalize elongated words.\n",
    "\n",
    "We saw `taak`, `saabaar` and another elongated words are not the original words, so we can use spelling correction to normalize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrector = malaya.spell.probability()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 14.4 s, sys: 2.17 s, total: 16.6 s\n",
      "Wall time: 17.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "preprocessing = malaya.preprocessing.preprocessing(speller = corrector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 71.5 ms, sys: 14.9 ms, total: 86.4 ms\n",
      "Wall time: 88.1 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<allcaps> tak boleh tunggu </allcaps> untuk yang baru musim daripada <hashtag> mahathir mohamad </hashtag> \\\\(^o^)/ ! <repeated> <hashtag> david lynch </hashtag> <hashtag> tv series </hashtag> <happy> , <allcaps> tak <elongated> sabar <elongated> </allcaps> ! <repeated>'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.01 ms, sys: 451 µs, total: 2.46 ms\n",
      "Wall time: 2.47 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'kecewa <hashtag> john doe </hashtag> filem dan ia sucks <elongated> ! <repeated> <allcaps> dibazir </allcaps> <money> . <repeated> <money> <hashtag> bad movies </hashtag> <annoyed>'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 969 µs, sys: 319 µs, total: 1.29 ms\n",
      "Wall time: 1.3 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<user> : boleh tidak tunggu untuk yang <date> <hashtag> sentimen </hashtag> talks ! <allcaps> yay <elongated> </allcaps> ! <repeated> :-d <url>'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.05 ms, sys: 731 µs, total: 4.78 ms\n",
      "Wall time: 4.77 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'ah <elongated> , malas nak pergi kerja hari ini <hashtag> isnin blues </hashtag>'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.05 ms, sys: 619 µs, total: 9.67 ms\n",
      "Wall time: 9.72 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<hashtag> dr mahathir </hashtag> <hashtag> najib razak </hashtag> <hashtag> 1 malaysia </hashtag> <hashtag> mahathir najib </hashtag>'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## disable hashtag expander\n",
    "\n",
    "Sometime we want to maintain hashtags because we want to study the frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 163 ms, sys: 36.4 ms, total: 199 ms\n",
      "Wall time: 206 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "preprocessing = malaya.preprocessing.preprocessing(expand_hashtags = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.25 ms, sys: 28 µs, total: 1.28 ms\n",
      "Wall time: 1.29 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<allcaps> tak boleh tunggu </allcaps> untuk yang baru musim daripada <hashtag> mahathirmohamad </hashtag> \\\\(^o^)/ ! <repeated> <hashtag> davidlynch </hashtag> <hashtag> tvseries </hashtag> <happy> , <allcaps> taak <elongated> saabaar <elongated> </allcaps> ! <repeated>'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 595 µs, sys: 24 µs, total: 619 µs\n",
      "Wall time: 628 µs\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<hashtag> drmahathir </hashtag> <hashtag> najibrazak </hashtag> <hashtag> 1 malaysia </hashtag> <hashtag> mahathirnajib </hashtag>'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## disable english translation\n",
    "\n",
    "But there are basic normalizations that cannot override, like, `for` automatically become `untuk`. You can check default entire normalizations at `from malaya.texts._tatabahasa import rules_normalizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 15 s, sys: 2.64 s, total: 17.6 s\n",
      "Wall time: 19.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "preprocessing = malaya.preprocessing.preprocessing(translate_english_to_bm = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 85.9 ms, sys: 2.74 ms, total: 88.7 ms\n",
      "Wall time: 92 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<allcaps> tak boleh wait </allcaps> untuk the new season of <hashtag> mahathir mohamad </hashtag> \\\\(^o^)/ ! <repeated> <hashtag> david lynch </hashtag> <hashtag> tv series </hashtag> <happy> , <allcaps> taak <elongated> saabaar <elongated> </allcaps> ! <repeated>'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.07 ms, sys: 942 µs, total: 5.01 ms\n",
      "Wall time: 4.66 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'kecewa <hashtag> john doe </hashtag> movie and it suucks <elongated> ! <repeated> <allcaps> wasted </allcaps> <money> . <repeated> <money> <hashtag> bad movies </hashtag> <annoyed>'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.38 ms, sys: 74 µs, total: 1.46 ms\n",
      "Wall time: 1.48 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<user> : can not wait untuk the <date> <hashtag> sentiment </hashtag> talks ! <allcaps> yaay <elongated> </allcaps> ! <repeated> :-d <url>'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "' '.join(preprocessing.process(string_3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizer\n",
    "\n",
    "It able to tokenize multiple regex pipelines, you can check the list from `malaya.preprocessing.get_normalize()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = malaya.preprocessing.SocialTokenizer().tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['CANT',\n",
       " 'WAIT',\n",
       " 'for',\n",
       " 'the',\n",
       " 'new',\n",
       " 'season',\n",
       " 'of',\n",
       " '#mahathirmohamad',\n",
       " '＼(^o^)／',\n",
       " '!',\n",
       " '!',\n",
       " '!',\n",
       " '#davidlynch',\n",
       " '#tvseries',\n",
       " ':)))',\n",
       " ',',\n",
       " 'TAAAK',\n",
       " 'SAAABAAR',\n",
       " '!',\n",
       " '!',\n",
       " '!']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(string_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['kecewa',\n",
       " '#johndoe',\n",
       " 'movie',\n",
       " 'and',\n",
       " 'it',\n",
       " 'suuuuucks',\n",
       " '!',\n",
       " '!',\n",
       " '!',\n",
       " 'WASTED',\n",
       " 'RM10',\n",
       " '.',\n",
       " '.',\n",
       " '.',\n",
       " 'rm10',\n",
       " '#badmovies',\n",
       " ':/']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(string_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['@husein',\n",
       " ':',\n",
       " 'can',\n",
       " \"'\",\n",
       " 't',\n",
       " 'wait',\n",
       " 'for',\n",
       " 'the',\n",
       " 'Nov 9',\n",
       " '#Sentiment',\n",
       " 'talks',\n",
       " '!',\n",
       " 'YAAAAAAY',\n",
       " '!',\n",
       " '!',\n",
       " '!',\n",
       " ':-D',\n",
       " 'http://sentimentsymposium.com/.']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(string_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['saya', 'nak', 'makan', 'ayam', 'harga', 'rm10k']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer('saya nak makan ayam harga rm10k')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Segmentation\n",
    "\n",
    "sometime when we want to clean social media texts or crawled texts, it lack of spaces, example, `sayanakmakannasiayam`, `DrMahathir berjalan`.\n",
    "\n",
    "We provide segmentation to split those sentences using Viterbi algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "segmenter = malaya.preprocessing.segmenter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'saya nak makan nasi ayam'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "segmenter.segment('sayanakmakannasiayam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'berjalan di'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "segmenter.segment('berjalandi')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dr mahathir'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "segmenter.segment('DrMahathir')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But it will lower the output, you can create a simple function to fix it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def segment(string):\n",
    "    segmented = segmenter.segment(string)\n",
    "    splitted = re.sub(r'[ ]+', ' ', segmented).strip().split()\n",
    "    splitted = [split.title() if string[string.lower().find(split)].isupper() else split for split in splitted]\n",
    "    return ' '.join(splitted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Dr Mahathir dan Najib Razak'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "segment('DrMahathir dan NajibRazak')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
