{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tidytweets\n",
    "Clean text extracted from social networks to perform various NLP tasks such as topic analysis, word embeddings, sentiment analysis, etc.\n",
    "\n",
    "## Table of contents\n",
    "1. `remove_repetitions`\n",
    "2. `remove_last_repetition`\n",
    "3. `remove_urls`\n",
    "4. `remove_RT`\n",
    "5. `remove_accents`\n",
    "6. `remove_hashtags`\n",
    "7. `remove_mentions`\n",
    "8. `remove_special_characters`\n",
    "9. `remove_extra_spaces`\n",
    "10. `space_between_emojis`\n",
    "11. `preprocess`\n",
    "12. `remove_words`\n",
    "13. `unnest_tokens`\n",
    "14. `spanish_lemmatizer`\n",
    "15. `create_bol`\n",
    "16. Tutorial: Topic Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: tidyX\n",
      "Version: 1.1.1\n",
      "Summary: Python package to clean raw tweets for ML applications\n",
      "Home-page: \n",
      "Author: Lucas Gómez Tobón, Jose Fernando Barrera\n",
      "Author-email: lucasgomeztobon@gmail.com, jf.barrera10@uniandes.edu.co\n",
      "License: MIT\n",
      "Location: c:\\users\\lucas\\anaconda3\\lib\\site-packages\n",
      "Requires: emoji, numpy, pandas, regex, spacy, thefuzz, Unidecode\n",
      "Required-by: \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution -rotobuf (c:\\users\\lucas\\anaconda3\\lib\\site-packages)\n"
     ]
    }
   ],
   "source": [
    "!pip show tidyX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\Lucas\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import tidyX.TextPreprocessor as tp"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. `remove_repetitions`\n",
    "**Description of the function**\n",
    "\n",
    "This function deletes any consecutive repetition of characters in a string. For example, the string 'coooroosooo' will be changed to 'coroso'. As in many languages it's common to have some special characters that can be repeated, for example the 'l' in spanish to form 'll', the exception argument could be used to specify which characters are allowed to repeat once.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "In social media, it is common for people to repeat certain characters of a word in order to add more emotion to a sentence. However, when we attempt to count the occurrences of a word, the various ways in which a word can be written make it difficult to uniquely identify each instance. One simple solution to this issue is to use the `remove_repetitions` function. Let's consider the following tweet:\n",
    "\n",
    "<center>\n",
    "<img src=\"img/remove_repetitions1.png\" alt=\"remove_repetitions1\" height=300px />\n",
    "</center>\n",
    "\n",
    "In this particular case, the author writes \"Goooal\" and \"Goal.\" Consequently, it becomes necessary for us to eliminate the repeated \"o\"s in the first word in order to make both words equal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: Goooal ⚽️⚽️⚽️ Christiano Ronaldo Amazing Goal Juventus vs Real Madrid 1-3 Champions League Final #JUVRMA #UCLFinal2017 #JuventusRealMadrid\n"
     ]
    }
   ],
   "source": [
    "string_example = \"Goooal ⚽️⚽️⚽️ Christiano Ronaldo Amazing Goal Juventus vs Real Madrid 1-3 Champions League Final #JUVRMA #UCLFinal2017 #JuventusRealMadrid\"\n",
    "print(\"Before:\", string_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After: Goal ⚽️⚽️⚽️ Christiano Ronaldo Amazing Goal Juventus vs Real Madrid 1-3 Champions League Final #JUVRMA #UCLFinal2017 #JuventusRealMadrid\n"
     ]
    }
   ],
   "source": [
    "string_without_repetitions = tp.remove_repetitions(string = string_example, exceptions = None)\n",
    "print(\"After:\", string_without_repetitions)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, it's worth noting that there exist numerous words that feature the repetition of a single character. To address this, the `remove_repetitions` function incorporates the `exceptions` parameter, which allows for specifying a list of characters that are permitted to appear twice. For instance, if we set `exceptions = ['p']`, words such as 'happpy' will be cleaned and transformed into 'happy'. The default value for this parameter is `['r', 'l', 'n', 'c', 'a', 'e', 'o']`. Let's see another example:\n",
    "\n",
    "<center>\n",
    "<img src=\"img/remove_repetitions2.png\" alt=\"remove_repetitions2\" width=300px />\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: HAPPPYYYYY GRADUATION TO US!! THANKYOUUUU LORD!!! 🫶🤍\n"
     ]
    }
   ],
   "source": [
    "string_example = \"HAPPPYYYYY GRADUATION TO US!! THANKYOUUUU LORD!!! 🫶🤍\"\n",
    "print(\"Before:\", string_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After: HAPPY GRADUATION TO US! THANKYOU LORD! 🫶🤍\n"
     ]
    }
   ],
   "source": [
    "string_without_repetitions = tp.remove_repetitions(string = string_example, \n",
    "    exceptions = [\"P\"])\n",
    "print(\"After:\", string_without_repetitions)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. `remove_last_repetition`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_last_repetition` function is designed to remove the repetition of the last character in each word of a given string. It's particularly useful when dealing with text that contains repeated characters at the end of words, a common occurrence in social media posts where users emphasize words for expression. This function helps clean and standardize the text by eliminating these last-character repetitions.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "Imagine you are analyzing text data from social media platforms, and you want to ensure consistency in your analysis by removing repetitive characters at the end of words. For example, in Spanish, words typically do not end with a repeated character, but social media users often add emphasis by repeating the last character. Let's explore a practical use case with a tweet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: Holaaaa amigooo\n",
      "After: Hola amigo\n"
     ]
    }
   ],
   "source": [
    "# Original tweet with last-character repetitions\n",
    "string_example = \"Holaaaa amigooo\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_last_repetition function to clean the text\n",
    "string_without_last_repetitions = tp.remove_last_repetition(string = string_example)\n",
    "print(\"After:\", string_without_last_repetitions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input string contains repeated characters at the end of words, like \"Holaaaa\" and \"amigooo.\" To ensure consistent analysis, you can use the `remove_last_repetition` function, which removes the last-character repetitions and transforms the text into \"Hola amigo.\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. `remove_urls`\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_urls` function is designed to remove all URLs that start with \"http\" from a given string. It's a handy tool for text processing when you want to eliminate URLs from a text dataset, making it cleaner and more focused on textual content. This function scans the entire string, identifies any sequences of characters that start with \"http\" and continue until a space or end of the line, and removes them.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "You may encounter situations where you want to analyze or visualize the textual content of a dataset, but the presence of URLs can clutter the text and skew your analysis. This is especially common in social media data, chat messages, or web scraping scenarios. Let's explore a practical use case with a sample text containing URLs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: Check out our website: http://example.com. For more info, visit http://example2.com\n",
      "After: Check out our website:  For more info, visit \n"
     ]
    }
   ],
   "source": [
    "# Original text with URLs\n",
    "string_example = \"Check out our website: http://example.com. For more info, visit http://example2.com\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_urls function to clean the text\n",
    "string_without_urls = tp.remove_urls(string = string_example)\n",
    "print(\"After:\", string_without_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input string contains two URLs, \"http://example.com\" and \"http://example2.com.\" To focus on the textual content without the distraction of URLs, you can use the `remove_urls` function, which removes them and results in cleaner text."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. `remove_RT`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_RT` function is designed to remove the \"RT\" prefix from tweets. In the context of social media, \"RT\" typically stands for \"Retweet\" and is often used as a prefix when users share or retweet content. This function is useful for cleaning and standardizing tweet text data by removing the \"RT\" prefix, accounting for varying amounts of white space after \"RT.\"\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "When you're working with tweet data and you want to analyze or visualize the content of tweets without the distraction of the \"RT\" prefix, the remove_RT function comes in handy. Retweets often have the \"RT\" prefix at the beginning, but the amount of white space after \"RT\" can vary. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: RT     @username: Check out this amazing article!\n",
      "After: @username: Check out this amazing article!\n"
     ]
    }
   ],
   "source": [
    "# Original tweet with \"RT\" prefix\n",
    "string_example = \"RT     @username: Check out this amazing article!\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_RT function to clean the tweet\n",
    "cleaned_tweet = tp.remove_RT(string = string_example)\n",
    "print(\"After:\", cleaned_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input tweet contains the \"RT\" prefix followed by varying amounts of white space before the actual content of the tweet. To focus on the tweet's content and remove the \"RT\" prefix, you can use the `remove_RT` function, which standardizes the text and results in a tweet without the \"RT\" prefix."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. `remove_accents`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_accents` function serves two purposes: it removes accent marks from characters in a given string and can optionally remove emojis. Accent marks can be common in languages like French or Spanish (this specific use case), and removing them can be helpful for text processing tasks. This function provides flexibility by allowing you to choose whether to remove emojis as well.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "The `remove_accents` function is particularly useful when working with text data that contains accented characters, and you want to simplify the text for analysis or comparison. Additionally, if your text data includes emojis that are not relevant to your analysis, you can choose to remove them as well. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: Café ☕️ à côté de l'hôtel. 😃\n",
      "After: Cafe  a cote de l'hotel. \n"
     ]
    }
   ],
   "source": [
    "# Original text with accents and emojis\n",
    "string_example = \"Café ☕️ à côté de l'hôtel. 😃\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_accents function to clean the text (removing emojis)\n",
    "cleaned_text = tp.remove_accents(string = string_example, delete_emojis = True)\n",
    "print(\"After:\", cleaned_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input text contains accented characters (e.g., \"é\") and emojis (e.g., \"☕️\" and \"😃\"). To simplify the text for analysis and remove emojis, you can use the `remove_accents` function with the `delete_emojis` option set to True, resulting in cleaned text without accents or emojis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is flexible over the total number of followed emojis on a text, let's process a Spanish common example:\n",
    "<center>\n",
    "<img src=\"img/remove_accents.png\" alt=\"remove_accents\" width=500px />\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: ‼️ La función de traductor no funciona así que este tweet es solo para nuestros seguidores hispanohablantes, siempre van a ser nuestros favoritos y ahora vamos a poner emojis tristes para que los que no hablan español se preocupen 😭  y también esta foto fuera de contexto 😔💔\n",
      "After: !! La funcion de traductor no funciona asi que este tweet es solo para nuestros seguidores hispanohablantes, siempre van a ser nuestros favoritos y ahora vamos a poner emojis tristes para que los que no hablan espanol se preocupen   y tambien esta foto fuera de contexto \n"
     ]
    }
   ],
   "source": [
    "# Original text with accents and emojis\n",
    "string_example = \"‼️ La función de traductor no funciona así que este tweet es solo para nuestros seguidores hispanohablantes, siempre van a ser nuestros favoritos y ahora vamos a poner emojis tristes para que los que no hablan español se preocupen 😭  y también esta foto fuera de contexto 😔💔\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_accents function to clean the text (removing emojis)\n",
    "cleaned_text = tp.remove_accents(string = string_example, delete_emojis = True)\n",
    "print(\"After:\", cleaned_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw, the method removed continuously repeated emojis, but passes over \"!!\" v2 class emojis (Link to the emoji: https://abs-0.twimg.com/emoji/v2/svg/203c.svg). This is due to the fact that it is considered an expression, rather not a direct emoji, when you type double exclamation on Twitter. You can see a full list of this wildcard emoji converter expressions on X's documentation in https://twemoji.twitter.com/ and some examples in https://twitter.com/FakeUnicode/status/1251505174348095488"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. `remove_hashtags`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_hashtags` function is designed to remove hashtags from a given string. In social media and text data, hashtags are often used to categorize or highlight content. This function scans the input string and removes any text that starts with a '#' and is followed by alphanumeric characters, effectively removing hashtags from the text.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "You might encounter situations where you want to analyze or visualize text data without the presence of hashtags. Hashtags can be prevalent in social media posts and may not be relevant to your analysis. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: Exploring the beauty of #nature in #springtime. #NaturePhotography 🌼\n",
      "After: Exploring the beauty of  in .  🌼\n"
     ]
    }
   ],
   "source": [
    "# Original text with hashtags\n",
    "string_example = \"Exploring the beauty of #nature in #springtime. #NaturePhotography 🌼\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_hashtags function to clean the text\n",
    "cleaned_text = tp.remove_hashtags(string = string_example)\n",
    "print(\"After:\", cleaned_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input text contains hashtags such as \"#nature,\" \"#springtime,\" and \"#NaturePhotography.\" To focus on the textual content without the distraction of hashtags, you can use the `remove_hashtags` function, which removes them and results in a cleaner text."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. `remove_mentions`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_mentions` function is designed to remove mentions (e.g., @username) from a given tweet string. In the context of social media, mentions are often used to reference or tag other users. This function scans the input tweet string and removes any text that starts with '@' followed by a username. Optionally, it can also return a list of unique mentions found in the tweet.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "You may encounter situations where you want to analyze or visualize tweet text data without the presence of mentions. Mentions can be common in social media posts and may not be relevant to your analysis. Additionally, you might want to extract and track mentioned accounts separately. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: Exploring the beauty of nature with @NatureExplorer and @WildlifeEnthusiast. #NaturePhotography 🌼\n",
      "After: Exploring the beauty of nature with  and . #NaturePhotography 🌼\n",
      "Extracted Mentions: ['@WildlifeEnthusiast', '@NatureExplorer']\n"
     ]
    }
   ],
   "source": [
    "# Original tweet with mentions\n",
    "string_example = \"Exploring the beauty of nature with @NatureExplorer and @WildlifeEnthusiast. #NaturePhotography 🌼\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_mentions function to clean the tweet and extract mentions\n",
    "cleaned_text, extracted_mentions = tp.remove_mentions(string=string_example, extract = True)\n",
    "print(\"After:\", cleaned_text)\n",
    "print(\"Extracted Mentions:\", extracted_mentions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input tweet text contains mentions such as \"@NatureExplorer\" and \"@WildlifeEnthusiast.\" To focus on the textual content without the distraction of mentions and to extract mentioned accounts, you can use the `remove_mentions` function, which removes mentions and provides a list of unique mentions found in the tweet."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. `remove_special_characters`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_special_characters` function is designed to remove all characters from a string except for lowercase letters and spaces. It's a useful tool for cleaning text data when you want to focus on the textual content while excluding punctuation marks, exclamation marks, special characters, numbers, and uppercase letters. This function scans the input string and removes any character that does not match the criteria.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "You may encounter situations where you want to preprocess text data and eliminate special characters and non-lowercase characters to make it more suitable for natural language processing tasks. Cleaning text in this way can help improve text analysis, topic modeling, or sentiment analysis. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: This is an example text! It contains special characters. 123\n",
      "After: his is an example text t contains special characters\n"
     ]
    }
   ],
   "source": [
    "string_example = \"This is an example text! It contains special characters. 123\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_special_characters function to clean the text\n",
    "cleaned_text = tp.remove_special_characters(string = string_example)\n",
    "print(\"After:\", cleaned_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input text contains special characters, punctuation marks, numbers, and uppercase letters. To focus on the textual content with lowercase letters and spaces only, you can use the `remove_special_characters` function, which removes the undesired characters and results in a cleaner text. Beware to lowercase your text before applying this method over your corpus, as you can see on the past example, it can remove useful strings."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. `remove_extra_spaces`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_extra_spaces` function is designed to remove extra spaces within and surrounding a given string. It's a valuable tool for cleaning text data when you want to standardize spaces, trim leading and trailing spaces, and replace consecutive spaces between words with a single space. This function helps improve the consistency and readability of text.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "You may encounter situations where you want to preprocess text data and ensure consistent spacing for better readability and analysis. Extra spaces can be common in unstructured text, and cleaning them can enhance text analysis, especially when dealing with natural language processing tasks. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: This is    an   example  text with extra   spaces.     \n",
      "After: This is an example text with extra spaces.\n"
     ]
    }
   ],
   "source": [
    "# Original text with extra spaces\n",
    "string_example = \"This is    an   example  text with extra   spaces.     \"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply remove_extra_spaces function to clean the text\n",
    "cleaned_text = tp.remove_extra_spaces(string = string_example)\n",
    "print(\"After:\", cleaned_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input text contains extra spaces between words and leading/trailing spaces. To standardize the spacing and remove the extra spaces, you can use the `remove_extra_spaces` function, which trims leading/trailing spaces and replaces consecutive spaces with a single space."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. `space_between_emojis`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `space_between_emojis` function is designed to insert spaces around emojis within a given string. It ensures that emojis are separated from other text or emojis in the string. This function is helpful for improving the readability of text containing emojis and ensuring proper spacing. It also removes any extra spaces resulting from the insertion of spaces around emojis.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "This function is particularly useful when you're working with text data that includes emojis and you want to enhance the visual presentation of the text. Emojis are often used for expressing emotions or conveying messages, and proper spacing ensures that emojis are distinct and do not run together. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: I love😍this place🌴It's amazing!👏\n",
      "After: I love 😍 this place 🌴 It's amazing! 👏\n"
     ]
    }
   ],
   "source": [
    "# Original text with emojis\n",
    "string_example = \"I love😍this place🌴It's amazing!👏\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply space_between_emojis function to add spaces around emojis\n",
    "cleaned_text = tp.space_between_emojis(string = string_example)\n",
    "print(\"After:\", cleaned_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input text contains emojis such as \"😍,\" \"🌴,\" and \"👏\" mixed with regular text. To ensure that emojis are separated from other text and from each other, you can use the `space_between_emojis` function, which inserts spaces around emojis and removes any extra spaces resulting from the insertion."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. `preprocess`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `preprocess` function is a comprehensive text preprocessing tool designed to clean and standardize tweet text. It applies a series of cleaning functions to perform tasks such as removing retweet prefixes, converting text to lowercase, removing accents and emojis, extracting or removing mentions, removing URLs, hashtags, special characters, extra spaces, and consecutive repeated characters with specified exceptions. This function offers extensive text cleaning capabilities and prepares tweet text for analysis or visualization.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "The `preprocess` function is particularly useful when you're working with tweet data and need to clean and standardize the text for various text analysis tasks. Tweet text can be messy and contain various elements such as mentions, URLs, emojis, and special characters that may need to be processed and standardized. Let's explore a practical use case:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: RT @user1: I love this place! 😍 Check out the link: https://example.com #travel #vacation!!!\n",
      "After: i love this place check out the link\n",
      "Extracted Mentions: ['@user1']\n"
     ]
    }
   ],
   "source": [
    "# Original tweet with various elements\n",
    "string_example = \"RT @user1: I love this place! 😍 Check out the link: https://example.com #travel #vacation!!!\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# Apply preprocess function to clean and preprocess the tweet\n",
    "cleaned_text, extracted_mentions = tp.preprocess(string = string_example, delete_emojis = True)\n",
    "print(\"After:\", cleaned_text)\n",
    "print(\"Extracted Mentions:\", extracted_mentions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input tweet text contains retweet prefixes, mentions, emojis, URLs, hashtags, and special characters. To standardize the tweet text for analysis, you can use the `preprocess` function, which performs a series of cleaning operations to remove or extract various elements and return cleaned text and mentions."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. `remove_words`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `remove_words` function is designed to remove all occurrences of specific words listed in the `bag_of_words` parameter from a given string. This function is particularly useful for removing stopwords or any other set of unwanted words from text data. It performs an exact match, meaning it will remove only the exact words listed in the `bag_of_words` and won't remove variations of those words that are not in the list.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "This function is valuable when you want to clean text data by removing specific words that are not relevant to your analysis or that you consider stopwords. It's commonly used in natural language processing tasks to improve the quality of text analysis, topic modeling, or sentiment analysis. Let's explore a practical use case:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: This is an example sentence with some unnecessary words like 'the', 'is', and 'with'.\n",
      "Stopwords to Remove: ['the', 'is', 'and', 'with']\n",
      "After: This an example sentence some unnecessary words like '', '', ''.\n"
     ]
    }
   ],
   "source": [
    "# Original text with stopwords\n",
    "string_example = \"This is an example sentence with some unnecessary words like 'the', 'is', and 'with'.\"\n",
    "print(\"Before:\", string_example)\n",
    "\n",
    "# List of stopwords to remove\n",
    "stopwords = [\"the\", \"is\", \"and\", \"with\"]\n",
    "print(\"Stopwords to Remove:\", stopwords)\n",
    "\n",
    "# Apply remove_words function to clean the text\n",
    "cleaned_text = tp.remove_words(string = string_example, bag_of_words = stopwords)\n",
    "print(\"After:\", cleaned_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input text contains stopwords such as \"the,\" \"is,\" and \"with.\" To clean the text by removing these stopwords, you can use the `remove_words` function, which removes the specified words from the text."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. `unnest_tokens`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `unnest_tokens` function is designed to flatten a pandas DataFrame by tokenizing a specified column. It takes a pandas DataFrame, the name of the column to tokenize, and an optional flag to create an \"id\" column based on the DataFrame's index. Each token in the specified column becomes a separate row in the resulting DataFrame, effectively \"exploding\" the data into a long format.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "This function is useful when you have text data stored in a DataFrame, and you want to transform it into a format that is more suitable for certain text analysis or modeling tasks. For instance, when working with natural language processing or text mining, you may need to tokenize text data and represent it in a format where each token corresponds to a separate row. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame:\n",
      "                     text_column\n",
      "0     This is a sample sentence.\n",
      "1  Another sentence with tokens.\n",
      "2  Text analysis is interesting.\n",
      "\n",
      "Tokenized DataFrame:\n",
      "   id   text_column\n",
      "0   0          This\n",
      "0   0            is\n",
      "0   0             a\n",
      "0   0        sample\n",
      "0   0     sentence.\n",
      "1   1       Another\n",
      "1   1      sentence\n",
      "1   1          with\n",
      "1   1       tokens.\n",
      "2   2          Text\n",
      "2   2      analysis\n",
      "2   2            is\n",
      "2   2  interesting.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "# Create a sample DataFrame with a text column\n",
    "data = {'text_column': [\"This is a sample sentence.\",\n",
    "                        \"Another sentence with tokens.\",\n",
    "                        \"Text analysis is interesting.\"]}\n",
    "df = pd.DataFrame(data)\n",
    "print(\"Original DataFrame:\")\n",
    "print(df)\n",
    "\n",
    "# Apply unnest_tokens function to tokenize the text column\n",
    "tokenized_df = tp.unnest_tokens(df=df, input_column='text_column')\n",
    "print(\"\\nTokenized DataFrame:\")\n",
    "print(tokenized_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the input DataFrame contains a column named 'text_column' with sentences. To tokenize the text and transform it into a long format where each token is a separate row, you can use the `unnest_tokens` function."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14. `spanish_lemmatizer`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `spanish_lemmatizer` function is designed to lemmatize a given Spanish language token using Spacy's Spanish language model. It takes a token (word) and a Spacy language model object as input and returns the lemmatized version of the token with accents removed. This function is valuable for text analysis tasks where you need to reduce words to their base or dictionary form.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "This function is useful when you're working with text data in Spanish and want to perform text analysis tasks such as sentiment analysis, topic modeling, or text classification. Lemmatization helps standardize words to their base form, reducing the complexity of text data. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m spacy download es_core_news_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Token: corriendo\n",
      "Lemmatized Token: correr\n"
     ]
    }
   ],
   "source": [
    "# Load Spacy's Spanish language model (you should have this model downloaded)\n",
    "nlp = spacy.load(\"es_core_news_sm\")\n",
    "\n",
    "# Input token to lemmatize\n",
    "token = \"corriendo\"  # Example token in Spanish\n",
    "print(\"Original Token:\", token)\n",
    "\n",
    "# Apply spanish_lemmatizer function to lemmatize the token\n",
    "lemmatized_token = tp.spanish_lemmatizer(token = token, model = nlp)\n",
    "print(\"Lemmatized Token:\", lemmatized_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we have an input token, \"corriendo,\" in Spanish that we want to lemmatize to its base form. We use the `spanish_lemmatizer` function to perform the lemmatization."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15. `create_bol`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `create_bol` function is designed to group lemmas based on Levenshtein distance to handle misspelled words in social media data. It takes a numpy array containing lemmas and an optional verbose flag for progress reporting. The function groups similar lemmas into bags of lemmas based on their Levenshtein distance. The result is a pandas DataFrame that contains information about the bags of lemmas, including their IDs, names, associated lemmas, and the similarity threshold used for grouping.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "This function is useful when you're dealing with text data, especially social media data, where misspelled or variations of words are common. Grouping similar lemmas together can help clean and organize text data for analysis, improving the accuracy of text-based tasks like sentiment analysis or topic modeling. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Lemmas:\n",
      "['apple' 'aple' 'apples' 'banana' 'banan' 'bananas' 'cherry' 'cheri'\n",
      " 'cherries']\n",
      "An error occurred: integer division or modulo by zero\n",
      "\n",
      "Bags of Lemmas DataFrame:\n",
      "   bow_id bow_name   lemma  similarity  threshold\n",
      "0       1    apple   apple         100         86\n",
      "1       1    apple    aple          89         86\n",
      "2       1    apple  apples          91         86\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create a numpy array of lemmas\n",
    "lemmas = np.array(['apple', 'aple', 'apples', 'banana', 'banan', 'bananas', 'cherry', 'cheri', 'cherries'])\n",
    "print(\"Original Lemmas:\")\n",
    "print(lemmas)\n",
    "\n",
    "# Apply create_bol function to group similar lemmas\n",
    "bol_df = tp.create_bol(lemmas=lemmas, verbose=True)\n",
    "print(\"\\nBags of Lemmas DataFrame:\")\n",
    "print(bol_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we have an array of lemmas representing fruits, but some of the lemmas are misspelled or have variations. We want to group similar lemmas together into bags of lemmas using the `create_bol` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 16. `get_most_common_strings`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `get_most_common_strings` function is designed to identify and retrieve the most common strings in a list of texts. It takes two arguments: a list of texts and an integer specifying the number of most common words to return. The function calculates word frequencies across the texts and returns a list of the most frequently occurring words along with their respective counts.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "This function is particularly useful when you want to gain insights into the content of a collection of texts. It helps you identify which words or strings are the most prevalent within the text data. You can use this information for various purposes, including data validation, descriptive analysis, or identifying significant terms in text data. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Common Strings:\n",
      "[('quick', 4), ('brown', 3), ('jumps', 3), ('over', 3), ('lazy', 3)]\n"
     ]
    }
   ],
   "source": [
    "# List of example texts\n",
    "texts = [\n",
    "    \"The quick brown fox jumps over the lazy dog.\",\n",
    "    \"A quick brown dog jumps over a lazy fox.\",\n",
    "    \"The quick brown dog jumps over the quick lazy fox.\"\n",
    "]\n",
    "\n",
    "# Number of most common strings to retrieve\n",
    "num_strings = 5\n",
    "\n",
    "# Apply get_most_common_strings function to find the most common words\n",
    "most_common_words = tp.get_most_common_strings(texts = texts, num_strings = num_strings)\n",
    "print(\"Most Common Strings:\")\n",
    "print(most_common_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we have a list of example texts, and we want to find the most common words within these texts using the `get_most_common_strings` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 17. `spacy_pipeline`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `spacy_pipeline` function is a comprehensive text preprocessing tool that leverages spaCy's capabilities to process a list of documents. It allows you to customize the spaCy pipeline, including options such as using a custom lemmatizer for Spanish, specifying stopwords language, choosing a spaCy model, and retrieving the most common words after preprocessing.\n",
    "\n",
    "The function takes several arguments, including a list of documents, a custom lemmatizer flag, pipeline components, stopwords language, spaCy model, and the number of most common words to return. It processes the documents by tokenizing, lemmatizing, and removing stopwords, providing you with well-preprocessed documents and a list of the most common words within them.\n",
    "\n",
    "**When is it useful to use it?**\n",
    "\n",
    "This function is highly useful when you need to preprocess a collection of text documents for natural language processing tasks. It offers flexibility by allowing you to configure the spaCy pipeline according to your specific requirements. Additionally, it provides insights into the most common words in the preprocessed documents, which can be valuable for data validation or descriptive analysis. Let's explore a practical use case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of example documents\n",
    "documents = [\n",
    "    \"El rápido zorro marrón salta sobre el perro perezoso.\",\n",
    "    \"Un veloz perro marrón salta sobre un zorro perezoso.\",\n",
    "    \"El rápido perro marrón salta sobre el veloz zorro perezoso.\"\n",
    "]\n",
    "\n",
    "# Specify preprocessing options\n",
    "custom_lemmatizer = False\n",
    "pipeline = ['tokenize', 'lemmatizer']\n",
    "stopwords_language = 'spanish'\n",
    "model = 'es_core_news_sm'\n",
    "num_strings = 5\n",
    "\n",
    "# Apply spacy_pipeline function to preprocess documents\n",
    "processed_documents, most_common_words = tp.spacy_pipeline(\n",
    "    documents=documents,\n",
    "    custom_lemmatizer=custom_lemmatizer,\n",
    "    pipeline=pipeline,\n",
    "    stopwords_language=stopwords_language,\n",
    "    model=model,\n",
    "    num_strings=num_strings\n",
    ")\n",
    "\n",
    "print(\"Processed Documents:\")\n",
    "for i, doc in enumerate(processed_documents):\n",
    "    print(f\"Document {i + 1}: {' '.join(doc)}\")\n",
    "\n",
    "print(\"\\nMost Common Words:\")\n",
    "print(most_common_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 18. `dependency_parse_visualizer_text`\n",
    "\n",
    "**Description of the function**\n",
    "\n",
    "The `dependency_parse_visualizer_text` function is designed to visualize the dependency parsing or named entity recognition (NER) of a single text document. It leverages spaCy's visualization tool, DisplaCy, to render a graphical representation of linguistic features. The function is configurable, allowing you to specify the visualization style, whether you're working within a Jupyter notebook environment, and which spaCy model to use for parsing.\n",
    "\n",
    "**When is it Useful to Use this Function?**\n",
    "\n",
    "This function is beneficial in multiple scenarios:\n",
    "\n",
    "1. **Exploratory Data Analysis (EDA):** During the initial stages of text analysis, understanding the syntactic structure of your documents can be crucial. The visualization helps you to quickly grasp the relationships between words in a sentence or identify named entities.\n",
    "\n",
    "2. **Debugging NLP Pipelines:** If you're building an NLP pipeline that includes dependency parsing or named entity recognition, this function serves as a helpful debugging tool. You can visually confirm whether the spaCy model is interpreting the text as expected.\n",
    "\n",
    "3. **Educational Purposes:** If you're learning about dependency parsing or named entity recognition, visual representations can significantly aid your understanding of these complex linguistic features.\n",
    "\n",
    "4. **Reporting and Presentation:** You can use this function to generate visualizations for reports or presentations, making your findings more accessible to those without a technical background in linguistics or NLP.\n",
    "\n",
    "Here a practical dependency example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example document in Spanish\n",
    "document = \"El perro saltó sobre el gato.\"\n",
    "\n",
    "# Visualizing the dependency parse\n",
    "tp.dependency_parse_visualizer_text(document, style='dep', jupyter=True, model='es_core_news_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's visualize the named entities instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example document in Spanish\n",
    "document = \"El Banco Mundial decidió contactar al gobierno de Argentina para ofrecerle ayuda. Varios países como Estados Unidos, China y Rusia también ofrecieron su ayuda.\"\n",
    "\n",
    "# Visualizing the named entities\n",
    "tp.dependency_parse_visualizer_text(document, style='ent', jupyter=True, model='es_core_news_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we have a list of Spanish documents, and we want to preprocess them using the `spacy_pipeline` function with specific configuration options."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 19. Tutorial: Topic Modelling\n",
    "**Introduction**\n",
    "\n",
    "In the age of social media, Twitter has become a fertile ground for data mining, sentiment analysis, and various other natural language processing (NLP) tasks. However, dealing with Spanish tweets adds another layer of complexity due to language-specific nuances, slang, abbreviations, and other colloquial expressions. 'TidyX' aims to streamline the preprocessing pipeline for Spanish tweets, making them ready for various NLP tasks such as text classification, topic modeling, sentiment analysis, and more. In this tutorial, we will focus on a classification task based on Topic Modelling, showing preprocessing, modeling and results with real data snippets.\n",
    "\n",
    "**Context**\n",
    "\n",
    "Using data provided by [Barómetro de Xenofobia](https://barometrodexenofobia.org/), a world-class, renowned non-profit organization that quantifies the amount of hate speech against migrants on social media, we aim to classify the overall conversation related to migrants. This is a **common NLP task** that involves preprocessing poorly-written social media posts. Subsequently, these processed posts are fed into an unsupervised Topic Classification Model (LDA) to identify an optimal number of cluster topics. This helps reveal the main discussion points concerning Venezuelan migrants in Colombia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PREPARATIONS\n",
    "# Environment set-up\n",
    "import sys\n",
    "sys.path.insert(1, r'C:\\Users\\JOSE\\Desktop\\Trabajo\\Paper_no_supervisado\\Tidytweets')\n",
    "from tidyX import TextPreprocessor as tt\n",
    "import pandas as pd\n",
    "import random\n",
    "# Getting the data:\n",
    "# In this tutorial, we use a sample dataset of 799053 tweets related to Venezuelan migrants in Colombia.\n",
    "# The dataset is available in the data folder of the repository.\n",
    "# For efficiency we will only use a random sample of 1000 tweets\n",
    "n = 799053 #number of records in file\n",
    "s = 1000 #desired sample size\n",
    "skip = sorted(random.sample(range(n),n-s))\n",
    "tweets = pd.read_excel(r\"C:\\Users\\JOSE\\Desktop\\Trabajo\\Paper_no_supervisado\\Tidytweets\\data\\Base_Para_Labels.xlsx\", skiprows=skip, header=None, names=['Snippet'])\n",
    "tweets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Preprocessing Tweets**\n",
    "\n",
    "We will then use `preprocess` function to clean the sample and prepare it for modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaning_process = lambda x: tp.preprocess(x, delete_emojis=True, extract=False)\n",
    "tweets['Clean_tweets'] = tweets['Snippet'].apply(cleaning_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a random sample of the before and after with specific Tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_tweets = tweets.sample(5, random_state=1)  # You can change the random_state for different samples\n",
    "print(\"Before and After Text Cleaning:\")\n",
    "print('-' * 40)\n",
    "for index, row in sample_tweets.iterrows():\n",
    "    print(f\"Original: {row['Snippet']}\")\n",
    "    print(f\"Cleaned: {row['Clean_tweets']}\")\n",
    "    print('-' * 40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tokenize the dataset**\n",
    "\n",
    "This representation of the dataset will return a list of tokens per document. `spacy_pipeline` function returns a list of lists of processed lemmatized and stopword absent tweets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_cleaned_tweets = tp.spacy_pipeline(tweets['Clean_tweets'].to_list(), custom_lemmatizer=True, pipeline=['tokenize', 'lemmatizer'], stopwords_language='spanish', model='es_core_news_sm', num_strings=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a random sample of the before and after with specific Tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets['lemmatized_tweets'] = tokenized_cleaned_tweets\n",
    "sample_tweets = tweets.sample(5, random_state=1)  # You can change the random_state for different samples\n",
    "print(\"Before and After Text Cleaning:\")\n",
    "print('-' * 40)\n",
    "for index, row in sample_tweets.iterrows():\n",
    "    print(f\"Original: {row['Snippet']}\")\n",
    "    print(f\"Cleaned: {row['lemmatized_tweets']}\")\n",
    "    print('-' * 40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Seemingly used words and social media bad writting addressing**\n",
    "\n",
    "May you saw in the previous proccesed tweets that there are seemingly used or Out-of-Vocabulary (OOV) words that became evident after processing and cleaning the tweets showed. This words can be a result of bad spelling, common in social media, abbreviations, or other language rules.\n",
    "\n",
    "Here we propose a method to handle this limitations, some research related to this topic establishes local solutions to this condition, we invite the user to try this approach and also find some other resources to proccess the resulted lemmas. Some additional research to handle OOV words can be found in:\n",
    "\n",
    "1. [FastText](https://github.com/facebookresearch/fastText)\n",
    "2. [Kaggle NER Bi-LSTM](https://www.kaggle.com/code/jatinmittal0001/ner-bi-lstm-dealing-with-oov-words)\n",
    "3. [Contextual Spell Check](https://github.com/R1j1t/contextualSpellCheck)\n",
    "\n",
    "We use our `create_bol` function to find distances between lemmas, we are based on the premise that seemingly used lemmas ar far away from the original corpus and don't have a big apperance on it. Warning: Expect long kernel runs, this method evaluates each distance from a lemma N-1 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import itertools\n",
    "from collections import Counter\n",
    "# We take our list of lists and convert it to a list of strings\n",
    "flattened_list = list(itertools.chain.from_iterable(tokenized_cleaned_tweets))\n",
    "# Now we count the number of times each lemma appears in the list and sort the list in descending order\n",
    "word_count = Counter(flattened_list)\n",
    "sorted_words = sorted(word_count.items(), key=lambda x: x[1], reverse=True)\n",
    "sorted_words_only = [word for word, count in sorted_words]\n",
    "numpy_array = np.array(sorted_words_only)\n",
    "# Now we create our bag of lemmas\n",
    "bol_df = tp.create_bol(numpy_array, verbose=True)\n",
    "bol_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to select a specific subset of words that does not include our probable OOV or NEW words in the text processing. We will replace words using 85% confidence treshold soo we can infer what was intended to be written."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace each lemma in the original list of lists with its bow_name\n",
    "lemma_to_bow = dict(zip(bol_df['lemma'], bol_df['bow_name']))\n",
    "replaced_lemmas = [[lemma_to_bow.get(lemma, lemma) for lemma in doc] for doc in tokenized_cleaned_tweets]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here some random examples with the new mapping, you can inspect the differences in lemmas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets['new_clean_lemmas'] = replaced_lemmas\n",
    "sample_tweets = tweets.sample(10, random_state=1)  # You can change the random_state for different samples\n",
    "print(\"Before and After Text Cleaning:\")\n",
    "print('-' * 40)\n",
    "for index, row in sample_tweets.iterrows():\n",
    "    print(f\"Original: {row['Snippet']}\")\n",
    "    print(f\"Cleaned: {row['new_clean_lemmas']}\")\n",
    "    print('-' * 40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here, you can use this processed tweets to train different models and make your own empirical applications of NLP using social media data. However, we will show you a simple application of Topic Modelling using the data we processed. For more information about this methodology, we deliver some links to help understanding this type of unsupervised classification.\n",
    "\n",
    "Now we can plug this processed documents in a toy model to see some topics about Venezuelan migrants in Colombia:\n",
    "\n",
    "This model resolves in some steps:\n",
    "1. We iterate over the best combination of hyperparameters alpha, beta, and number of topics.\n",
    "2. We filter the results and pick the model with best coherence. We calculate Coherence Score and Perplexity of each LDA Topic Modeling implementation.\n",
    "3. We display a visualization of the topics found in the toy model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we create our initial variables for Topic Modeling\n",
    "import gensim\n",
    "from gensim import corpora \n",
    "import tqdm\n",
    "from gensim.models import CoherenceModel\n",
    "# Create Dictionary\n",
    "dictionary = corpora.Dictionary(replaced_lemmas)\n",
    "corpus = [dictionary.doc2bow(text) for text in replaced_lemmas]\n",
    "# A function that resolves our hyperparameters using a corpus and a dictionary\n",
    "def compute_coherence_perplexity_values(corpus, dictionary, k, a, b):\n",
    "    \n",
    "    lda_model = gensim.models.LdaMulticore(corpus=corpus,\n",
    "                                           id2word=dictionary,\n",
    "                                           num_topics=k, \n",
    "                                           random_state=100,\n",
    "                                           chunksize=100,\n",
    "                                           passes=10,\n",
    "                                           alpha=a,\n",
    "                                           eta=b,\n",
    "                                           workers=7)\n",
    "    \n",
    "    coherence_model_lda = CoherenceModel(model=lda_model, texts=replaced_lemmas, dictionary=dictionary, coherence='c_v')\n",
    "    \n",
    "    return (coherence_model_lda.get_coherence(),lda_model.log_perplexity(corpus))\n",
    "grid = {}\n",
    "grid['Validation_Set'] = {}\n",
    "# Topics range\n",
    "min_topics = 2\n",
    "max_topics = 11\n",
    "step_size = 1\n",
    "topics_range = range(min_topics, max_topics, step_size)\n",
    "# Alpha parameter\n",
    "alpha = list(np.arange(0.01, 1, 0.3))\n",
    "alpha.append('symmetric')\n",
    "alpha.append('asymmetric')\n",
    "# Beta parameter\n",
    "beta = list(np.arange(0.01, 1, 0.3))\n",
    "beta.append('symmetric')\n",
    "# Validation sets\n",
    "num_of_docs = len(corpus)\n",
    "corpus_sets = [# gensim.utils.ClippedCorpus(corpus, num_of_docs*0.25), \n",
    "               # gensim.utils.ClippedCorpus(corpus, num_of_docs*0.5), \n",
    "               gensim.utils.ClippedCorpus(corpus, int(num_of_docs*0.75)), \n",
    "               corpus]\n",
    "corpus_title = ['75% Corpus', '100% Corpus']\n",
    "model_results = {'Validation_Set': [],\n",
    "                 'Topics': [],\n",
    "                 'Alpha': [],\n",
    "                 'Beta': [],\n",
    "                 'Coherence': [],\n",
    "                 'Perplexity': []\n",
    "                }\n",
    "# Can take a long time to run\n",
    "if 1 == 1:\n",
    "    # This is the number of times we want to iterate to find optimal hyperparameters\n",
    "    pbar = tqdm.tqdm(total=20)\n",
    "    \n",
    "    # iterate through validation corpuses\n",
    "    for i in range(len(corpus_sets)):\n",
    "        # iterate through number of topics\n",
    "        for k in topics_range:\n",
    "            # iterate through alpha values\n",
    "            for a in alpha:\n",
    "                # iterare through beta values\n",
    "                for b in beta:\n",
    "                    # get the coherence score for the given parameters\n",
    "                    (cv, pp) = compute_coherence_perplexity_values(corpus=corpus_sets[i], dictionary=dictionary, \n",
    "                                                  k=k, a=a, b=b)\n",
    "                    # Save the model results\n",
    "                    model_results['Validation_Set'].append(corpus_title[i])\n",
    "                    model_results['Topics'].append(k)\n",
    "                    model_results['Alpha'].append(a)\n",
    "                    model_results['Beta'].append(b)\n",
    "                    model_results['Coherence'].append(cv)\n",
    "                    model_results['Perplexity'].append(pp)\n",
    "                    pbar.update(1)\n",
    "    pd.DataFrame(model_results).to_csv(r\"C:\\Users\\JOSE\\Desktop\\Trabajo\\Paper_no_supervisado\\Tidytweets\\data\\lda_tuning_results.csv\", index=False)\n",
    "    pbar.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to find the optimal model to train, let's see the results of our trainning pocess:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tabla_tunning = pd.read_csv(r\"C:\\Users\\JOSE\\Desktop\\Trabajo\\Paper_no_supervisado\\Tidytweets\\data\\lda_tuning_results.csv\")\n",
    "tabla_tunning = tabla_tunning.sort_values(by = 'Coherence', ascending = False)\n",
    "tabla_tunning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the model! We now pick the best result from the validation table created on the last step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pprint\n",
    "import pyLDAvis\n",
    "pyLDAvis.enable_notebook()\n",
    "import pyLDAvis.gensim_models\n",
    "lda_final_model = gensim.models.LdaMulticore(corpus=corpus,\n",
    "                                             id2word=dictionary,\n",
    "                                             num_topics=9,\n",
    "                                             random_state=100,\n",
    "                                             chunksize=100,\n",
    "                                             passes=30,\n",
    "                                             alpha='asymmetric',\n",
    "                                             eta=0.61,\n",
    "                                             workers=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have trained an optimized version of our toy model, we want to visually inspect the derived topics and see if we find some interesting patterns giving information related to the way people speaks about Venezuelan migrants in Colombia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[[(dictionary[id], freq) for id, freq in cp] for cp in corpus[:1]]\n",
    "\n",
    "pprint(lda_final_model.print_topics())\n",
    "doc_lda = lda_final_model[corpus]\n",
    "\n",
    "visxx = pyLDAvis.gensim_models.prepare(topic_model=lda_final_model, corpus=corpus, dictionary=dictionary)\n",
    "pyLDAvis.display(visxx)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "BX",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
