{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Elasticsearch index\n",
    "\n",
    "An [Elasicsearch](https://www.elastic.co/what-is/elasticsearch) index is a collection of one or more documents (JSON objects) with the same storage settings and data structrue.\n",
    "\n",
    "Elasticsearch can automatically assign datatypes to the documents' fields. However, when documents have more complex structure, i.e., they are nested, it is importand do define mapping (of data struture of documents hat will be stored in an index) before hand.\n",
    "\n",
    "In this example we change the way the documents are stored to improve search relevance.\n",
    "\n",
    "In this tutorial we use an [official Elasticsearch python](https://elasticsearch-py.readthedocs.io/en/master/) wrapper around the Elasticsearch's REST API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import dependencies\n",
    "from elasticsearch import Elasticsearch\n",
    "import wikipediaapi\n",
    "from slugify import slugify\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a connection instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Elasticsearch(\"http://localhost:9200\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with _indices and _cat APIs\n",
    "\n",
    "Information about existing indices can be retrieved from Elasticsearch _indices API. Here is how we can get document counts in individual indices, multiple indices and all indicies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['machine-learning',\n",
       " 'marvel-comics',\n",
       " 'science-fiction-television',\n",
       " 'pandemics',\n",
       " 'natural-language-processing',\n",
       " 'presidents-of-the-united-states',\n",
       " 'coronaviridae',\n",
       " 'american-science-fiction-television-series',\n",
       " 'marvel-comics-editors-in-chief',\n",
       " 'american-comics-writers']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indices = list(client.indices.get_alias(\"_all\").keys())\n",
    "indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refresh index and count documents in each of existing indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "machine-learning [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '222'}]\n",
      "marvel-comics [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '14'}]\n",
      "science-fiction-television [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '2'}]\n",
      "pandemics [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '24'}]\n",
      "natural-language-processing [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '180'}]\n",
      "presidents-of-the-united-states [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '48'}]\n",
      "coronaviridae [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '17'}]\n",
      "american-science-fiction-television-series [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '48'}]\n",
      "marvel-comics-editors-in-chief [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '14'}]\n",
      "american-comics-writers [{'epoch': '1587668551', 'timestamp': '19:02:31', 'count': '929'}]\n"
     ]
    }
   ],
   "source": [
    "client.indices.refresh('marvel-comics')\n",
    "for i in indices:\n",
    "    print(i, client.cat.count(i, params={\"format\": \"json\"}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'epoch': '1587668589', 'timestamp': '19:03:09', 'count': '62'}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Count documents in multiple indices\n",
    "client.cat.count(['marvel-comics', 'presidents-of-the-united-states'], params={\"format\": \"json\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'epoch': '1587668589', 'timestamp': '19:03:09', 'count': '1498'}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Count documents in all indices\n",
    "client.cat.count(\"_all\", params={\"format\": \"json\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check an index mapping - default\n",
    "pprint(client.indices.get_mapping(\"marvel-comics\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check an index mapping - nested\n",
    "pprint(client.indices.get_mapping(\"coronaviridae\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete all documents in a single index\n",
    "client.indices.delete(\"coronaviridae\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'marvel-comics': {'aliases': {}},\n",
       " 'presidents-of-the-united-states': {'aliases': {}},\n",
       " 'american-comics-writers': {'aliases': {}}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check currently available indices\n",
    "client.indices.get_alias(\"_all\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# client.indices.delete(index=[\"marvel-comics-editors-in-chief\", \"21st-century-american-comedians\"])\n",
    "# client.indices.get_alias(\"_all\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.delete(\"_all\")\n",
    "client.indices.get_alias(\"_all\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create new indices and documents\n",
    "\n",
    "We will load wikipedia articles from selected categories using [Wikipedia-API](https://pypi.org/project/Wikipedia-API/) and create a few Elasticsearch indices with default index settings and mapping.\n",
    "Elasticsearch is smart enough to figure out what datatype each field of the document is, as long as the data structure of the document s not too complex.\n",
    "\n",
    "Let us concider an example of a document with the following structure:\n",
    "\n",
    "```\n",
    "        {title='some string'\n",
    "         page_id='numerical datatype'\n",
    "         source='string, url'\n",
    "         text='some string'}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = ['Presidents of the United States', \n",
    "              'Marvel Comics', \n",
    "              'American comics writers',\n",
    "              'Marvel Comics editors-in-chief']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Document:\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.title = ''\n",
    "        self.page_id = None\n",
    "        self.source = ''\n",
    "        self.text = ''\n",
    "        \n",
    "    def __if_exists(self, page_id, index=\"\"):\n",
    "        '''\n",
    "        A private method to check if the article already exists in the database\n",
    "        with a goal to avoid duplication\n",
    "        '''\n",
    "        \n",
    "        return client.search(index=index, \n",
    "                             body={\"query\": \n",
    "                                   {\"match\": \n",
    "                                    {\"page_id\": page_id}\n",
    "                                   }})['hits']['total']['value']\n",
    "        \n",
    "    def insert(self, title, page_id, url, text, index):\n",
    "        ''' Add a new document to the index'''\n",
    "        \n",
    "        self.title=title\n",
    "        self.page_id=page_id\n",
    "        self.source=url\n",
    "        self.text=text\n",
    "        self.body = {'title': self.title,\n",
    "            'page_id': self.page_id,\n",
    "            'source':self.source,\n",
    "            'text': self.text}\n",
    "        \n",
    "        if self.__if_exists(page_id) == 0:\n",
    "        \n",
    "            try:\n",
    "                client.index(index=index, body=self.body)\n",
    "#                 print(f\"Sucess! The article {self.title} was added to index {index}\")\n",
    "            except error:\n",
    "                print(\"Something went wrong\", error)\n",
    "                \n",
    "        else:\n",
    "            print(f\"Article {self.title} is already in the database\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Article Los Bros Hernandez is already in the database\n",
      "Article C. B. Cebulski is already in the database\n",
      "Article Gerry Conway is already in the database\n",
      "Article Tom DeFalco is already in the database\n",
      "Article Stan Lee is already in the database\n",
      "Article Jim Shooter is already in the database\n",
      "Article Joe Simon is already in the database\n",
      "Article Roy Thomas is already in the database\n",
      "Article Len Wein is already in the database\n",
      "Article Marv Wolfman is already in the database\n"
     ]
    }
   ],
   "source": [
    "def simple_wiki_doc(category):\n",
    "    \n",
    "    if type(category) is not list: category = [ category ]\n",
    "\n",
    "    wiki_wiki = wikipediaapi.Wikipedia('en')\n",
    "    \n",
    "    for c in category:\n",
    "\n",
    "        cat = wiki_wiki.page(f\"Category:{c}\")\n",
    "\n",
    "        for key in cat.categorymembers.keys():\n",
    "            page = wiki_wiki.page(key)\n",
    "\n",
    "            if not \"Category:\" in page.title:\n",
    "                \n",
    "                doc = Document()\n",
    "                doc.insert(page.title, page.pageid, page.fullurl, page.text, index=slugify(c))\n",
    "                \n",
    "simple_wiki_doc(categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if the indices were added and what is the document count in each index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "marvel-comics [{'epoch': '1585537470', 'timestamp': '03:04:30', 'count': '14'}]\n",
      "marvel-comics-editors-in-chief [{'epoch': '1585537470', 'timestamp': '03:04:30', 'count': '5'}]\n",
      "presidents-of-the-united-states [{'epoch': '1585537470', 'timestamp': '03:04:30', 'count': '48'}]\n",
      "american-comics-writers [{'epoch': '1585537470', 'timestamp': '03:04:30', 'count': '938'}]\n"
     ]
    }
   ],
   "source": [
    "indices1 = list(client.indices.get_alias(\"_all\").keys())\n",
    "for i in indices1:\n",
    "    print(i, client.cat.count(i, params={\"format\": \"json\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indexing nested documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default data structure may not be sufficient. We need our documents to be store in sections to improve search accuracy, like so:\n",
    "\n",
    "\n",
    "```\n",
    "        {title='text'\n",
    "         page_id='numerical datatype'\n",
    "         source='text'\n",
    "         text= [{\n",
    "                 'section_title': 'text',\n",
    "                 'section_num': 'integer'\n",
    "                 'section_content': 'text'\n",
    "                 },\n",
    "                 {\n",
    "                 'section_title': 'text',\n",
    "                 'section_num': 'integer'\n",
    "                 'section_content': 'text'\n",
    "                 },\n",
    "                 {\n",
    "                 'section_title': 'text',\n",
    "                 'section_num': 'integer'\n",
    "                 'section_content': 'text'\n",
    "                 },\n",
    "            }\n",
    "```\n",
    "It this a nested data structure. To be able to add such a document to an index, the data structure of the index has to be mapped first. We will havo to create an index before we add documents document with teh following call:\n",
    "\n",
    "```\n",
    "    client.indices.create(index='pandemics', body={\"mappings\":mapping})\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapping = {\n",
    "    \"properties\": {\n",
    "        \n",
    "            \"text\": {\n",
    "                \"type\": \"nested\",\n",
    "                \"properties\":{\n",
    "                    \"section_num\": {\"type\":\"integer\"},\n",
    "                    \"section_title\": {\"type\":\"text\"},\n",
    "                    \"section_content\": {\"type\":\"text\"}\n",
    "                }\n",
    "            },\n",
    "        \n",
    "            \"title\": {\n",
    "                \"type\": \"text\"\n",
    "            },\n",
    "        \n",
    "            \"source\": {\n",
    "                \"type\": \"text\"\n",
    "            },\n",
    "        \n",
    "            \"page_id\": {\n",
    "                \"type\": \"long\"\n",
    "            },\n",
    "            \n",
    "        }\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create new indices with nested data structure\n",
    "\n",
    "Some wikipedia articles in categories are large and may have multiple levels of subsections. We chose to parse the data from full text instead of drawing sections and subsections from wikipedia API to achieve uniform depth of nested dictionaries within a single index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load wikipedia articles from the following categories:\n",
    "categories2 = ['Machine learning',\n",
    "              'Natural language processing',\n",
    "              'Coronaviridae',\n",
    "              '21st-century American comedians',\n",
    "              'Pandemics']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_article(article):\n",
    "    ''' Parce wikipedia articles from the full article text'''\n",
    "    \n",
    "    text = article.text\n",
    "    # get section titles for the existing sections\n",
    "    section_titles = [sec.title for sec in article.sections]\n",
    "    \n",
    "    # initiate the sections dictionary with a summary (0th section) \n",
    "    sections = [{'section_num': 0},\n",
    "                {'section_title': \"Summary\"},\n",
    "                {'section_content': article.summary}]\n",
    "    \n",
    "    for i, title in enumerate(section_titles[::-1]):\n",
    "\n",
    "        num = len(section_titles)-i\n",
    "        if len(text.split(f\"\\n\\n{title}\")) == 2:\n",
    "            section_dict = {\"section_num\": num,\n",
    "                            \"section_title\": title,\n",
    "                            \"section_content\": text.split(f\"\\n\\n{title}\")[-1]}\n",
    "            sections.append(section_dict)\n",
    "            text = text.split(f\"\\n\\n{title}\")[0]\n",
    "        else:\n",
    "            pass\n",
    "            \n",
    "        \n",
    "    return sections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Article Multiple instance learning is already in the database\n",
      "Article Training, validation, and test sets is already in the database\n",
      "Article Bag-of-words model is already in the database\n",
      "Article Deeplearning4j is already in the database\n",
      "Article Document classification is already in the database\n",
      "Article Documenting Hate is already in the database\n",
      "Article Grammar induction is already in the database\n",
      "Article Multimodal sentiment analysis is already in the database\n",
      "Article Native-language identification is already in the database\n",
      "Article Semantic folding is already in the database\n",
      "Article Coronavirus is already in the database\n",
      "Article Sean Clements is already in the database\n",
      "Article Ernest Cline is already in the database\n",
      "Article Hayes Davenport is already in the database\n",
      "Article Adam Felber is already in the database\n",
      "Article Rashida Jones is already in the database\n",
      "Article Taran Killam is already in the database\n",
      "Article Steve Melcher is already in the database\n",
      "Article Patton Oswalt is already in the database\n",
      "Article Brian Posehn is already in the database\n",
      "Article Carlos Saldaña is already in the database\n",
      "Article Edward Savio is already in the database\n",
      "Article Rob Schrab is already in the database\n",
      "Article Larry Siegel is already in the database\n",
      "Article Kevin Smith is already in the database\n",
      "Article Johns Hopkins Center for Health Security is already in the database\n"
     ]
    }
   ],
   "source": [
    "def search_insert_wiki(category, mapping):\n",
    "    \n",
    "    if type(category) is not list: category = [ category ]\n",
    "\n",
    "    wiki_wiki = wikipediaapi.Wikipedia('en')\n",
    "    \n",
    "    for c in category:\n",
    "        \n",
    "        try:\n",
    "                    \n",
    "            '''Create and empty index with predefined data structure'''\n",
    "            client.indices.create(index=slugify(c), body={\"mappings\":mapping})\n",
    "            \n",
    "            '''Access the list of wikipedia articles in category c'''\n",
    "            cat = wiki_wiki.page(f\"Category:{c}\")\n",
    "            \n",
    "            ''' Parse and add articles in the category to database'''\n",
    "            for key in cat.categorymembers.keys():\n",
    "                page = wiki_wiki.page(key)\n",
    "\n",
    "                if not \"Category:\" in page.title:\n",
    "\n",
    "                    text = parse_article(page)\n",
    "                    doc = Document()\n",
    "                    doc.insert(page.title, page.pageid, page.fullurl, text, index=slugify(c))\n",
    "\n",
    "\n",
    "        except error:\n",
    "            '''Skip category if it alredy exists in indices'''\n",
    "            print(\"Something went wrong\", error)\n",
    "            \n",
    "search_insert_wiki(categories2, mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pandemics': {'aliases': {}},\n",
       " 'coronaviridae': {'aliases': {}},\n",
       " 'presidents-of-the-united-states': {'aliases': {}},\n",
       " 'natural-language-processing': {'aliases': {}},\n",
       " 'marvel-comics-editors-in-chief': {'aliases': {}},\n",
       " 'american-comics-writers': {'aliases': {}},\n",
       " 'marvel-comics': {'aliases': {}},\n",
       " '21st-century-american-comedians': {'aliases': {}},\n",
       " 'machine-learning': {'aliases': {}}}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.get_alias(\"_all\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:nbooks] *",
   "language": "python",
   "name": "conda-env-nbooks-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
