{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Improving search accuracy in Elastisearch\n",
    "\n",
    "Elasticsearch ranks documents found in response to a quary by a score, roughly described as a term frequency normalized by the field length. A detailed explanation of how relevancy scores are calculated in Elasticsearch can be found [here](https://qbox.io/blog/practical-guide-elasticsearch-scoring-relevancy)\n",
    "\n",
    "In this tutorial we experiment with different types of Elasticsearch queries and their performance with different document indexing approaches. We touch on indexing basics [here](https://ovbondarenko.github.io/elasticsearch/index.html).\n",
    "\n",
    "Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import dependencies\n",
    "from elasticsearch import Elasticsearch\n",
    "import wikipediaapi\n",
    "from slugify import slugify\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start a new Elasticsearch connection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Elasticsearch(\"http://localhost:9200\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our small Elasticsearch library contains several indices. One group of indices has a default data structure, and the second has a nested structure with predefined field mappings.\n",
    "\n",
    "Here is all the indices in the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'wikipedia-simple': {'aliases': {}},\n",
       " 'natural-language-processing': {'aliases': {}},\n",
       " 'pandemics': {'aliases': {}},\n",
       " 'coronaviridae': {'aliases': {}},\n",
       " 'science-fiction-television': {'aliases': {}},\n",
       " 'american-comics-writers': {'aliases': {}},\n",
       " 'machine-learning': {'aliases': {}},\n",
       " 'american-science-fiction-television-series': {'aliases': {}},\n",
       " 'marvel-comics': {'aliases': {}},\n",
       " 'marvel-comics-editors-in-chief': {'aliases': {}},\n",
       " 'presidents-of-the-united-states': {'aliases': {}}}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.get_alias(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index called 'presidents-of-the-united-states' has the fillowing structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'wikipedia-simple': {'mappings': {'properties': {'page_id': {'type': 'long'},\n",
       "    'source': {'type': 'text',\n",
       "     'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}},\n",
       "    'text': {'type': 'text',\n",
       "     'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}},\n",
       "    'title': {'type': 'text',\n",
       "     'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}}}}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.get_mapping('wikipedia-simple')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Elasticsearch(\"http://localhost:9200\") # or name of local network, e.g. client = Elasticsearch(\"es:9200\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'wikipedia-simple': {'mappings': {'properties': {'page_id': {'type': 'long'},\n",
      "                                                  'source': {'fields': {'keyword': {'ignore_above': 256,\n",
      "                                                                                    'type': 'keyword'}},\n",
      "                                                             'type': 'text'},\n",
      "                                                  'text': {'fields': {'keyword': {'ignore_above': 256,\n",
      "                                                                                  'type': 'keyword'}},\n",
      "                                                           'type': 'text'},\n",
      "                                                  'title': {'fields': {'keyword': {'ignore_above': 256,\n",
      "                                                                                   'type': 'keyword'}},\n",
      "                                                            'type': 'text'}}}}}\n"
     ]
    }
   ],
   "source": [
    "pprint(client.indices.get_mapping('wikipedia-simple'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'machine-learning': {'mappings': {'properties': {'page_id': {'type': 'long'},\n",
      "                                                  'references': {'properties': {'section_content': {'type': 'text'},\n",
      "                                                                                'section_num': {'type': 'integer'},\n",
      "                                                                                'section_title': {'type': 'text'}},\n",
      "                                                                 'type': 'nested'},\n",
      "                                                  'source': {'type': 'text'},\n",
      "                                                  'text': {'properties': {'section_content': {'type': 'text'},\n",
      "                                                                          'section_num': {'type': 'integer'},\n",
      "                                                                          'section_title': {'type': 'text'}},\n",
      "                                                           'type': 'nested'},\n",
      "                                                  'title': {'type': 'text'}}}}}\n"
     ]
    }
   ],
   "source": [
    "pprint(client.indices.get_mapping('machine-learning'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'presidents-of-the-united-states': {'mappings': {'properties': {'page_id': {'type': 'long'},\n",
       "    'source': {'type': 'text',\n",
       "     'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}},\n",
       "    'text': {'type': 'text',\n",
       "     'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}},\n",
       "    'title': {'type': 'text',\n",
       "     'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}}}}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.get_mapping('presidents-of-the-united-states')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is an example of an index with nested structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'coronaviridae': {'mappings': {'properties': {'page_id': {'type': 'long'},\n",
       "    'source': {'type': 'text'},\n",
       "    'text': {'type': 'nested',\n",
       "     'properties': {'section_content': {'type': 'text'},\n",
       "      'section_num': {'type': 'integer'},\n",
       "      'section_title': {'type': 'text'}}},\n",
       "    'title': {'type': 'text'}}}}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.indices.get_mapping('coronaviridae')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will show how to build queries for both types of indices to get better text search results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General query types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Elaticsearch has three types of queries: match, term and range.\n",
    "\n",
    "- **Match query is a standard query for full text search. Performed against analyzed text.** We will focus on match queries, because they are most useful for free text search.\n",
    "- Term query is looking for an eaxact match\n",
    "- Range is used for finding numerical values\n",
    "\n",
    "We will start with search in indices that have a default data structure. Let us check what this means in terms of mapping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query indices with single text fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "index1 = ['presidents-of-the-united-states', \n",
    "         'marvel-comics-editors-in-chief', \n",
    "         'marvel-comics',\n",
    "         'american-comics-writers']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "question1 = \"When stan Lee was born?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple full text search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'question1' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-31787c149c13>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m body = {\"query\": \n\u001b[0;32m      2\u001b[0m         {\"match\": \n\u001b[1;32m----> 3\u001b[1;33m          \u001b[1;33m{\u001b[0m\u001b[1;34m\"text\"\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mquestion1\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m         },\n\u001b[0;32m      5\u001b[0m         \"highlight\": {\n",
      "\u001b[1;31mNameError\u001b[0m: name 'question1' is not defined"
     ]
    }
   ],
   "source": [
    "body1 = {\"query\": {\"match\": {\"text\": question1}}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_search API response to a query includes several metafields, but we are mostly interest in the relevance score ('_score')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example data strucure\n",
      "----------------------\n",
      "dict_keys(['_index', '_type', '_id', '_score', '_source'])\n",
      "dict_keys(['title', 'page_id', 'source', 'text'])\n"
     ]
    }
   ],
   "source": [
    "docs = client.search(body1, index=index1)\n",
    "\n",
    "print(\"Example data strucure\")\n",
    "print(\"----------------------\")\n",
    "pprint(docs['hits']['hits'][0].keys())\n",
    "pprint(docs['hits']['hits'][0]['_source'].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: When stan Lee was born?\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Stan Lee | Relevance score 9.7426195 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Stan_Lee\n",
      "Result 1: Leon Lazarus | Relevance score 9.079921 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Leon_Lazarus\n",
      "Result 2: Roy Thomas | Relevance score 8.832645 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Roy_Thomas\n",
      "Result 3: Jim Salicrup | Relevance score 8.592315 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Jim_Salicrup\n",
      "Result 4: Danny Fingeroth | Relevance score 8.055832 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Danny_Fingeroth\n",
      "Result 5: Jack Kirby | Relevance score 7.9232183 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Jack_Kirby\n",
      "Result 6: Steve Ditko | Relevance score 7.883254 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Steve_Ditko\n",
      "Result 7: Daniel Keyes | Relevance score 7.6234007 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Daniel_Keyes\n",
      "Result 8: Jim Steranko | Relevance score 7.4930425 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Jim_Steranko\n",
      "Result 9: Colleen Doran | Relevance score 7.2435718 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Colleen_Doran\n"
     ]
    }
   ],
   "source": [
    "print(f\"Question: {question1}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "\n",
    "docs1 = client.search(body, index=\"\")\n",
    "\n",
    "for i, doc in enumerate(docs1[\"hits\"][\"hits\"]):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    idx = doc['_index']\n",
    "    url = doc['_source']['source']\n",
    "    print(f'Result {i}: {title} | Relevance score {score} | Index {idx}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the search returns the most relevant article first. However it is goot to run another test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: When Barack Obama was inaugurated?\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Jeff Mariotte | Relevance score 12.4784 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Jeff_Mariotte\n",
      "Result 1: Ta-Nehisi Coates | Relevance score 11.486142 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Ta-Nehisi_Coates\n",
      "Result 2: Amber Benson | Relevance score 9.384043 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Amber_Benson\n",
      "Result 3: Eric Millikin | Relevance score 8.75001 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Eric_Millikin\n",
      "Result 4: Jason Rubin | Relevance score 8.663446 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Jason_Rubin\n",
      "Result 5: Rashida Jones | Relevance score 7.785064 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Rashida_Jones\n",
      "Result 6: Barack Obama | Relevance score 7.342956 | Index presidents-of-the-united-states\n",
      "    Link: https://en.wikipedia.org/wiki/Barack_Obama\n",
      "Result 7: Alex Ross | Relevance score 6.783415 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Alex_Ross\n",
      "Result 8: Bill Clinton | Relevance score 6.571047 | Index presidents-of-the-united-states\n",
      "    Link: https://en.wikipedia.org/wiki/Bill_Clinton\n",
      "Result 9: Floyd Gottfredson | Relevance score 6.5376973 | Index american-comics-writers\n",
      "    Link: https://en.wikipedia.org/wiki/Floyd_Gottfredson\n"
     ]
    }
   ],
   "source": [
    "question2 = \"When Barack Obama was inaugurated?\"\n",
    "body2 = {\"query\": {\"match\": {\"text\": question2}}}\n",
    "\n",
    "print(f\"Question: {question2}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "\n",
    "docs2 = client.search(body2, index=index1)\n",
    "\n",
    "for i, doc in enumerate(docs2[\"hits\"][\"hits\"]):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    idx = doc['_index']\n",
    "    url = doc['_source']['source']\n",
    "    print(f'Result {i}: {title} | Relevance score {score} | Index {idx}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the highest score calculated by Elasticsearch does not always correspond to the most relevant article. How can we improve the results?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Match phrase query\n",
    "\n",
    "* All the terms must appear in the field, and in the same order\n",
    "* Can add custom query analyzer\n",
    "\n",
    "It may be very useful of we are looking for a specific set of words, like \"Barack Obama\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Barack Obama\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Barack Obama | Relevance score 7.8283415\n",
      "    Link: https://en.wikipedia.org/wiki/Barack_Obama\n"
     ]
    }
   ],
   "source": [
    "phrase = \"Barack Obama\"\n",
    "\n",
    "body3 = \\\n",
    "    {\n",
    "      \"query\": {\"match_phrase\": \n",
    "                    {\"title\": phrase}\n",
    "               }\n",
    "    }\n",
    "docs3 = client.search(body3, index=index1)\n",
    "\n",
    "print(f\"Question: {phrase}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "for i, doc in enumerate(docs3['hits']['hits']):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    url = doc['_source']['source']\n",
    "    print(f'Result {i}: {title} | Relevance score {score}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: When Barack Obama was born?\n",
      "\n",
      "Search results:\n",
      "----------------------\n"
     ]
    }
   ],
   "source": [
    "phrase = \"When Barack Obama was born?\"\n",
    "\n",
    "body3 = \\\n",
    "    {\n",
    "      \"query\": {\"match_phrase\": \n",
    "                    {\"title\": phrase}\n",
    "               }\n",
    "    }\n",
    "docs3 = client.search(body3, index=index1)\n",
    "\n",
    "print(f\"Question: {phrase}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "for i, doc in enumerate(docs3['hits']['hits']):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    url = doc['_source']['source']\n",
    "    print(f'Result {i}: {title} | Relevance score {score}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, it does not work when our question is has more natural and uncertain form."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-field search\n",
    "\n",
    "We can try to improve the scoring by searching in multiple fields. Our indices in this example have only two text fields - text and title, we will search in both of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: When Barack Obama was inaugurated?\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Jeff Mariotte | Relevance score 12.4784\n",
      "    Link: https://en.wikipedia.org/wiki/Jeff_Mariotte\n",
      "Result 1: Ta-Nehisi Coates | Relevance score 11.486142\n",
      "    Link: https://en.wikipedia.org/wiki/Ta-Nehisi_Coates\n",
      "Result 2: Amber Benson | Relevance score 9.384043\n",
      "    Link: https://en.wikipedia.org/wiki/Amber_Benson\n",
      "Result 3: Eric Millikin | Relevance score 8.75001\n",
      "    Link: https://en.wikipedia.org/wiki/Eric_Millikin\n",
      "Result 4: Jason Rubin | Relevance score 8.663446\n",
      "    Link: https://en.wikipedia.org/wiki/Jason_Rubin\n",
      "Result 5: Barack Obama | Relevance score 7.8283415\n",
      "    Link: https://en.wikipedia.org/wiki/Barack_Obama\n",
      "Result 6: Rashida Jones | Relevance score 7.785064\n",
      "    Link: https://en.wikipedia.org/wiki/Rashida_Jones\n",
      "Result 7: Alex Ross | Relevance score 6.783415\n",
      "    Link: https://en.wikipedia.org/wiki/Alex_Ross\n",
      "Result 8: Bill Clinton | Relevance score 6.571047\n",
      "    Link: https://en.wikipedia.org/wiki/Bill_Clinton\n",
      "Result 9: Floyd Gottfredson | Relevance score 6.5376973\n",
      "    Link: https://en.wikipedia.org/wiki/Floyd_Gottfredson\n"
     ]
    }
   ],
   "source": [
    "body4 = \\\n",
    "{\n",
    "  \"query\": {\n",
    "    \"multi_match\" : {\n",
    "      \"query\":    question2, \n",
    "      \"fields\": [ \"title\", \"text\" ] \n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "docs4 = client.search(body4, index=index1)\n",
    "\n",
    "print(f\"Question: {question2}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "for i, doc in enumerate(docs4[\"hits\"][\"hits\"]):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    url = doc['_source']['source']\n",
    "    print(f'Result {i}: {title} | Relevance score {score}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the most relevant \"Barack Obama\" article is ranked 6th instead of 7th, it is still not at the top of the list. The improvement is pretty minor. This is because other articles may have multiples mentions of Barack Obama too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dynamic boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dynamic boosting allows to boost some document fields during query. We can boost the score calculated from the title field, since we believe that the article that has the keywords from our query in its title is more relevant than the one that doesn't. In our Barack Obama example this is the correct assumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: When Barack Obama was inaugurated?\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Barack Obama | Relevance score 30.827982\n",
      "    Link: https://en.wikipedia.org/wiki/Barack_Obama\n",
      "Result 1: Jeff Mariotte | Relevance score 12.4784\n",
      "    Link: https://en.wikipedia.org/wiki/Jeff_Mariotte\n",
      "Result 2: Ta-Nehisi Coates | Relevance score 11.486142\n",
      "    Link: https://en.wikipedia.org/wiki/Ta-Nehisi_Coates\n",
      "Result 3: Amber Benson | Relevance score 9.384043\n",
      "    Link: https://en.wikipedia.org/wiki/Amber_Benson\n",
      "Result 4: Eric Millikin | Relevance score 8.75001\n",
      "    Link: https://en.wikipedia.org/wiki/Eric_Millikin\n",
      "Result 5: Jason Rubin | Relevance score 8.663446\n",
      "    Link: https://en.wikipedia.org/wiki/Jason_Rubin\n",
      "Result 6: Rashida Jones | Relevance score 7.785064\n",
      "    Link: https://en.wikipedia.org/wiki/Rashida_Jones\n",
      "Result 7: Alex Ross | Relevance score 6.783415\n",
      "    Link: https://en.wikipedia.org/wiki/Alex_Ross\n",
      "Result 8: Bill Clinton | Relevance score 6.571047\n",
      "    Link: https://en.wikipedia.org/wiki/Bill_Clinton\n",
      "Result 9: Floyd Gottfredson | Relevance score 6.5376973\n",
      "    Link: https://en.wikipedia.org/wiki/Floyd_Gottfredson\n"
     ]
    }
   ],
   "source": [
    "body5 = \\\n",
    "    {\n",
    "      \"query\": {\n",
    "        \"bool\": {\n",
    "          \"should\": [\n",
    "            {\n",
    "              \"match\": {\n",
    "                \"title\": {\n",
    "                  \"query\": question2,\n",
    "                  \"boost\": 3\n",
    "                }\n",
    "              }\n",
    "            },\n",
    "            {\n",
    "              \"match\": { \n",
    "                \"text\": question2\n",
    "              }\n",
    "            }\n",
    "          ]\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "\n",
    "docs5 = client.search(body5, index=index1)\n",
    "\n",
    "print(f\"Question: {question2}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "\n",
    "for i, doc in enumerate(docs5[\"hits\"][\"hits\"]):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    url = doc['_source']['source']\n",
    "    print(f'Result {i}: {title} | Relevance score {score}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better! Now the most useful article s at the top of the list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multifield search with boosing - short version\n",
    "\n",
    "This is just another way to make a multi-field query with boosting which is more concise then the first one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: When Barack Obama was inaugurated?\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Barack Obama | Relevance score 23.485025\n",
      "    Link: https://en.wikipedia.org/wiki/Barack_Obama\n",
      "Result 1: Jeff Mariotte | Relevance score 12.4784\n",
      "    Link: https://en.wikipedia.org/wiki/Jeff_Mariotte\n",
      "Result 2: Ta-Nehisi Coates | Relevance score 11.486142\n",
      "    Link: https://en.wikipedia.org/wiki/Ta-Nehisi_Coates\n",
      "Result 3: Amber Benson | Relevance score 9.384043\n",
      "    Link: https://en.wikipedia.org/wiki/Amber_Benson\n",
      "Result 4: Eric Millikin | Relevance score 8.75001\n",
      "    Link: https://en.wikipedia.org/wiki/Eric_Millikin\n",
      "Result 5: Jason Rubin | Relevance score 8.663446\n",
      "    Link: https://en.wikipedia.org/wiki/Jason_Rubin\n",
      "Result 6: Rashida Jones | Relevance score 7.785064\n",
      "    Link: https://en.wikipedia.org/wiki/Rashida_Jones\n",
      "Result 7: Alex Ross | Relevance score 6.783415\n",
      "    Link: https://en.wikipedia.org/wiki/Alex_Ross\n",
      "Result 8: Bill Clinton | Relevance score 6.571047\n",
      "    Link: https://en.wikipedia.org/wiki/Bill_Clinton\n",
      "Result 9: Floyd Gottfredson | Relevance score 6.5376973\n",
      "    Link: https://en.wikipedia.org/wiki/Floyd_Gottfredson\n"
     ]
    }
   ],
   "source": [
    "body6 = \\\n",
    "{\n",
    "  \"query\": {\n",
    "    \"multi_match\" : {\n",
    "      \"query\":    question2, \n",
    "      \"fields\": [ \"title^3\", \"text\" ] \n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "docs6 = client.search(body6, index=\"\")\n",
    "\n",
    "print(f\"Question: {question2}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "\n",
    "for i, doc in enumerate(docs6[\"hits\"][\"hits\"]):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    url = doc['_source']['source']\n",
    "    print(f'Result {i}: {title} | Relevance score {score}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Default Elasticsearch tokenizer is N-gram tokenizer. The tokenizer analyzes the input text and produces letter N-grams with N of 1 and 2:\n",
    "```\n",
    "{\n",
    "  \"tokenizer\": \"ngram\",\n",
    "  \"text\": \"Quick Fox\"\n",
    "}\n",
    "\n",
    "[ Q, Qu, u, ui, i, ic, c, ck, k, \"k \", \" \", \" F\", F, Fo, o, ox, x ]\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying nested fields\n",
    "\n",
    "Here is the indices in our library with a nested structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "index2 = [\n",
    "    'natural-language-processing',\n",
    "    'machine-learning',\n",
    "    'coronaviridae',\n",
    "    'pandemics'\n",
    "]"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "By confining query to a section that we know to be particularly relevant, in this case Summary, we can get good results without boosting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "question3 = \"What is natural language processing?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: What is natural language processing?\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Conditional random field | Relevance score 21.414635 | 4118276 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Conditional_random_field\n",
      "Result 1: Paraphrasing (computational linguistics) | Relevance score 17.06732 | 56142183 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Paraphrasing_(computational_linguistics)\n",
      "Result 2: Documenting Hate | Relevance score 16.588968 | 54994687 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Documenting_Hate\n",
      "Result 3: Bag-of-words model | Relevance score 16.155718 | 14003441 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Bag-of-words_model\n",
      "Result 4: Constrained conditional model | Relevance score 15.544483 | 28255458 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Constrained_conditional_model\n",
      "Result 5: Structured sparsity regularization | Relevance score 15.375017 | 48844125 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Structured_sparsity_regularization\n",
      "Result 6: Knowledge distillation | Relevance score 15.303329 | 62295363 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Knowledge_distillation\n",
      "Result 7: Convolutional neural network | Relevance score 14.754961 | 40409788 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Convolutional_neural_network\n",
      "Result 8: Semantic folding | Relevance score 14.433018 | 50222574 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Semantic_folding\n",
      "Result 9: Time series | Relevance score 14.091452 | 406624 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Time_series\n"
     ]
    }
   ],
   "source": [
    "body7={\n",
    "    \"query\": {\n",
    "        \"nested\":{\n",
    "            \"path\":\"text\",\n",
    "            \"query\":{\n",
    "                \"bool\": {\n",
    "                \"must\": [\n",
    "                    {\"match\":{\"text.section_content\":{'query': question3}}},\n",
    "                    {\"match\":{\"text.section_title\":{'query': 'Summary', \"boost\": 3}}},\n",
    "                    ]\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "docs7 = client.search(body7, index=index2)\n",
    "\n",
    "print(f\"Question: {question3}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "\n",
    "for i, doc in enumerate(docs7[\"hits\"][\"hits\"]):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    url = doc['_source']['source']\n",
    "    pageid =doc['_source']['page_id']\n",
    "    ind = doc[\"_index\"]\n",
    "    print(f'Result {i}: {title} | Relevance score {score} | {pageid} | {ind}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, it didn't quite work out, because the article titled \"Natural language processing\" came up only on the 8th place. Let's try the multi-field search with the title field boosting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: What is natural language processing?\n",
      "\n",
      "Search results:\n",
      "----------------------\n",
      "Result 0: Natural language processing | Relevance score 21.272564 | 21652 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/Natural_language_processing\n",
      "Result 1: Studies in Natural Language Processing | Relevance score 16.254246 | 6650456 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/Studies_in_Natural_Language_Processing\n",
      "Result 2: Semantic decomposition (natural language processing) | Relevance score 16.254246 | 57932194 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/Semantic_decomposition_(natural_language_processing)\n",
      "Result 3: Outline of natural language processing | Relevance score 16.254246 | 37764426 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/Outline_of_natural_language_processing\n",
      "Result 4: History of natural language processing | Relevance score 16.254246 | 27837170 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/History_of_natural_language_processing\n",
      "Result 5: Natural language | Relevance score 14.911903 | 21173 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/Natural_language\n",
      "Result 6: Empirical Methods in Natural Language Processing | Relevance score 14.539296 | 43771647 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/Empirical_Methods_in_Natural_Language_Processing\n",
      "Result 7: Data pre-processing | Relevance score 14.150436 | 12386904 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Data_pre-processing\n",
      "Result 8: Native-language identification | Relevance score 12.701324 | 45627703 | machine-learning\n",
      "    Link: https://en.wikipedia.org/wiki/Native-language_identification\n",
      "Result 9: Controlled natural language | Relevance score 12.6099615 | 563439 | natural-language-processing\n",
      "    Link: https://en.wikipedia.org/wiki/Controlled_natural_language\n"
     ]
    }
   ],
   "source": [
    "body7={\n",
    "  \"query\": {\n",
    "    \"multi_match\" : {\n",
    "      \"query\": question3,\n",
    "      \"type\": \"best_fields\",\n",
    "      \"fields\": [ \"title^3\", \"text.section_content\"],\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "docs7 = client.search(body7, index=index2)\n",
    "\n",
    "print(f\"Question: {question3}\")\n",
    "print(\"\")\n",
    "print(\"Search results:\")\n",
    "print(\"----------------------\")\n",
    "\n",
    "for i, doc in enumerate(docs7[\"hits\"][\"hits\"]):\n",
    "    title = doc['_source']['title']\n",
    "    score = doc['_score']\n",
    "    url = doc['_source']['source']\n",
    "    pageid =doc['_source']['page_id']\n",
    "    ind = doc[\"_index\"]\n",
    "    print(f'Result {i}: {title} | Relevance score {score} | {pageid} | {ind}')\n",
    "    print(f'    Link: {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, this approach helped to bring the most relevant document to the top of our list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:nbooks] *",
   "language": "python",
   "name": "conda-env-nbooks-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
