{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started and Usage Examples\n",
    "\n",
    "## Basic Operations\n",
    "\n",
    "Create a new document within Python from a given string, set a document feature\n",
    "and add an annotation that spans the whole document to the default\n",
    "annotation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Annotation(0,33,Document,features=Features({}),id=0)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gatenlp import Document\n",
    "\n",
    "# Create the document\n",
    "doc1 = Document(\"This is the text of the document.\")\n",
    "\n",
    "# set a document feature\n",
    "doc1.features[\"purpose\"] = \"simple illustration of gatenlp basics\"\n",
    "\n",
    "# get the default annotation set\n",
    "defset = doc1.annset()\n",
    "\n",
    "# add an annotation that spans the whole document, no features\n",
    "defset.add(0, len(doc1), \"Document\", {})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now save the document in bdocjson format and load back from the saved\n",
    "document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{\"annotation_sets\": {\"\": {\"name\": \"\", \"annotations\": [{\"type\": \"Document\", \"start\": 0, \"end\": 33, \"id\": 0, \"features\": {}}], \"next_annid\": 1}}, \"text\": \"This is the text of the document.\", \"features\": {\"purpose\": \"simple illustration of gatenlp basics\"}, \"offset_type\": \"p\", \"name\": \"\"}\n"
     ]
    }
   ],
   "source": [
    "# save the document in bdocjson format, this can be read into Java GATE\n",
    "# using the format-bdoc plugin.\n",
    "doc1.save(\"testdoc.bdocjs\")\n",
    "# Read back the document\n",
    "doc2 = Document.load(\"testdoc.bdocjs\")\n",
    "\n",
    "\n",
    "# print the json representation of the loaded document\n",
    "print(doc2.save_mem(fmt=\"json\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenize and create annotations for the tokens using a NLTK tokenizer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gatenlp.processing.tokenizer import NLTKTokenizer\n",
    "from nltk.tokenize import TreebankWordTokenizer\n",
    "\n",
    "nltk_tok = TreebankWordTokenizer()\n",
    "ann_tok = NLTKTokenizer(nltk_tokenizer=nltk_tok, out_set=\"NLTK\")\n",
    "doc1 = ann_tok(doc1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get all the annotation for the NLTK tokens which are in annotation set \"NLTK\":\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "8\n",
      "8\n",
      "Annotation(0,4,Token,features=Features({}),id=0)\n",
      "Annotation(5,7,Token,features=Features({}),id=1)\n",
      "Annotation(8,11,Token,features=Features({}),id=2)\n",
      "Annotation(12,16,Token,features=Features({}),id=3)\n",
      "Annotation(17,19,Token,features=Features({}),id=4)\n",
      "Annotation(20,23,Token,features=Features({}),id=5)\n",
      "Annotation(24,32,Token,features=Features({}),id=6)\n",
      "Annotation(32,33,Token,features=Features({}),id=7)\n"
     ]
    }
   ],
   "source": [
    "# get the annotation set with the name \"NLTK\" and print the number of annotations\n",
    "set_nltk = doc1.annset(\"NLTK\")\n",
    "print(len(set_nltk))\n",
    "\n",
    "# get only annotations of annotation type \"Token\" from the set and print the number of annotations\n",
    "# since there are no other annotations in the original set, the number should be the same\n",
    "set_tokens = set_nltk.with_type(\"Token\")\n",
    "print(len(set_tokens))\n",
    "\n",
    "# print all the annotations in order\n",
    "for ann in set_tokens:\n",
    "    print(ann)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The annotation set `set_nltk` is an original annotation set and reflects directly what is stored with the \n",
    "document: it is an \"attached\" annotation set. Adding or removing annotations in that set will change what is stored with the document. \n",
    "\n",
    "On the other hand, the annotation set `set_tokens` is the result of a filtering operation and is \"detached\". \n",
    "By default it is immutable, it cannot be modified. It can be made mutable, but when annotations are then added or removed from the set, this will not change what is stored with the document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "set_nltk is detached:  False\n",
      "set_nltk is immutable:  False\n",
      "set_tokens is detached:  True\n",
      "set_tokens is immutable:  True\n",
      "Error: Cannot add an annotation to an immutable annotation set\n",
      "set_nltk: size= 9 , annotation: AnnotationSet([Annotation(0,4,Token,features=Features({}),id=0), Annotation(3,5,New1,features=Features({}),id=8), Annotation(5,7,Token,features=Features({}),id=1), Annotation(8,11,Token,features=Features({}),id=2), Annotation(12,16,Token,features=Features({}),id=3), Annotation(17,19,Token,features=Features({}),id=4), Annotation(20,23,Token,features=Features({}),id=5), Annotation(24,32,Token,features=Features({}),id=6), Annotation(32,33,Token,features=Features({}),id=7)])\n",
      "set_tokens: size= 9 , annotations:  AnnotationSet([Annotation(0,4,Token,features=Features({}),id=0), Annotation(5,7,Token,features=Features({}),id=1), Annotation(5,7,New2,features=Features({}),id=8), Annotation(8,11,Token,features=Features({}),id=2), Annotation(12,16,Token,features=Features({}),id=3), Annotation(17,19,Token,features=Features({}),id=4), Annotation(20,23,Token,features=Features({}),id=5), Annotation(24,32,Token,features=Features({}),id=6), Annotation(32,33,Token,features=Features({}),id=7)])\n"
     ]
    }
   ],
   "source": [
    "# check if the set_nltk is detached\n",
    "print(\"set_nltk is detached: \", set_nltk.isdetached())  # no it is attached! \n",
    "print(\"set_nltk is immutable: \", set_nltk.immutable )  # no\n",
    "# add an annotation to the set\n",
    "set_nltk.add(3,5,\"New1\")\n",
    "\n",
    "# check if the set_tokens is detached\n",
    "print(\"set_tokens is detached: \", set_tokens.isdetached())  # yes!\n",
    "# check if it is immutable as well\n",
    "print(\"set_tokens is immutable: \", set_tokens.immutable )  # yes\n",
    "try:\n",
    "    set_tokens.add(5,7,\"New2\")\n",
    "except Exception as ex:\n",
    "    print(\"Error:\", ex)\n",
    "    \n",
    "# ok, let's make the set mutable and add the annotation\n",
    "set_tokens.immutable = False\n",
    "set_tokens.add(5,7,\"New2\")\n",
    "\n",
    "print(\"set_nltk: size=\", len(set_nltk), \", annotation:\", set_nltk)\n",
    "print(\"set_tokens: size=\", len(set_tokens), \", annotations: \",   set_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding the annotation with type \"New1\" to `set_nltk` actually stored the annotation with the document, but did not affect the filtered set `set_tokens`, and adding the annotation with type \"New2\" to the filtered set did not affect the set stored with the document. \n",
    "\n",
    "The document and document features and the annotation sets and annotation as well as their features can be shown in a Jupyter notebook by simply showing a document value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>#GDPFPTPQAZ-wrapper { color: black !important; }</style>\n",
       "<div id=\"GDPFPTPQAZ-wrapper\">\n",
       "\n",
       "<div>\n",
       "<style>\n",
       "#GDPFPTPQAZ-content {\n",
       "    width: 100%;\n",
       "    height: 100%;\n",
       "    font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-row {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "    flex-direction: row;\n",
       "    flex-wrap: nowrap;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-col {\n",
       "    border: 1px solid grey;\n",
       "    display: inline-block;\n",
       "    min-width: 200px;\n",
       "    padding: 5px;\n",
       "    /* white-space: normal; */\n",
       "    /* white-space: pre-wrap; */\n",
       "    overflow-y: auto;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-hdr {\n",
       "    font-size: 1.2rem;\n",
       "    font-weight: bold;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-label {\n",
       "    margin-bottom: -15px;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-input {\n",
       "    vertical-align: middle;\n",
       "    position: relative;\n",
       "    *overflow: hidden;\n",
       "}\n",
       "\n",
       "#GDPFPTPQAZ-popup {\n",
       "    display: none;\n",
       "    color: black;\n",
       "    position: absolute;\n",
       "    margin-top: 10%;\n",
       "    margin-left: 10%;\n",
       "    background: #aaaaaa;\n",
       "    width: 60%;\n",
       "    height: 60%;\n",
       "    z-index: 50;\n",
       "    padding: 25px 25px 25px;\n",
       "    border: 1px solid black;\n",
       "    overflow: auto;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-selection {\n",
       "    margin-bottom: 5px;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-featuretable {\n",
       "    margin-top: 10px;\n",
       "}\n",
       "\n",
       ".GDPFPTPQAZ-fname {\n",
       "    text-align: left !important;\n",
       "    font-weight: bold;\n",
       "    margin-right: 10px;\n",
       "}\n",
       ".GDPFPTPQAZ-fvalue {\n",
       "    text-align: left !important;\n",
       "}\n",
       "</style>\n",
       "  <div id=\"GDPFPTPQAZ-content\">\n",
       "        <div id=\"GDPFPTPQAZ-popup\" style=\"display: none;\">\n",
       "        </div>\n",
       "        <div class=\"GDPFPTPQAZ-row\" id=\"GDPFPTPQAZ-row1\" style=\"height:67vh; min-height:100px;\">\n",
       "            <div id=\"GDPFPTPQAZ-text-wrapper\" class=\"GDPFPTPQAZ-col\" style=\"width:70%;\">\n",
       "                <div class=\"GDPFPTPQAZ-hdr\" id=\"GDPFPTPQAZ-dochdr\"></div>\n",
       "                <div id=\"GDPFPTPQAZ-text\">\n",
       "                </div>\n",
       "            </div>\n",
       "            <div id=\"GDPFPTPQAZ-chooser\" class=\"GDPFPTPQAZ-col\" style=\"width:30%; border-left-width: 0px;\"></div>\n",
       "        </div>\n",
       "        <div class=\"GDPFPTPQAZ-row\" id=\"GDPFPTPQAZ-row2\" style=\"height:30vh; min-height: 100px;\">\n",
       "            <div id=\"GDPFPTPQAZ-details\" class=\"GDPFPTPQAZ-col\" style=\"width:100%; border-top-width: 0px;\">\n",
       "            </div>\n",
       "        </div>\n",
       "    </div>\n",
       "    <script src=\"https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js\"></script><script src=\"https://unpkg.com/gatenlp-ann-viewer@1.0.10/gatenlp-ann-viewer.js\"></script>\n",
       "    <script type=\"application/json\" id=\"GDPFPTPQAZ-data\">\n",
       "    {\"annotation_sets\": {\"\": {\"name\": \"detached-from:\", \"annotations\": [{\"type\": \"Document\", \"start\": 0, \"end\": 33, \"id\": 0, \"features\": {}}], \"next_annid\": 1}, \"NLTK\": {\"name\": \"detached-from:NLTK\", \"annotations\": [{\"type\": \"Token\", \"start\": 0, \"end\": 4, \"id\": 0, \"features\": {}}, {\"type\": \"Token\", \"start\": 5, \"end\": 7, \"id\": 1, \"features\": {}}, {\"type\": \"Token\", \"start\": 8, \"end\": 11, \"id\": 2, \"features\": {}}, {\"type\": \"Token\", \"start\": 12, \"end\": 16, \"id\": 3, \"features\": {}}, {\"type\": \"Token\", \"start\": 17, \"end\": 19, \"id\": 4, \"features\": {}}, {\"type\": \"Token\", \"start\": 20, \"end\": 23, \"id\": 5, \"features\": {}}, {\"type\": \"Token\", \"start\": 24, \"end\": 32, \"id\": 6, \"features\": {}}, {\"type\": \"Token\", \"start\": 32, \"end\": 33, \"id\": 7, \"features\": {}}, {\"type\": \"New1\", \"start\": 3, \"end\": 5, \"id\": 8, \"features\": {}}], \"next_annid\": 9}}, \"text\": \"This is the text of the document.\", \"features\": {\"purpose\": \"simple illustration of gatenlp basics\"}, \"offset_type\": \"j\", \"name\": \"\"}\n",
       "    </script>\n",
       "    <script type=\"text/javascript\">\n",
       "        gatenlp_run(\"GDPFPTPQAZ-\");\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "</div></div>"
      ],
      "text/plain": [
       "Document(This is the text of the document.,features=Features({'purpose': 'simple illustration of gatenlp basics'}),anns={'': AnnotationSet([Annotation(0,33,Document,features=Features({}),id=0)]), 'NLTK': AnnotationSet([Annotation(0,4,Token,features=Features({}),id=0), Annotation(3,5,New1,features=Features({}),id=8), Annotation(5,7,Token,features=Features({}),id=1), Annotation(8,11,Token,features=Features({}),id=2), Annotation(12,16,Token,features=Features({}),id=3), Annotation(17,19,Token,features=Features({}),id=4), Annotation(20,23,Token,features=Features({}),id=5), Annotation(24,32,Token,features=Features({}),id=6), Annotation(32,33,Token,features=Features({}),id=7)])})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
