{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extraction of Nibelungen texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is really hard to find Nibelungenlied texts online in a form that is:\n",
    "- free access\n",
    "- made by scholars\n",
    "- easily parsable\n",
    "\n",
    "**Our aim is to make semantic analysis from Nibelungen texts and Völsunga saga.**\n",
    "\n",
    "I found PDF files from [Universität Wien](https://www.univie.ac.at/nibelungenwerkstatt/) which contained only the raw texts, however, parsing them is unfeasible because no spaces remained after extraction and tokenizing words is too hard for Mittelhochdeutsch.\n",
    "\n",
    "I found HTML files from [Augsburg Hochschule](https://www.hs-augsburg.de/~harsch/germanica/Chronologie/d_chrono.html) and it was easier to extract them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From PDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "link_nibelungen = \"https://www.univie.ac.at/nibelungenwerkstatt/files/wrkst_codices.zip\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The zip file is downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/clementbesnier/.virtualenvs/old_norse_notebook/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (2.3.0) doesn't match a supported version!\n",
      "  RequestsDependencyWarning)\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "r = requests.get(link_nibelungen)\n",
    "\n",
    "with open(link_nibelungen.split(\"/\")[-1], \"wb\") as f:\n",
    "    f.write(r.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The zip file is unzipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile(link_nibelungen.split(\"/\")[-1], \"r\") as f:\n",
    "    f.extractall(\".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PDF files are read and texts are xtracted. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 270 pages.\n"
     ]
    }
   ],
   "source": [
    "import PyPDF2\n",
    "l_text = []\n",
    "with open(\"gr-A_nib.pdf\", \"rb\") as f:\n",
    "    pdf_reader = PyPDF2.PdfFileReader(f)\n",
    "    print(\"There are \"+str(pdf_reader.getNumPages())+\" pages.\")\n",
    "    for page_index in range(pdf_reader.getNumPages()):\n",
    "        page = pdf_reader.getPage(page_index)\n",
    "        l_text.append(page.extractText())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'2A1Unsistinalten\\nmærenwndersvilgeseitvon\\nheldenlobebærnvon\\ngrozzerchnheitvon\\nfroudenhochgeziten\\nvon\\nweinen\\nundvon\\nklagenvon\\nchnerrechenstritemugetirnuwunderhoerensagenA2EzwhsinBurgondeneinscho\\nenemagedindazinallenlandennihtschoenersmohte\\nsinChriemhiltwas\\nsigeheizzen\\n\\nundewas\\neinscho\\nenewipdarumbemsendegenevilverliesen\\ndenlipA3Derminnechlichenmeidetrtenwol\\ngezaminmteknerreckenniemenwas\\nirgramanemazen\\nscho\\nenesowas\\niredellipderjunchfrouwen\\ntugende\\nliertenanderiu\\nwipA4Irphlagendrikunigeedelunderich\\nGunthereundeGernot\\ndiereckenlobelichundeGiselherderjungeeinzerwelter\\ndegendiufrouwewas\\nirswester\\n\\ndiefurstenhetensinirpflegenA5Dieherrenwarn\\nmiltevon\\nartehoh\\ngebornmitkrefteunmazzen\\nkuenediereckenzerkorndazenBurgondensowas\\nirlantgenant\\nsifrumdenstarkiuwndersitinEzelen\\nlantA6ZeWornitz\\nbidemRinesiwonden\\nmitirkraftindiendevon\\nirlandenvilstolziuriterschaft\\nmitstoltzlicheneren\\nunzanirendes\\nzitsitsturbensijamerlichevon\\nzweier\\nedelenfrouwen\\nnitA7Einrichiu\\nkuniginnefrouteirmterhiezirvater\\nhiezDanchratder\\nindiuerbeliezsitnachsime\\nlebneeinellensrichermanderouchinsinerjugendegrozzererenvilgewan\\nA8Diedrikunigewaren\\nalsichgesagethanvon\\nvihohemelleninwarn\\nundertanouchdiebestenrechenvon\\ndenman\\nhatgesagetstarkundevilchneinallenstritenunverzaget\\nA9Dazwas\\nvon\\nTronyn\\nHagene\\nundouchderbrdersinDanchwart\\ndervilsnelleundevon\\nMecenOrtwindiezwene\\nmarchgraven\\nGereundeEckewart\\n'\n"
     ]
    }
   ],
   "source": [
    "print(repr(l_text[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words are not tokenized so it's useless for what we want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "main_links = [\n",
    "    \"https://www.hs-augsburg.de/~harsch/germanica/Chronologie/12Jh/Nibelungen/nib_c_00.html\",\n",
    "    \"https://www.hs-augsburg.de/~harsch/germanica/Chronologie/12Jh/Nibelungen/nib_b_00.html\",\n",
    "    \"https://www.hs-augsburg.de/~harsch/germanica/Chronologie/12Jh/Nibelungen/nib_a_00.html\",\n",
    "    \"https://www.hs-augsburg.de/~harsch/germanica/Chronologie/12Jh/Nibelungen/nib_n_00.html\"\n",
    "]\n",
    "\n",
    "n_pages = 39"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Making links "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def int_to_string(i):\n",
    "    if 0 <= i < 10:\n",
    "        return \"0\"+str(i)\n",
    "    else:\n",
    "        return str(i)\n",
    "\n",
    "links = {}\n",
    "for link in main_links:\n",
    "    links[link] = []\n",
    "    for i in range(n_pages+1):\n",
    "        link.split(\"/\")\n",
    "        links[link].append(\"/\".join(link.split(\"/\")[:-1])+\"/\"+\n",
    "                           link.split(\"/\")[-1].split(\".\")[0][:-2]+int_to_string(i)+\".html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Retrieving parts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "texts = {}\n",
    "for link in links:\n",
    "    texts[link] = []\n",
    "    for page_link in links[link]:\n",
    "        r = requests.get(page_link)\n",
    "        time.sleep(1)\n",
    "        texts[link].append(r.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Saving part"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "for main_link in main_links:\n",
    "    directory = main_link.split(\"/\")[-1].split(\".\")[0]\n",
    "    if not os.path.exists(directory):\n",
    "        os.mkdir(directory)\n",
    "    for i, text in enumerate(texts[main_link]):\n",
    "        filename = os.path.join(directory, str(i)+\".html\")\n",
    "        with open(filename, \"w\") as f:\n",
    "            f.write(text.replace(b\"s\\x8d\", b\"i\").decode(\"utf-8\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reading part"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_texts = {}\n",
    "for main_link in main_links:\n",
    "    directory = main_link.split(\"/\")[-1].split(\".\")[0]\n",
    "    retrieved_texts[main_link] = []\n",
    "    for i, text in enumerate(texts[main_link]):\n",
    "        filename = os.path.join(directory, str(i)+\".html\")\n",
    "        with open(filename, \"r\") as f:\n",
    "            text = f.read()\n",
    "            tree = BeautifulSoup(text, \"lxml\")\n",
    "            retrieved_texts[main_link].append(tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "bibliotheca Augustana\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "<<< Übersicht  <<< vorige Seite  nächste Seite >>>\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "BIBLIOTHECA AUGUSTANA\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "Das Nibelungenlied\n",
      "1190/1200\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\t \n",
      "\n",
      "\n",
      "Handschrift C\n",
      " \n",
      "1. Aventiure\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "___________________________________________________\n",
      " \n",
      " \n",
      "Auenture von den Nibelungen\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "UNS IST> In alten     mæren wnders vil geseit\n",
      "von heleden lobebæren     von grozer arebeit\n",
      "von frevde vn– hochgeciten     von weinen vn– klagen\n",
      "von kvner recken striten     mvget ir nv wnder horen sagen\n",
      "Ez whs <inBvregonden>     ein vil edel magedin\n",
      "daz in allen landen     niht schoners mohte sin\n",
      "Chriemhilt geheizen     div wart ein schone wip\n",
      "dar vmbe mvsin degene     vil verliesen den lip\n",
      "Ir pflagen dri kunige     edel un– rich\n",
      "Gunther un– Gernot     die rechen lobelich\n",
      "vn– Giselher der iunge     ein wetlicher degen\n",
      "div frowe was ir swester     die helde hetens inir pflegen\n",
      "4Ein richiv chuniginne     frov Vte ir mvter hiez\n",
      "ir vater der hiez Dancrât     der in div erbe liez\n",
      "sit nach sime lebene     ein ellens richer man\n",
      "der ovch insiner iugende     grozer eren vil gewan\n",
      "5Die herren waren milte     von arde hoh erborn\n",
      "mit kraft vn– mazen chvne     die rechen vz erchorn\n",
      "da zen Bvrgonden     so was ir lant genant\n",
      "si frvmten starchiv wnder     sit in Etzelen lant\n",
      "6Ze Wormze bi dem Rine     si wonten mit ir chraft\n",
      "in dienten von ir landen     vil stolziv ritterschaft\n",
      "mit lobelichen eren     vnz an ir endes zit\n",
      "si sturben iæmerliche     sit von zweier frowen nit\n",
      "Die dri kunige waren     als ich gesaget <han>\n",
      "von vil hohem ellen     in waren vndertan\n",
      "ovch die besten rechen     von den man hat gesaget\n",
      "starch vn– vil chvone     inscharpfen striten vnverzaget\n",
      "8Daz was von tronege hagene     vn– ovoch der bruder sin\n",
      "Danchwart der snelle     von Metzzen Ortwin\n",
      "die zwene marcgrauen     Gere vn– Ekkewart\n",
      "Volker von Alzeye     mit ganzem ellen wol be wart\n",
      "9Rvomolt der chvchen meister     ein vz erwelter degen\n",
      "Sindolt vn– Hvnolt     dise herren mvsin pflegen\n",
      "des hoves vn– der eren     der drier kunige man\n",
      "si heten noch manigen rechen     des ich genennen nienen kan\n",
      "10Danchwart der was marschalch     do was der nefe sin\n",
      "trvhsetze des kuniges     von Mezzen Ortwin\n",
      "Sindolt der was schenche     ein wetlicher degen\n",
      "Hvnolt was chame[1v]rære     si chunden hoher eren pflegen\n",
      "11Von des hofes ere     vn– von ir witen chraft\n",
      "von ir vil hohen werdekeit     vn– von ir ritterschaft\n",
      "der die herren pflagen     mit frevden al ir leben\n",
      "des enchunde iv ze ware     niemen gar ein ende geben\n",
      "\n",
      "\n",
      "INDisen hohen eren     trvmte Chriemilde\n",
      "wie si zvge einen valchen     starch schon vn– wilde\n",
      "den ir zwene arn erchrvmmen     daz si daz mvste sehen\n",
      "ir enkunde indirre werlde     leider nimmer <geschehen>\n",
      "13Den trvom si do sagete     ir <mvoter> voten\n",
      "sine chundes niht beschaiden     baz der gvten\n",
      "der valche den dv zivhest     daz ist ein edel man\n",
      "in welle got behvten     dv mvst in schier vloren han\n",
      "14Waz saget ir mir von manne     vil liebiv mvter min\n",
      "ane <rechen> minne     so wil ich immer sin\n",
      "svs schon ich wil beliben     vnz an minen tot\n",
      "daz ich von rechen minne     sol gewinnen nimmer not\n",
      "15Nvne versprich ez niht zesere     sprach ir mvter do\n",
      "soltv immer hercenliche     zer werlde werden vro\n",
      "daz chvmt von mannes minne     dv wirst ein schone wip\n",
      "ob dir got gefuget     eins rehte gvten ritters lip\n",
      "16Die rede lat beliben     vil liebiv frowe min\n",
      "ez ist an manigen wiben     vil diche worden schin\n",
      "wie liebe mit leide     zeiungest lonen chan\n",
      "ich sol si miden beide     sone chan mir nimmer missegan\n",
      "Chriemhilt in ir mvte     sich minne gar bewach\n",
      "sit lebete div uil gvte     vil manigen lieben tac\n",
      "daz sine wesse niemen     den minnen wolde ir lip\n",
      "sit wart si mit eren     eines vil werden rechen wip\n",
      "18Der was der selbe valche     den si in ir trvome sach\n",
      "den ir beschiet ir mvoter     wie sere si daz rach\n",
      "an ir nehsten magen     die in slvgen sint\n",
      "durch sin eines sterben     starp vil manich mvoter kint\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "<<< Übersicht  <<< vorige Seite  nächste Seite >>>\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(retrieved_texts[main_links[0]][1].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```verbatim\n",
    "We select texts between the first <h4> and teh first <<<< occurrences.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'UNS IST> In alten'\n"
     ]
    }
   ],
   "source": [
    "def extract_text(html_text):\n",
    "    lines = [i.text.replace(\"\\xa0\", \"\") for i in html_text.find(\"div\", attrs={\"class\": \"contentus\"}).findAll(\"h3\")]\n",
    "    return [line.split(\"  \") for line in lines]\n",
    "    \n",
    "\n",
    "print(repr(extract_text(retrieved_texts[main_links[0]][1])[0][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extracted text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import codecs\n",
    "\n",
    "for main_link in main_links:\n",
    "    directory = \"extracted_\"+main_link.split(\"/\")[-1].split(\".\")[0][:-3]\n",
    "    if not os.path.exists(directory):\n",
    "        os.mkdir(directory)\n",
    "    for i, text in enumerate(retrieved_texts[main_link]):\n",
    "        filename = os.path.join(directory, str(i)+\".txt\")\n",
    "        extracted_text = extract_text(text)\n",
    "        if len(extracted_text) > 0:\n",
    "            with codecs.open(filename, mode=\"w\", encoding=\"utf-8\") as f:\n",
    "                lines = [\"\\t\".join(line) for line in extracted_text]\n",
    "                final_text = \"\\n\".join(lines)\n",
    "                f.write(final_text)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
