{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from webrefine.core import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# webrefine\n",
    "\n",
    "> A workflow for refining web pages into useful datasets.\n",
    "> webrefine does the work of executing transformations on web data, with the ability to access Common Crawl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the [full documentation](https://edwardjross.github.io/webrefine/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pip install webrefine`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll go through an example of getting some titles from my blog at [skeptric.com](https://skeptric.com).\n",
    "\n",
    "The process consists of:\n",
    "\n",
    "* Defining Queries\n",
    "* Defining Extraction and Filters\n",
    "* Running the process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Querying data\n",
    "\n",
    "To start we'll need some captures of my blog, and so we'll get them from the Internet Archive's Wayback Machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from webrefine.query import WaybackQuery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could get some HTML pages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "skeptric_wb = WaybackQuery('skeptric.com/*', start='2020', end='2020', mime='text/html')\n",
    "sample = list(skeptric_wb.query(limit=20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get some sample records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WaybackRecord(url='https://skeptric.com/', timestamp=datetime.datetime(2020, 11, 26, 6, 41, 2), mime='text/html', status=200, digest='WDYU3RU7ZMFFSZPAPE56PC4L3EK4FE3D')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WaybackRecord(url='https://skeptric.com/casper-2-to-3/', timestamp=datetime.datetime(2020, 11, 26, 7, 52, 8), mime='text/html', status=200, digest='3XDBGHY77ZEA2Z7IVBARXEQT6UDLYAL7')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And view them on the Wayback Machine to work out how to get the information we want"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href=\"http://web.archive.org/web/20201126075208/https://skeptric.com/casper-2-to-3/\">http://web.archive.org/web/20201126075208/https://skeptric.com/casper-2-to-3/</a>"
      ],
      "text/plain": [
       "URL(url='http://web.archive.org/web/20201126075208/https://skeptric.com/casper-2-to-3/')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample[1].preview()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also query CommonCrawl similarly with a `CommonCrawlQuery`.\n",
    "This has more captures but takes a bit longer to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from webrefine.query import CommonCrawlQuery\n",
    "skeptric_cc = CommonCrawlQuery('skeptric.com/*')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option is to add local Warc Files (e.g. produced using [`warcio`](https://github.com/webrecorder/warcio) or `wget` with `warc` parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from webrefine.query import WarcFileQuery\n",
    "test_data = '../resources/test/skeptric.warc.gz'\n",
    "\n",
    "skeptric_file_query = WarcFileQuery(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://skeptric.com/pagination-wayback-cdx/',\n",
       " 'https://skeptric.com/robots.txt',\n",
       " 'https://skeptric.com/style.main.min.5ea2f07be7e07e221a7112a3095b89d049b96c48b831f16f1015bf2d95d914e5.css',\n",
       " 'https://skeptric.com/',\n",
       " 'https://skeptric.com/about/',\n",
       " 'https://skeptric.com/tags/data',\n",
       " 'https://skeptric.com/tags/data/',\n",
       " 'https://skeptric.com/images/wayback_empty_returns.png',\n",
       " 'https://skeptric.com/searching-100b-pages-cdx',\n",
       " 'https://skeptric.com/searching-100b-pages-cdx/',\n",
       " 'https://skeptric.com/fast-web-data-workflow/',\n",
       " 'https://skeptric.com/key-web-captures/',\n",
       " 'https://skeptric.com/emacs-tempfile-hugo/']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[r.url for r in skeptric_file_query.query()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering and Extracting the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From Inspecting some web results we can see that the titles are written like:\n",
    "\n",
    "```html\n",
    "<h1 class=\"post-full-title\">{TITLE}</h1>\n",
    "```\n",
    "\n",
    "In a real example we'd parse the HTML, but for simplicity we'll extract it with a regular expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def skeptric_extract(content, record):\n",
    "    html = content.decode('utf-8')\n",
    "    title = next(re.finditer('<h1 class=\"post-full-title\">([^<]+)</h1>', html)).group(1)\n",
    "    return {\n",
    "        'title': title,\n",
    "        'url': record.url,\n",
    "        'timestamp': record.timestamp\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then test it on some content we fetch from the Wayback Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'Hugo Casper 2 to 3',\n",
       " 'url': 'https://skeptric.com/casper-2-to-3/',\n",
       " 'timestamp': datetime.datetime(2020, 11, 26, 7, 52, 8)}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skeptric_extract(sample[1].content, sample[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some pages don't have it so we filter them out, and we remove duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def skeptric_filter(records):\n",
    "    last_url = None\n",
    "    for record in records:\n",
    "        # Only use ok HTML captures\n",
    "        if record.mime != 'text/html' or record.status != 200:\n",
    "            continue\n",
    "        # Pages that are not articles (and so do not have a title)\n",
    "        if record.url == 'https://skeptric.com/' or '/tags/' in record.url:\n",
    "            continue\n",
    "        # Duplicates (using the fact that here the posts come in order)\n",
    "        if record.url == last_url:\n",
    "            continue\n",
    "        last_url = record.url\n",
    "        yield record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://skeptric.com/casper-2-to-3/',\n",
       " 'https://skeptric.com/common-crawl-index-athena/',\n",
       " 'https://skeptric.com/common-crawl-job-ads/',\n",
       " 'https://skeptric.com/considering-vscode/',\n",
       " 'https://skeptric.com/decorating-pandas-tables/',\n",
       " 'https://skeptric.com/drive-metrics/',\n",
       " 'https://skeptric.com/emacs-buffering/',\n",
       " 'https://skeptric.com/ngram-python/',\n",
       " 'https://skeptric.com/portable-custom-config/',\n",
       " 'https://skeptric.com/searching-100b-pages-cdx/',\n",
       " 'https://skeptric.com/text-meta-data-commoncrawl/']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[r.url for r in skeptric_filter(sample)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the process\n",
    "\n",
    "Now we've written all the logic we need, we can collect it all in a process to run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from webrefine.runners import ProcessMemory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "skeptric_process = ProcessMemory(\n",
    "    queries=[skeptric_file_query,\n",
    "             # commented out to make faster\n",
    "             #skeptric_wb,\n",
    "             #skeptric_cc,\n",
    "          ],\n",
    "    filter=skeptric_filter,\n",
    "    steps = [skeptric_extract])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can run it all with `.run`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5da0ea6d5b5e43d1ab8caf66b2b32391",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "filter: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce2c2c00569f4fe984409d628d67a0b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "transform: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2434487fdad8475d9f8ecf9c7ffce247",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "fetch: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "45902867fcf84a5499b30a2a2bc3cef9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "query:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 248 ms, sys: 11.1 ms, total: 259 ms\n",
      "Wall time: 285 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'title': 'Pagination in Internet Archive&#39;s Wayback Machine with CDX',\n",
       "  'url': 'https://skeptric.com/pagination-wayback-cdx/',\n",
       "  'timestamp': datetime.datetime(2021, 11, 26, 11, 28, 34)},\n",
       " {'title': 'About Skeptric',\n",
       "  'url': 'https://skeptric.com/about/',\n",
       "  'timestamp': datetime.datetime(2021, 11, 26, 11, 28, 37)},\n",
       " {'title': 'Searching 100 Billion Webpages Pages With Capture Index',\n",
       "  'url': 'https://skeptric.com/searching-100b-pages-cdx/',\n",
       "  'timestamp': datetime.datetime(2021, 11, 26, 11, 28, 39)},\n",
       " {'title': 'Fast Web Dataset Extraction Worfklow',\n",
       "  'url': 'https://skeptric.com/fast-web-data-workflow/',\n",
       "  'timestamp': datetime.datetime(2021, 11, 26, 11, 28, 39)},\n",
       " {'title': 'Unique Key for Web Captures',\n",
       "  'url': 'https://skeptric.com/key-web-captures/',\n",
       "  'timestamp': datetime.datetime(2021, 11, 26, 11, 28, 40)},\n",
       " {'title': 'Hugo Readdir Error with Emacs',\n",
       "  'url': 'https://skeptric.com/emacs-tempfile-hugo/',\n",
       "  'timestamp': datetime.datetime(2021, 11, 26, 11, 28, 40)}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "data = list(skeptric_process.run())\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in the case of errors in the steps the process keeps going, and logs the errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "skeptric_error_process = ProcessMemory(\n",
    "    queries=[skeptric_file_query,\n",
    "             # commented out to make faster\n",
    "             #skeptric_wb,\n",
    "             #skeptric_cc,\n",
    "          ],\n",
    "    filter=lambda x: x,\n",
    "    steps = [skeptric_extract])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a784a1480e2f404f91260c19432bcbc3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "filter: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d20507edb4e496ba84071a833398722",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "transform: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fff50d5d31cb46cdac4e1f5013a8abb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "fetch: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f3642e0b541f4fbf8eec5bfe3ca36c9c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "query:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR:root:Error processing WarcFileRecord(url='https://skeptric.com/robots.txt', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 34), mime='text/html', status=404, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=5804, digest='QRNGXIUXE4LAI3XR5RVATIUX5GTB33HX') at step skeptric_extract: \n",
      "ERROR:root:Error processing WarcFileRecord(url='https://skeptric.com/style.main.min.5ea2f07be7e07e221a7112a3095b89d049b96c48b831f16f1015bf2d95d914e5.css', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 35), mime='text/css', status=200, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=7197, digest='LINCDTSPQGAQGZZ6LY2XFXZHG2X476H6') at step skeptric_extract: \n",
      "ERROR:root:Error processing WarcFileRecord(url='https://skeptric.com/', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 36), mime='text/html', status=200, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=17122, digest='JJVB3MQERHRZJCHOJNKS5VDOODXPZAV2') at step skeptric_extract: \n",
      "ERROR:root:Error processing WarcFileRecord(url='https://skeptric.com/tags/data', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 37), mime='text/html', status=302, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=129093, digest='ZZZXDZTTV2KTABRO64ESHVWFPNKB4I5H') at step skeptric_extract: \n",
      "ERROR:root:Error processing WarcFileRecord(url='https://skeptric.com/tags/data/', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 38), mime='text/html', status=200, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=130269, digest='R7CLAACFU5L7T5LKI5G53RZSMCNUNV6F') at step skeptric_extract: \n",
      "ERROR:root:Error processing WarcFileRecord(url='https://skeptric.com/images/wayback_empty_returns.png', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 38), mime='image/png', status=200, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=160971, digest='SU7JRTHNW6KFCJQFL5PMMKV33U2VLV7T') at step skeptric_extract: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte\n",
      "ERROR:root:Error processing WarcFileRecord(url='https://skeptric.com/searching-100b-pages-cdx', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 39), mime='text/html', status=302, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=173368, digest='AYVHQLVFIVGZGUYPEHX46CHMZ5NUDDBF') at step skeptric_extract: \n"
     ]
    }
   ],
   "source": [
    "data = list(skeptric_error_process.run())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could then investigate them to see what happened"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "from pathlib import PosixPath\n",
    "from webrefine.query import WarcFileRecord\n",
    "\n",
    "record = WarcFileRecord(url='https://skeptric.com/tags/data/', timestamp=datetime.datetime(2021, 11, 26, 11, 28, 38), mime='text/html', status=200, path=PosixPath('../resources/test/skeptric.warc.gz'), offset=130269, digest='R7CLAACFU5L7T5LKI5G53RZSMCNUNV6F')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
