---

title: Query


keywords: fastai
sidebar: home_sidebar



nb_path: "nbs/01_query.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/01_query.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%</span><span class="k">load_ext</span> autoreload
<span class="o">%</span><span class="k">autoreload</span> 2
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is an interface to <em>querying</em> resources that can be fetched.</p>
<p>It covers local files, the <a href="http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/">Common Crawl Index</a> and the <a href="https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server">Internet Archive's CDX Server</a> for finding archived resources based on URL patterns.
The latter two have  a lot of overlap with <a href="https://github.com/cocrawler/cdx_toolkit">cdx_toolkit</a> but this is more optimised for speed and ease of use.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Local-WARC-Files">Local WARC Files<a class="anchor-link" href="#Local-WARC-Files"> </a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Object">Object<a class="anchor-link" href="#Object"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="WarcFileRecord" class="doc_header"><code>class</code> <code>WarcFileRecord</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L35" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>WarcFileRecord</code>(<strong><code>url</code></strong>:<code>str</code>, <strong><code>timestamp</code></strong>:<code>datetime</code>, <strong><code>mime</code></strong>:<code>str</code>, <strong><code>status</code></strong>:<code>int</code>, <strong><code>path</code></strong>:<code>Path</code>, <strong><code>offset</code></strong>:<code>int</code>, <strong><code>digest</code></strong>:<code>str</code>)</p>
</blockquote>
<p>WarcFileRecord(url: 'str', timestamp: 'datetime', mime: 'str', status: 'int', path: 'Path', offset: 'int', digest: 'str')</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_warc_url" class="doc_header"><code>get_warc_url</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L60" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_warc_url</code>(<strong><code>record</code></strong>:<code>ArcWarcRecord</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_warc_timestamp" class="doc_header"><code>get_warc_timestamp</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L64" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_warc_timestamp</code>(<strong><code>record</code></strong>:<code>ArcWarcRecord</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_warc_mime" class="doc_header"><code>get_warc_mime</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L67" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_warc_mime</code>(<strong><code>record</code></strong>:<code>ArcWarcRecord</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_warc_status" class="doc_header"><code>get_warc_status</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L70" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_warc_status</code>(<strong><code>record</code></strong>:<code>ArcWarcRecord</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_warc_digest" class="doc_header"><code>get_warc_digest</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L73" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_warc_digest</code>(<strong><code>record</code></strong>:<code>ArcWarcRecord</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="WarcFileQuery" class="doc_header"><code>class</code> <code>WarcFileQuery</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L80" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>WarcFileQuery</code>(<strong><code>path</code></strong>:<code>Union[str, Path]</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Testing">Testing<a class="anchor-link" href="#Testing"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Generated some test data with:</p>

<pre><code>wget -r -Q1M --domains skeptric.com --warc-file=skeptric --delete-after --no-directories https://skeptric.com/pagination-wayback-cdx/</code></pre>
<p>See <a href="https://github.com/webrecorder/warcio#writing-warc-records">warcio library</a> for how to do this in Python.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test_data</span> <span class="o">=</span> <span class="s1">&#39;../resources/test/skeptric.warc.gz&#39;</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">warc</span> <span class="o">=</span> <span class="n">WarcFileQuery</span><span class="p">(</span><span class="n">test_data</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">warc</span><span class="o">.</span><span class="n">query</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[WarcFileRecord(url=&#39;https://skeptric.com/pagination-wayback-cdx/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 34), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=889, digest=&#39;Q6HKH563Z7HF2333QILSSSHY2K3B6NOK&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/robots.txt&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 34), mime=&#39;text/html&#39;, status=&#39;404&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=5804, digest=&#39;QRNGXIUXE4LAI3XR5RVATIUX5GTB33HX&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/style.main.min.5ea2f07be7e07e221a7112a3095b89d049b96c48b831f16f1015bf2d95d914e5.css&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 35), mime=&#39;text/css&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=7197, digest=&#39;LINCDTSPQGAQGZZ6LY2XFXZHG2X476H6&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 36), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=17122, digest=&#39;JJVB3MQERHRZJCHOJNKS5VDOODXPZAV2&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/about/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 37), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=125261, digest=&#39;Z5NRUTRW3XTKZDCJFDKGPJ5BWIBNQCG7&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/tags/data&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 37), mime=&#39;text/html&#39;, status=&#39;302&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=129093, digest=&#39;ZZZXDZTTV2KTABRO64ESHVWFPNKB4I5H&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/tags/data/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 38), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=130269, digest=&#39;R7CLAACFU5L7T5LKI5G53RZSMCNUNV6F&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/images/wayback_empty_returns.png&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 38), mime=&#39;image/png&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=160971, digest=&#39;SU7JRTHNW6KFCJQFL5PMMKV33U2VLV7T&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/searching-100b-pages-cdx&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 39), mime=&#39;text/html&#39;, status=&#39;302&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=173368, digest=&#39;AYVHQLVFIVGZGUYPEHX46CHMZ5NUDDBF&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/searching-100b-pages-cdx/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 39), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=174558, digest=&#39;CMT3ZNELTRC7H7ICVCGAYYS6GQ2NSZGP&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/fast-web-data-workflow/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 39), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=188608, digest=&#39;UMQCERJOQ3AGE3Z576ABKKKNDGSNBQAX&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/key-web-captures/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 40), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=195651, digest=&#39;5GLUI5DL5PFUM6QY53NWUCMJBC2CTXKS&#39;),
 WarcFileRecord(url=&#39;https://skeptric.com/emacs-tempfile-hugo/&#39;, timestamp=datetime.datetime(2021, 11, 26, 11, 28, 40), mime=&#39;text/html&#39;, status=&#39;200&#39;, path=PosixPath(&#39;../resources/test/skeptric.warc.gz&#39;), offset=201243, digest=&#39;JY2UE6SKCYPHXSUN5IR6SQVMZRRAGU64&#39;)]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Try fetching a record</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">image_record</span> <span class="o">=</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">mime</span> <span class="o">==</span> <span class="s1">&#39;image/png&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>

<span class="n">content</span> <span class="o">=</span> <span class="n">image_record</span><span class="o">.</span><span class="n">content</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="n">Image</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_png output_subarea output_execute_result">
<img src="data:image/png;base64,[encoded data, 16196 characters]
"
>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Check the digests</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">for</span> <span class="n">result</span> <span class="ow">in</span> <span class="n">results</span><span class="p">:</span>
    <span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">digest</span> <span class="o">==</span> <span class="n">sha1_digest</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">get_content</span><span class="p">())</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Internet-Archive">Internet Archive<a class="anchor-link" href="#Internet-Archive"> </a></h1><h2 id="Querying-CDX">Querying CDX<a class="anchor-link" href="#Querying-CDX"> </a></h2><p>The Internet Archive runs it's own <a href="https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server">Java CDX Server</a>. <a href="https://hackernoon.com/guide-to-handling-internet-archives-cdx-server-api-response-c469df5b81f4">Hakernoon has an article</a> that gives a good overview of querying it.</p>
<p>See the documentation for all the paramemters; here are the ones that I find most interesting</p>
<ul>
<li>url: The URL to query, a wildcard <code>*</code> can implicitly define it (which means we don't need <code>matchType</code>)</li>
<li>output: "json" returns JSON instead of space separated</li>
<li>fl: Comma separated list of fields to return</li>
<li>from, to: Date filtering 1-14 digits yyyyMMddhhmmss, inclusive (this seems to do the right thing for truncation).</li>
<li>filter: <code>[!]field:regex</code> filter a returned field with <a href="https://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html">Java Regex</a>, inverting with <code>!</code></li>
<li>limit/offset: For getting a small number of results (sampling). Internal limit is 150000 results.</li>
</ul>
<p>Default fields returned: ["urlkey","timestamp","original","mimetype","statuscode","digest","length"]</p>
<p>Pagination isn't really useful because it's applied <em>before</em> filtering (including date filtering) so most pages are empty with a date filter.
Unpaginated requests are fast enough anyway.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Exporting-the-data">Exporting the data<a class="anchor-link" href="#Exporting-the-data"> </a></h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The data is a JSON array or arrays; the first line contains the headers and the subsequent the data.
Let's transform it into a list of dictionaries of keys to values which make it a bit easier to work with (although less efficient in memory).</p>
<p>An alternative would be to directly use something like Pandas</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="header_and_rows_to_dict" class="doc_header"><code>header_and_rows_to_dict</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L104" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>header_and_rows_to_dict</code>(<strong><code>rows</code></strong>:<code>Iterable[list[Any]]</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Check it on some data</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">header_and_rows_to_dict</span><span class="p">([[</span><span class="s1">&#39;col_1&#39;</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;a&#39;</span><span class="p">],</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">]])</span> <span class="o">==</span> <span class="p">[</span>
    <span class="p">{</span><span class="s1">&#39;col_1&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">:</span> <span class="s1">&#39;a&#39;</span><span class="p">},</span>
    <span class="p">{</span><span class="s1">&#39;col_1&#39;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">:</span> <span class="s1">&#39;b&#39;</span><span class="p">}]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">header_and_rows_to_dict</span><span class="p">([[</span><span class="s1">&#39;col_1&#39;</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">]])</span> <span class="o">==</span> <span class="p">[]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">header_and_rows_to_dict</span><span class="p">([])</span> <span class="o">==</span> <span class="p">[]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Making-a-query">Making a query<a class="anchor-link" href="#Making-a-query"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="mimetypes_to_regex" class="doc_header"><code>mimetypes_to_regex</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L121" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>mimetypes_to_regex</code>(<strong><code>mime</code></strong>:<code>list[str]</code>, <strong><code>prefix</code></strong>=<em><code>'mimetype:'</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="query_wayback_cdx" class="doc_header"><code>query_wayback_cdx</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L124" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>query_wayback_cdx</code>(<strong><code>url</code></strong>:<code>str</code>, <strong><code>start</code></strong>:<code>Optional[str]</code>, <strong><code>end</code></strong>:<code>Optional[str]</code>, <strong><code>status_ok</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>mime</code></strong>:<code>Optional[Union[str, Iterable[str]]]</code>=<em><code>None</code></em>, <strong><code>limit</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>offset</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>
<p>Get references to Wayback Machine Captures for url.</p>
<p>Queries the Internet Archive Capture Index (CDX) for url.</p>
<p>Arguments:</p>
<ul>
<li>start: Minimum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>end: Maximum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>status_ok: Only return those with a HTTP status 200</li>
<li>mime: Filter on mimetypes, '<em>' is a wildcard (e.g. 'image/</em>')</li>
<li>limit: Only return first limit records</li>
<li>offset: Skip the first offset records, combine with limit</li>
<li>session: Session to use when making requests
Filters results between start and end inclusive, in format YYYYmmddHHMMSS or any substring
(e.g. start="202001", end="202001" will get all captures in January 2020)</li>
</ul>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-querying">Test querying<a class="anchor-link" href="#Test-querying"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">full_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 22.6 ms, sys: 3.83 ms, total: 26.5 ms
Wall time: 1.15 s
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-statuscode-filter">Test statuscode filter<a class="anchor-link" href="#Test-statuscode-filter"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">ok_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span> <span class="c1">#status_ok=True</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 16.5 ms, sys: 596 µs, total: 17.1 ms
Wall time: 808 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">ok_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;statuscode&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;200&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-mimetypes">Test mimetypes<a class="anchor-link" href="#Test-mimetypes"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">html_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 10.2 ms, sys: 183 µs, total: 10.4 ms
Wall time: 611 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">html_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;mimetype&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;text/html&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">image_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;image/*&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 5.91 ms, sys: 2.61 ms, total: 8.52 ms
Wall time: 698 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">image_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;mimetype&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">&#39;image/&#39;</span><span class="p">)]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">prog_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> 
                                 <span class="n">mime</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;text/css&#39;</span><span class="p">,</span> <span class="s1">&#39;application/javascript&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 5.76 ms, sys: 454 µs, total: 6.21 ms
Wall time: 446 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">prog_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;mimetype&#39;</span><span class="p">]</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">&#39;text/css&#39;</span><span class="p">,</span> <span class="s1">&#39;application/javascript&#39;</span><span class="p">]]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-date-filters">Test date filters<a class="anchor-link" href="#Test-date-filters"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_2020</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="nb">len</span><span class="p">(</span><span class="n">sample_2020</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 4.68 ms, sys: 6.3 ms, total: 11 ms
Wall time: 444 ms
</pre>
</div>
</div>

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>106</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_2020</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="s1">&#39;2020&#39;</span> <span class="o">&lt;=</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&lt;</span> <span class="s1">&#39;2021&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_to_2020</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 11.5 ms, sys: 343 µs, total: 11.8 ms
Wall time: 441 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_to_2020</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&lt;</span> <span class="s1">&#39;2021&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_from_2020</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 16.6 ms, sys: 684 µs, total: 17.3 ms
Wall time: 975 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_from_2020</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&gt;=</span> <span class="s1">&#39;2020&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-limits">Test limits<a class="anchor-link" href="#Test-limits"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_10</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">limit</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 8.4 ms, sys: 382 µs, total: 8.78 ms
Wall time: 415 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">sample_10</span> <span class="o">==</span> <span class="n">full_sample</span><span class="p">[:</span><span class="mi">10</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>True</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_10_offset_20</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">limit</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">offset</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 8.54 ms, sys: 366 µs, total: 8.91 ms
Wall time: 711 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_10_offset_20</span>  <span class="o">==</span> <span class="n">full_sample</span><span class="p">[</span><span class="mi">20</span><span class="p">:</span><span class="mi">20</span><span class="o">+</span><span class="mi">10</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Fetching-Content">Fetching Content<a class="anchor-link" href="#Fetching-Content"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The original content is at <code>http://web.archive.org/web/{timestamp}_id/{url}</code>, and the Wayback Machine version is the same excluding <code>_id</code>.
The Wayback Machine version <a href="https://skeptric.com/restoring-wayback-html/">makes some changes</a> that make it better for interactive viewing, but the content is different.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="wayback_url" class="doc_header"><code>wayback_url</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L170" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>wayback_url</code>(<strong><code>timestamp</code></strong>:<code>str</code>, <strong><code>url</code></strong>:<code>str</code>, <strong><code>wayback</code></strong>:<code>bool</code>=<em><code>False</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="fetch_wayback_content" class="doc_header"><code>fetch_wayback_content</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L174" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>fetch_wayback_content</code>(<strong><code>timestamp</code></strong>:<code>str</code>, <strong><code>url</code></strong>:<code>str</code>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="n">image_sample</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">record</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;urlkey&#39;: &#39;com,skeptric)/favicon.ico&#39;,
 &#39;timestamp&#39;: &#39;20180819051126&#39;,
 &#39;original&#39;: &#39;http://skeptric.com/favicon.ico&#39;,
 &#39;mimetype&#39;: &#39;image/vnd.microsoft.icon&#39;,
 &#39;statuscode&#39;: &#39;200&#39;,
 &#39;digest&#39;: &#39;R6YE2GPPT4BM4IAMHGUJPDJF6BGKKHDA&#39;,
 &#39;length&#39;: &#39;851&#39;}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span> <span class="o">=</span> <span class="n">fetch_wayback_content</span><span class="p">(</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;original&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="n">Image</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_png output_subarea output_execute_result">
<img src="data:image/png;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgAAACYAAAAmwAAAIwAAABjAAAAFgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHoAAAD/AAAAQQAAAAEAAAABAAAANAAAAOoAAAAnAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAD/AAAAewAAAAAAAAAAAAAAAAAAAAAAAAB7AAAA6AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACoAAAA9gAAAJQAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAP8AAABPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAzQAAAPwAAAD/AAAAAAAAAAAAAAAAAAAAAAAAACUAAAD7AAAAkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKoAAAD/AAAA/wAAAAAAAABUAAAAAAAAAAAAAAAmAAAA/QAAAJwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAuAAAA/wAAAP8AAAACAAAA0gAAAAAAAAAAAAAAYgAAAP8AAABlAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGEAAAD/AAAAkgAAAPwAAAAAAAAAAgAAAPkAAAD8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXwAAAPEAAAD/AAAACgAAAMwAAAD/AAAASgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPAAAAqwAAAOwAAAD/AAAAYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQwAAAPoAAADqAAAAPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYwAAAP8AAADEAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGAAAAPIAAAC5AAAAAQAAAAAAAAADAAAAHgAAAAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC8AAADxAAAAAAAAAAAAAAABAAAArgAAAPgAAABgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAAAA5gAAACAAAAAbAAAAfQAAAO4AAACbAAAABQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACUAAAB7AAAAegAAAE8AAAAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/H8AAPvfAAD37wAA4+8AAOPnAADj5wAA8u8AAPjPAAD8nwAA/j8AAP5/AAD8/wAA+f8AAPufAAD7nwAA//8AAA==
"
>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;digest&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="n">sha1_digest</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>And fetching a HTML webpage</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="n">ok_sample</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span> <span class="o">=</span> <span class="n">fetch_wayback_content</span><span class="p">(</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;original&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;digest&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="n">sha1_digest</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Get the Wayback URL for preview</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">URL</span><span class="p">(</span><span class="n">wayback_url</span><span class="p">(</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;original&#39;</span><span class="p">],</span> <span class="n">wayback</span><span class="o">=</span><span class="kc">True</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<a href="http://web.archive.org/web/20180805132731/http://skeptric.com/">http://web.archive.org/web/20180805132731/http://skeptric.com/</a>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Wayback-Record-Object">Wayback Record Object<a class="anchor-link" href="#Wayback-Record-Object"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="WaybackRecord" class="doc_header"><code>class</code> <code>WaybackRecord</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L190" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>WaybackRecord</code>(<strong><code>url</code></strong>:<code>str</code>, <strong><code>timestamp</code></strong>:<code>datetime</code>, <strong><code>mime</code></strong>:<code>str</code>, <strong><code>status</code></strong>:<code>Optional[int]</code>, <strong><code>digest</code></strong>:<code>str</code>)</p>
</blockquote>
<p>WaybackRecord(url: 'str', timestamp: 'datetime', mime: 'str', status: 'Optional[int]', digest: 'str')</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="n">WaybackRecord</span><span class="o">.</span><span class="n">from_dict</span><span class="p">(</span><span class="n">full_sample</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="n">record</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>WaybackRecord(url=&#39;http://skeptric.com/&#39;, timestamp=datetime.datetime(2018, 8, 5, 13, 27, 31), mime=&#39;text/html&#39;, status=200, digest=&#39;2N5QQYZFAM36CSESTGSDKRTRV7I5HEXJ&#39;)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span><span class="o">.</span><span class="n">preview</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<a href="http://web.archive.org/web/20180805132731/http://skeptric.com/">http://web.archive.org/web/20180805132731/http://skeptric.com/</a>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">content</span>
<span class="n">content</span><span class="p">[:</span><span class="mi">100</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 8.38 ms, sys: 3.03 ms, total: 11.4 ms
Wall time: 718 ms
</pre>
</div>
</div>

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>b&#39;&lt;!DOCTYPE html&gt;\n&lt;html lang=&#34;en&#34;&gt;\n  &lt;head&gt;\n    &lt;meta charset=&#34;utf-8&#34; /&gt;\n    &lt;title&gt;Edward Ross - An \xc9&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sha1_digest</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="o">==</span> <span class="n">record</span><span class="o">.</span><span class="n">digest</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Wayback-Query">Wayback Query<a class="anchor-link" href="#Wayback-Query"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="WaybackQuery" class="doc_header"><code>class</code> <code>WaybackQuery</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L225" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>WaybackQuery</code>(<strong><code>url</code></strong>:<code>str</code>, <strong><code>start</code></strong>:<code>Optional[str]</code>, <strong><code>end</code></strong>:<code>Optional[str]</code>, <strong><code>status_ok</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>mime</code></strong>:<code>Optional[Union[str, Iterable[str]]]</code>=<em><code>None</code></em>)</p>
</blockquote>
<p>WaybackQuery(url: 'str', start: 'Optional[str]', end: 'Optional[str]', status_ok: 'bool' = True, mime: 'Optional[Union[str, Iterable[str]]]' = None)</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-fetching-one">Test fetching one<a class="anchor-link" href="#Test-fetching-one"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">wb</span> <span class="o">=</span> <span class="n">WaybackQuery</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">items</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">wb</span><span class="o">.</span><span class="n">query</span><span class="p">())</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 53.9 ms, sys: 3.47 ms, total: 57.3 ms
Wall time: 1.15 s
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">items</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">50</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span> <span class="o">=</span> <span class="n">items</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">content</span>
<span class="k">assert</span> <span class="n">sha1_digest</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="o">==</span> <span class="n">items</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">digest</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-fetching-many">Test fetching many<a class="anchor-link" href="#Test-fetching-many"> </a></h4>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>I have no idea whether Session is <a href="https://github.com/urllib3/urllib3/issues/1252">actually thread-safe</a>; if it's not we should have 1 session per thread. As far as I can tell the issue occurs when you have lots of hosts, so in this case it should be ok.</p>
<p>I guess we'll try it and see. Maybe long term we're better going with asyncio.
Using a Session makes things slightly faster; we can always turn it off by passing session=False.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">joblib</span> <span class="kn">import</span> <span class="n">delayed</span><span class="p">,</span> <span class="n">Parallel</span>

<span class="k">def</span> <span class="nf">wayback_fetch_parallel</span><span class="p">(</span><span class="n">items</span><span class="p">,</span> <span class="n">threads</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">session</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
    <span class="k">if</span> <span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
        <span class="n">session</span> <span class="o">=</span> <span class="n">make_session</span><span class="p">(</span><span class="n">threads</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">Parallel</span><span class="p">(</span><span class="n">n_jobs</span><span class="o">=</span><span class="n">threads</span><span class="p">,</span> <span class="n">prefer</span><span class="o">=</span><span class="s1">&#39;threads&#39;</span><span class="p">)(</span><span class="n">delayed</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">get_content</span><span class="p">)(</span><span class="n">session</span><span class="o">=</span><span class="n">session</span><span class="p">)</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">items</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">contents_16</span> <span class="o">=</span> <span class="n">wayback_fetch_parallel</span><span class="p">(</span><span class="n">items</span><span class="p">[:</span><span class="mi">16</span><span class="p">],</span> <span class="n">threads</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 165 ms, sys: 16.7 ms, total: 181 ms
Wall time: 846 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">contents_16</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">content</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Common-Crawl">Common Crawl<a class="anchor-link" href="#Common-Crawl"> </a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Common Crawl is similar, but slightly different, because it is split over many indexes, roughly monthly covering 2 weeks.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Indexes">Indexes<a class="anchor-link" href="#Indexes"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Cache within a session, but not between sessions since over months the indexes change.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_cc_indexes" class="doc_header"><code>get_cc_indexes</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L241" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_cc_indexes</code>()</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">indexes</span> <span class="o">=</span> <span class="n">get_cc_indexes</span><span class="p">()</span>
<span class="n">indexes</span><span class="p">[:</span><span class="mi">3</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[{&#39;id&#39;: &#39;CC-MAIN-2021-43&#39;,
  &#39;name&#39;: &#39;October 2021 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-43/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-43-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2021-39&#39;,
  &#39;name&#39;: &#39;September 2021 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-39/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-39-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2021-31&#39;,
  &#39;name&#39;: &#39;July 2021 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-31/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-31-index&#39;}]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Notice the oldest ones have a different format</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">indexes</span><span class="p">[</span><span class="o">-</span><span class="mi">4</span><span class="p">:]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[{&#39;id&#39;: &#39;CC-MAIN-2013-20&#39;,
  &#39;name&#39;: &#39;Summer 2013 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2013-20/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2013-20-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2012&#39;,
  &#39;name&#39;: &#39;Index of 2012 ARC files&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2012/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2012-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2009-2010&#39;,
  &#39;name&#39;: &#39;Index of 2009 - 2010 ARC files&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2009-2010/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2009-2010-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2008-2009&#39;,
  &#39;name&#39;: &#39;Index of 2008 - 2009 ARC files&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2008-2009/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2008-2009-index&#39;}]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Parsing-the-times">Parsing the times<a class="anchor-link" href="#Parsing-the-times"> </a></h3><p>We can parse approximate periods for the crawls using the same logic as <a href="https://github.com/cocrawler/cdx_toolkit/blob/0.9.33/cdx_toolkit/timeutils.py#L68">cdx_toolkit</a>.</p>
<p>However in reality <a href="https://skeptric.com/common-crawl-time-ranges/">it's complicated</a></p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="parse_cc_crawl_date" class="doc_header"><code>parse_cc_crawl_date</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L249" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>parse_cc_crawl_date</code>(<strong><code>crawl_id</code></strong>:<code>str</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">dates</span> <span class="o">=</span> <span class="p">[</span><span class="n">parse_cc_crawl_date</span><span class="p">(</span><span class="n">i</span><span class="p">[</span><span class="s1">&#39;id&#39;</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">indexes</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="cc_index_by_time" class="doc_header"><code>cc_index_by_time</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L260" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>cc_index_by_time</code>(<strong><code>start</code></strong>:<code>Optional[datetime]</code>=<em><code>None</code></em>, <strong><code>end</code></strong>:<code>Optional[datetime]</code>=<em><code>None</code></em>, <strong><code>indexes</code></strong>:<code>Optional[list[str]]</code>=<em><code>None</code></em>)</p>
</blockquote>
<p>Gets all indexes that may contain entries between start and end</p>
<p>Generally errs on the side of giving an additional index</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc_index_by_time</span><span class="p">(</span><span class="n">start</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2021</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span> <span class="n">end</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2021</span><span class="p">,</span><span class="mi">12</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[&#39;CC-MAIN-2021-43&#39;,
 &#39;CC-MAIN-2021-39&#39;,
 &#39;CC-MAIN-2021-31&#39;,
 &#39;CC-MAIN-2021-25&#39;,
 &#39;CC-MAIN-2021-21&#39;,
 &#39;CC-MAIN-2021-17&#39;,
 &#39;CC-MAIN-2021-10&#39;]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc_index_by_time</span><span class="p">(</span><span class="n">end</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2010</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[&#39;CC-MAIN-2009-2010&#39;, &#39;CC-MAIN-2008-2009&#39;]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc_index_by_time</span><span class="p">(</span><span class="n">start</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2021</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[&#39;CC-MAIN-2021-43&#39;,
 &#39;CC-MAIN-2021-39&#39;,
 &#39;CC-MAIN-2021-31&#39;,
 &#39;CC-MAIN-2021-25&#39;,
 &#39;CC-MAIN-2021-21&#39;,
 &#39;CC-MAIN-2021-17&#39;,
 &#39;CC-MAIN-2021-10&#39;,
 &#39;CC-MAIN-2021-04&#39;,
 &#39;CC-MAIN-2020-50&#39;]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Querying-CDX">Querying CDX<a class="anchor-link" href="#Querying-CDX"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p><a href="https://pywb.readthedocs.io/en/latest/manual/cdxserver_api.html">https://pywb.readthedocs.io/en/latest/manual/cdxserver_api.html</a></p>
<p><a href="http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/">http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/</a></p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc_sample_api</span> <span class="o">=</span> <span class="s1">&#39;https://index.commoncrawl.org/CC-MAIN-2021-43-index&#39;</span>
<span class="k">assert</span> <span class="n">cc_sample_api</span> <span class="ow">in</span> <span class="p">[</span><span class="n">i</span><span class="p">[</span><span class="s1">&#39;cdx-api&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">indexes</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The Common Crawl CDX is <em>much slower</em> than the Wayback Machines.
It also uses the zipped CDX by default which automatically paginates.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="jsonl_loads" class="doc_header"><code>jsonl_loads</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L287" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>jsonl_loads</code>(<strong><code>jsonl</code></strong>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test_jline</span> <span class="o">=</span> <span class="sa">b</span><span class="s1">&#39;{&quot;status&quot;: &quot;200&quot;, &quot;mime&quot;: &quot;text/html&quot;}</span><span class="se">\n</span><span class="s1">{&quot;status&quot;: &quot;301&quot;, &quot;mime&quot;: &quot;-&quot;}</span><span class="se">\n</span><span class="s1">&#39;</span>
<span class="n">test_jdict</span> <span class="o">=</span> <span class="p">[{</span><span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;200&#39;</span><span class="p">,</span> <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">},</span> <span class="p">{</span><span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;301&#39;</span><span class="p">,</span> <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;-&#39;</span><span class="p">}]</span>
<span class="k">assert</span> <span class="n">jsonl_loads</span><span class="p">(</span><span class="n">test_jline</span><span class="p">)</span> <span class="o">==</span> <span class="n">test_jdict</span>
<span class="k">assert</span> <span class="n">jsonl_loads</span><span class="p">(</span><span class="n">test_jline</span><span class="o">.</span><span class="n">rstrip</span><span class="p">())</span> <span class="o">==</span> <span class="n">test_jdict</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="query_cc_cdx_num_pages" class="doc_header"><code>query_cc_cdx_num_pages</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L297" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>query_cc_cdx_num_pages</code>(<strong><code>api</code></strong>:<code>str</code>, <strong><code>url</code></strong>:<code>str</code>, <strong><code>page_size</code></strong>:<code>int</code>=<em><code>5</code></em>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="query_cc_cdx_page" class="doc_header"><code>query_cc_cdx_page</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L309" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>query_cc_cdx_page</code>(<strong><code>api</code></strong>:<code>str</code>, <strong><code>url</code></strong>:<code>str</code>, <strong><code>page</code></strong>:<code>int</code>, <strong><code>start</code></strong>:<code>Optional[str]</code>=<em><code>None</code></em>, <strong><code>end</code></strong>:<code>Optional[str]</code>=<em><code>None</code></em>, <strong><code>status_ok</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>mime</code></strong>:<code>Optional[Union[str, Iterable[str]]]</code>=<em><code>None</code></em>, <strong><code>limit</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>offset</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>page_size</code></strong>:<code>int</code>=<em><code>5</code></em>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>
<p>Get references to Common Crawl Captures for url.</p>
<p>Queries the Common Crawl Capture Index (CDX) for url.</p>
<p>Filters:</p>
<ul>
<li>api: API endpoint to use (e.g. '<a href="https://index.commoncrawl.org/CC-MAIN-2021-43-index">https://index.commoncrawl.org/CC-MAIN-2021-43-index</a>')</li>
<li>start: Minimum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>end: Maximum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>status_ok: Only return those with a HTTP status 200</li>
<li>mime: Filter on mimetypes, '<em>' is a wildcard (e.g. 'image/</em>')</li>
<li>limit: Only return first limit records</li>
<li>offset: Skip the first offset records, combine with limit</li>
<li>session: Session to use when making requests
Filters results between start and end inclusive, in format YYYYmmddHHMMSS or any substring
(e.g. start="202001", end="202001" will get all captures in January 2020)</li>
</ul>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Testing-pagination">Testing pagination<a class="anchor-link" href="#Testing-pagination"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test_url</span> <span class="o">=</span> <span class="s1">&#39;rmy.wikipedia.org/*&#39;</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">pages</span> <span class="o">=</span> <span class="n">query_cc_cdx_num_pages</span><span class="p">(</span><span class="n">cc_sample_api</span><span class="p">,</span> <span class="n">test_url</span><span class="p">,</span> <span class="n">page_size</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">pages</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>2</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">records</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">page</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">pages</span><span class="p">):</span>
    <span class="n">records</span> <span class="o">+=</span> <span class="n">query_cc_cdx_page</span><span class="p">(</span><span class="n">cc_sample_api</span><span class="p">,</span> <span class="n">test_url</span><span class="p">,</span> <span class="n">page</span><span class="o">=</span><span class="n">page</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">,</span> <span class="n">page_size</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="nb">len</span><span class="p">(</span><span class="n">records</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>576</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">timestamps</span> <span class="o">=</span> <span class="p">[</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">records</span><span class="p">]</span>
<span class="nb">min</span><span class="p">(</span><span class="n">timestamps</span><span class="p">),</span> <span class="nb">max</span><span class="p">(</span><span class="n">timestamps</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>(&#39;20211017144648&#39;, &#39;20211026191502&#39;)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We should get an error querying past the last page (400: Invalid Request)</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">requests.exceptions</span> <span class="kn">import</span> <span class="n">HTTPError</span>

<span class="k">try</span><span class="p">:</span>
    <span class="n">query_cc_cdx_page</span><span class="p">(</span><span class="n">cc_sample_api</span><span class="p">,</span> <span class="n">test_url</span><span class="p">,</span> <span class="n">page</span><span class="o">=</span><span class="n">pages</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">,</span> <span class="n">page_size</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
    <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="s1">&#39;Expected Failure&#39;</span><span class="p">)</span>
<span class="k">except</span> <span class="n">HTTPError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
    <span class="n">status</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span>
    <span class="k">if</span> <span class="n">status</span> <span class="o">!=</span> <span class="mi">400</span><span class="p">:</span>
        <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Expected 400, got </span><span class="si">{</span><span class="n">status</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Error-for-missing-fields">Error for missing fields<a class="anchor-link" href="#Error-for-missing-fields"> </a></h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>For CC-MAIN-2015-11 and CC-MAIN-2015-06 both the status and mime are missing from the index which leads to <em>problems</em>.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">api_2015_11</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">idx</span><span class="p">[</span><span class="s1">&#39;cdx-api&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">idx</span> <span class="ow">in</span> <span class="n">get_cc_indexes</span><span class="p">()</span> <span class="k">if</span> <span class="n">idx</span><span class="p">[</span><span class="s1">&#39;id&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;CC-MAIN-2015-11&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This turns up in this particular case as a 404</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">try</span><span class="p">:</span>
    <span class="n">results</span> <span class="o">=</span> <span class="n">query_cc_cdx_page</span><span class="p">(</span><span class="n">api_2015_11</span><span class="p">,</span> <span class="n">test_url</span><span class="p">,</span> <span class="n">page</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">page_size</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
    <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">&quot;Has it been fixed? Remove the blacklist&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="n">requests</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
    <span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">!=</span> <span class="mi">404</span><span class="p">:</span>
        <span class="k">raise</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Which doesn't happen when we remove the filters</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">query_cc_cdx_page</span><span class="p">(</span><span class="n">api_2015_11</span><span class="p">,</span> <span class="n">test_url</span><span class="p">,</span> <span class="n">page</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">page_size</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Downloading">Downloading<a class="anchor-link" href="#Downloading"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="fetch_cc" class="doc_header"><code>fetch_cc</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L369" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>fetch_cc</code>(<strong><code>filename</code></strong>:<code>str</code>, <strong><code>offset</code></strong>:<code>int</code>, <strong><code>length</code></strong>:<code>int</code>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">record</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;urlkey&#39;: &#39;org,wikipedia,rmy)/wiki/%c3%93lvega&#39;,
 &#39;timestamp&#39;: &#39;20211019031358&#39;,
 &#39;url&#39;: &#39;https://rmy.wikipedia.org/wiki/%C3%93lvega&#39;,
 &#39;mime&#39;: &#39;text/html&#39;,
 &#39;mime-detected&#39;: &#39;text/html&#39;,
 &#39;status&#39;: &#39;200&#39;,
 &#39;digest&#39;: &#39;QZSQHAQSLMBHHCJSADEUS3I4U3EJALH5&#39;,
 &#39;length&#39;: &#39;11604&#39;,
 &#39;offset&#39;: &#39;641473227&#39;,
 &#39;filename&#39;: &#39;crawl-data/CC-MAIN-2021-43/segments/1634323585231.62/warc/CC-MAIN-20211019012407-20211019042407-00526.warc.gz&#39;,
 &#39;languages&#39;: &#39;ron,nno,vie&#39;,
 &#39;encoding&#39;: &#39;UTF-8&#39;}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;urlkey&#39;</span><span class="p">:</span> <span class="s1">&#39;org,wikipedia,en)/?curid=3516101&#39;</span><span class="p">,</span>
 <span class="s1">&#39;timestamp&#39;</span><span class="p">:</span> <span class="s1">&#39;20211024051554&#39;</span><span class="p">,</span>
 <span class="s1">&#39;url&#39;</span><span class="p">:</span> <span class="s1">&#39;https://en.wikipedia.org/?curid=3516101&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime-detected&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;200&#39;</span><span class="p">,</span>
 <span class="s1">&#39;digest&#39;</span><span class="p">:</span> <span class="s1">&#39;TFZFWXWKS3NFJLLLXLYU6JTNZ77D3IVD&#39;</span><span class="p">,</span>
 <span class="s1">&#39;length&#39;</span><span class="p">:</span> <span class="s1">&#39;8922&#39;</span><span class="p">,</span>
 <span class="s1">&#39;offset&#39;</span><span class="p">:</span> <span class="s1">&#39;332329771&#39;</span><span class="p">,</span>
 <span class="s1">&#39;filename&#39;</span><span class="p">:</span> <span class="s1">&#39;crawl-data/CC-MAIN-2021-43/segments/1634323585911.17/warc/CC-MAIN-20211024050128-20211024080128-00689.warc.gz&#39;</span><span class="p">,</span>
 <span class="s1">&#39;languages&#39;</span><span class="p">:</span> <span class="s1">&#39;eng&#39;</span><span class="p">,</span>
 <span class="s1">&#39;encoding&#39;</span><span class="p">:</span> <span class="s1">&#39;UTF-8&#39;</span><span class="p">}</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">fetch_cc</span><span class="p">(</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;filename&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;offset&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;length&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 25.2 ms, sys: 56 µs, total: 25.3 ms
Wall time: 1.17 s
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span><span class="p">[:</span><span class="mi">1000</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>b&#39;&lt;!DOCTYPE html&gt;\n&lt;html class=&#34;client-nojs&#34; lang=&#34;en&#34; dir=&#34;ltr&#34;&gt;\n&lt;head&gt;\n&lt;meta charset=&#34;UTF-8&#34;/&gt;\n&lt;title&gt;Category:Tad Morose albums - Wikipedia&lt;/title&gt;\n&lt;script&gt;document.documentElement.className=&#34;client-js&#34;;RLCONF={&#34;wgBreakFrames&#34;:!1,&#34;wgSeparatorTransformTable&#34;:[&#34;&#34;,&#34;&#34;],&#34;wgDigitTransformTable&#34;:[&#34;&#34;,&#34;&#34;],&#34;wgDefaultDateFormat&#34;:&#34;dmy&#34;,&#34;wgMonthNames&#34;:[&#34;&#34;,&#34;January&#34;,&#34;February&#34;,&#34;March&#34;,&#34;April&#34;,&#34;May&#34;,&#34;June&#34;,&#34;July&#34;,&#34;August&#34;,&#34;September&#34;,&#34;October&#34;,&#34;November&#34;,&#34;December&#34;],&#34;wgRequestId&#34;:&#34;362a73a3-175d-4af3-92bb-4707797d6067&#34;,&#34;wgCSPNonce&#34;:!1,&#34;wgCanonicalNamespace&#34;:&#34;Category&#34;,&#34;wgCanonicalSpecialPageName&#34;:!1,&#34;wgNamespaceNumber&#34;:14,&#34;wgPageName&#34;:&#34;Category:Tad_Morose_albums&#34;,&#34;wgTitle&#34;:&#34;Tad Morose albums&#34;,&#34;wgCurRevisionId&#34;:906041584,&#34;wgRevisionId&#34;:906041584,&#34;wgArticleId&#34;:3516101,&#34;wgIsArticle&#34;:!0,&#34;wgIsRedirect&#34;:!1,&#34;wgAction&#34;:&#34;view&#34;,&#34;wgUserName&#34;:null,&#34;wgUserGroups&#34;:[&#34;*&#34;],&#34;wgCategories&#34;:[&#34;Set categories&#34;,&#34;Albums by artist&#34;,&#34;Power metal albums by Swedish artists&#34;],&#34;wgPageContentLanguage&#34;:&#34;en&#34;,&#34;wgPageContentModel&#34;:&#34;wik&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Check the digest</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sha1_digest</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="o">==</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;digest&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Put-it-into-an-Object">Put it into an Object<a class="anchor-link" href="#Put-it-into-an-Object"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="CommonCrawlRecord" class="doc_header"><code>class</code> <code>CommonCrawlRecord</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L394" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>CommonCrawlRecord</code>(<strong><code>url</code></strong>:<code>str</code>, <strong><code>timestamp</code></strong>:<code>datetime</code>, <strong><code>filename</code></strong>:<code>str</code>, <strong><code>offset</code></strong>:<code>int</code>, <strong><code>length</code></strong>:<code>int</code>, <strong><code>mime</code></strong>:<code>Optional[str]</code>, <strong><code>status</code></strong>:<code>Optional[int]</code>, <strong><code>digest</code></strong>:<code>Optional[str]</code>)</p>
</blockquote>
<p>CommonCrawlRecord(url: 'str', timestamp: 'datetime', filename: 'str', offset: 'int', length: 'int', mime: 'Optional[str]', status: 'Optional[int]', digest: 'Optional[str]')</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;urlkey&#39;: &#39;org,wikipedia,en)/?curid=3516101&#39;,
 &#39;timestamp&#39;: &#39;20211024051554&#39;,
 &#39;url&#39;: &#39;https://en.wikipedia.org/?curid=3516101&#39;,
 &#39;mime&#39;: &#39;text/html&#39;,
 &#39;mime-detected&#39;: &#39;text/html&#39;,
 &#39;status&#39;: &#39;200&#39;,
 &#39;digest&#39;: &#39;TFZFWXWKS3NFJLLLXLYU6JTNZ77D3IVD&#39;,
 &#39;length&#39;: &#39;8922&#39;,
 &#39;offset&#39;: &#39;332329771&#39;,
 &#39;filename&#39;: &#39;crawl-data/CC-MAIN-2021-43/segments/1634323585911.17/warc/CC-MAIN-20211024050128-20211024080128-00689.warc.gz&#39;,
 &#39;languages&#39;: &#39;eng&#39;,
 &#39;encoding&#39;: &#39;UTF-8&#39;}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="n">_cc_cdx_to_record</span><span class="p">(</span><span class="n">record</span><span class="p">)</span>
<span class="n">x</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>CommonCrawlRecord(url=&#39;https://en.wikipedia.org/?curid=3516101&#39;, timestamp=datetime.datetime(2021, 10, 24, 5, 15, 54), filename=&#39;crawl-data/CC-MAIN-2021-43/segments/1634323585911.17/warc/CC-MAIN-20211024050128-20211024080128-00689.warc.gz&#39;, offset=&#39;332329771&#39;, length=&#39;8922&#39;, mime=&#39;text/html&#39;, status=200, digest=&#39;TFZFWXWKS3NFJLLLXLYU6JTNZ77D3IVD&#39;)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">x</span><span class="o">.</span><span class="n">content</span><span class="p">[:</span><span class="mi">100</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>b&#39;&lt;!DOCTYPE html&gt;\n&lt;html class=&#34;client-nojs&#34; lang=&#34;en&#34; dir=&#34;ltr&#34;&gt;\n&lt;head&gt;\n&lt;meta charset=&#34;UTF-8&#34;/&gt;\n&lt;title&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s1">&#39;wb&#39;</span><span class="p">,</span> <span class="nb">dir</span><span class="o">=</span><span class="s1">&#39;.&#39;</span><span class="p">,</span> <span class="n">suffix</span><span class="o">=</span><span class="s1">&#39;.html&#39;</span><span class="p">)</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">)</span><span class="o">.</span><span class="n">relative_to</span><span class="p">(</span><span class="n">Path</span><span class="p">(</span><span class="s1">&#39;.&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">absolute</span><span class="p">())</span>
<span class="n">x</span><span class="o">.</span><span class="n">preview</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<a href='tmpn7cny9ks.html' target='_blank'>tmpn7cny9ks.html</a><br>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Common-Crawl-Query">Common Crawl Query<a class="anchor-link" href="#Common-Crawl-Query"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="CommonCrawlQuery" class="doc_header"><code>class</code> <code>CommonCrawlQuery</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/query.py#L441" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>CommonCrawlQuery</code>(<strong><code>url</code></strong>:<code>str</code>, <strong><code>start</code></strong>:<code>Optional[str]</code>=<em><code>None</code></em>, <strong><code>end</code></strong>:<code>Optional[str]</code>=<em><code>None</code></em>, <strong><code>apis</code></strong>:<code>Optional[list[str]]</code>=<em><code>None</code></em>, <strong><code>status_ok</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>mime</code></strong>:<code>Optional[Union[str, Iterable[str]]]</code>=<em><code>None</code></em>)</p>
</blockquote>
<p>CommonCrawlQuery(url: 'str', start: 'Optional[str]' = None, end: 'Optional[str]' = None, apis: 'Optional[list[str]]' = None, status_ok: 'bool' = True, mime: 'Optional[Union[str, Iterable[str]]]' = None)</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-querying-on-blacklist-APIs">Test querying on blacklist APIs<a class="anchor-link" href="#Test-querying-on-blacklist-APIs"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">CC_API_FILTER_BLACKLIST</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[&#39;CC-MAIN-2015-11&#39;, &#39;CC-MAIN-2015-06&#39;]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">query_cc_all</span> <span class="o">=</span> <span class="n">CommonCrawlQuery</span><span class="p">(</span><span class="s1">&#39;www.commoncrawl.org/*&#39;</span><span class="p">,</span>
                           <span class="n">start</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span> <span class="n">end</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">query_cc_all</span><span class="o">.</span><span class="n">cdx_apis</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 4.73 ms, sys: 307 µs, total: 5.04 ms
Wall time: 5.02 ms
</pre>
</div>
</div>

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;CC-MAIN-2015-11&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2015-11-index&#39;,
 &#39;CC-MAIN-2015-06&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2015-06-index&#39;}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">query_cc_all</span><span class="o">.</span><span class="n">query</span><span class="p">())</span>
<span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>13</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Notice there is no Mime or Status. We <em>could</em> fill it in from the archive record.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>CommonCrawlRecord(url=&#39;http://commoncrawl.org/&#39;, timestamp=datetime.datetime(2015, 3, 2, 3, 27, 5), filename=&#39;crawl-data/CC-MAIN-2015-11/segments/1424936462700.28/warc/CC-MAIN-20150226074102-00159-ip-10-28-5-156.ec2.internal.warc.gz&#39;, offset=&#39;53235662&#39;, length=&#39;2526&#39;, mime=None, status=None, digest=&#39;QE4UUUWUJWEZBBK6PUG3CHFAGEKDMDBZ&#39;)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">content</span><span class="p">[:</span><span class="mi">100</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>b&#39;\n&lt;!doctype html&gt;\n&lt;html class=&#34;no-js&#34; lang=&#34;en&#34;&gt;\n&lt;head&gt;\n&lt;meta charset=&#34;utf-8&#34;/&gt;\n&lt;meta name=&#34;viewport&#34;&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-querying-on-happy-case-data">Test querying on happy case data<a class="anchor-link" href="#Test-querying-on-happy-case-data"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test_url</span> <span class="o">=</span> <span class="s1">&#39;upload.wikimedia.org/wikipedia/commons/*&#39;</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc_archive_all</span> <span class="o">=</span> <span class="n">CommonCrawlQuery</span><span class="p">(</span><span class="n">test_url</span><span class="p">,</span>
                                  <span class="n">apis</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">],</span>
                                  <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>

<span class="n">results</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">cc_archive_all</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">page_size</span><span class="o">=</span><span class="mi">5</span><span class="p">))</span>

<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">Counter</span>
<span class="n">Counter</span><span class="p">((</span><span class="n">r</span><span class="o">.</span><span class="n">mime</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">status</span><span class="p">)</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>Counter({(&#39;application/pdf&#39;, 200): 561,
         (&#39;text/html&#39;, 404): 181,
         (&#39;image/vnd.djvu&#39;, 200): 125,
         (&#39;image/jpeg&#39;, 200): 65,
         (&#39;image/png&#39;, 200): 20,
         (&#39;unk&#39;, 301): 31,
         (&#39;application/ogg&#39;, 200): 2,
         (&#39;application/sla&#39;, 200): 2,
         (&#39;image/svg+xml&#39;, 200): 4,
         (&#39;image/gif&#39;, 200): 2,
         (&#39;image/x-xcf&#39;, 200): 2,
         (&#39;text/html&#39;, 412): 2,
         (&#39;warc/revisit&#39;, 304): 1,
         (&#39;text/html&#39;, 400): 154})</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-status">Test status<a class="anchor-link" href="#Test-status"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results_ok</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">CommonCrawlQuery</span><span class="p">(</span><span class="n">test_url</span><span class="p">,</span>
                                  <span class="n">apis</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">])</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">page_size</span><span class="o">=</span><span class="mi">5</span><span class="p">))</span>

<span class="k">assert</span> <span class="n">results_ok</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">status</span> <span class="o">==</span> <span class="mi">200</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-mimetypes">Test mimetypes<a class="anchor-link" href="#Test-mimetypes"> </a></h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>These would fail in the BLACKLIST apis.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results_html</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">CommonCrawlQuery</span><span class="p">(</span><span class="n">test_url</span><span class="p">,</span>
                                  <span class="n">apis</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">],</span>
                                  <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
                                  <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">page_size</span><span class="o">=</span><span class="mi">5</span><span class="p">))</span>

<span class="k">assert</span> <span class="n">results_html</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">mime</span> <span class="o">==</span> <span class="s1">&#39;text/html&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">mimes</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;application/pdf&#39;</span><span class="p">,</span> <span class="s1">&#39;image/vnd.djvu&#39;</span><span class="p">,</span> <span class="s1">&#39;application/ogg&#39;</span><span class="p">]</span>
<span class="n">results_mime</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">CommonCrawlQuery</span><span class="p">(</span><span class="n">test_url</span><span class="p">,</span>
                                  <span class="n">apis</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">],</span>
                                  <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
                                  <span class="n">mime</span><span class="o">=</span><span class="n">mimes</span><span class="p">)</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">page_size</span><span class="o">=</span><span class="mi">5</span><span class="p">))</span>

<span class="k">assert</span> <span class="n">results_mime</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">mime</span> <span class="ow">in</span> <span class="n">mimes</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results_image</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">CommonCrawlQuery</span><span class="p">(</span><span class="n">test_url</span><span class="p">,</span>
                                  <span class="n">apis</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">],</span>
                                  <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
                                  <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;image/*&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">page_size</span><span class="o">=</span><span class="mi">5</span><span class="p">))</span>

<span class="k">assert</span> <span class="n">results_image</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">mime</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">&#39;image/&#39;</span><span class="p">)]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Check-fetching">Check fetching<a class="anchor-link" href="#Check-fetching"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">result</span> <span class="o">=</span> <span class="n">_cc_cdx_to_record</span><span class="p">({</span><span class="s1">&#39;urlkey&#39;</span><span class="p">:</span> <span class="s1">&#39;com,skeptric)/&#39;</span><span class="p">,</span>
 <span class="s1">&#39;timestamp&#39;</span><span class="p">:</span> <span class="s1">&#39;20211028110756&#39;</span><span class="p">,</span>
 <span class="s1">&#39;url&#39;</span><span class="p">:</span> <span class="s1">&#39;https://skeptric.com/&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime-detected&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;200&#39;</span><span class="p">,</span>
 <span class="s1">&#39;digest&#39;</span><span class="p">:</span> <span class="s1">&#39;7RBLUZ55MD4FUPDRPVVJVTM7YDYQXRS3&#39;</span><span class="p">,</span>
 <span class="s1">&#39;length&#39;</span><span class="p">:</span> <span class="s1">&#39;107238&#39;</span><span class="p">,</span>
 <span class="s1">&#39;offset&#39;</span><span class="p">:</span> <span class="s1">&#39;650789450&#39;</span><span class="p">,</span>
 <span class="s1">&#39;filename&#39;</span><span class="p">:</span> <span class="s1">&#39;crawl-data/CC-MAIN-2021-43/segments/1634323588284.71/warc/CC-MAIN-20211028100619-20211028130619-00465.warc.gz&#39;</span><span class="p">,</span>
 <span class="s1">&#39;languages&#39;</span><span class="p">:</span> <span class="s1">&#39;eng&#39;</span><span class="p">,</span>
 <span class="s1">&#39;encoding&#39;</span><span class="p">:</span> <span class="s1">&#39;UTF-8&#39;</span><span class="p">})</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sha1_digest</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">content</span><span class="p">)</span> <span class="o">==</span> <span class="n">result</span><span class="o">.</span><span class="n">digest</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Check-fetching-in-parallel">Check fetching in parallel<a class="anchor-link" href="#Check-fetching-in-parallel"> </a></h3><p>Since they're hosted on AWS S3 we can go wild with parallelism.
This code is actually identical to the wayback version</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">joblib</span> <span class="kn">import</span> <span class="n">delayed</span><span class="p">,</span> <span class="n">Parallel</span>


<span class="k">def</span> <span class="nf">cc_fetch_parallel</span><span class="p">(</span><span class="n">items</span><span class="p">,</span> <span class="n">threads</span><span class="o">=</span><span class="mi">64</span><span class="p">,</span> <span class="n">session</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
    <span class="k">if</span> <span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
        <span class="n">session</span> <span class="o">=</span> <span class="n">make_session</span><span class="p">(</span><span class="n">threads</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">Parallel</span><span class="p">(</span><span class="n">n_jobs</span><span class="o">=</span><span class="n">threads</span><span class="p">,</span> <span class="n">prefer</span><span class="o">=</span><span class="s1">&#39;threads&#39;</span><span class="p">)(</span><span class="n">delayed</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">get_content</span><span class="p">)(</span><span class="n">session</span><span class="o">=</span><span class="n">session</span><span class="p">)</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">items</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">many_content</span> <span class="o">=</span> <span class="n">cc_fetch_parallel</span><span class="p">(</span><span class="n">results_html</span><span class="p">[:</span><span class="mi">100</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

</div>
 

