---

title: CDX


keywords: fastai
sidebar: home_sidebar



nb_path: "nbs/01_cdx.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/01_cdx.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is an interface to the <a href="http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/">Common Crawl Index</a> and the <a href="https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server">Internet Archive's CDX Server</a> for finding archived resources based on URL patterns.
It has a lot of overlap with <a href="https://github.com/cocrawler/cdx_toolkit">cdx_toolkit</a> but is more optimised for speed and ease of use.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test_cache</span> <span class="o">=</span> <span class="s1">&#39;../data/test/cache&#39;</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Internet-Archive">Internet Archive<a class="anchor-link" href="#Internet-Archive"> </a></h1><h2 id="Querying-CDX">Querying CDX<a class="anchor-link" href="#Querying-CDX"> </a></h2><p>The Internet Archive runs it's own <a href="https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server">Java CDX Server</a>. <a href="https://hackernoon.com/guide-to-handling-internet-archives-cdx-server-api-response-c469df5b81f4">Hakernoon has an article</a> that gives a good overview of querying it.</p>
<p>See the documentation for all the paramemters; here are the ones that I find most interesting</p>
<ul>
<li>url: The URL to query, a wildcard <code>*</code> can implicitly define it (which means we don't need <code>matchType</code>)</li>
<li>output: "json" returns JSON instead of space separated</li>
<li>fl: Comma separated list of fields to return</li>
<li>from, to: Date filtering 1-14 digits yyyyMMddhhmmss, inclusive (this seems to do the right thing for truncation).</li>
<li>filter: <code>[!]field:regex</code> filter a returned field with <a href="https://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html">Java Regex</a>, inverting with <code>!</code></li>
<li>limit/offset: For getting a small number of results (sampling). Internal limit is 150000 results.</li>
</ul>
<p>Default fields returned: ["urlkey","timestamp","original","mimetype","statuscode","digest","length"]</p>
<p>Pagination isn't really useful because it's applied <em>before</em> filtering (including date filtering) so most pages are empty with a date filter.
Unpaginated requests are fast enough anyway.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Exporting-the-data">Exporting the data<a class="anchor-link" href="#Exporting-the-data"> </a></h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The data is a JSON array or arrays; the first line contains the headers and the subsequent the data.
Let's transform it into a list of dictionaries of keys to values which make it a bit easier to work with (although less efficient in memory).</p>
<p>An alternative would be to directly use something like Pandas</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="header_and_rows_to_dict" class="doc_header"><code>header_and_rows_to_dict</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L22" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>header_and_rows_to_dict</code>(<strong><code>rows</code></strong>:<code>Iterable[list[Any]]</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Check it on some data</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">header_and_rows_to_dict</span><span class="p">([[</span><span class="s1">&#39;col_1&#39;</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;a&#39;</span><span class="p">],</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">]])</span> <span class="o">==</span> <span class="p">[</span>
    <span class="p">{</span><span class="s1">&#39;col_1&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">:</span> <span class="s1">&#39;a&#39;</span><span class="p">},</span>
    <span class="p">{</span><span class="s1">&#39;col_1&#39;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">:</span> <span class="s1">&#39;b&#39;</span><span class="p">}]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">header_and_rows_to_dict</span><span class="p">([[</span><span class="s1">&#39;col_1&#39;</span><span class="p">,</span> <span class="s1">&#39;col_2&#39;</span><span class="p">]])</span> <span class="o">==</span> <span class="p">[]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">header_and_rows_to_dict</span><span class="p">([])</span> <span class="o">==</span> <span class="p">[]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Making-a-query">Making a query<a class="anchor-link" href="#Making-a-query"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="mimetypes_to_regex" class="doc_header"><code>mimetypes_to_regex</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L39" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>mimetypes_to_regex</code>(<strong><code>mime</code></strong>:<code>list[str]</code>, <strong><code>prefix</code></strong>=<em><code>'mimetype:'</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="query_wayback_cdx" class="doc_header"><code>query_wayback_cdx</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L42" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>query_wayback_cdx</code>(<strong><code>url</code></strong>:<code>str</code>, <strong><code>start</code></strong>:<code>Optional[str]</code>, <strong><code>end</code></strong>:<code>Optional[str]</code>, <strong><code>status_ok</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>mime</code></strong>:<code>Optional[Union[str, Iterable[str]]]</code>=<em><code>None</code></em>, <strong><code>limit</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>offset</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>
<p>Get references to Wayback Machine Captures for url.</p>
<p>Queries the Internet Archive Capture Index (CDX) for url.</p>
<p>Arguments:</p>
<ul>
<li>start: Minimum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>end: Maximum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>status_ok: Only return those with a HTTP status 200</li>
<li>mime: Filter on mimetypes, '<em>' is a wildcard (e.g. 'image/</em>')</li>
<li>limit: Only return first limit records</li>
<li>offset: Skip the first offset records, combine with limit</li>
<li>session: Session to use when making requests
Filters results between start and end inclusive, in format YYYYmmddHHMMSS or any substring
(e.g. start="202001", end="202001" will get all captures in January 2020)</li>
</ul>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-querying">Test querying<a class="anchor-link" href="#Test-querying"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">full_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 16.8 ms, sys: 0 ns, total: 16.8 ms
Wall time: 1.53 s
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">full_sample</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1000</span>
<span class="nb">len</span><span class="p">(</span><span class="n">full_sample</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>1579</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">full_sample</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>urlkey</th>
      <th>timestamp</th>
      <th>original</th>
      <th>mimetype</th>
      <th>statuscode</th>
      <th>digest</th>
      <th>length</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>com,skeptric)/</td>
      <td>20180805132731</td>
      <td>http://skeptric.com/</td>
      <td>text/html</td>
      <td>200</td>
      <td>2N5QQYZFAM36CSESTGSDKRTRV7I5HEXJ</td>
      <td>1982</td>
    </tr>
    <tr>
      <th>1</th>
      <td>com,skeptric)/</td>
      <td>20190610142553</td>
      <td>http://skeptric.com/</td>
      <td>warc/revisit</td>
      <td>-</td>
      <td>2N5QQYZFAM36CSESTGSDKRTRV7I5HEXJ</td>
      <td>580</td>
    </tr>
    <tr>
      <th>2</th>
      <td>com,skeptric)/</td>
      <td>20201126064102</td>
      <td>https://skeptric.com/</td>
      <td>text/html</td>
      <td>200</td>
      <td>WDYU3RU7ZMFFSZPAPE56PC4L3EK4FE3D</td>
      <td>82985</td>
    </tr>
    <tr>
      <th>3</th>
      <td>com,skeptric)/</td>
      <td>20210109075617</td>
      <td>https://skeptric.com/</td>
      <td>text/html</td>
      <td>200</td>
      <td>M6GZ4ZVD7U5L2TUXS5POHN36SSLOEPO7</td>
      <td>92869</td>
    </tr>
    <tr>
      <th>4</th>
      <td>com,skeptric)/</td>
      <td>20210417050827</td>
      <td>https://skeptric.com/</td>
      <td>text/html</td>
      <td>200</td>
      <td>ZOYZF2RVY44PBSNNXMVH27MHUPGGO4OK</td>
      <td>101114</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>1574</th>
      <td>com,skeptric)/wsl2-xserver</td>
      <td>20201216173655</td>
      <td>https://skeptric.com/wsl2-xserver/</td>
      <td>warc/revisit</td>
      <td>-</td>
      <td>UAS4RFHUDI6TNMPWVTD4N6RZWLKQAOGT</td>
      <td>720</td>
    </tr>
    <tr>
      <th>1575</th>
      <td>com,skeptric)/wsl2-xserver</td>
      <td>20201216185608</td>
      <td>https://skeptric.com/wsl2-xserver/</td>
      <td>warc/revisit</td>
      <td>-</td>
      <td>UAS4RFHUDI6TNMPWVTD4N6RZWLKQAOGT</td>
      <td>721</td>
    </tr>
    <tr>
      <th>1576</th>
      <td>com,skeptric)/wsl2-xserver</td>
      <td>20210128160531</td>
      <td>https://skeptric.com/wsl2-xserver/</td>
      <td>text/html</td>
      <td>200</td>
      <td>BBSHS372N5MIYDLGSQWQKF6TKNBEDD35</td>
      <td>5794</td>
    </tr>
    <tr>
      <th>1577</th>
      <td>com,skeptric)/wsl2-xserver</td>
      <td>20210520180504</td>
      <td>https://skeptric.com/wsl2-xserver/</td>
      <td>text/html</td>
      <td>200</td>
      <td>BBSHS372N5MIYDLGSQWQKF6TKNBEDD35</td>
      <td>4821</td>
    </tr>
    <tr>
      <th>1578</th>
      <td>com,skeptric)/wsl2-xserver</td>
      <td>20211011012618</td>
      <td>https://skeptric.com/wsl2-xserver/</td>
      <td>text/html</td>
      <td>200</td>
      <td>LZPYUW2572JR5QP3W3JLQ4SAHQWCY7PQ</td>
      <td>5603</td>
    </tr>
  </tbody>
</table>
<p>1579 rows × 7 columns</p>
</div>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">timestamps</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span><span class="p">]</span>
<span class="nb">min</span><span class="p">(</span><span class="n">timestamps</span><span class="p">),</span> <span class="nb">max</span><span class="p">(</span><span class="n">timestamps</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>(&#39;20170915060403&#39;, &#39;20211123083615&#39;)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">full_sample</span><span class="p">)</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s1">&#39;mimetype&#39;</span><span class="p">,</span> <span class="s1">&#39;statuscode&#39;</span><span class="p">])[</span><span class="s1">&#39;urlkey&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>mimetype                  statuscode
application/javascript    200             1
application/octet-stream  200             3
image/gif                 200             3
image/jpeg                200            59
image/png                 200           292
image/svg+xml             200            51
image/vnd.microsoft.icon  200             1
image/x-icon              200             2
text/css                  200            10
text/html                 200           393
                          302             1
                          404             5
warc/revisit              -             758
Name: urlkey, dtype: int64</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-statuscode-filter">Test statuscode filter<a class="anchor-link" href="#Test-statuscode-filter"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">ok_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span> <span class="c1">#status_ok=True</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 14.6 ms, sys: 2.33 ms, total: 16.9 ms
Wall time: 829 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">ok_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;statuscode&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;200&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-mimetypes">Test mimetypes<a class="anchor-link" href="#Test-mimetypes"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">html_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 6.73 ms, sys: 0 ns, total: 6.73 ms
Wall time: 634 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">html_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;mimetype&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;text/html&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">image_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;image/*&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 9.46 ms, sys: 0 ns, total: 9.46 ms
Wall time: 701 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">image_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;mimetype&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">&#39;image/&#39;</span><span class="p">)]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">prog_sample</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> 
                                 <span class="n">mime</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;text/css&#39;</span><span class="p">,</span> <span class="s1">&#39;application/javascript&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 0 ns, sys: 4.06 ms, total: 4.06 ms
Wall time: 472 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">prog_sample</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;mimetype&#39;</span><span class="p">]</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">&#39;text/css&#39;</span><span class="p">,</span> <span class="s1">&#39;application/javascript&#39;</span><span class="p">]]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-date-filters">Test date filters<a class="anchor-link" href="#Test-date-filters"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_2020</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="nb">len</span><span class="p">(</span><span class="n">sample_2020</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 3.84 ms, sys: 0 ns, total: 3.84 ms
Wall time: 505 ms
</pre>
</div>
</div>

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>106</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_2020</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="s1">&#39;2020&#39;</span> <span class="o">&lt;=</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&lt;</span> <span class="s1">&#39;2021&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_to_2020</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 5.08 ms, sys: 0 ns, total: 5.08 ms
Wall time: 447 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_to_2020</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&lt;</span> <span class="s1">&#39;2021&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_from_2020</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="s1">&#39;2020&#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 15.9 ms, sys: 0 ns, total: 15.9 ms
Wall time: 1.34 s
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_from_2020</span> <span class="o">==</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">full_sample</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&gt;=</span> <span class="s1">&#39;2020&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-limits">Test limits<a class="anchor-link" href="#Test-limits"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_10</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">limit</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 4.59 ms, sys: 0 ns, total: 4.59 ms
Wall time: 442 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">sample_10</span> <span class="o">==</span> <span class="n">full_sample</span><span class="p">[:</span><span class="mi">10</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>True</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">sample_10_offset_20</span> <span class="o">=</span> <span class="n">query_wayback_cdx</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">limit</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">offset</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 0 ns, sys: 4.72 ms, total: 4.72 ms
Wall time: 432 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">sample_10_offset_20</span>  <span class="o">==</span> <span class="n">full_sample</span><span class="p">[</span><span class="mi">20</span><span class="p">:</span><span class="mi">20</span><span class="o">+</span><span class="mi">10</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Fetching-Content">Fetching Content<a class="anchor-link" href="#Fetching-Content"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="fetch_internet_archive_content" class="doc_header"><code>fetch_internet_archive_content</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L88" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>fetch_internet_archive_content</code>(<strong><code>timestamp</code></strong>:<code>str</code>, <strong><code>url</code></strong>:<code>str</code>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="n">image_sample</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">record</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;urlkey&#39;: &#39;com,skeptric)/favicon.ico&#39;,
 &#39;timestamp&#39;: &#39;20180819051126&#39;,
 &#39;original&#39;: &#39;http://skeptric.com/favicon.ico&#39;,
 &#39;mimetype&#39;: &#39;image/vnd.microsoft.icon&#39;,
 &#39;statuscode&#39;: &#39;200&#39;,
 &#39;digest&#39;: &#39;R6YE2GPPT4BM4IAMHGUJPDJF6BGKKHDA&#39;,
 &#39;length&#39;: &#39;851&#39;}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span> <span class="o">=</span> <span class="n">fetch_internet_archive_content</span><span class="p">(</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;original&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="n">Image</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_png output_subarea output_execute_result">
<img src="data:image/png;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgAAACYAAAAmwAAAIwAAABjAAAAFgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHoAAAD/AAAAQQAAAAEAAAABAAAANAAAAOoAAAAnAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAD/AAAAewAAAAAAAAAAAAAAAAAAAAAAAAB7AAAA6AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACoAAAA9gAAAJQAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAP8AAABPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAzQAAAPwAAAD/AAAAAAAAAAAAAAAAAAAAAAAAACUAAAD7AAAAkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKoAAAD/AAAA/wAAAAAAAABUAAAAAAAAAAAAAAAmAAAA/QAAAJwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAuAAAA/wAAAP8AAAACAAAA0gAAAAAAAAAAAAAAYgAAAP8AAABlAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGEAAAD/AAAAkgAAAPwAAAAAAAAAAgAAAPkAAAD8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXwAAAPEAAAD/AAAACgAAAMwAAAD/AAAASgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPAAAAqwAAAOwAAAD/AAAAYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQwAAAPoAAADqAAAAPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYwAAAP8AAADEAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGAAAAPIAAAC5AAAAAQAAAAAAAAADAAAAHgAAAAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC8AAADxAAAAAAAAAAAAAAABAAAArgAAAPgAAABgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAAAA5gAAACAAAAAbAAAAfQAAAO4AAACbAAAABQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACUAAAB7AAAAegAAAE8AAAAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/H8AAPvfAAD37wAA4+8AAOPnAADj5wAA8u8AAPjPAAD8nwAA/j8AAP5/AAD8/wAA+f8AAPufAAD7nwAA//8AAA==
"
>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="nb">len</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>1150</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="n">ok_sample</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">record</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;urlkey&#39;: &#39;com,skeptric)/&#39;,
 &#39;timestamp&#39;: &#39;20180805132731&#39;,
 &#39;original&#39;: &#39;http://skeptric.com/&#39;,
 &#39;mimetype&#39;: &#39;text/html&#39;,
 &#39;statuscode&#39;: &#39;200&#39;,
 &#39;digest&#39;: &#39;2N5QQYZFAM36CSESTGSDKRTRV7I5HEXJ&#39;,
 &#39;length&#39;: &#39;1982&#39;}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span> <span class="o">=</span> <span class="n">fetch_internet_archive_content</span><span class="p">(</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;original&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span><span class="p">[:</span><span class="mi">100</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>b&#39;&lt;!DOCTYPE html&gt;\n&lt;html lang=&#34;en&#34;&gt;\n  &lt;head&gt;&lt;script src=&#34;//archive.org/includes/analytics.js?v=cf34f82&#34;&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Wrap-it-in-a-fast-Object">Wrap it in a fast Object<a class="anchor-link" href="#Wrap-it-in-a-fast-Object"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>I have no idea whether Session is <a href="https://github.com/urllib3/urllib3/issues/1252">actually thread-safe</a>; if it's not we should have 1 session per thread. As far as I can tell the issue occurs when you have lots of hosts, so in this case it should be ok.</p>
<p>I guess we'll try it and see. Maybe long term we're better going with asyncio.
Using a Session makes things slightly faster; we can always turn it off by passing session=False.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="make_session" class="doc_header"><code>make_session</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L106" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>make_session</code>(<strong><code>pool_maxsize</code></strong>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We cache everything with joblib and make it fast.</p>
<p>Note we only use 8 threads to avoid overloading the servers.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="WaybackResult" class="doc_header"><code>class</code> <code>WaybackResult</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L117" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>WaybackResult</code>(<strong><code>params</code></strong>, <strong><code>session</code></strong>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="WaybackMachine" class="doc_header"><code>class</code> <code>WaybackMachine</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L129" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>WaybackMachine</code>(<strong><code>location</code></strong>:<code>Optional[str]</code>, <strong><code>session</code></strong>:<code>Union[bool, Session]</code>=<em><code>True</code></em>, <strong><code>threads</code></strong>:<code>int</code>=<em><code>8</code></em>, <strong><code>verbose</code></strong>=<em><code>0</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-fetching-one">Test fetching one<a class="anchor-link" href="#Test-fetching-one"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">wb</span> <span class="o">=</span> <span class="n">WaybackMachine</span><span class="p">(</span><span class="n">test_cache</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">wb</span><span class="o">.</span><span class="n">memory</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stderr output_text">
<pre>WARNING:root:[Memory(location=data/test/cache/joblib)]: Flushing completely the cache
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">items</span> <span class="o">=</span> <span class="n">wb</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>________________________________________________________________________________
[Memory] Calling __main__--tmp-ipykernel-2773891608.query_wayback_cdx...
query_wayback_cdx(&#39;skeptric.com/*&#39;, None, None, True, None, None, None, session=&lt;requests.sessions.Session object at 0x7f81dbb45100&gt;)
________________________________________________query_wayback_cdx - 0.9s, 0.0min
CPU times: user 49 ms, sys: 0 ns, total: 49 ms
Wall time: 894 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">items2</span> <span class="o">=</span> <span class="n">wb</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 22.3 ms, sys: 158 µs, total: 22.4 ms
Wall time: 19.4 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">items2</span> <span class="o">==</span> <span class="n">items</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">items3</span> <span class="o">=</span> <span class="n">wb</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;skeptric.com/*&#39;</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="n">force</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>________________________________________________________________________________
[Memory] Calling __main__--tmp-ipykernel-2773891608.query_wayback_cdx...
query_wayback_cdx(&#39;skeptric.com/*&#39;, None, None, True, None, None, None, session=&lt;requests.sessions.Session object at 0x7f81dbb45100&gt;)
________________________________________________query_wayback_cdx - 0.7s, 0.0min
CPU times: user 41.8 ms, sys: 4.34 ms, total: 46.1 ms
Wall time: 751 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">items3</span> <span class="o">==</span> <span class="n">items</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">wb</span><span class="o">.</span><span class="n">fetch_one</span><span class="p">(</span><span class="n">items</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>________________________________________________________________________________
[Memory] Calling __main__--tmp-ipykernel-2187831798.fetch_internet_archive_content...
fetch_internet_archive_content(&#39;20180805132731&#39;, &#39;http://skeptric.com/&#39;, session=&lt;requests.sessions.Session object at 0x7f81dbb45100&gt;)
___________________________________fetch_internet_archive_content - 0.2s, 0.0min
CPU times: user 5.63 ms, sys: 2.69 ms, total: 8.32 ms
Wall time: 199 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">content2</span> <span class="o">=</span> <span class="n">wb</span><span class="o">.</span><span class="n">fetch_one</span><span class="p">(</span><span class="n">items</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 1.5 ms, sys: 714 µs, total: 2.21 ms
Wall time: 1.18 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">content2</span> <span class="o">==</span> <span class="n">content</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">content3</span> <span class="o">=</span> <span class="n">wb</span><span class="o">.</span><span class="n">fetch_one</span><span class="p">(</span><span class="n">items</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">force</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>________________________________________________________________________________
[Memory] Calling __main__--tmp-ipykernel-2187831798.fetch_internet_archive_content...
fetch_internet_archive_content(&#39;20180805132731&#39;, &#39;http://skeptric.com/&#39;, session=&lt;requests.sessions.Session object at 0x7f81dbb45100&gt;)
___________________________________fetch_internet_archive_content - 0.2s, 0.0min
CPU times: user 4.82 ms, sys: 2.29 ms, total: 7.11 ms
Wall time: 189 ms
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">content3</span> <span class="o">==</span> <span class="n">content</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Test-fetching-many">Test fetching many<a class="anchor-link" href="#Test-fetching-many"> </a></h4>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">wb</span> <span class="o">=</span> <span class="n">WaybackMachine</span><span class="p">(</span><span class="n">test_cache</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">contents_16</span> <span class="o">=</span> <span class="n">wb</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">items</span><span class="p">[:</span><span class="mi">16</span><span class="p">],</span> <span class="n">threads</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 197 ms, sys: 9.6 ms, total: 207 ms
Wall time: 4.24 s
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="n">contents_16</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">content</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Common-Crawl">Common Crawl<a class="anchor-link" href="#Common-Crawl"> </a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Common Crawl is similar, but slightly different, because it is split over many indexes, roughly monthly covering 2 weeks.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Indexes">Indexes<a class="anchor-link" href="#Indexes"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Cache within a session, but not between sessions since over months the indexes change.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_cc_indexes" class="doc_header"><code>get_cc_indexes</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L163" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_cc_indexes</code>()</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">indexes</span> <span class="o">=</span> <span class="n">get_cc_indexes</span><span class="p">()</span>
<span class="n">indexes</span><span class="p">[:</span><span class="mi">3</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[{&#39;id&#39;: &#39;CC-MAIN-2021-43&#39;,
  &#39;name&#39;: &#39;October 2021 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-43/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-43-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2021-39&#39;,
  &#39;name&#39;: &#39;September 2021 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-39/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-39-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2021-31&#39;,
  &#39;name&#39;: &#39;July 2021 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-31/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2021-31-index&#39;}]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Notice the oldest ones have a different format</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">indexes</span><span class="p">[</span><span class="o">-</span><span class="mi">4</span><span class="p">:]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[{&#39;id&#39;: &#39;CC-MAIN-2013-20&#39;,
  &#39;name&#39;: &#39;Summer 2013 Index&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2013-20/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2013-20-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2012&#39;,
  &#39;name&#39;: &#39;Index of 2012 ARC files&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2012/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2012-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2009-2010&#39;,
  &#39;name&#39;: &#39;Index of 2009 - 2010 ARC files&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2009-2010/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2009-2010-index&#39;},
 {&#39;id&#39;: &#39;CC-MAIN-2008-2009&#39;,
  &#39;name&#39;: &#39;Index of 2008 - 2009 ARC files&#39;,
  &#39;timegate&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2008-2009/&#39;,
  &#39;cdx-api&#39;: &#39;https://index.commoncrawl.org/CC-MAIN-2008-2009-index&#39;}]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Parsing-the-times">Parsing the times<a class="anchor-link" href="#Parsing-the-times"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">re</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">year_week</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;^CC-MAIN-(\d</span><span class="si">{4}</span><span class="s1">-\d</span><span class="si">{2}</span><span class="s1">)$&#39;</span><span class="p">,</span> <span class="n">indexes</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s1">&#39;id&#39;</span><span class="p">])</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">year_week</span>  <span class="o">+</span> <span class="s1">&#39;-6&#39;</span><span class="p">,</span> <span class="s1">&#39;%Y-%W-%w&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>datetime.datetime(2021, 10, 30, 0, 0)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">indexes</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s1">&#39;id&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>&#39;CC-MAIN-2021-43&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Querying-CDX">Querying CDX<a class="anchor-link" href="#Querying-CDX"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p><a href="https://pywb.readthedocs.io/en/latest/manual/cdxserver_api.html">https://pywb.readthedocs.io/en/latest/manual/cdxserver_api.html</a></p>
<p><a href="http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/">http://commoncrawl.org/2015/04/announcing-the-common-crawl-index/</a></p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc_sample_api</span> <span class="o">=</span> <span class="s1">&#39;https://index.commoncrawl.org/CC-MAIN-2021-43-index&#39;</span>
<span class="k">assert</span> <span class="n">cc_sample_api</span> <span class="ow">in</span> <span class="p">[</span><span class="n">i</span><span class="p">[</span><span class="s1">&#39;cdx-api&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">indexes</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The Common Crawl CDX is <em>much slower</em> than the Wayback Machines.
It also uses the zipped CDX by default which automatically paginates.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="jsonl_loads" class="doc_header"><code>jsonl_loads</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L172" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>jsonl_loads</code>(<strong><code>jsonl</code></strong>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test_jline</span> <span class="o">=</span> <span class="sa">b</span><span class="s1">&#39;{&quot;status&quot;: &quot;200&quot;, &quot;mime&quot;: &quot;text/html&quot;}</span><span class="se">\n</span><span class="s1">{&quot;status&quot;: &quot;301&quot;, &quot;mime&quot;: &quot;-&quot;}</span><span class="se">\n</span><span class="s1">&#39;</span>
<span class="n">test_jdict</span> <span class="o">=</span> <span class="p">[{</span><span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;200&#39;</span><span class="p">,</span> <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">},</span> <span class="p">{</span><span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;301&#39;</span><span class="p">,</span> <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;-&#39;</span><span class="p">}]</span>
<span class="k">assert</span> <span class="n">jsonl_loads</span><span class="p">(</span><span class="n">test_jline</span><span class="p">)</span> <span class="o">==</span> <span class="n">test_jdict</span>
<span class="k">assert</span> <span class="n">jsonl_loads</span><span class="p">(</span><span class="n">test_jline</span><span class="o">.</span><span class="n">rstrip</span><span class="p">())</span> <span class="o">==</span> <span class="n">test_jdict</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="query_cc_cdx_num_pages" class="doc_header"><code>query_cc_cdx_num_pages</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L176" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>query_cc_cdx_num_pages</code>(<strong><code>api</code></strong>:<code>str</code>, <strong><code>url</code></strong>:<code>str</code>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="query_cc_cdx_page" class="doc_header"><code>query_cc_cdx_page</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L186" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>query_cc_cdx_page</code>(<strong><code>api</code></strong>:<code>str</code>, <strong><code>url</code></strong>:<code>str</code>, <strong><code>page</code></strong>:<code>int</code>, <strong><code>start</code></strong>:<code>Optional[str]</code>=<em><code>None</code></em>, <strong><code>end</code></strong>:<code>Optional[str]</code>=<em><code>None</code></em>, <strong><code>status_ok</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>mime</code></strong>:<code>Optional[Union[str, Iterable[str]]]</code>=<em><code>None</code></em>, <strong><code>limit</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>offset</code></strong>:<code>Optional[int]</code>=<em><code>None</code></em>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>
<p>Get references to Common Crawl Captures for url.</p>
<p>Queries the Common Crawl Capture Index (CDX) for url.</p>
<p>Filters:</p>
<ul>
<li>api: API endpoint to use (e.g. '<a href="https://index.commoncrawl.org/CC-MAIN-2021-43-index">https://index.commoncrawl.org/CC-MAIN-2021-43-index</a>')</li>
<li>start: Minimum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>end: Maximum date in format YYYYmmddHHMMSS (or any substring) inclusive</li>
<li>status_ok: Only return those with a HTTP status 200</li>
<li>mime: Filter on mimetypes, '<em>' is a wildcard (e.g. 'image/</em>')</li>
<li>limit: Only return first limit records</li>
<li>offset: Skip the first offset records, combine with limit</li>
<li>session: Session to use when making requests
Filters results between start and end inclusive, in format YYYYmmddHHMMSS or any substring
(e.g. start="202001", end="202001" will get all captures in January 2020)</li>
</ul>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Testing-pagination">Testing pagination<a class="anchor-link" href="#Testing-pagination"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test_url</span> <span class="o">=</span> <span class="s1">&#39;mn.wikipedia.org/*&#39;</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">pages</span> <span class="o">=</span> <span class="n">query_cc_cdx_num_pages</span><span class="p">(</span><span class="n">cc_sample_api</span><span class="p">,</span> <span class="n">test_url</span><span class="p">)</span>
<span class="n">pages</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>2</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">records</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">page</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">pages</span><span class="p">):</span>
    <span class="n">records</span> <span class="o">+=</span> <span class="n">query_cc_cdx_page</span><span class="p">(</span><span class="n">cc_sample_api</span><span class="p">,</span> <span class="n">test_url</span><span class="p">,</span> <span class="n">page</span><span class="o">=</span><span class="n">page</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">)</span>
<span class="nb">len</span><span class="p">(</span><span class="n">records</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_text output_error">
<pre>
<span class="ansi-red-fg">---------------------------------------------------------------------------</span>
<span class="ansi-red-fg">KeyboardInterrupt</span>                         Traceback (most recent call last)
<span class="ansi-green-fg">/tmp/ipykernel_17414/431236362.py</span> in <span class="ansi-cyan-fg">&lt;module&gt;</span>
<span class="ansi-green-intense-fg ansi-bold">      2</span> records <span class="ansi-blue-fg">=</span> <span class="ansi-blue-fg">[</span><span class="ansi-blue-fg">]</span>
<span class="ansi-green-intense-fg ansi-bold">      3</span> <span class="ansi-green-fg">for</span> page <span class="ansi-green-fg">in</span> range<span class="ansi-blue-fg">(</span>pages<span class="ansi-blue-fg">)</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">----&gt; 4</span><span class="ansi-red-fg">     </span>records <span class="ansi-blue-fg">+=</span> query_cc_cdx_page<span class="ansi-blue-fg">(</span>cc_sample_api<span class="ansi-blue-fg">,</span> test_url<span class="ansi-blue-fg">,</span> page<span class="ansi-blue-fg">=</span>page<span class="ansi-blue-fg">,</span> mime<span class="ansi-blue-fg">=</span><span class="ansi-blue-fg">&#39;text/html&#39;</span><span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">      5</span> len<span class="ansi-blue-fg">(</span>records<span class="ansi-blue-fg">)</span>

<span class="ansi-green-fg">/tmp/ipykernel_17414/1284078378.py</span> in <span class="ansi-cyan-fg">query_cc_cdx_page</span><span class="ansi-blue-fg">(api, url, page, start, end, status_ok, mime, limit, offset, session)</span>
<span class="ansi-green-intense-fg ansi-bold">     56</span> 
<span class="ansi-green-intense-fg ansi-bold">     57</span>     params <span class="ansi-blue-fg">=</span> <span class="ansi-blue-fg">{</span>k<span class="ansi-blue-fg">:</span>v <span class="ansi-green-fg">for</span> k<span class="ansi-blue-fg">,</span>v <span class="ansi-green-fg">in</span> params<span class="ansi-blue-fg">.</span>items<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span> <span class="ansi-green-fg">if</span> v<span class="ansi-blue-fg">}</span>
<span class="ansi-green-fg">---&gt; 58</span><span class="ansi-red-fg">     </span>response <span class="ansi-blue-fg">=</span> session<span class="ansi-blue-fg">.</span>get<span class="ansi-blue-fg">(</span>api<span class="ansi-blue-fg">,</span> params<span class="ansi-blue-fg">=</span>params<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">     59</span>     response<span class="ansi-blue-fg">.</span>raise_for_status<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">     60</span>     <span class="ansi-green-fg">return</span> jsonl_loads<span class="ansi-blue-fg">(</span>response<span class="ansi-blue-fg">.</span>content<span class="ansi-blue-fg">)</span>

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/requests/api.py</span> in <span class="ansi-cyan-fg">get</span><span class="ansi-blue-fg">(url, params, **kwargs)</span>
<span class="ansi-green-intense-fg ansi-bold">     73</span>     &#34;&#34;&#34;
<span class="ansi-green-intense-fg ansi-bold">     74</span> 
<span class="ansi-green-fg">---&gt; 75</span><span class="ansi-red-fg">     </span><span class="ansi-green-fg">return</span> request<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">&#39;get&#39;</span><span class="ansi-blue-fg">,</span> url<span class="ansi-blue-fg">,</span> params<span class="ansi-blue-fg">=</span>params<span class="ansi-blue-fg">,</span> <span class="ansi-blue-fg">**</span>kwargs<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">     76</span> 
<span class="ansi-green-intense-fg ansi-bold">     77</span> 

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/requests/api.py</span> in <span class="ansi-cyan-fg">request</span><span class="ansi-blue-fg">(method, url, **kwargs)</span>
<span class="ansi-green-intense-fg ansi-bold">     59</span>     <span class="ansi-red-fg"># cases, and look like a memory leak in others.</span>
<span class="ansi-green-intense-fg ansi-bold">     60</span>     <span class="ansi-green-fg">with</span> sessions<span class="ansi-blue-fg">.</span>Session<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span> <span class="ansi-green-fg">as</span> session<span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">---&gt; 61</span><span class="ansi-red-fg">         </span><span class="ansi-green-fg">return</span> session<span class="ansi-blue-fg">.</span>request<span class="ansi-blue-fg">(</span>method<span class="ansi-blue-fg">=</span>method<span class="ansi-blue-fg">,</span> url<span class="ansi-blue-fg">=</span>url<span class="ansi-blue-fg">,</span> <span class="ansi-blue-fg">**</span>kwargs<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">     62</span> 
<span class="ansi-green-intense-fg ansi-bold">     63</span> 

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/requests/sessions.py</span> in <span class="ansi-cyan-fg">request</span><span class="ansi-blue-fg">(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)</span>
<span class="ansi-green-intense-fg ansi-bold">    540</span>         }
<span class="ansi-green-intense-fg ansi-bold">    541</span>         send_kwargs<span class="ansi-blue-fg">.</span>update<span class="ansi-blue-fg">(</span>settings<span class="ansi-blue-fg">)</span>
<span class="ansi-green-fg">--&gt; 542</span><span class="ansi-red-fg">         </span>resp <span class="ansi-blue-fg">=</span> self<span class="ansi-blue-fg">.</span>send<span class="ansi-blue-fg">(</span>prep<span class="ansi-blue-fg">,</span> <span class="ansi-blue-fg">**</span>send_kwargs<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">    543</span> 
<span class="ansi-green-intense-fg ansi-bold">    544</span>         <span class="ansi-green-fg">return</span> resp

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/requests/sessions.py</span> in <span class="ansi-cyan-fg">send</span><span class="ansi-blue-fg">(self, request, **kwargs)</span>
<span class="ansi-green-intense-fg ansi-bold">    653</span> 
<span class="ansi-green-intense-fg ansi-bold">    654</span>         <span class="ansi-red-fg"># Send the request</span>
<span class="ansi-green-fg">--&gt; 655</span><span class="ansi-red-fg">         </span>r <span class="ansi-blue-fg">=</span> adapter<span class="ansi-blue-fg">.</span>send<span class="ansi-blue-fg">(</span>request<span class="ansi-blue-fg">,</span> <span class="ansi-blue-fg">**</span>kwargs<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">    656</span> 
<span class="ansi-green-intense-fg ansi-bold">    657</span>         <span class="ansi-red-fg"># Total elapsed time of the request (approximately)</span>

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/requests/adapters.py</span> in <span class="ansi-cyan-fg">send</span><span class="ansi-blue-fg">(self, request, stream, timeout, verify, cert, proxies)</span>
<span class="ansi-green-intense-fg ansi-bold">    437</span>         <span class="ansi-green-fg">try</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">    438</span>             <span class="ansi-green-fg">if</span> <span class="ansi-green-fg">not</span> chunked<span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">--&gt; 439</span><span class="ansi-red-fg">                 resp = conn.urlopen(
</span><span class="ansi-green-intense-fg ansi-bold">    440</span>                     method<span class="ansi-blue-fg">=</span>request<span class="ansi-blue-fg">.</span>method<span class="ansi-blue-fg">,</span>
<span class="ansi-green-intense-fg ansi-bold">    441</span>                     url<span class="ansi-blue-fg">=</span>url<span class="ansi-blue-fg">,</span>

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/urllib3/connectionpool.py</span> in <span class="ansi-cyan-fg">urlopen</span><span class="ansi-blue-fg">(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)</span>
<span class="ansi-green-intense-fg ansi-bold">    697</span> 
<span class="ansi-green-intense-fg ansi-bold">    698</span>             <span class="ansi-red-fg"># Make the request on the httplib connection object.</span>
<span class="ansi-green-fg">--&gt; 699</span><span class="ansi-red-fg">             httplib_response = self._make_request(
</span><span class="ansi-green-intense-fg ansi-bold">    700</span>                 conn<span class="ansi-blue-fg">,</span>
<span class="ansi-green-intense-fg ansi-bold">    701</span>                 method<span class="ansi-blue-fg">,</span>

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/urllib3/connectionpool.py</span> in <span class="ansi-cyan-fg">_make_request</span><span class="ansi-blue-fg">(self, conn, method, url, timeout, chunked, **httplib_request_kw)</span>
<span class="ansi-green-intense-fg ansi-bold">    443</span>                     <span class="ansi-red-fg"># Python 3 (including for exceptions like SystemExit).</span>
<span class="ansi-green-intense-fg ansi-bold">    444</span>                     <span class="ansi-red-fg"># Otherwise it looks like a bug in the code.</span>
<span class="ansi-green-fg">--&gt; 445</span><span class="ansi-red-fg">                     </span>six<span class="ansi-blue-fg">.</span>raise_from<span class="ansi-blue-fg">(</span>e<span class="ansi-blue-fg">,</span> <span class="ansi-green-fg">None</span><span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">    446</span>         <span class="ansi-green-fg">except</span> <span class="ansi-blue-fg">(</span>SocketTimeout<span class="ansi-blue-fg">,</span> BaseSSLError<span class="ansi-blue-fg">,</span> SocketError<span class="ansi-blue-fg">)</span> <span class="ansi-green-fg">as</span> e<span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">    447</span>             self<span class="ansi-blue-fg">.</span>_raise_timeout<span class="ansi-blue-fg">(</span>err<span class="ansi-blue-fg">=</span>e<span class="ansi-blue-fg">,</span> url<span class="ansi-blue-fg">=</span>url<span class="ansi-blue-fg">,</span> timeout_value<span class="ansi-blue-fg">=</span>read_timeout<span class="ansi-blue-fg">)</span>

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/urllib3/packages/six.py</span> in <span class="ansi-cyan-fg">raise_from</span><span class="ansi-blue-fg">(value, from_value)</span>

<span class="ansi-green-fg">~/.venv/webdata/lib/python3.8/site-packages/urllib3/connectionpool.py</span> in <span class="ansi-cyan-fg">_make_request</span><span class="ansi-blue-fg">(self, conn, method, url, timeout, chunked, **httplib_request_kw)</span>
<span class="ansi-green-intense-fg ansi-bold">    438</span>                 <span class="ansi-red-fg"># Python 3</span>
<span class="ansi-green-intense-fg ansi-bold">    439</span>                 <span class="ansi-green-fg">try</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">--&gt; 440</span><span class="ansi-red-fg">                     </span>httplib_response <span class="ansi-blue-fg">=</span> conn<span class="ansi-blue-fg">.</span>getresponse<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">    441</span>                 <span class="ansi-green-fg">except</span> BaseException <span class="ansi-green-fg">as</span> e<span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">    442</span>                     <span class="ansi-red-fg"># Remove the TypeError from the exception chain in</span>

<span class="ansi-green-fg">/usr/lib/python3.8/http/client.py</span> in <span class="ansi-cyan-fg">getresponse</span><span class="ansi-blue-fg">(self)</span>
<span class="ansi-green-intense-fg ansi-bold">   1345</span>         <span class="ansi-green-fg">try</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">   1346</span>             <span class="ansi-green-fg">try</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">-&gt; 1347</span><span class="ansi-red-fg">                 </span>response<span class="ansi-blue-fg">.</span>begin<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">   1348</span>             <span class="ansi-green-fg">except</span> ConnectionError<span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">   1349</span>                 self<span class="ansi-blue-fg">.</span>close<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span>

<span class="ansi-green-fg">/usr/lib/python3.8/http/client.py</span> in <span class="ansi-cyan-fg">begin</span><span class="ansi-blue-fg">(self)</span>
<span class="ansi-green-intense-fg ansi-bold">    305</span>         <span class="ansi-red-fg"># read until we get a non-100 response</span>
<span class="ansi-green-intense-fg ansi-bold">    306</span>         <span class="ansi-green-fg">while</span> <span class="ansi-green-fg">True</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">--&gt; 307</span><span class="ansi-red-fg">             </span>version<span class="ansi-blue-fg">,</span> status<span class="ansi-blue-fg">,</span> reason <span class="ansi-blue-fg">=</span> self<span class="ansi-blue-fg">.</span>_read_status<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">    308</span>             <span class="ansi-green-fg">if</span> status <span class="ansi-blue-fg">!=</span> CONTINUE<span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">    309</span>                 <span class="ansi-green-fg">break</span>

<span class="ansi-green-fg">/usr/lib/python3.8/http/client.py</span> in <span class="ansi-cyan-fg">_read_status</span><span class="ansi-blue-fg">(self)</span>
<span class="ansi-green-intense-fg ansi-bold">    266</span> 
<span class="ansi-green-intense-fg ansi-bold">    267</span>     <span class="ansi-green-fg">def</span> _read_status<span class="ansi-blue-fg">(</span>self<span class="ansi-blue-fg">)</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">--&gt; 268</span><span class="ansi-red-fg">         </span>line <span class="ansi-blue-fg">=</span> str<span class="ansi-blue-fg">(</span>self<span class="ansi-blue-fg">.</span>fp<span class="ansi-blue-fg">.</span>readline<span class="ansi-blue-fg">(</span>_MAXLINE <span class="ansi-blue-fg">+</span> <span class="ansi-cyan-fg">1</span><span class="ansi-blue-fg">)</span><span class="ansi-blue-fg">,</span> <span class="ansi-blue-fg">&#34;iso-8859-1&#34;</span><span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">    269</span>         <span class="ansi-green-fg">if</span> len<span class="ansi-blue-fg">(</span>line<span class="ansi-blue-fg">)</span> <span class="ansi-blue-fg">&gt;</span> _MAXLINE<span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">    270</span>             <span class="ansi-green-fg">raise</span> LineTooLong<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">&#34;status line&#34;</span><span class="ansi-blue-fg">)</span>

<span class="ansi-green-fg">/usr/lib/python3.8/socket.py</span> in <span class="ansi-cyan-fg">readinto</span><span class="ansi-blue-fg">(self, b)</span>
<span class="ansi-green-intense-fg ansi-bold">    667</span>         <span class="ansi-green-fg">while</span> <span class="ansi-green-fg">True</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">    668</span>             <span class="ansi-green-fg">try</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">--&gt; 669</span><span class="ansi-red-fg">                 </span><span class="ansi-green-fg">return</span> self<span class="ansi-blue-fg">.</span>_sock<span class="ansi-blue-fg">.</span>recv_into<span class="ansi-blue-fg">(</span>b<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">    670</span>             <span class="ansi-green-fg">except</span> timeout<span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">    671</span>                 self<span class="ansi-blue-fg">.</span>_timeout_occurred <span class="ansi-blue-fg">=</span> <span class="ansi-green-fg">True</span>

<span class="ansi-green-fg">/usr/lib/python3.8/ssl.py</span> in <span class="ansi-cyan-fg">recv_into</span><span class="ansi-blue-fg">(self, buffer, nbytes, flags)</span>
<span class="ansi-green-intense-fg ansi-bold">   1239</span>                   <span class="ansi-blue-fg">&#34;non-zero flags not allowed in calls to recv_into() on %s&#34;</span> <span class="ansi-blue-fg">%</span>
<span class="ansi-green-intense-fg ansi-bold">   1240</span>                   self.__class__)
<span class="ansi-green-fg">-&gt; 1241</span><span class="ansi-red-fg">             </span><span class="ansi-green-fg">return</span> self<span class="ansi-blue-fg">.</span>read<span class="ansi-blue-fg">(</span>nbytes<span class="ansi-blue-fg">,</span> buffer<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">   1242</span>         <span class="ansi-green-fg">else</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">   1243</span>             <span class="ansi-green-fg">return</span> super<span class="ansi-blue-fg">(</span><span class="ansi-blue-fg">)</span><span class="ansi-blue-fg">.</span>recv_into<span class="ansi-blue-fg">(</span>buffer<span class="ansi-blue-fg">,</span> nbytes<span class="ansi-blue-fg">,</span> flags<span class="ansi-blue-fg">)</span>

<span class="ansi-green-fg">/usr/lib/python3.8/ssl.py</span> in <span class="ansi-cyan-fg">read</span><span class="ansi-blue-fg">(self, len, buffer)</span>
<span class="ansi-green-intense-fg ansi-bold">   1097</span>         <span class="ansi-green-fg">try</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">   1098</span>             <span class="ansi-green-fg">if</span> buffer <span class="ansi-green-fg">is</span> <span class="ansi-green-fg">not</span> <span class="ansi-green-fg">None</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-fg">-&gt; 1099</span><span class="ansi-red-fg">                 </span><span class="ansi-green-fg">return</span> self<span class="ansi-blue-fg">.</span>_sslobj<span class="ansi-blue-fg">.</span>read<span class="ansi-blue-fg">(</span>len<span class="ansi-blue-fg">,</span> buffer<span class="ansi-blue-fg">)</span>
<span class="ansi-green-intense-fg ansi-bold">   1100</span>             <span class="ansi-green-fg">else</span><span class="ansi-blue-fg">:</span>
<span class="ansi-green-intense-fg ansi-bold">   1101</span>                 <span class="ansi-green-fg">return</span> self<span class="ansi-blue-fg">.</span>_sslobj<span class="ansi-blue-fg">.</span>read<span class="ansi-blue-fg">(</span>len<span class="ansi-blue-fg">)</span>

<span class="ansi-red-fg">KeyboardInterrupt</span>: </pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">timestamps</span> <span class="o">=</span> <span class="p">[</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">records</span><span class="p">]</span>
<span class="nb">min</span><span class="p">(</span><span class="n">timestamps</span><span class="p">),</span> <span class="nb">max</span><span class="p">(</span><span class="n">timestamps</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We should get an error querying past the last page (400: Invalid Request)</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">requests.exceptions</span> <span class="kn">import</span> <span class="n">HTTPError</span>

<span class="k">try</span><span class="p">:</span>
    <span class="n">query_cc_cdx_page</span><span class="p">(</span><span class="n">cc_sample_api</span><span class="p">,</span> <span class="n">test_url</span><span class="p">,</span> <span class="n">page</span><span class="o">=</span><span class="n">pages</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">)</span>
    <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="s1">&#39;Expected Failure&#39;</span><span class="p">)</span>
<span class="k">except</span> <span class="n">HTTPError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
    <span class="n">status</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span>
    <span class="k">if</span> <span class="n">status</span> <span class="o">!=</span> <span class="mi">400</span><span class="p">:</span>
        <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Expected 400, got </span><span class="si">{</span><span class="n">status</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Downloading">Downloading<a class="anchor-link" href="#Downloading"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="fetch_cc" class="doc_header"><code>fetch_cc</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L241" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>fetch_cc</code>(<strong><code>filename</code></strong>:<code>str</code>, <strong><code>offset</code></strong>:<code>int</code>, <strong><code>length</code></strong>:<code>int</code>, <strong><code>session</code></strong>:<code>Optional[Session]</code>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">record</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">record</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;urlkey&#39;</span><span class="p">:</span> <span class="s1">&#39;org,wikipedia,en)/?curid=3516101&#39;</span><span class="p">,</span>
 <span class="s1">&#39;timestamp&#39;</span><span class="p">:</span> <span class="s1">&#39;20211024051554&#39;</span><span class="p">,</span>
 <span class="s1">&#39;url&#39;</span><span class="p">:</span> <span class="s1">&#39;https://en.wikipedia.org/?curid=3516101&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime-detected&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;200&#39;</span><span class="p">,</span>
 <span class="s1">&#39;digest&#39;</span><span class="p">:</span> <span class="s1">&#39;TFZFWXWKS3NFJLLLXLYU6JTNZ77D3IVD&#39;</span><span class="p">,</span>
 <span class="s1">&#39;length&#39;</span><span class="p">:</span> <span class="s1">&#39;8922&#39;</span><span class="p">,</span>
 <span class="s1">&#39;offset&#39;</span><span class="p">:</span> <span class="s1">&#39;332329771&#39;</span><span class="p">,</span>
 <span class="s1">&#39;filename&#39;</span><span class="p">:</span> <span class="s1">&#39;crawl-data/CC-MAIN-2021-43/segments/1634323585911.17/warc/CC-MAIN-20211024050128-20211024080128-00689.warc.gz&#39;</span><span class="p">,</span>
 <span class="s1">&#39;languages&#39;</span><span class="p">:</span> <span class="s1">&#39;eng&#39;</span><span class="p">,</span>
 <span class="s1">&#39;encoding&#39;</span><span class="p">:</span> <span class="s1">&#39;UTF-8&#39;</span><span class="p">}</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%%time</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">fetch_cc</span><span class="p">(</span><span class="n">record</span><span class="p">[</span><span class="s1">&#39;filename&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;offset&#39;</span><span class="p">],</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;length&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">content</span><span class="p">[:</span><span class="mi">1000</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Check the digest</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">hashlib</span> <span class="kn">import</span> <span class="n">sha1</span>
<span class="kn">from</span> <span class="nn">base64</span> <span class="kn">import</span> <span class="n">b32encode</span>

<span class="k">def</span> <span class="nf">get_digest</span><span class="p">(</span><span class="n">content</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
    <span class="k">return</span> <span class="n">b32encode</span><span class="p">(</span><span class="n">sha1</span><span class="p">(</span><span class="n">content</span><span class="p">)</span><span class="o">.</span><span class="n">digest</span><span class="p">())</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;ascii&#39;</span><span class="p">)</span>

<span class="k">assert</span> <span class="n">get_digest</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="o">==</span> <span class="n">record</span><span class="p">[</span><span class="s1">&#39;digest&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Make-it-go-fast">Make it go fast<a class="anchor-link" href="#Make-it-go-fast"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="CommonCrawl" class="doc_header"><code>class</code> <code>CommonCrawl</code><a href="https://github.com/EdwardJRoss/webrefine/tree/master/webrefine/cdx.py#L267" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>CommonCrawl</code>(<strong><code>location</code></strong>:<code>Optional[str]</code>, <strong><code>session</code></strong>:<code>Union[bool, Session]</code>=<em><code>True</code></em>, <strong><code>threads</code></strong>:<code>int</code>=<em><code>64</code></em>, <strong><code>verbose</code></strong>=<em><code>0</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-querying">Test querying<a class="anchor-link" href="#Test-querying"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc</span> <span class="o">=</span> <span class="n">CommonCrawl</span><span class="p">(</span><span class="n">test_cache</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc</span><span class="o">.</span><span class="n">memory</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">assert</span> <span class="s1">&#39;CC-MAIN-2021-43&#39;</span> <span class="ow">in</span> <span class="n">cc</span><span class="o">.</span><span class="n">cdx_apis</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">,</span> <span class="s1">&#39;archive.org/s*&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>


<span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">results</span><span class="p">)</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s1">&#39;mime&#39;</span><span class="p">,</span> <span class="s1">&#39;status&#39;</span><span class="p">])[</span><span class="s1">&#39;url&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results2</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">,</span> <span class="s1">&#39;archive.org/s*&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>

<span class="k">assert</span> <span class="n">results</span> <span class="o">==</span> <span class="n">results2</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results3</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">,</span> <span class="s1">&#39;archive.org/s*&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">force</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>

<span class="k">assert</span> <span class="n">results</span> <span class="o">==</span> <span class="n">results3</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-status">Test status<a class="anchor-link" href="#Test-status"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results_ok</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">,</span> <span class="s1">&#39;archive.org/s*&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>

<span class="k">assert</span> <span class="n">results_ok</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="p">[</span><span class="s1">&#39;status&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;200&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-mimetypes">Test mimetypes<a class="anchor-link" href="#Test-mimetypes"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results_html</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">,</span> <span class="s1">&#39;archive.org/s*&#39;</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;text/html&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>

<span class="k">assert</span> <span class="n">results_html</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="p">[</span><span class="s1">&#39;mime&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;text/html&#39;</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">mimes</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;audio/x-mpegurl&#39;</span><span class="p">,</span> <span class="s1">&#39;image/x.djvu&#39;</span><span class="p">,</span> <span class="s1">&#39;text/xml&#39;</span><span class="p">]</span>
<span class="n">results_mime</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">,</span> <span class="s1">&#39;archive.org/s*&#39;</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="n">mimes</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>

<span class="k">assert</span> <span class="n">results_mime</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="p">[</span><span class="s1">&#39;mime&#39;</span><span class="p">]</span> <span class="ow">in</span> <span class="n">mimes</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">results_image</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s1">&#39;CC-MAIN-2021-43&#39;</span><span class="p">,</span> <span class="s1">&#39;archive.org/s*&#39;</span><span class="p">,</span> <span class="n">mime</span><span class="o">=</span><span class="s1">&#39;image/*&#39;</span><span class="p">,</span> <span class="n">status_ok</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>

<span class="k">assert</span> <span class="n">results_image</span> <span class="o">==</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">r</span><span class="p">[</span><span class="s1">&#39;mime&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">&#39;image/&#39;</span><span class="p">)]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Check-fetching">Check fetching<a class="anchor-link" href="#Check-fetching"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">cc</span> <span class="o">=</span> <span class="n">CommonCrawl</span><span class="p">(</span><span class="n">test_cache</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">result</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;urlkey&#39;</span><span class="p">:</span> <span class="s1">&#39;com,skeptric)/&#39;</span><span class="p">,</span>
 <span class="s1">&#39;timestamp&#39;</span><span class="p">:</span> <span class="s1">&#39;20211028110756&#39;</span><span class="p">,</span>
 <span class="s1">&#39;url&#39;</span><span class="p">:</span> <span class="s1">&#39;https://skeptric.com/&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;mime-detected&#39;</span><span class="p">:</span> <span class="s1">&#39;text/html&#39;</span><span class="p">,</span>
 <span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;200&#39;</span><span class="p">,</span>
 <span class="s1">&#39;digest&#39;</span><span class="p">:</span> <span class="s1">&#39;7RBLUZ55MD4FUPDRPVVJVTM7YDYQXRS3&#39;</span><span class="p">,</span>
 <span class="s1">&#39;length&#39;</span><span class="p">:</span> <span class="s1">&#39;107238&#39;</span><span class="p">,</span>
 <span class="s1">&#39;offset&#39;</span><span class="p">:</span> <span class="s1">&#39;650789450&#39;</span><span class="p">,</span>
 <span class="s1">&#39;filename&#39;</span><span class="p">:</span> <span class="s1">&#39;crawl-data/CC-MAIN-2021-43/segments/1634323588284.71/warc/CC-MAIN-20211028100619-20211028130619-00465.warc.gz&#39;</span><span class="p">,</span>
 <span class="s1">&#39;languages&#39;</span><span class="p">:</span> <span class="s1">&#39;eng&#39;</span><span class="p">,</span>
 <span class="s1">&#39;encoding&#39;</span><span class="p">:</span> <span class="s1">&#39;UTF-8&#39;</span><span class="p">}</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">contents</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">fetch</span><span class="p">([</span><span class="n">result</span><span class="p">])</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">cc</span><span class="o">.</span><span class="n">fetch_one</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">contents</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span>
<span class="k">assert</span> <span class="n">content</span> <span class="o">==</span> <span class="n">contents</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

</div>
 

