---

title: Dataset Utils


keywords: fastai
sidebar: home_sidebar

summary: "Utils for dataset processing"
description: "Utils for dataset processing"
nb_path: "nbs/data_datasets__utils.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/data_datasets__utils.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Download-Utils">Download Utils<a class="anchor-link" href="#Download-Utils"> </a></h1>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="download_file" class="doc_header"><code>download_file</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/data/datasets/utils.py#L23" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>download_file</code>(<strong><code>directory</code></strong>:<code>str</code>, <strong><code>source_url</code></strong>:<code>str</code>, <strong><code>decompress</code></strong>:<code>bool</code>=<em><code>False</code></em>)</p>
</blockquote>
<p>Download data from source_ulr inside directory.</p>
<h2 id="Parameters">Parameters<a class="anchor-link" href="#Parameters"> </a></h2><p>directory: str, Path
    Custom directory where data will be downloaded.
source_url: str
    URL where data is hosted.
decompress: bool
    Wheter decompress downloaded file. Default False.</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="Info" class="doc_header"><code>class</code> <code>Info</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/data/datasets/utils.py#L75" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>Info</code>(<strong><code>groups</code></strong>:<code>Tuple</code>[<code>str</code>], <strong><code>class_groups</code></strong>:<code>Tuple</code>[<code>dataclass</code>])</p>
</blockquote>
<p>Info Dataclass of datasets.
Args:
    groups (Tuple): Tuple of str groups
    class_groups (Tuple): Tuple of dataclasses.</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="TimeSeriesDataclass" class="doc_header"><code>class</code> <code>TimeSeriesDataclass</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/data/datasets/utils.py#L106" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>TimeSeriesDataclass</code>(<strong><code>S</code></strong>:<code>DataFrame</code>, <strong><code>X</code></strong>:<code>DataFrame</code>, <strong><code>Y</code></strong>:<code>DataFrame</code>, <strong><code>idx_categorical_static</code></strong>:<code>Optional</code>[<code>List</code>[<code>T</code>]]=<em><code>None</code></em>, <strong><code>group</code></strong>:<code>Union</code>[<code>str</code>, <code>List</code>[<code>str</code>]]=<em><code>None</code></em>)</p>
</blockquote>
<p>Args:
    S (pd.DataFrame): DataFrame of static features of shape
        (n_time_series, n_features).
    X (pd.DataFrame): DataFrame of exogenous variables of shape
        (sum n_periods_i for i=1..n_time_series, n_exogenous).
    Y (pd.DataFrame): DataFrame of target variable of shape
        (sum n_periods_i for i=1..n_time_series, 1).
    idx_categorical_static (list, optional): List of categorical indexes
        of S.
    group (str, optional): Group name if applies.
        Example: 'Yearly'</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="US-Calendar-Utils">US Calendar Utils<a class="anchor-link" href="#US-Calendar-Utils"> </a></h1><ul>
<li><a href="https://www.opm.gov/policy-data-oversight/pay-leave/federal-holidays/">US Official Federal definitions</a></li>
<li><a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html">Pandas calendar utility</a></li>
</ul>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_holiday_dates" class="doc_header"><code>get_holiday_dates</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/data/datasets/utils.py#L151" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_holiday_dates</code>(<strong><code>holiday</code></strong>, <strong><code>dates</code></strong>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="holiday_kernel" class="doc_header"><code>holiday_kernel</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/data/datasets/utils.py#L158" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>holiday_kernel</code>(<strong><code>holiday</code></strong>, <strong><code>dates</code></strong>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="create_calendar_variables" class="doc_header"><code>create_calendar_variables</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/data/datasets/utils.py#L172" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>create_calendar_variables</code>(<strong><code>X_df</code></strong>:<code>DataFrame</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="create_us_holiday_distance_variables" class="doc_header"><code>create_us_holiday_distance_variables</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/data/datasets/utils.py#L178" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>create_us_holiday_distance_variables</code>(<strong><code>X_df</code></strong>:<code>DataFrame</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>

<span class="n">ds</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">date_range</span><span class="p">(</span><span class="n">start</span><span class="o">=</span><span class="s1">&#39;2010-01-01&#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;2012-12-31&#39;</span><span class="p">)</span>
<span class="n">holiday_dist_new_year</span> <span class="o">=</span> <span class="n">holiday_kernel</span><span class="p">(</span><span class="n">holiday</span><span class="o">=</span><span class="s1">&#39;new_year&#39;</span><span class="p">,</span> <span class="n">dates</span><span class="o">=</span><span class="n">ds</span><span class="p">)</span>
<span class="n">holiday_dist_independence</span> <span class="o">=</span> <span class="n">holiday_kernel</span><span class="p">(</span><span class="n">holiday</span><span class="o">=</span><span class="s1">&#39;independence&#39;</span><span class="p">,</span> <span class="n">dates</span><span class="o">=</span><span class="n">ds</span><span class="p">)</span>

<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">4</span><span class="p">))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="n">holiday_dist_new_year</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">&#39;new_year&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="n">holiday_dist_independence</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">&#39;independence&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">ds</span><span class="p">)))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s1">&#39;Holiday Kernels&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_png output_subarea ">
<img src="data:image/png;base64,[encoded data, 42424 characters]
"
>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

</div>
 

