---

title: dataloader


keywords: fastai
sidebar: home_sidebar

summary: "Now deprecated, please use datagenerator instead."
description: "Now deprecated, please use datagenerator instead."
nb_path: "nbs/04_dataloader.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/04_dataloader.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_basename" class="doc_header"><code>get_basename</code><a href="https://github.com/aniketmaurya/chitra/tree/master/chitra/dataloader.py#L22" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_basename</code>(<strong><code>path</code></strong>:<code>&lt;dtype: 'string'&gt;</code>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="show_batch" class="doc_header"><code>show_batch</code><a href="https://github.com/aniketmaurya/chitra/tree/master/chitra/dataloader.py#L27" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>show_batch</code>(<strong><code>clf</code></strong>, <strong><code>limit</code></strong>:<code>int</code>, <strong><code>figsize</code></strong>:<code>tuple</code>=<em><code>(10, 10)</code></em>)</p>
</blockquote>
<p>Visualize image and labels</p>
<p><a href="https://www.tensorflow.org/tutorials/load_data/images#load_using_keraspreprocessing">https://www.tensorflow.org/tutorials/load_data/images#load_using_keraspreprocessing</a></p>
<p>Args:
    data: tf.data.Dataset containing image, label
    limit: number of images to display
    figsize: size of visualization
Returns:
    Displays images and labels</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="CLF-DLoader">CLF DLoader<a class="anchor-link" href="#CLF-DLoader"> </a></h1><p><code>DataLoader class for loading dataset for image classification tasks.</code></p>
<h2 id="clf.load_from_folder-(TODOs)">clf.load_from_folder (TODOs)<a class="anchor-link" href="#clf.load_from_folder-(TODOs)"> </a></h2><ol>
<li><code>__len__</code> method impl</li>
<li><code>image augmentation</code> impl</li>
</ol>
<h2 id="folder-structure">folder structure<a class="anchor-link" href="#folder-structure"> </a></h2><blockquote><pre>/root
    /class1_folder
        /img0.jpg img1.jpg img2.jpg ....
    /class2_folder
        /img0.jpg...</pre>
</blockquote>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="Clf" class="doc_header"><code>class</code> <code>Clf</code><a href="https://github.com/aniketmaurya/chitra/tree/master/chitra/dataloader.py#L60" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>Clf</code>()</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Detect">Detect<a class="anchor-link" href="#Detect"> </a></h1><h2 id="from_xml">from_xml<a class="anchor-link" href="#from_xml"> </a></h2><h3 id="Steps:">Steps:<a class="anchor-link" href="#Steps:"> </a></h3>
<pre><code>. list annotations
. read and parse annotations
. read images
. return images and annotations
</code></pre>
<h3 id="folder-structure">folder structure<a class="anchor-link" href="#folder-structure"> </a></h3><blockquote><pre>/root
    /image_folder
    /annotation_folder</pre>
</blockquote>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">class</span> <span class="nc">Detect</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>

    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">CLASS_NAMES</span> <span class="o">=</span> <span class="kc">None</span>

    <span class="k">def</span> <span class="nf">from_xml</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">pathlib</span><span class="o">.</span><span class="n">Path</span><span class="p">]):</span>
        <span class="sd">&quot;&quot;&quot;Load dataset from given path.</span>
<span class="sd">        Args:</span>
<span class="sd">            path: string, path of folder containing dataset.</span>
<span class="sd">        Returns: image, label -&gt; tf.data.Dataset prefetched with tf.data.AUTOTUNE</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">pathlib</span><span class="o">.</span><span class="n">Path</span><span class="p">))</span>
        <span class="n">path</span> <span class="o">=</span> <span class="n">pathlib</span><span class="o">.</span><span class="n">Path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
        <span class="n">remove_dsstore</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>

        <span class="n">list_folders</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">Dataset</span><span class="o">.</span><span class="n">list_files</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">path</span> <span class="o">/</span> <span class="s1">&#39;*&#39;</span><span class="p">))</span>
        <span class="n">list_images</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_image_list</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">path</span><span class="p">))</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">CLASS_NAMES</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">get_basename</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="o">.</span><span class="n">numpy</span><span class="p">()</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">list_folders</span><span class="p">)</span>

        <span class="n">data</span> <span class="o">=</span> <span class="n">list_images</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_process_path</span><span class="p">,</span> <span class="n">num_parallel_calls</span><span class="o">=</span><span class="n">AUTOTUNE</span><span class="p">)</span>
        <span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">prefetch</span><span class="p">(</span><span class="n">AUTOTUNE</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">data</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

</div>
 

