---

title: Optimizers

keywords: fastai
sidebar: home_sidebar

summary: "The basics for building and training models are contained in this module."
description: "The basics for building and training models are contained in this module."
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: notebooks/06_optimizers.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%</span><span class="k">load_ext</span> autoreload
<span class="o">%</span><span class="k">autoreload</span> 2
<span class="o">%</span><span class="k">matplotlib</span> inline
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># Used in notebook but not needed in package.</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span>
<span class="kn">from</span> <span class="nn">torch.utils.data</span> <span class="k">import</span> <span class="n">Dataset</span><span class="p">,</span> <span class="n">DataLoader</span>

<span class="kn">from</span> <span class="nn">htools</span> <span class="k">import</span> <span class="n">assert_raises</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Optimizers">Optimizers<a class="anchor-link" href="#Optimizers"> </a></h2><p>Optimizers like Adam or RMSProp can contain multiple "parameter groups", each with a different learning rate. (Other hyperparameters can vary as well, but we ignore that for now.) The functions below allow us to get a new optimizer or update an existing one. It allows us to easily use differential learning rate, but that is not required: it can also use the same LR for each parameter group.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="variable_lr_optimizer" class="doc_header"><code>variable_lr_optimizer</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/optimizers.py#L16" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>variable_lr_optimizer</code>(<strong><code>model</code></strong>, <strong><code>lr</code></strong>=<em><code>0.003</code></em>, <strong><code>lr_mult</code></strong>=<em><code>1.0</code></em>, <strong><code>optimizer</code></strong>=<em><code>'Adam'</code></em>, <strong><code>eps</code></strong>=<em><code>0.001</code></em>, <strong>**<code>kwargs</code></strong>)</p>
</blockquote>

<pre><code>Get an optimizer that uses different learning rates for different layer
groups. Additional keyword arguments can be used to alter momentum and/or
weight decay, for example, but for the sake of simplicity these values
will be the same across layer groups.

Parameters
-----------
model: nn.Module
    A model object. If you intend to use differential learning rates,
    the model must have an attribute `groups` containing a ModuleList of
    layer groups in the form of Sequential objects. The number of layer
    groups must match the number of learning rates passed in.
lr: float, Iterable[float]
    A number of list of numbers containing the learning rates to use for
    each layer group. There should generally be one LR for each layer group
    in the model. If fewer LR's are provided, lr_mult will be used to
    compute additional LRs. See `update_optimizer` for details.
lr_mult: float
    If you pass in fewer LRs than layer groups, `lr_mult` will be used to
    compute additional learning rates from the one that was passed in.
optimizer: torch optimizer
    The Torch optimizer to be created (Adam by default).
eps: float
    Hyperparameter used by optimizer. The default of 1e-8 can lead to
    exploding gradients, so we typically override this.

Examples
---------
optim = variable_lr_optimizer(model, lrs=[3e-3, 3e-2, 1e-1])</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="update_optimizer" class="doc_header"><code>update_optimizer</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/optimizers.py#L57" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>update_optimizer</code>(<strong><code>optim</code></strong>, <strong><code>lrs</code></strong>, <strong><code>lr_mult</code></strong>=<em><code>1.0</code></em>)</p>
</blockquote>

<pre><code>Pass in 1 or more learning rates, 1 for each layer group, and update the
optimizer accordingly. The optimizer is updated in place so nothing is
returned.

Parameters
----------
optim: torch.optim
    Optimizer object.
lrs: float, Iterable[float]
    One or more learning rates. If using multiple values, usually the
    earlier values will be smaller and later values will be larger. This
    can be achieved by passing in a list of LRs that is the same length as
    the number of layer groups in the optimizer, or by passing in a single
    LR and a value for lr_mult.
lr_mult: float
    If you pass in fewer LRs than layer groups, `lr_mult` will be used to
    compute additional learning rates from the one that was passed in.

Returns
-------
None

Examples
--------
If optim has 3 layer groups, this will result in LRs of [3e-5, 3e-4, 3e-3]
in that order:
update_optimizer(optim, lrs=3e-3, lr_mult=0.1)

Again, optim has 3 layer groups. We leave the default lr_mult of 1.0 so
each LR will be 3e-3.
update_optimizer(optim, lrs=3e-3)

Again, optim has 3 layer groups. 3 LRs are passed in so lr_mult is unused.
update_optimizer(optim, lrs=[1e-3, 1e-3, 3e-3])</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

</div>
 

