--- title: Layers keywords: fastai sidebar: home_sidebar summary: "Custom activations, layers, and layer blocks are contained in this module." ---

    %load_ext autoreload
    %autoreload 2
    %matplotlib inline

    # Used in notebook but not needed in package.
    import matplotlib.pyplot as plt

## Activations¶

## `class` `GRelu`[source]

> `GRelu`(`leak`=`0.0`, `max`=`inf`, `sub`=`0.0`) :: `Module`

Generic ReLU.

## `class` `Mish`[source]

> `Mish`() :: `Module`

OOP form of mish activation.

Mish: A Self Regularized Non-Monotonic Neural Activation Function https://arxiv.org/pdf/1908.08681v1.pdf

#### `mish`[source]

> `mish`(`x`)

Functional form of mish activation.

Mish: A Self Regularized Non-Monotonic Neural Activation Function https://arxiv.org/pdf/1908.08681v1.pdf

## Parameters¶

x: torch.Tensor[float] Input tensor.

## Returns¶

torch.Tensor[float]: Tensor of same shape as input x.

    def plot_activations(z, a, mode='scatter', **kwargs):
        """Plot an input tensor and its corresponding activations.  Both tensors
        will be flattened for plotting.

        Parameters
        ----------
        z: tf.Tensor
            Tensor containing values to plot on the x axis (we can often think of
            this as the output of a linear layer, where z=f(x) and a=mish(z)).
        a: tf.Tensor
            Tensor containing values to plot on y axis.
        mode: str
            'scatter' for scatter plot or 'plot' for line plot.
        kwargs: Values to be passed to the matplotlib plotting function, such as
            's' when in 'scatter' mode or 'lw' in 'plot' mode.

        Returns
        -------
        None
        """
        plt_func = getattr(plt, mode)
        kwargs = kwargs or {}
        if mode == 'scatter' and not kwargs:
            kwargs = {'s': .75}
        plt_func(z.numpy().flatten(), a.numpy().flatten(), **kwargs)
        plt.axvline(0, lw=.5, alpha=.5)
        plt.axhline(0, lw=.5, alpha=.5)
        plt.show()

    x = torch.arange(-5, 5, .05)
    a = mish(x)

    plot_activations(x, a, 'plot')

## Layer Blocks¶

## `class` `ConvBlock`[source]

> `ConvBlock`(`c_in`, `c_out`, `kernel_size`=`3`, `norm`=`True`, `activation`=`GReLU(leak=0.1, max=6.0, sub=0.4)`, **`kwargs`) :: `Module`

Create a convolutional block optionally followed by a batch norm layer.

    ConvBlock(3, 5, norm=False)

    ConvBlock(
      (block): Sequential(
        (0): Conv2d(3, 5, kernel_size=(3, 3), stride=(1, 1))
        (1): GReLU(leak=0.1, max=6.0, sub=0.4)
      )
    )

## `class` `ResBlock`[source]

> `ResBlock`(`c_in`, `activation`=`GReLU(leak=0.1, max=6.0, sub=0.4)`, `f`=`3`, `stride`=`1`, `pad`=`1`, `skip_size`=`2`, `norm`=`True`) :: `Module`

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

    import torch.nn as nn
    import torch.nn.functional as F

    class Model(nn.Module):
        def __init__(self):
            super(Model, self).__init__()
            self.conv1 = nn.Conv2d(1, 20, 5)
            self.conv2 = nn.Conv2d(20, 20, 5)

        def forward(self, x):
            x = F.relu(self.conv1(x))
            return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:`to`, etc.

    ResBlock(4)

    ResBlock(
      (layers): ModuleList(
        (0): ConvBlock(
          (block): Sequential(
            (0): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
            (1): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (1): ConvBlock(
          (block): Sequential(
            (0): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
            (1): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
      )
      (activation): GReLU(leak=0.1, max=6.0, sub=0.4)
    )

    ResBlock(4, norm=False)

    ResBlock(
      (layers): ModuleList(
        (0): ConvBlock(
          (block): Sequential(
            (0): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          )
        )
        (1): ConvBlock(
          (block): Sequential(
            (0): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          )
        )
      )
      (activation): GReLU(leak=0.1, max=6.0, sub=0.4)
    )