********
Examples
********

.. _the_mnist_example:

MNIST: Handwriting Recognition
==============================

.. include:: mnist-example.rst

.. _continue_examples:

.. note::

	Clever readers will notice that each training epoch only used 320 training
	samples. But MNIST provides 60,000 training samples total, so what gives?
	Simple: lots of us our running this code on consumer hardware; in fact, I'm
	running this example on my tiny ultrabook on an Intel Core m7 CPU. As you'll
	see in `Under the Hood`_, I truncate the training process to only train
	on 10 batches of 32 samples each, just to make the training loop finish in
	a reasonable amount of time. It's not cheating: you still get 90% accuracy!
	But if you have awesome hardware, or just want to see how good your accuracy
	can get, then by all means read on and we'll show you how to modify that.

Under the Hood
--------------

So what exactly is going on here? Let's take a look at the MNIST example
specification file:

.. literalinclude:: ../../examples/mnist.yml
   :language: yaml
   :caption: mnist.yml

This is just plain, old `YAML <yaml.org>`_, a markup language meant to be easy
for humans to interpret (for a good overview of YAML language features, look at
the `Ansible overview <https://docs.ansible.com/ansible/YAMLSyntax.html>`_).

The first block of the file is the ``train`` section, which describes the data
that your model uses to train. Here, we just point it to the MNIST data online,
denoted by the ``images`` and ``labels`` tags. Cool, data!

The only other part you need is the model itself. What's in a Kur model? Just a
nice, gentle list of things you want your deep learning model to do. Let's break
it down:

- We have an ``input`` called ``images`` (yep, it's the same ``images`` from our
  ``train`` section).
- We pass the input to a ``convolution`` layer.
- We add a regularized linear unit ("ReLU") activation.
- We collapse (``flatten``) the high-dimensional output of a convolution into a
  nice, flat, 1-dimensional shape appropriate for sending into the
  fully-connected layers.
- We add a fully-connected (``dense``) layer with 10 outputs.
- We add a softmax activation (appropriate for classification tasks like MNIST),
  and mark it as producing labels (``name: labels``).

And that's it! It's pretty naïve: one convolution + activation +
fully-connected + activation.  But it works: we got 90% accuracy after only
showing it a small subset of the training set.

But let's think about make it more complicated. What if we want two
convolutional layers instead? Easy! Just add another ``convolution`` section to
the model.  We'll also add in another non-linearity (ReLU activation) between
the two convolutions.

.. code-block:: yaml

	model:
	  - input: images
	  - convolution:
		  kernels: 64
		  size: [2, 2]
	  - activation: relu
	  - convolution:
		  kernels: 64
		  size: [2, 2]
	  - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

We can also add more dense (fully-connected) layers. You probably want them
separated by activation layers, too. So if we add a 32-node fully-connected
layer to our model, it now looks like this:

.. code-block:: yaml

	model:
	  - input: images
	  - convolution:
		  kernels: 64
		  size: [2, 2]
	  - activation: relu
	  - convolution:
		  kernels: 64
		  size: [2, 2]
	  - activation: relu
	  - flatten:
	  - dense: 32
	  - activation: relu
	  - dense: 10
	  - activation: softmax
		name: labels

Let's give it a try! Save your changes, a just run the same ``kur train
mnist.yml`` and ``kur evaluate mnist.yml`` commands from before.

.. note::

	A more complex model will likely need more data. So be sure to look at the
	tip in `More Advanced Things`_ to train on more of the data set.

More Advanced Things
--------------------

The one line in the ``mnist.yml`` specification that we didn't cover is the
``include: mnist-defaults.yml`` line. This is just a convenient way for us to
separate out the default behavior of the MNIST example.

If you tweak this file, probably the big thing you want to remove is the
``num_batches: 10`` line, which is what limits training to just the first 10
batches every epoch. Just delete the line or comment it out, and Kur will train
on the whole dataset.

.. _cifar_10:

CIFAR-10: Image Classification
==============================

Okay, MNIST was pretty cool, but Kur can do much, much more. Imagine if you
wanted to have an arbitrary number of convolution layers. Imagine if each
convolution should have a different number of kernels. Imagine if you truly
want *flexibility*. You've come to the right place.

Flexibility: Variables
----------------------

Kur uses an *engine* to determine how do variable substitution. `Jinja2
<jinja.pocoo.org>`_ is the default templating engine, and it is very powerful
and extensible. Let's see how to use it!

Let's look at the `CIFAR-10 <https://www.cs.toronto.edu/~kriz/cifar.html>`_
dataset. This is a image classification dataset of small 32 by 32 pixel color
(RGB) images, each with one of ten classes (airplane, automobile, bird, cat,
deer, dog, frog, horse, ship, truck). You might decide to start with a very
similar model to the MNIST example:

.. code-block:: yaml

	model:
	  - input: images
	  - convolution:
		  kernels: 64
		  size: [2, 2]
	  - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

We will start with a simple modification: let's make the convolution `size` a
variable, so we can easily change it later. We can do it like this:

.. code-block:: yaml
   :emphasize-lines: 1-3,9

	settings:
	  cnn:
	    size: [2, 2]

	model:
	  - input: images
	  - convolution:
		  kernels: 64
		  size: "{{ cnn.size }}"
	  - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

Okay, what just happened? First, we added a ``settings:`` section. This section
is the appropriate place to declare variables, settings, and hyperparameters
that will be used by the model (or for training, evaluation, etc.). We declared
a variable named ``cnn`` with a nested ``size`` variable. In Python, this would
be equivalent to a dictionary: ``{"cnn": {"size": [2, 2]}}``.

Then we used the variable in the model's convolution layer: ``size: "{{
cnn.size }}"``.  This is standard Jinja2 grammar. The double-brackets indicate
that variable substitution should take place (without the brackets, we would
accidently assign ``size`` to the literal string "cnn.size", which doesn't make
sense). The variable we grab is ``cnn.size``, corresponding to the variables we
added in the ``settings`` section.

Cool! So we can use variables now. But how does that help us? It seems like we
just made it more complicated. Well, let's imagine if we added another
convolution layer. We already know how to add extra convolutions by just adding
another `convolution` block (and usually you want another `activation: relu`
layer, too). So this would look like:

.. code-block:: yaml
   :emphasize-lines: 11-14

	settings:
	  cnn:
	    size: [2, 2]

	model:
	  - input: images
	  - convolution:
		  kernels: 64
		  size: "{{ cnn.size }}"
	  - activation: relu
	  - convolution:
		  kernels: 64
		  size: "{{ cnn.size }}"
	  - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

Ah! So now we can see why variablizing the convolution size was nice: if we
want to play with a model that uses different size kernels, we only need to
edit one line instead of two.

But there are still two problems we might encounter:

- What if we wanted to try out lots of models with different numbers of
  convolutions?
- What if we wanted to use *different* ``size`` or ``kernel`` values in each
  convolution?

Kur can do it!

Flexibility: Loops
------------------

Let's address the first problem: what if we want to make the number of
convolutions?  Kur supports many "meta-layers" that it calls "operators." A
very simple operator is the classic `"for" loop
<https://en.wikipedia.org/wiki/For_loop>`_. This allows us to add many
convolution + activation layers at once. It looks like this:

.. code-block:: yaml
   :emphasize-lines: 7-13

	settings:
	  cnn:
	    size: [2, 2]

	model:
	  - input: images
	  - for:
	      range: 2
	      iterate:
	        - convolution:
	            kernels: 64
	            size: "{{ cnn.size }}"
	        - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

This is equivalent to the version without the "for" loop. The ``for:`` loop
tells us to do everything in the ``iterate:`` section twice. (Why twice?
Because ``range: 2``.) And of course, we can variabilize the number of
iterations like this:

.. code-block:: yaml
   :emphasize-lines: 4,9

	settings:
	  cnn:
	    size: [2, 2]
	    layers: 2

	model:
	  - input: images
	  - for:
	      range: "{{ cnn.layers }}"
	      iterate:
	        - convolution:
	            kernels: 64
	            size: "{{ cnn.size }}"
	        - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

Think about this for a minute. Does it make sense? It should. The model looks
like this:

- An ``input`` layer of images.
- A number of ``convolution`` and ``activation`` layers. How many?
  ``cnn.layers``, so 2.
- The rest of the model is as expected: a dense operation followed by an
  activation.

Flexibility: Variable-length Loops
----------------------------------

So we solved the problem of allowing for a variable number of convolutions. But
what if each convolution should use a different number of kernels (or sizes,
etc.)?  Well, Kur can happily handle this, too. In fact, the ``for:`` loop
already does most of the work. Every ``for:`` loop creates its own "local"
variable to let you know which iteration it is on. The default name for this
variable is ``index``. So if we want to use a different number of kernels for
each convolution, we can do this:

.. code-block:: yaml
   :emphasize-lines: 4,13

	settings:
	  cnn:
	    size: [2, 2]
	    kernels: [64, 32]
	    layers: 2

	model:
	  - input: images
	  - for:
	      range: "{{ cnn.layers }}"
	      iterate:
	        - convolution:
	            kernels: "{{ cnn.kernels[index] }}"
	            size: "{{ cnn.size }}"
	        - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

Again, this is just Jinja2 substitution: we are asking for the ``index``-th
element of the ``cnn.kernels`` list. Each iteration of the ``for:`` loop
therefore grabs a different value for ``kernels:``. Cool, huh?

But we can do one better.

Flexibility: Filters
--------------------

The annoying thing about our current model is that nothing forces the ``layers``
value to be the same as the length of the ``kernels`` variable. If you make
really long (like, length seventeen) but leave ``layers`` at two, you probably
made a mistake. (Why did you put in seventeen layers but then only use the first
two in the loop?) What you really want is to make sure that ``layers`` is set to
the length of the ``kernels`` list. Or put another way, you want add as many
convolutions as you have kernels in the list.

Jinja2 supports a concept called "filters," which are basically functions that
you can apply to objects. You can even define your own filters. But what we
want right now is a way to get the length of a list. It's easy and it looks
like this:

.. code-block:: yaml
   :emphasize-lines: 9

	settings:
	  cnn:
	    size: [2, 2]
	    kernels: [64, 32]

	model:
	  - input: images
	  - for:
	      range: "{{ cnn.kernels|length }}"
	      iterate:
	        - convolution:
	            kernels: "{{ cnn.kernels[index] }}"
	            size: "{{ cnn.size }}"
	        - activation: relu
	  - flatten:
	  - dense: 10
	  - activation: softmax
		name: labels

You'll notice that the ``layers`` variable is gone, and we have this funky
``|length`` thing in the "for" loop's ``range``. This is standard Jinja2: the
``length`` filter returns the length of a list. So now we are asking the "for"
loop to iterate as many times as we have another kernel size.

This is really cool if you think about it. You want to add another convolution
to the network? *All you do is add it's size to the* ``kernels`` *list*. And
look!  You're model is now more general, more reuseable. You could have used
the same model for MNIST! Or CIFAR! Or many different applications.

This is the heart of the **Kur philosophy: you should describe your model once
and simply.** The specification *describes** your model: a bunch of
convolutions and then a fully-connected layer. You can specify the details (how
many convolutions, their parameters, etc.) elsewhere. The model should stay
elegant.

.. note::

	Of course, it isn't always easy to write reusable models. And the learning
	curve can get in the way. When we say that models should be "simple," we
	don't mean that you don't need to think about it. We mean that it should be
	simple to use, simple to modify, and simple to share. A more general model
	is elegant: making changes to it is easy (you only modify the settings). And
	this makes it easier to reuse in new contexts or to share with the
	community. Simplicity is power.

Actually Training a CIFAR-10 Model
----------------------------------

Great, we now have a simple, but powerful and general model. Let's train it. As
before, you'll need to ``cd examples`` first.

.. code-block:: bash

	$ kur train cifar.yml

Again, evaluation is just as simple:

.. code-block:: bash

	$ kur evaluate cifar.yml

Advanced Features
-----------------

The ``cifar.yml`` specification file is more complicated than the MNIST one,
mostly to expose you to some more knobs you can tweak. For example, you'll see
these lines in the ``train`` section:

.. code-block:: yaml

	provider:
	  batch_size: 32
	  num_batches: 2

As in the MNIST case, ``num_batches`` tells Kur to only train on that many
batches of data each epoch (mostly so that if you don't have a nice GPU, the
example still finishes in a reasonable amount of time). The ``batch_size`` value
indicates the number of training samples that should be used in each batch.

.. _using_binary_logger:

The ``train`` section also has a ``log: cifar-log`` line. This tells Kur to
save a log file to ``cifar-log`` (in the current working directory). This log
contains lots of interesting information about current training loss, batch
loss, and the number of epochs. By default, they are binary-encoded files, but
you can load them using the Kur API (in Python 3):

.. code-block:: python

	from kur.loggers import BinaryLogger
	data = BinaryLogger.load_column(LOG_PATH, STATISTIC)

where ``LOG_PATH`` is the path to the log file (e.g., ``cifar-log``) and
``STATISTIC`` is one of the logged statistics. ``data`` will be a `Numpy
<http://www.numpy.org/>`_ array. To find available statistics, just list the
available files in the ``LOG_PATH``, like this:

.. code-block:: bash

	$ ls cifar-log
	training_loss_labels
	training_loss_total
	validation_loss_labels
	validation_loss_total

For an example of using this log data, see the :doc:`tutorial`.

Another difference from the MNIST examples is that there are more files
referring to weights in the CIFAR specification. For example, in the
``validate`` section there is:

.. code-block:: yaml

	weights: cifar.best.valid.w

This tells Kur to save the best models weights (corresponding to the lowest
loss on the *validation* set) to ``cifar.best.valid.w``. Similarly, in the
``train`` section there is this:

.. code-block:: yaml

  	weights:
	  initial: cifar.best.valid.w
	  save_best: cifar.best.train.w
	  last: cifar.last.w

The ``initial`` key tells Kur to try and load ``cifar.best.valid.w`` (the best
weights with respect to the *validation* loss) at the beginning of training. If
this file doesn't exist, nothing happens. This means that if you run the
training cycle many times (with many calls to ``kur train cifar.yml``), you
always "restart" from the best model weights.

We are also saving the best weights (with respect to the *training* loss) to
``cifar.best.train.w``.  The most recent weights are saved to ``cifar.last.w``. 

.. note::

	The weights depend on the model architecture. Say you you train CIFAR and
	produce ``cifar.best.valid.w``. Then you tweak the model in the
	specification file. If you try to resume training (``kur train
	cifar.yml``), Kur will try to load ``cifar.best.valid.w``. But the weights
	many not fit the new architecture!
	
	To be safe, you should always delete (or backup) your weight files before
	trying to train a fresh, tweaked model. In a production environment, you
	probably want to have different sub-directories for each variation/tweak to
	the model so that you never run into this problem.

The CIFAR-10 example also explicitly specifies an optimizer in the ``train``
section:

.. code-block:: yaml

	optimizer:
	  name: adam
	  learning_rate: 0.001

The optimizer function is set in the ``name`` field and all other parameters
(such as ``learning_rate``) are defined in the other fields. You can safely
change the optimizer without breaking backwards-compatibility with older weight
files.
