Autoencoders
==============================================

An autoencoder is a feed forward neural network which is trained to map
its input to itself via the representation formed by the hidden units. The optimisation
problem for input data :math:`\vec{x}_1,\dots,\vec{x}_N` is stated as:

.. math ::
	\min_{\theta} \frac 1 N \sum_{i=1}^N (\vec x_i - f_{\theta}(\vec x_i))^2 \enspace .

Of course, without any constraints this is a simple task as the model
will just try to learn the identity. It becomes a bit more challenging
when we restrict the size of the intermediate representation (i.e.,
the number of hidden units). An image with several hundred input
points can not be squeezed in a representation of a few hidden
neurons. Thus, it is assumed that this intermediate representation
learns something meaningful about the problem.  Of course, using this
simple technique without any additional regularization
only works if the number of hidden neurons is smaller than
the number of dimensions of the image.

As a dataset for this tutorial, we use a subset of the MNIST dataset which needs to
be unzipped first. It can be found in ``examples/Supervised/data/mnist_subset.zip``.
The full example program can be found in  :doxy:`AutoEncoderTutorial.cpp`.

The following includes are needed for this tutorial::

..sharkcode<Unsupervised/AutoEncoderTutorial.tpp,includes>

Training autoencoders
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Training an autoencoder is straight forward in shark. We just setup two neural networks,
one for encoding and one for decoding. Those are then concatenated to form one autoencoder
network::

..sharkcode<Unsupervised/AutoEncoderTutorial.tpp,model_creation>
Note that for the deeper layers we use the shape of the output of the 
previous layers (in this case just a 1-d shape with the number of neurons) to
specify the shape of the input of the next layer.

Next, we set up the objective function. This should by now be looking
quite familiar.  We set up an :doxy:`ErrorFunction` with the model and
the squared loss. Here we enable minibatch training to speed up 
the training progress.
We create the :doxy:`LabeledData` object from the
input data by setting the labels to be the same as the inputs. Finally
we add  two-norm regularisation by creating an instance of the
:doxy:`TwoNormRegularizer` class::

..sharkcode<Unsupervised/AutoEncoderTutorial.tpp,objective>

Lastly, we optimize the objective using :doxy:`Adam`::

..sharkcode<Unsupervised/AutoEncoderTutorial.tpp,optimizer>

Visualizing the autoencoder
^^^^^^^^^^^^^^^^^^^^^^^^^^^

After training the different architectures, we printed the feature maps of the first layer
(i.e., the input weights of the hidden neurons ordered according to the pixels they are connected to). Let's have a look.

..sharkcode<Unsupervised/AutoEncoderTutorial.tpp,visualize>

.. figure:: ../images/featuresAutoencoder.png
  :alt: Plot of features learned by the normal autoencoders

What next?
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The next tutorials covers :doc:`variational_autoencoders`