
==============================
Deep Neural Network Library
==============================

It is for eliminating repeat jobs of machine learning. Also it can makes your code more beautifully and Pythonic.

.. contents:: Table of Contents

Building Deep Neural Network 
==============================

Please see my several examples_. It contains below networks using MNIST dataset:

- Logistic Regression
- Association Learning
- GAN: Generative Adversarial Network
- VAE: Variational Autoencoder
- AAE: Adversal Autoencoder

.. _examples: https://gitlab.com/hansroh/dnn/tree/master/examples


Data Normalization
=====================

Data normalization and standardization,

.. code-block:: python

  train_xs = net.normalize (train_xs, normalize = True, standardize = True)

To show cumulative sum of explained_variance_ratio\_ of sklearn PCA. 

.. code-block:: python

  train_xs = net.normalize (train_xs, normalize = True, standardize = True, pca_k = -1)

Then you can decide n_components for PCA.

.. code-block:: python
  
  train_xs = net.normalize (train_xs, normalize = True, standardize = True, axis = 0, pca_k = 500)

Test dataset will be nomalized by factors of train dataset.

.. code-block:: python  
  
  test_xs = net.normalize (test_xs)

This parameters will be pickled at your train directory named as *normfactors*. You can use this pickled file for serving your model.
   

Export Model
==========================


To Saved Model
-------------------------
 
For serving model,

.. code-block:: python

  import mydnn
  
  net = mydnn.MyDNN ()
  net.restore ('./checkpoint')
  version = net.to_save_model ( 
    './export', 
    'predict_something', 
    inputs = {'x': net.x},
    outputs={'label': net.label, 'logit': net.logit}
  )
  print ("version {} has been exported".format (version))
 
For testing your model,

.. code-block:: python

  from dnn import save_model
  
  interpreter = save_model.load (model_dir, sess, graph)
  y = interpreter.run (x)
   
 
You can serve the expoted model with `TensorFlow Serving`_ or tfserver_.

Note: If you use net.normalize (train_xs), normalizing factors (mean, std, max and etc) willl be pickled and saved to model directory with tensorflow model. 
If you can use this file for normalizing new x data at real service.

.. code-block:: python
  
  from dnn import _normalize 
  
  def normalize (x):    
    norm_file = os.path.join (model_dir, "normfactors")
    with open (norm_file, "rb") as f:
      norm_factor = pickle.load (f)
    return _normalize (x, *norm_factor)


.. _`TensorFlow Serving`: https://github.com/tensorflow/serving 
.. _tfserver: https://pypi.python.org/pypi/tfserver


To Tensorflow Lite Flat Buffer Model
-------------------------------------------------------

* Required Tensorflow version 1.9*

For exporting tensorflow lite you should convert your model to save model first.

.. code-block:: python

  net.to_tflite (
      "model.tflite", 
      save_model_dir
  )

If you want to convert to quntized model, it will be needed additional parameters. 

.. code-block:: python

  net.to_tflite (
      "model.tflite", 
      save_model_dir,
      True, # quantize
      (128, 128), # mean/std stats of input value 
      (-1, 6) # min/max range output value of logit
  )

For testing tflite model,

.. code-block:: python
  
  from dnn import tflite
  
  interpreter = tflite.load ("model.tflite")
  y = interpreter.run (x)
  
If your model is quantized, it need mean/std stats of input value,

.. code-block:: python
  
  from dnn import tflite
  
  interpreter = tflite.load ("model.tflite", (128, 128))
  y = interpreter.run (x)

If your input value range -1.0 ~ 1.0, its will be translated into 0 - 255 for qunatized model by mean and std parameters.
So (128, 128) means your inout value range is -1.0 ~ 1.0. Then interpreter will qunatize x to uint8 by this parameter.

.. code-block:: python

  unit8 = (float32 x * std) + mean

And tflite will reverse this uinit8 to float value by,

.. code-block:: python

  float32 x = (uint8 x - mean) / std
  
Helpers
============

There're several helper modules.

Generic DNN Model Helper
------------------------------

.. code-block:: python

  from dnn import costs, predutil


Data Processing Helper
------------------------------

.. code-block:: python
  
  from dnn import split, vector
  import dnn.video
  import dnn.audio
  import dnn.image
  import dnn.text


dnn Class  Methods & Properties
====================================

You can override or add anything. If it looks good, contribute to this project please.

Predefined Operations & Creating
---------------------------------------------------

You should or could create these operations by overriding methods,
 
- train_op: create with 'make_optimizer'
- logit: create with 'DNN.make_logit'
- cost: create with 'DNN.make_cost'
- accuracy: create with 'DNN.calculate_accuracy'

Predefined Place Holders
--------------------------------

- dropout_rate: if negative value, dropout rate will be selected randomly. 
- is_training
- n_sample: Numner of x (or y) set. This value will be fed automatically, do not feed.


Optimizers
-----------------

You can use predefined optimizers.

.. code-block:: python

  def make_optimizer (self):
    return self.optimizer ("adam")
    # Or
    return self.optimizer ("rmsprob", mometum = 0.01)
    
Available optimizer names are,

- "adam"    
- "rmsprob"
- "momentum"
- "clip"
- "grad"
- "adagrad"
- "adagradDA"
- "adadelta"
- "ftrl"
- "proxadagrad"
- "proxgrad"

see dnn/optimizers.py


Model 
------------

- save
- restore
- to_save_model
- to_tflite
- reset_dir
- set_train_dir
- eval


Tensor Board
-----------------------

- set_tensorboard_dir
- make_writers
- write_summary


History
=========

- 0.3:

  - remove trainale ()
  - add set_learning_rate ()
  - add argument to set_train_dir () for saving chcekpoit
  - make compatible with tf 1.12.0

- 0.2

  - add tensorflow lite conversion and interpreting
   
- 0.1: project initialized
