# THUNET: A simple deep learning framework for scientific and education purpose.
1. **Neural networks[1]**  
    * Layers / Layer-wise ops
        - Add
        - Flatten
        - Multiply
        - Softmax
        - Fully-connected/Dense
        - Sparse evolutionary connections
        - LSTM
        - Elman-style RNN
        - Max + average pooling
        - Dot-product attention
        - Embedding layer
        - Restricted Boltzmann machine (w. CD-n training)
        - 2D deconvolution (w. padding and stride)
        - 2D convolution (w. padding, dilation, and stride)
        - 1D convolution (w. padding, dilation, stride, and causality)
    * Modules
        - Bidirectional LSTM
        - ResNet-style residual blocks (identity and convolution)
        - WaveNet-style residual blocks with dilated causal convolutions
        - Transformer-style multi-headed scaled dot product attention
    * Regularizers
        - Dropout
    * Normalization
        - Batch normalization (spatial and temporal)
        - Layer normalization (spatial and temporal)
    * Optimizers
        - SGD w/ momentum
        - AdaGrad
        - RMSProp
        - Adam
    * Learning Rate Schedulers
        - Constant
        - Exponential
        - Noam/Transformer
        - Dlib scheduler
    * Weight Initializers
        - Glorot/Xavier uniform and normal
        - He/Kaiming uniform and normal
        - Standard and truncated normal
    * Losses
        - Cross entropy
        - Squared error
        - Bernoulli VAE loss
        - Wasserstein loss with gradient penalty
        - Noise contrastive estimation loss
    * Activations
        - ReLU
        - Tanh
        - Affine
        - Sigmoid
        - Leaky ReLU
        - ELU
        - SELU
        - Exponential
        - Hard Sigmoid
        - Softplus
    * Models
        - Bernoulli variational autoencoder
        - Wasserstein GAN with gradient penalty
        - word2vec encoder with skip-gram and CBOW architectures
    * Utilities
        - `col2im` (MATLAB port)
        - `im2col` (MATLAB port)
        - `conv1D`
        - `conv2D`
        - `deconv2D`
        - `minibatch`
2. **BERT** 
   * Vanilla BERT
   * Simple BERT


3. **REFERENCE**
Our contribution is implementation of the vanilla BERT and simple BERT.  
All other codes following the licence claimed by (ddbourgin)[https://github.com/ddbourgin]  in his (Numpy_ML)![https://github.com/ddbourgin/numpy-ml]  project.


4. **Release Frequent Ased Questions**

* Q: Python2.7:    LookupError: unknown encoding: cp0  
* A: Setting environment in the shell: set PYTHONIOENCODING=UTF-8

5. **Product Release**

Supported Python versions:

| Python |
|--------|
| 2.7    |
| 3.5    |
| 3.6    |
| 3.7    |
| 3.8    |
| 3.9    |
| 3.10   |

[1] David Bourgin. Machine learning, in numpy. https://github.com/ddbourgin/numpy-ml.