The dataframes generated by `dsmtools.preprocessing` can be used for the models in this module.
You can learn how to make your dataset before you proceed in here.

This module exposes three classes, each with one model. For the deep learning application,
the models you might be mostly interested in are `DSMAutoencoder`(AE) for unsupervised learning,
and `DSMHierarchicalAttentionNetwork`(HAN) for the supervised. For both network, we have trained
a model on 1282 SEU neurons, and you can use them to either encode your data or predict your
cell type out of the box.

Another model, word2vec, is used by `DSMDataconverter` to convert the brain abbreviation field
of the dataframes from the preprocessing into quantifiable vectors. It's already trained
and don't have to be changed if you are using CCFv3 abbreviations of brain regions.
If you need your own word2vec model, you can learn to train one following one of our
[jupyter notebook scripts](https://github.com/xiongfengNJ/neuron2seq/blob/e11cb567d9ee3ea259f0b8f4dcc6517f29c7c6e2/jupyter/Word2vec.ipynb).

# Data Conversion

Before the data can be fed to the networks, some further conversion are needed. The
reason why this is not included in preprocessing is we deem it necessary to expose
a readable intermediate dataset that can be analyzed and recombined.

There are two methods for conversion, one for the AE, the other for HAN. Some processing
steps are common, like converting the region abbreviation into vectors using the word2vec
model, turning multilevel fields into dummies, and scaling the coordinates.

## AE

The data conversion for AE is simple. As AE only accepts input of a certain length,
you need to give the length and that's all. This length must be the same with the model
you are using, otherwise train a new AE with this new piece of data.

## HAN

The data conversion for HAN is different from that for AE in two ways. 

First, its sequence is hierarchical, comparable to a word-sentence structure. 
By using some feature (here we use terminal annotations from the 'node_tye'
field), we can sort out words, and stacking up the words we have a sentence, so there's
one more dimension. You only need to specify the lengths for the word and the sentence
like for AE and these two dimensions will be unified.

Second, it can accept a label field to encode them into one-hot matrix, useful for training.
The label should be a dict so that the method can find the cell type for each neuron.
The length of the one-hot will be determined by the exact number of cell types in your
dataset.
The label encoder will also be returned as you need to convert the result back to the
type name. If this field is not provided, only the input data will be output.

# Deep Sequential Models

We use the [Keras API](https://keras.io/api/) to make the model classes, so they are
basically Keras. We provide the most basic functions for using and training our models, which is very simple even
for a starter. 

If you are familiar with Keras, we also expose the model reference, so you can play with it the way you like.
The most important part is how we build. the model building is in the `compile` method for each class. Our
trained model is provided as a staticmethod of the class, and you can get it without compiling and loading.

The class structures of the 2 models are similar, only differ in their model compilation.

## AE

AE is typically a class of network used for nonlinear dimensionality reduction, a method that is often compared with others like
PCA, UMAP, etc. As a result, you get a latent variable with a specific dimension lower than the original but retain the most
important information to map it back. In our model, since our data are sequences of different lengths, we have to unify them
in the data conversion step.

What makes it complicated is the length of sequence is coupled with the feature dimension. You can see the real number of
dimension as sequence length * #feature, but it's untrue, because sequence can't be aligned so plainly like features do.

Here, it's solved by using GRU to build the encoder and the decoder. The latent space representation is the hidden variable
of the last GRU in encoder. To decode it back, the latent result is repeated to be a sequence of the original length.

We provide you with an already trained AE on a neuron dataset of 1282 manual reconstructions from Southeast University. 
Follow the default parameter settings of data conversion, and you can get your input array for it. 

## HAN

HAN uses a two-level encoding strategy to embed sequences. Actually, it takes sequences(sentence) of sequences(word) of
features, and turns them, from word to sentence, into the latent representation similar to that of the AE.

First, it uses GRU to get latent variables for word, so that the data become sentences with a certain dimension
of latent features. After coupling with an attention mechanism, it's fed to a GRU again to become variables of
pure latent features. Then, it operates like a typical supervised classification model.

We provide you with an already trained HAN on a neuron dataset of 1282 manual reconstructions from Southeast University. 
Follow the default parameter settings of data conversion, and you can get your input array for it. To convert the
prediction to labels, we also offer a corresponding sklearn label encoder. To use this, you need to find the place of the
biggest prediction result for each occurrence and map it back to the label name.

---

You can follow the API documentation in this page to learn how to use these classes.