# micrograd2023


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

  
  

<div class="flushright">

> *“What I cannot create, I do not understand.”*
>
> – Richard Feynman (1918-1988)

</div>

  
  

# What is micrograd2023?

I developed `micrograd2023` based on Andrej Karpathy’s `micrograd`
implementation with added functionalities for teaching purposes using
Jeremy Howard’s and Hamel Husain’s `nbdev`, which enables package
development, testing, documentation, and dissemination all in one place.

<img src='media/mArtificialNeuralNetwork_title.gif' width=100% height=auto>

# Literate Programming

``` mermaid
flowchart LR
  A(Andrej's micrograd) --> C((Combination))
  B(Jeremy's nbdev) --> C
  C -->|Literate Programming| D(micrograd2023)
```

# Overview of micrograd2023

- A detailed demonstration of `micrograd2023` for training and
  integrating MLP can be found in this [MLP
  DEMO](https://hdocmsu.github.io/micrograd2023/mlp_demo.html).

- A demonstration of `micrograd2023` for Physics for
  auto-differentiation of a popular cosine function can be found in this
  [Physics Cosine
  DEMO](https://hdocmsu.github.io/micrograd2023/phys_demo_cos.html).

  - Comparing the `micrograd2023` results with the analytical solutions,
    `pytorch`’s autograd, and `jax`’s autograd.
  - Additionally, second-order derivatives are calculated using `jax`’s
    autograd.
  - it is possible to use `jax`’s autograd to calculate higher-order
    derivatives.

- A demonstration of `micrograd2023` for Physics for
  auto-differentiation of a popular exponential decay function can be
  found in this [Physics Exp.
  DEMO](https://hdocmsu.github.io/micrograd2023/phys_demo_exp.html).

- A demonstration of `micrograd2023` for Physics for
  auto-differentiation of a damping function can be found in this
  [Physics Damp
  DEMO](https://hdocmsu.github.io/micrograd2023/phys_demo_damp.html).

- A demonstration of `micrograd2023` for MRI for auto-differentiation of
  a T2\* decay model of data acquired from a multi-echo UTE sequence.
  Additionally, the auto-differentiations then be used to calculate the
  Fisher Information Matrix (FIM), which then allows calculations of
  Cramer-Rao Lower Bound (CRLB) of an un-bias estimator of T2\*. Details
  can be seen at [MRI T2\* Decay
  DEMO](https://hdocmsu.github.io/micrograd2023/mri_demo_expdecay.html).

- A demonstration of `micrograd2023` for MRI for auto-differentiation of
  a T1 recovery model of data acquired from a myocardial MOLLI T1
  mapping sequence. Additionally, the auto-differentiations then be used
  to calculate the Fisher Information Matrix (FIM), which then allows
  calculations of Cramer-Rao Lower Bound (CRLB) of an un-bias estimator
  of T1. Details can be seen at [MRI T1 Recovery
  DEMO](https://hdocmsu.github.io/micrograd2023/mri_demo_exprec.html).

# Added features

Compared to Andrej’s `micrograd`, `micrograd2023` has many extensions
such as:

- Adding more and extensive unit and integration tests.

- Adding more methods for
  [`Value`](https://hdocmsu.github.io/micrograd2023/engine.html#value)
  object such as `tanh()`, `exp()`, and `log()`. In principle, any
  method/function with known derivative or can be broken into primitive
  operations can be added to the
  [`Value`](https://hdocmsu.github.io/micrograd2023/engine.html#value)
  object. Examples are `sin()`, `sigmoid()`, `cos()`, etc., which I left
  as exercises 😄.

- Refactoring Andrej’s demo code make it easier to demonstrate many
  fundamental concepts and/or best engineering practices when training
  neural network. The concepts/best-practices are listed below. Some
  concepts were demonstrated while the rest are left as exercises 😄.

  - Always implemented a simplest and most intuitive solution as a
    baseline to compare with whatever fancy implementations we want to
    achieve

  - Data preparation - train, validation, and test sets are disjointed

  - Over-fitting

  - Gradient Descent vs. Stochastic Gradient Descent (SGD)

  - Develop and experiment with different optimizations i.e. SGD, SGD
    with momentum, rmsProp, Adam, etc.

  - SGD with momentum

  - Non-Optimal learning rate

  - How to find the optimal learning rate

  - Learning rate decay and learning rate schedule

  - Role of nonlinearity

  - Linear separable and non-separable data

  - Out of distribution shift

  - Under-fitting

  - The importance and trade-off between width and depth of the MLP

  - Over-fitting a single-batch

  - Hyperparameter tuning and optimizing

  - Weights initialization

  - Inspect and visualize statistics of weights, gradients, gradient to
    data ratios, and update to data ratios

  - Forward and backward dynamics of shallow and deep linear and
    non-linear Neural Network

  - etc.

# Related Projects

Below are a few of my projects related to optimization and Deep
Learning:

- In 2019, I led the clinical validation of Canon Medical Systems’s Deep
  Learning-based denoising reconstruction technology. It was the first
  fully-integrated Deep Learning-based reconstruction technology to
  receive FDA 510(k) clearance to be used in clinical environment
  in 2020. See more details in this
  <a href="https://hdocmsu.github.io/projects/canon_1_dlr/"
  target="_blank">DLR-MRI project</a>

- Diploma Research on Crystal Structure using Gradient-based
  Optimization <a href="https://hdocmsu.github.io/projects/ictp_thesis/"
  target="_blank">SLIDES</a>

- Deep Convolution Neural Network (DCNN) for MRI image segmentation with
  uncertainty quantification and controllable tradeoff between False
  Positive and False Negative.
  <a href="https://hdocmsu.github.io/assets/pdf/papers/do_mrm2019.pdf"
  target="_blank">Journal Paper PDF</a>; <a
  href="https://hdocmsu.github.io/assets/pdf/slides/2018-10-28-HungDo_MLworkshop2018_web.pdf"
  target="_blank">Conference Talk SLIDES</a>;
  <a href="https://youtu.be/Qt5SdPFkAg4" target="_blank">Conference Talk
  VIDEO</a>

- Deep Learning-based Denoising for quantitative MRI. <a
  href="https://hdocmsu.github.io/assets/pdf/slides/2019-02-06-HungDo_dnoiseNET_web.pdf"
  target="_blank">Conference Talk SLIDES</a>

# How to install

Simply run the following command in your terminal to install
`micrograd2023`:

`pip install micrograd2023`

# Examples of using micrograd2023

Below are some examples of using `micrograd2023`.

## Automatic Differentiation

``` python
# import necessary objects and functions
from micrograd2023.engine import Value
from micrograd2023.nn import Neuron, Layer, MLP
from micrograd2023.utils import draw_dot
import random
```

``` python
# inputs xs, weights ws, and bias b
w1 = Value(1.1)
x1 = Value(0.5)
w2 = Value(0.12)
x2 = Value(1.7)
b = Value(0.34)

# pre-activation
s = w1*x1 + x2*w2 + b

# activation
y = s.tanh()

# automatic differentiation
y.backward()

# show the computation graph of the perceptron
draw_dot(y)
```

![](index_files/figure-commonmark/cell-4-output-1.svg)

``` python
# added random seed for reproducibility
random.seed(1234)
n = Neuron(3)
x = [Value(0.15), Value(-0.21), Value(-0.91) ]
y = n(x)
y.backward()
draw_dot(y)
```

![](index_files/figure-commonmark/cell-5-output-1.svg)

## Multilayer Perceptron (MLP) Training

You can use `micrograd2023` to train a MLP and learn fundamental
concepts such as overfilling, optimal learning rate, etc.

### Good training

<img src='media/MPL_good_training_decision_boundary.png' width=100% height=auto >

<img src='media/MPL_good_training_loss_acc_plots.png' width=100% height=auto >

### Overfitting

<img src='media/MPL_overfitting_decision_boundary.png' width=100% height=auto >

<img src='media/MPL_overfitting_loss_acc_plots.png' width=100% height=auto >

# Running unit tests

To perform unit testing, using terminal to navigate to the directory,
which contains `tests` folder, then simply type `python -m pytest` on
the terminal. Note that, `PyTorch` is needed for the test to run since
derivatives calculated using `micrograd2023` are compared against those
calculated using `PyTorch` as references.

`python -m pytest`

# References

`micrograd2023`, an automatic differentiation software, was developed
based on Andrej Karpathy’s
[micrograd](https://github.com/karpathy/micrograd), and was written
using Jeremy Howard’s and Hamel Husain’s
[nbdev](https://nbdev.fast.ai/).
