{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `cyanDiff` -- a forward-mode AD library\n",
    "\n",
    "## Introduction\n",
    "\n",
    "`cyanDiff` will implement forward-mode automatic differentiation in Python using Dual Numbers to differentiate many functions.\n",
    "\n",
    "Solving the automatic differentiation problem is useful not only because it is a core mathematical operation that can be used in scenarios like CASes (computer algebra systems), but because it has key applications in tasks such as machine learning and data analysis. In particular, automatic differentiation is a critical component of the ML model training process. By modeling a generalized differentation scheme within `cyanDiff`, users will be able to differentiate all sorts of functions composed from basic elementary functions and variables, which is exactly the type of thing that appears in complex ML models with heterogeneous architectures.\n",
    "\n",
    "## Background\n",
    "\n",
    "Automatic differentiation and our package rely on the chain rule in calculus. For scalars, the chain rule is given by:\n",
    "\n",
    "$$\\frac{d}{dx} (f \\circ g)(x) = f'(g(x)) \\cdot g'(x).$$\n",
    "\n",
    "For example, if we are differentiating the elementary function, $\\sin x^2$, we would get $2x \\cdot \\cos x^2$ because $\\cos$ is the derivative of $\\sin$ and $2x$ is the derivative of $x^2.$\n",
    "\n",
    "At its core, automatic differentiation allows for a program to differentiate an elementary function (or the composition of elementary functions, including $\\exp$, trig functions, polynomials, $\\log$ etc) by repeatedly applying the chain rule to the compositions of functions. AD also keeps track of the directional derivative, which is given by multiplying the currently computed derivative by a direction vector commonly denoted $p.$\n",
    "\n",
    "This necessitates the use of a computational graph. Each node in the graph keeps track of the derivative and the directional derivative. Then, edges between the nodes are represented by functions (addition, multiplication, $e^x$ etc). It is worth noting that while we gave scalar functions above, we will support the differentiation of elementary vector valued functions $f: \\mathbb{R}^m \\to \\mathbb{R}^n$.\n",
    "\n",
    "Now we describe the technical details about implementing automatic differentiation. Specifically, we talk a bit about dual numbers and how they are used in the automatic differentiation process. Dual numbers are similar to complex numbers, except instead of using the imaginary unit $i$, we use a value $\\varepsilon$ such that $\\varepsilon^2 = 0$. So, dual numbers are of the form $a + b\\varepsilon$, where $a, b$ are real numbers.\n",
    "\n",
    "What is the advantage of using dual numbers? There is a clever way of using the properties of $\\varepsilon$ so that we can store both the primal and tangent traces of a computation within a *single* dual number. This drastically simplifies computations, and combined with overloaded DualNumber-supporting functions, forward-mode AD becomes a simple task.\n",
    "\n",
    "To understand how DualNumbers can be used to streamline AD, we first discuss the idea of a computational graph. All mathematical functions can be represented in terms of simpler \"elementary\" functions. These functions are things such as addition, exponentiation, and cosine. We then represent a mathematical function as a composition of a bunch of these elementary functions, on top of a set of input \"independent\" variables. Let's consider the function $f(x, y) = x*y + 2y$ for the remainder of this section. This function takes two input variables $x, y$ and is computed as follows, in terms of elementary functions: we multiply $x$ and $y$, multiply $2$ and $y$, and then add those two results together in order to get $f(x, y)$. We can then represent these chained operations in the following graph, where the $v_i$ represent intermediate computed values when $i > 0$, and for $i < 0$ we simply have the independent input variables:\n",
    "\n",
    "![](https://i.imgur.com/JBKTduD.jpg)\n",
    "\n",
    "Now, suppose we want to use this computational graph to help us calculate the gradient of $v_3$, the function value, with respect to the inputs $v_{-1}$ and $v_0$. The trick is to store both the imtermediate value and (directional) derivative in the form of a single Dual number at every node of the computational graph. So, suppose we want to compute the directional derivative in a direction vector $p$ at some point $(x_0, y_0) = \\mathbf{x}$. We would store at every node $v_i$ the value $v_i(\\mathbf{x}) + D_p(v_i)(\\mathbf{x})\\varepsilon$, where $D_p$ denotes the directional derivative in direction $p$ (the Jacobian multiplied by $p$).\n",
    "\n",
    "Then, by storing both the function value $v_i(\\mathbf{x})$ and the derivative $D_p(v_i)(\\mathbf{x})$ in a single Dual number, we can compute successive function values and derivatives simply by performing the corresponding elementary function operations on the dual numbers themselves. For example: suppose we're in the above graph and want to compute $v_3$'s node from $v_1$ and $v_2$. We simply add their Dual number values to get $v_1(\\mathbf{x}) + v_2(\\mathbf{x}) + \\left[D_p(v_1)(\\mathbf{x}) + D_p(v_2)(\\mathbf{x})\\right]\\varepsilon$, which is exactly $v_3(\\mathbf{x}) + D_p(v_3)(\\mathbf{x})\\varepsilon$, by the addition rule for derivatives.\n",
    "\n",
    "This holds true for more complex scenarios; suppose we wanted to multiply nodes $v_1$ and $v_2$. The product rule is required when computing the derivative of the resultant node $v_3 = v_1\\cdot v_2$. But when using Dual numbers, simply multiplying the Dual number representations works: $(v_1(\\mathbf{x}) + D_p(v_1)(\\mathbf{x})\\varepsilon)\\cdot (v_2(\\mathbf{x}) + D_p(v_2)(\\mathbf{x})\\varepsilon) = v_1(\\mathbf{x})v_2(\\mathbf{x}) + \\left[v_1D_p(v_2)(\\mathbf{x}) + v_2D_p(v_1)(\\mathbf{x})\\right]\\varepsilon$, which is exactly what we'd get from the product rule. Notice how the term with $\\varepsilon^2$ cancels out by definition.\n",
    "\n",
    "## How to install `cyanDiff` \n",
    "We provide two methods of installing `cyanDiff` package. The following installation details will walk you through the creation of a virtual environment.\n",
    "#### Installation Method 1: Clone from `GitHub`\n",
    "1. Download our package from GitHub to users' local directory by typing the following command in the terminal:\n",
    "```\n",
    "$ git clone https://code.harvard.edu/CS107/team40.git\n",
    "```\n",
    "2. Create and Activate a `virtual environment` in project directory by the commands:\n",
    "```\n",
    "$ cd team40\n",
    "$ python -m venv test_env\n",
    "```\n",
    "To activate it run:\n",
    "```\n",
    "source test_env/bin/activate\n",
    " ```\n",
    "To install the dependencies run:\n",
    "```\n",
    "$ pip install -r requirements.txt\n",
    "```\n",
    "\n",
    "3. Go to your Python interpreter on the virtual environment to import the module:\n",
    "```\n",
    "import cyanDiff as cd\n",
    "```\n",
    "\n",
    "#### Install Method 2: Install via `PyPI`\n",
    "We will distribute `cyanDiff` via PyPI, with all the necessary package configuration files. The package can thus be installed via\n",
    " ```\n",
    "python3 -m pip install cyanDiff\n",
    "```\n",
    "\n",
    "## Usage demonstration\n",
    "\n",
    "We now give an example demonstration on how to use `cyanDiff` to differentiate a simple function.\n",
    "\n",
    "Suppose we have the function $$f(x) = \\frac{e^x}{\\sqrt{1 + \\sin(x)}}.$$ Analytically, using the quotient and chain rules, we have:\n",
    "\n",
    "$$f'(x) = \\frac{\\sqrt{1 + \\sin x} \\cdot e^x - e^x \\cdot \\frac{1}{2}(1 + \\sin(x))^{-1/2} \\cdot \\cos(x)}{1 + \\sin x}.$$\n",
    "\n",
    "Now, suppose we want to find the derivative at an arbitrary value like $$x = 3.1$$ Analytically, all we need to do is compute:\n",
    "\n",
    "$$\\frac{\\sqrt{1 + \\sin(3.1)} \\cdot e^{3.1} - e^{3.1} \\cdot 0.5 (1 + \\sin(3.1))^{-1/2} \\cdot \\cos(3.1)}{1 + \\sin(3.1)} \\approx 32.182364199.$$\n",
    "\n",
    "We now compute this derivative using the forward mode AD functionality of our package. To do this, we simply import our package `cyanDiff`. We then instantiate a variable object using the `make_vars()` function, passing an argument of 1 to specify we want a single variable.\n",
    "\n",
    "```\n",
    "import cyanDiff as cd\n",
    "\n",
    "x = make_vars(1)\n",
    "```\n",
    "\n",
    "Next, we construct the function $f(x)$ with this new variable object.\n",
    "\n",
    "```\n",
    "f = cd.exp(x) / ((1 + cd.sin(x)) ** 0.5)\n",
    "```\n",
    "\n",
    "Now, we specify the point at which the derivative should be taken, in the form of a dictionary.\n",
    "\n",
    "```\n",
    "point = {x: 3.1}\n",
    "```\n",
    "\n",
    "Finally, we can take the derivative:\n",
    "\n",
    "```\n",
    "derivative_value = f.diff_at(point)\n",
    "```\n",
    "\n",
    "If we do this and check the value of `derivative_value`, we get $32.182364199$, which matches our analytical calculation.\n",
    "\n",
    "An additional simple example: we can also evaluate functions, simply by calling them on points. For example, below we construct a function $g$ and evaluate it at $(y, z) = (2, 3)$:\n",
    "\n",
    "```\n",
    "y, z = make_vars(2)\n",
    "g = y * z + y ** z\n",
    "point2 = {y: 2, z: 3}\n",
    "res1 = g(point2)\n",
    "```\n",
    "\n",
    "And printing out the value of `res1` will result in 14, as expected.\n",
    "\n",
    "## Software organization\n",
    "\n",
    "```\n",
    "team40\n",
    "|----docs/\n",
    "      |----documentation_files/\n",
    "      |----milestone_files/\n",
    "|----src/cyanDiff\n",
    "      |----__init__.py\n",
    "      |----ad_helpers.py\n",
    "      |----ad_overloads.py\n",
    "      |----ad_types.py\n",
    "|----tst/cyanDiff\n",
    "      |----test_ad_overloads.py\n",
    "      |----test_ad_types.py\n",
    "...\n",
    "```\n",
    "\n",
    "The main important files are shown in the diagram above. Other ancillary files, such as the LICENSE, are not shown above (represented by the ellipsis). A quick description of our modules and what they do:\n",
    "\n",
    "- `ad_types.py` implements the core data types for automatic differentiation. These are the DualNumber class, the Function class (used to represent functions such as $f = x + y$), and the VectorFunction class (used to represent functions which can output vectors).\n",
    "\n",
    "- `ad_overloads.py` implements overloaded functions that can operate on standard integers/floats, dual numbers, and functions. For example, overloaded trignometric functions are implemented here.\n",
    "\n",
    "- `ad_helpers.py` implements functions that simplify the user's usage of the `cyanDiff` package. For example, the `make_vars()` function is implemented here which abstracts away the creation of variables.\n",
    "\n",
    "The tests are located in the `tst` folder. They can be run by using the test harness, `run_tests.sh` within the `tst` folder. The tests that are run are `test_ad_types.py` which takes care of the core data structures/types that are used in forward mode AD. Next, the `test_ad_overloads.py` test file runs tests on overloaded functions, performing integration tests with the data structures like `DualNumber` and `Function`. Finally, `test_ad_helpers.py` tests some user functions that are designed to abstract away some of the logic involved in creating variable objects (like the `x`, `y` used in our sample code above).\n",
    "\n",
    "## Implementation Details\n",
    "\n",
    "We first outline the data structures that we use in our implementation of forward mode AD. Our most basic data type we use is dual numbers, implemented in our `DualNumber` class. It stores two attributes, the real part of the dual number and the dual part of the dual number. To store the dual part we store the real coefficient of the dual part of the number. The second data structure we use is given by the `Function` type, defined in the `Function` class. `Function` objects store evaluators for the math functions that the user is differentiating as an attribute called `evaluator`. These `Function` objects can be combined in various ways through math operators and functions, thus combining their evaluators, in order to create representations of more complex functions made from more basic components. The variables that we provide the user are in fact `Function` objects with simply an identity evaluator, which the user can then use to define the math expressions they would like to evaluate and differentiate at specified points.\n",
    "\n",
    "We now explain how the `DualNumber` and `Function` classes are used together in forward mode AD. The `make_vars()` produce some variables, which are actually `Function` objects which can then be composed together using basic math operations as well as the elementary functions we provide to make a math function. Then, to find a derivative of this function at a particular point, we calculate the directional derivatives for each variable, i.e. we calculate for each direction where each coordinate is 0 except for one. To do this, we create a dual number representation for each variable, with the real part being the value of the variable at the point given by the user, and the dual part being the derivative with respect to the direction. As described in the above background section, dual numbers can be used to streamline forward mode AD since the real part stores the primal trace while the dual part tracks the tangent trace. Our basic operations, such as addition, multiplication, subtraction, division, and exponentiation for dual numbers, as well as basic trig functions, exponentiation, and logarithm are defined so that they are overloaded to correctly compute for dual numbers, so by just using the `evaluator` attribute in the function (a `Function` object) defined by the user on the dual number inputs, we get our calculated primal and tangent traces, and we return the tangent trace as the derivative desired.\n",
    "\n",
    "We now discuss our implementation of elementary functions. Here, we make use of our only external dependency, Numpy. We use Numpy because they provide functions such as the basic trig functions that are already useable with array (useful for when we finish implementing differentiation of vector functions). In our implementation of elementary functions, we must overload them to work with objects from our `DualNumber` and `Function` classes. Since the `DualNumber` class represents dual numbers, we use the math rules for dual numbers to determine how our overloaded functions should change the `real` and `dual` attributes of `DualNumber` objects. For functions represented by the `Function` type, we modify the `evaluator` attribute to add on the evaluator for the math function.\n",
    "\n",
    "We still have to implement our handling of vector functions, which we plan to differentiate using our `jacobian_at()` function which is part of the `Function` class. We also have to update our dunder methods to work with vector functions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.8.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
