# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['high_order_layers_torch']

package_data = \
{'': ['*']}

install_requires = \
['hydra-core>=1.2.0,<2.0.0',
 'pytorch-lightning>=2.0.0,<3.0.0',
 'torch-optimizer>=0.3.0,<0.4.0',
 'torch==2.1.1',
 'torchmetrics>=0.10.2,<0.11.0',
 'torchvision>=0.16.1,<0.17.0']

setup_kwargs = {
    'name': 'high-order-layers-torch',
    'version': '2.2.2',
    'description': 'High order layers in pytorch',
    'long_description': '\n# Piecewise Polynomial in PyTorch\n\nThis is a PyTorch implementation of my tensorflow [repository](https://github.com/jloveric/high-order-layers) and is more complete due to the flexibility of PyTorch.\n\nLagrange Polynomial, Piecewise Lagrange Polynomial, Discontinuous Piecewise Lagrange Polynomial, Fourier Series, sum and product layers in PyTorch.  The sparsity of using piecewise polynomial layers means that by adding new segments the representational power of your network increases, but the time to complete a forward step remains constant. Implementation includes simple fully connected layers, convolution layers and deconvolutional layers using these models. This is a PyTorch implementation of this [paper](https://www.researchgate.net/publication/276923198_Discontinuous_Piecewise_Polynomial_Neural_Networks) including extension to Fourier Series and convolutional neural networks.\n\n## Collab Notebook\nUsing simple high order layers\n[Simple function approximation](https://colab.research.google.com/drive/1kew0Kz4v5GB5D59-wP1rHZuCdhYknz4s?usp=sharing)\n\nUsing simple high order MLP\n[2d function approximation](https://colab.research.google.com/drive/14wSNzBUFYk-1o6fuqiux_y33aV9VuwkF?usp=sharing)\n\n\n## Idea\n\nThe idea is extremely simple - instead of a single weight at the synapse, use n-weights.  The n-weights describe a piecewise polynomial (or other complex function) and each of the n-weights can be updated independently. A Lagrange polynomial and Gauss Lobatto points are used to minimize oscillations of the polynomial.  The same approach can be applied to any "functional" synapse, and I also have Fourier series synapses in this repo as well.  This can be implemented as construction of a polynomial or Fourier kernel followed by a standard pytorch layer where a linear activation is used.\n\nIn the image below each "link" instead of being a single weight, is a function of both x and a set of weights.  These functions can consist of an orthogonal basis functions for efficient approximation.\n\n<img src="plots/NetworkZoom.png" width=50% height=50% style="display: block; margin: 0 auto">\n\n## Why\n\nUsing higher order polynomial representations might allow networks with much fewer total weights.\n\n## Fully Connected Layer Types\nAll polynomials are Lagrange polynomials with Chebyshev interpolation points.\n\nA helper function is provided in selecting and switching between these layers\n\n```python\nfrom high_order_layers_torch.layers import *\nlayer1 = high_order_fc_layers(\n    layer_type=layer_type,\n    n=n,\n    in_features=784,\n    out_features=100,\n    segments=segments,\n)\n```\n\nwhere `layer_type` is one of\n| layer_type          | representation\n|--------------------|-------------------------|\n|continuous         |  piecewise polynomial using sum at the neuron |\n|continuous_prod    |  piecewise polynomial using products at the neuron |\n|discontinuous      |  discontinuous piecewise polynomial with sum at the neuron|\n|discontinuous_prod | discontinous piecewise polynomial with product at the neuron|\n|polynomial | single polynomial (non piecewise) with sum at the neuron|\n|polynomial_prod | single polynomial (non piecewise) with product at the neuron|\n|product | Product |\n|fourier | fourier series with sum at the neuron |\n\n\n\n`n` is the number of interpolation points per segment for polynomials or the number of frequencies for fourier series, `segments` is the number of segments for piecewise polynomials, `alpha` is used in product layers and when set to 1 keeps the linear part of the product, when set to 0 it subtracts the linear part from the product.\n\n## Convolutional Layer Types\n\n```python\nconv_layer = high_order_convolution_layers(layer_type=layer_type, n=n, in_channels=3, out_channels=6, kernel_size=5, segments=segments, rescale_output=rescale_output, periodicity=periodicity)\n```\n\nAll polynomials are Lagrange polynomials with Chebyshev interpolation points.\n| layer_type   | representation       |\n|--------------|----------------------|\n|continuous(1d,2d)   | piecewise continuous polynomial\n|discontinuous(1d,2d) | piecewise discontinuous polynomial\n|polynomial(1d,2d) | single polynomial\n|fourier(1d,2d) | fourier series convolution\n\n## h and p refinement\np refinement is taking an existing network and increasing the polynomial order of that network without changing the network output.  This allow the user to train a network at low polynomial order and then use that same network to initialize a network with higher polynomial order.  This is particularly useful since a high order polynomial network will often converge poorly without the right initialization, the lower order network provides a good initial solution.  The function for changing the order of a network is\n```\nfrom high_order_layers_torch.networks import interpolate_high_order_mlp\ninterpolate_high_order_mlp(\n    network_in: HighOrderMLP, network_out: HighOrderMLP\n```\ncurrent implementation only works with high order MLPs, not with convnets.  A similar function exists for h refinement.  h refinement is\nrefining the number of segments in a layer, and is used for similar reasoning.  Layers with lots of segments may be slow to converge\nso the user starts with a small number of segments (1 or 2) and then increases the number of segments (h) using the lower initialization.  The following function currently only works for high order MLPs, not with convnets\n```\nfrom high_order_layers_torch.network import hp_refine_high_order_mlp\nhp_refine_high_order_mlp(\n    network_in: HighOrderMLP, network_out: HighOrderMLP\n)\n```\n# Installing\n\n## Installing locally\n\nThis repo uses poetry, so run\n\n```\npoetry install\n```\n\nand then\n\n```\npoetry shell\n```\n\n## Installing from pypi\n\n```bash\npip install high-order-layers-torch\n```\n\nor\n\n```\npoetry add high-order-layers-torch\n```\n# Examples\n\n## Simple function approximation\n\nApproximating a simple function using a single input and single output (single layer) with no hidden layers\nto approximate a function using continuous and discontinuous piecewise polynomials (with 5 pieces) and simple\npolynomials and fourier series.  The standard approach using ReLU is non competitive.  To see more complex see\nthe implicit representation page [here](https://github.com/jloveric/high-order-implicit-representation).\n\n![piecewise continuous polynomial](plots/piecewise_continuous.png)\n![piecewise discontinuous polynomial](plots/piecewise_discontinuous.png)\n![polynomial](plots/polynomial.png)\n![fourier series](plots/fourier_series.png)\n\n```python\npython examples/function_example.py\n```\n\n## XOR : 0.5 for x*y > 0 else -0.5\nSimple XOR problem using the standard network structure (2 inputs 2 hidden 1 output) this will also work with no hidden layers. The function is discontinuous along the axis and we try and fit that function. Using piecewise discontinuous layers the model can match the function exactly.\n![piecewise discontinuous polynomial](plots/xor_discontinuous.png)\nWith piecewise continuous it doesn\'t work quite as well.\n![piecewise continuous polynomial](plots/xor_continuous.png)\nPolynomial doesn\'t work well at all (expected).\n![polynomial](plots/xor_polynomial.png)\n\n## MNIST (convolutional)\n\n```python\npython examples/mnist.py max_epochs=1 train_fraction=0.1 layer_type=continuous2d n=4 segments=2\n```\n\n## CIFAR100 (convolutional)\n\n```\npython examples/cifar100.py -m max_epochs=20 train_fraction=1.0 layer_type=polynomial segments=2 n=7 nonlinearity=False rescale_output=False periodicity=2.0 lr=0.001 linear_output=False\n```\n\n## Variational Autoencoder\nStill a WIP.  Does work, but needs improvement.\n```\npython examples/variational_autoencoder.py -m max_epochs=300 train_fraction=1.0\n```\nrun with nevergrad for parameter tuning\n```\npython examples/variational_autoencoder.py -m\n```\n\n## Invariant MNIST (fully connected)\nWithout polynomial refinement\n```python\npython examples/invariant_mnist.py max_epochs=100 train_fraction=1 mlp.layer_type=continuous mlp.n=5 mlp.p_refine=False mlp.hidden.layers=4\n```\nwith polynomial refinement (p-refinement)\n```\npython examples/invariant_mnist.py max_epochs=100 train_fraction=1 layer_type=mlp.continuous mlp.n=2 mlp.target_n=5 mlp.p_refine=True\n```\nI\'ve also added hp refinement, but it needs a lot of testing.\n\n## Implicit Representation\n\nAn example of implicit representation for image compression, language generation can be found [here](https://github.com/jloveric/high-order-implicit-representation).  I intend to explore generative models in natural language further [here](https://github.com/jloveric/language-interpolation)\n\n## PDEs in Fluid Dynamics\n\nAn example using implicit representation to solve hyperbolic (nonlinear) wave equations can be found [here](https://github.com/jloveric/neural-network-pdes)\n\n## Natural Language Generation\n\nExamples using these networks for natural language generation can be found\n[here](https://github.com/jloveric/language-interpolation)\n\n## Generative music\n\nNo real progress here\n[here](https://github.com/jloveric/high-order-generative-music)\n\n\n## Test and Coverage\n\nAfter installing and running\n```\npoetry shell\n```\nrun\n```\npytest\n```\nfor coverage, run\n```\ncoverage run -m pytest\n```\nand then\n```\ncoverage report\n```\n## A note on the product unit (I rarely use anymore)\nThe layers used here do not require additional activation functions and use a simple sum or product in place of the activation.\nI almost always use sum units, but product units are performed in this manner\n\n$$ product=-1+\\prod_{i}(1 + f_{i})+(1-\\alpha)\\sum_{i}f_{i} $$\n\nThe 1 is added to each function output to as each of the sub products is also computed.  The linear part is controlled by\nthe alpha parameter.\n\n## Notes on normalization\nAlthough you can use batchnorm, layernorm etc... I\'ve found that you can actually just use the infinity norm ("max_abs" norm) which has no parameters\nfor this formulation (same approach seems not to work very well for standard relu networks - but need to investigate this further).\nThe max_abs normalization is defined this way\n```\nnormalized_x = x/(max(abs(x))+eps)\n```\nwhere the normalization is done per sample (as opposed to per batch).  The way the layers are formulated, we don\'t want the neuron\nvalues to extend beyond [-1, 1] as the polynomial values grow rapidly beyond that range.  You can also use mirror periodicity to keep the\nvalues within from growing rapidly. We want the values to cover the entire range [-1, 1] of the polynomials as the weights\nare packed towards the edges of each segment (though using even number of segments means you\'ll have a lot of weights near the origin).\n\n\n## Reference\n```\n@misc{Loverich2020,\n  author = {Loverich, John},\n  title = {High Order Layers Torch},\n  year = {2020},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/jloveric/high-order-layers-torch}},\n}\n```\n',
    'author': 'jloverich',
    'author_email': 'john.loverich@gmail.com',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.9',
}


setup(**setup_kwargs)
