# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['mango', 'mango.domain', 'mango.optimizer']

package_data = \
{'': ['*']}

install_requires = \
['attrdict>=2.0.1',
 'numpy>=1.17.0',
 'scikit_learn>=0.21.3',
 'scipy>=1.4.1',
 'tqdm>=4.36.1']

setup_kwargs = {
    'name': 'arm-mango',
    'version': '1.3.1',
    'description': 'parallel bayesian optimization over complex search spaces',
    'long_description': '# Mango: A parallel hyperparameter tuning library\n\n**Mango** is a python library to find the optimal hyperparameters for machine learning classifiers. \nMango enables parallel optimization over complex search spaces of continuous/discrete/categorical values.\n\n**Check out the quick 12 seconds demo** of Mango approximating a complex decision boundary of SVM\n\n[![AirSim Drone Demo Video](documents/demo_video.png)](https://youtu.be/hFmSdDLLUfY)\n\nMango has the following salient features:\n- Easily define complex search spaces compatible with the scikit-learn.\n- A novel state-of-the-art gradient-free optimizer for continuous/discrete/categorical values.\n- Modular design to schedule objective function on local, cluster,  or cloud infrastructure.\n- Failure detection in the application layer for scalability on commodity hardware.\n- New features are continuously added due to the testing and usage in production settings.\n\n\n## Index\n1. [Installation](#setup)\n2. [Getting started ](#getting-started)\n3. [Hyperparameter tuning example ](#knnexample)\n4. [Search space definitions](#DomainSpace)\n5. [Scheduler](#scheduler)\n6. [Optional configurations](#MangoConfigurations)\n7. [Additional features](#AdditionalFeatures)\n8. [CASH feature](#CASHFeature)\n9. [Platform-aware neural architecture search](https://github.com/ARM-software/mango/tree/master/examples/THIN-Bayes)\n10. [Mango introduction slides](https://github.com/ARM-software/mango/blob/master/documents/Mango_github_slides.pdf) & [Mango production usage slides](https://github.com/ARM-software/mango/blob/master/documents/Mango_cogml_slides.pdf).\n11. [Core Mango research papers to cite](#CorePapers) and [novel applications built over Mango](#ApplicationPapers)\n\n<!--\n11. [Mango paper (ICASSP 2020)](https://arxiv.org/pdf/2005.11394.pdf) & [Mango paper (CogMI 2021)](https://github.com/ARM-software/mango/blob/master/documents/Mango_CogMI_paper.pdf).\n-->\n\n<!--\n7. [Schedule Objective Function on Celery](#Celery)\n8. [Algorithms](#mangoAlgorithms)\n9. [ Tune Hyperparameters of Facebook Prophet ](https://github.com/ARM-software/mango/blob/master/examples/Prophet_Classifier.ipynb)\n10. [ Tune Hyperparameters of xgboost XGBRegressor ](https://github.com/ARM-software/mango/blob/master/examples/Xgboost_Example.ipynb)\n11. [ Tune Hyperparameters of xgboost XGBClassifier ](https://github.com/ARM-software/mango/blob/master/examples/Xgboost_XGBClassifier.ipynb)\n12. [Tune Hyperparameters of SVM](https://github.com/ARM-software/mango/blob/master/examples/SVM_Example.ipynb)\n13. [ More Examples](https://github.com/ARM-software/mango/tree/master/examples)\n14. [ Contact & Questions ](#contactDetails)\n-->\n\n<a name="setup"></a>\n## 1. Installation\nUsing `pip`:\n```\npip install arm-mango\n```\nFrom source:\n```\n$ git clone https://github.com/ARM-software/mango.git\n$ cd mango\n$ pip3 install .\n```\n\n<!--\n- Mango requires scikit-learn and is developed for python 3, some other packages are installed which required to optimize xgboost classifiers and fbprophet.\n!-->\n\n\n<a name="getting-started"></a>\n## 2. Getting Started\nMango is straightforward to use. Following example minimizes the quadratic function whose input is an integer between -10 and 10.\n\n```python\nfrom mango import scheduler, Tuner\n\n# Search space\nparam_space = dict(x=range(-10,10))\n\n# Quadratic objective Function\n@scheduler.serial\ndef objective(x):\n    return x * x\n\n# Initialize and run Tuner\ntuner = Tuner(param_space, objective)\nresults = tuner.minimize()\n\nprint(f\'Optimal value of parameters: {results["best_params"]} and objective: {results["best_objective"]}\')```\n# => Optimal value of parameters: {\'x\': 0}  and objective: 0\n```\n\n<a name="knnexample"></a>\n## 3. Hyperparameter Tuning Example\n\n```python\nfrom sklearn import datasets\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.model_selection import cross_val_score\n\nfrom mango import Tuner, scheduler\n\n# search space for KNN classifier\'s hyperparameters\n# n_neighbors can vary between 1 and 50, with different choices of algorithm\nparam_space = dict(n_neighbors=range(1, 50),\n                   algorithm=[\'auto\', \'ball_tree\', \'kd_tree\', \'brute\'])\n\n\n@scheduler.serial\ndef objective(**params):\n    X, y = datasets.load_breast_cancer(return_X_y=True)\n    clf = KNeighborsClassifier(**params)\n    score = cross_val_score(clf, X, y, scoring=\'accuracy\').mean()\n    return score\n\n\ntuner = Tuner(param_space, objective)\nresults = tuner.maximize()\nprint(\'best parameters:\', results[\'best_params\'])\nprint(\'best accuracy:\', results[\'best_objective\'])\n# => best parameters: {\'algorithm\': \'ball_tree\', \'n_neighbors\': 11}\n# => best accuracy: 0.9332401800962584\n```\nNote that best parameters may be different but accuracy should be ~ 0.93. More examples are available\nin the `examples` directory ([Facebook\'s Prophet](https://github.com/ARM-software/mango/blob/master/examples/Prophet_Classifier.ipynb),\n[XGBoost](https://github.com/ARM-software/mango/blob/master/examples/Xgboost_XGBClassifier.ipynb), [SVM](https://github.com/ARM-software/mango/blob/master/examples/SVM_Example.ipynb)).\n\n\n<a name="DomainSpace"></a>\n## 4. Search Space\nThe search space defines the range and distribution of input parameters to the objective function.\nMango search space is compatible with scikit-learn\'s parameter space definitions used in\n[RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n or [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html).\nThe search space is defined as a dictionary with keys being the parameter names (string) and values being\nlist of discreet choices, range of integers or the distributions. Example of some common search spaces are:\n\n### Integer\nFollowing space defines `x` as an integer parameters with values in `range(-10, 11)` (11 is not included):\n```python\nparam_space = dict(x=range(-10, 11)) #=> -10, -9, ..., 10\n# you can use steps for sparse ranges\nparam_space = dict(x=range(0, 101, 10)) #=> 0, 10, 20, ..., 100\n```\nIntegers are uniformly sampled from the given range and are assumed to be ordered and treated as continuous variables.\n\n### Categorical\nDiscreet categories can be defined as lists. For example:\n```python\n# string\nparam_space = dict(color=[\'red\', \'blue\', \'green\'])\n# float\nparam_space = dict(v=[0.2, 0.1, 0.3])\n# mixed\nparam_space = dict(max_features=[\'auto\', 0.2, 0.3])\n```\nLists are uniformly sampled and are assumed to be unordered. They are one-hot encoded internally.\n\n### Distributions\nAll the distributions supported by [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/stats.html) are supported.\nIn general, distributions must provide a `rvs` method for sampling.\n\n#### Uniform distribution\nUsing `uniform(loc, scale)` one obtains the uniform distribution on `[loc, loc + scale]`.\n```python\nfrom scipy.stats import uniform\n\n# uniformly distributed between -1 and 1\nparam_space = dict(a=uniform(-1, 2))\n```\n\n#### Log uniform distribution\nWe have added [loguniform](https://github.com/ARM-software/mango/blob/master/mango/domain/distribution.py) distribution by extending the `scipy.stats.distributions` constructs.\nUsing `loguniform(loc, scale)` one obtains the loguniform distribution on <code>[10<sup>loc</sup>, 10<sup>loc + scale</sup>]</code>.\n```python\nfrom mango.domain.distribution import loguniform\n\n# log uniformly distributed between 10^-3 and 10^-1\nparam_space = dict(learning_rate=loguniform(-3, 2))\n```\n\n\n### Hyperparameter search space examples\nExample hyperparameter search space for [Random Forest Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html):\n```python\nparam_space =  dict(\n    max_features=[\'sqrt\', \'log2\', .1, .3, .5, .7, .9],\n    n_estimators=range(10, 1000, 50), # 10 to 1000 in steps of 50\n    bootstrap=[True, False],\n    max_depth=range(1, 20),\n    min_samples_leaf=range(1, 10)\n)\n```\n\n\nExample search space for [XGBoost Classifier](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier):\n```python\nfrom scipy.stats import uniform\nfrom mango.domain.distribution import loguniform\n\nparam_space = {\n    \'n_estimators\': range(10, 2001, 100), # 10 to 2000 in steps of 100\n    \'max_depth\': range(1, 15), # 1 to 14\n    \'reg_alpha\': loguniform(-3, 6),  # 10^-3 to 10^3\n    \'booster\': [\'gbtree\', \'gblinear\'],\n    \'colsample_bylevel\': uniform(0.05, 0.95), # 0.05 to 1.0\n    \'colsample_bytree\': uniform(0.05, 0.95), # 0.05 to 1.0\n    \'learning_rate\': loguniform(-3, 3),  # 0.001 to 1\n    \'reg_lambda\': loguniform(-3, 6),  # 10^-3 to 10^3\n    \'min_child_weight\': loguniform(0, 2), # 1 to 100\n    \'subsample\': uniform(0.1, 0.89) # 0.1 to 0.99\n}\n```\n\nExample search space for [SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html):\n```python\nfrom scipy.stats import uniform\nfrom mango.domain.distribution import loguniform\n\nparam_dict = {\n    \'kernel\': [\'rbf\', \'sigmoid\'],\n    \'gamma\': uniform(0.1, 4), # 0.1 to 4.1\n    \'C\': loguniform(-7, 8) # 10^-7 to 10\n}\n```\n\n<a name="scheduler"></a>\n## 5. Scheduler\n\nMango is designed to take advantage of distributed computing. The objective function can be scheduled to\nrun locally or on a cluster with parallel evaluations. Mango is designed to allow the use of any distributed\ncomputing framework (like Celery or Kubernetes). The `scheduler` module comes with some pre-defined\nschedulers.\n\n### Serial scheduler\nSerial scheduler runs locally with one objective function evaluation at a time\n```python\nfrom mango import scheduler\n\n@scheduler.serial\ndef objective(x):\n    return x * x\n```\n\n### Parallel scheduler\nParallel scheduler runs locally and uses `joblib` to evaluate the objective functions in parallel\n```python\nfrom mango import scheduler\n\n@scheduler.parallel(n_jobs=2)\ndef objective(x):\n    return x * x\n```\n`n_jobs` specifies the number of parallel evaluations. `n_jobs = -1` uses all the available cpu cores\non the machine. See [simple_parallel](https://github.com/ARM-software/mango/tree/master/examples/simple_parallel.py)\nfor full working example.\n\n### Custom distributed scheduler\nUsers can define their own distribution strategies using `custom` scheduler. To do so, users need to define\nan objective function that takes a list of parameters and returns the list of results:\n```python\nfrom mango import scheduler\n\n@scheduler.custom(n_jobs=4)\ndef objective(params_batch):\n    """ Template for custom distributed objective function\n    Args:\n        params_batch (list): Batch of parameter dictionaries to be evaluated in parallel\n\n    Returns:\n        list: Values of objective function at given parameters\n    """\n    # evaluate the objective on a distributed framework\n    ...\n    return results\n```\n\nFor example the following snippet uses [Celery](http://www.celeryproject.org/):\n```python\nimport celery\nfrom mango import Tuner, scheduler\n\n# connect to celery backend\napp = celery.Celery(\'simple_celery\', backend=\'rpc://\')\n\n# remote celery task\n@app.task\ndef remote_objective(x):\n    return x * x\n\n@scheduler.custom(n_jobs=4)\ndef objective(params_batch):\n    jobs = celery.group(remote_objective.s(params[\'x\']) for params in params_batch)()\n    return jobs.get()\n\nparam_space = dict(x=range(-10, 10))\n\ntuner = Tuner(param_space, objective)\nresults = tuner.minimize()\n```\nA working example to tune hyperparameters of KNN using Celery is [here](https://github.com/ARM-software/mango/tree/master/examples/knn_celery.py).\n\n<!--\n<a name="ObjectiveFunction"></a>\n## 5. More on Objective Function\nThe serial objective function has the following structure.\n\n```python\ndef objective_function(params_list):\n    evaluations = []\n    for hyper_par in params_list:\n        result =  evaluate_function_on_hyper_par\n        evaluations.append(result)\n    return evaluations\n```\nThe objective function is called with the input list of hyper parameters. Each element of the list is the dictionary which is a sample drawn from the domain space of variables. Mango expects the objective function to return the list of\nevaluations which has the same size as the args_list. Each value of the evaluations list is the function evaluated at hyperparameters\nof params_list in the same order. A rich set of objective functions are shown in the [examples](https://github.com/ARM-software/mango/tree/master/examples). The size of the params_list is controlled by the batch_size configuration parameter of Mango. By default,\nbatch_size is 1. The configuration parameters of Mango are explained in the [Mango Configurations](#MangoConfigurations) section.\n\nThe sample skeleton of the Celery based parallel objective function in Mango is as following.\n\n```python\ndef objective_celery(params_list):\n    process_queue = []\n    for par in params_list:\n        process = train_clf.delay(par)\n        process_queue.append((process, par))\n    evals = []\n    params = []\n    for process, par in process_queue:\n        result = process.get()\n        evals.append(result)\n        params.append(par)\n    return params, evals\n```\n-->\n\n<a name="MangoConfigurations"></a>\n## 6. Optional configurations\n\nThe default configuration parameters used by the Mango as below:\n```python\n{\'param_dict\': ...,\n \'userObjective\': ...,\n \'domain_size\': 5000,\n \'initial_random\': 1,\n \'num_iteration\': 20,\n \'batch_size\': 1}\n```\nThe configuration parameters are:\n- *domain_size*: The size which is explored in each iteration by the gaussian process. Generally, a larger size is preferred if higher dimensional functions are optimized. More on this will be added with details about the internals of bayesian optimization.\n- *initial_random*: The number of random samples tried. Note: Mango returns all the random samples together. Users can exploit this to parallelize the random runs without any constraint.\n- *num_iteration*: The total number of iterations used by Mango to find the optimal value.\n- *batch_size*: The size of args_list passed to the objective function for parallel evaluation. For larger batch sizes, Mango internally uses intelligent sampling to decide the optimal samples to evaluate.\n- *early_stopping*: A Callable to specify custom stopping criteria. The callback has the following signature:\n   ```python\n  def early_stopping(results):\n      \'\'\'\n          results is the same as dict returned by tuner\n          keys available: params_tries, objective_values,\n              best_objective, best_params\n      \'\'\'\n      ...\n      return True/False\n  ```\n    Early stopping is one of Mango\'s important features that allow to early terminate the current parallel search\n    based on the custom user-designed criteria, such as the total optimization time spent, current validation accuracy\n    achieved, or improvements in the past few iterations. For usage see early stopping examples [notebook](https://github.com/ARM-software/mango/blob/master/examples/EarlyStopping.ipynb).\n- *constraint*: A callable to specify constraints on parameter space. It has the following\n  signature:\n  ```python\n  def constraint(samples: List[dict]) -> List[bool]:\n    \'\'\'\n        Given a list of samples (each sample is a dict with parameter names as keys)\n        Returns a list of True/False elements indicating whether the corresponding sample\n        satisfies the constraints or not\n    \'\'\'\n  ```\n  See [this](<examples/Test functions for constrained optimization.ipynb>) notebook for an example.\n- *initial_custom*: A list of initial evaluation points to warm up the optimizer instead of random sampling. For example, for a search space with two parameters `x1` and `x2` the input could be:   `[{\'x1\': 10, \'x2\': -5}, {\'x1\': 0, \'x2\': 10}]`. This allows the user to customize the initial evaluation points and therefore guide the optimization process. If this option is given then `initial_random` is ignored.  \n\n\nThe default configuration parameters can be modified, as shown below. Only the parameters whose values need to adjusted can be passed as the dictionary.\n\n```python\nconf_dict = dict(num_iteration=40, domain_size=10000, initial_random=3)\n\ntuner = Tuner(param_dict, objective, conf_dict)\n```\n\n<a name="AdditionalFeatures"></a>\n## 7. Additional Features\n### Handling runtime failed evaluation\nAt runtime, failed evaluations are widespread in production deployments. Mango abstractions enable users to make progress even in the presence of failures by only using the correct evaluations. The syntax can return the successful evaluation, and the user can flexibly keep track of failures, for example, using timeouts. Examples showing the usage of Mango in the presence of failures: [serial execution](https://github.com/ARM-software/mango/blob/master/examples/Failure_Handling.ipynb) and [parallel execution](https://github.com/ARM-software/mango/blob/master/examples/Failure_Handling_Parallel.ipynb)\n\n### Neural Architecture Search\nMango can also do an efficient neural architecture search. An example on the MNIST dataset to search for optimal filter sizes, the number of filters, etc., is [available](https://github.com/ARM-software/mango/blob/master/examples/NAS_Mnist.ipynb).\n\nMore extensive examples are available in the [THIN-Bayes](https://github.com/ARM-software/mango/tree/master/examples/THIN-Bayes) folder doing *Neural Architecture Search* for a class of neural networks and classical models for different regression and classification tasks. \n\n<!--\n<a name="Celery"></a>\n## 7. Schedule Objective Function on Celery\nUser-defined objective function can be scheduled on local, cluster or cloud infrastructure. The objective function scheduler\nis entirely independent of the Mango. This design was chosen to enable the scheduling of varied resource objective function according to developer needs. We have included examples using [Celery](http://www.celeryproject.org/). In the sample\nexamples, celery workers are used to evaluate the objective function in parallel. These examples assume celery is installed, and workers\nare running. Default celery configurations can be modified in the [file](https://github.com/ARM-software/mango/blob/master/examples/classifiers/celery.py).\n\n- [KNN example using celery workers](https://github.com/ARM-software/mango/blob/master/examples/KNN_Celery.ipynb)\n- [Prophet example using celery workers](https://github.com/ARM-software/mango/blob/master/examples/Prophet_Celery.ipynb)\n\nMore examples will be included to show the scheduling of objective function using local threads/processes. By default examples schedule\nthe objective function on the local machine itself.\n\n<a name ="mangoAlgorithms"></a>\n## 8. Algorithms\nThe optimization algorithms in Mango are based on widely used Bayesian optimization techniques, extended to sample a batch of configurations in parallel. Currently, Mango provides two parallel optimization algorithms that use the upper confidence bound as the acquisition function. The first algorithm uses hallucination combined with exponential rescaling of the surrogate function to select a batch. In the second algorithm, we create clusters of acquisition function in spatially distinct search spaces, and select the maximum value within each cluster to create the batch.\n\n<a name="contactDetails"></a>\n## More Details\nDetails about specifying parameter/variable domain space, user objective function, and internals of Mango will be added.\nPlease stay tuned.\n-->\n\n<a name="CASHFeature"></a>\n## 8. Combiner Classifier Selection and Optimization (CASH)\nMango now provides a novel functionality of combined classifier selection and optimization. It allows developers to directly specify a set of classifiers along with their different hyperparameter spaces. Mango internally finds the best classifier along with the optimal parameters with the least possible number of overall iterations. The examples are  available [here](https://github.com/ARM-software/mango/tree/master/benchmarking/MetaTuner_Examples)\n\nThe important parts in the skeletion code are as below.\n\n```python\nfrom mango import MetaTuner\n\n#define search spaces and objective functions as done for tuner.\n\nparam_space_list = [param_space1, param_space2, param_space3, param_space4, ..]\nobjective_list = [objective_1, objective_2, objective_3, objective_4, ..]\n\nmetatuner = MetaTuner(param_space_list, objective_list)\n\nresults = metatuner.run()\n\nprint(\'best_objective:\',results[\'best_objective\'])\nprint(\'best_params:\',results[\'best_params\'])\nprint(\'best_objective_fid:\',results[\'best_objective_fid\'])\n\n```\n\n## Participate\n\n<a name="CorePapers"></a>\n### Core Papers to Cite Mango\n\nMore technical details are available in the [Mango paper-1 (ICASSP 2020)](https://arxiv.org/pdf/2005.11394.pdf) and [Mango paper-2 (CogMI 2021)](https://drive.google.com/file/d/1uzcTUfLM3JSc47RLQJin-YzybwNl6BZO/view)\nPlease cite them as:\n```\n@inproceedings{sandha2020mango,\n  title={Mango: A Python Library for Parallel Hyperparameter Tuning},\n  author={Sandha, Sandeep Singh and Aggarwal, Mohit and Fedorov, Igor and Srivastava, Mani},\n  booktitle={ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},\n  pages={3987--3991},\n  year={2020},\n  organization={IEEE}\n}\n```\n```\n@inproceedings{sandha2021mango,\n  title={Enabling Hyperparameter Tuning of Machine Learning Classifiers in Production},\n  author={Sandha, Sandeep Singh and Aggarwal, Mohit and Saha, Swapnil Sayan and Srivastava, Mani},\n  booktitle={CogMI 2021, IEEE International Conference on Cognitive Machine Intelligence},\n  year={2021},\n  organization={IEEE}\n}\n```\n\n<a name="ApplicationPapers"></a>\n### Novel Applications built over Mango\n```\n@article{saha2022auritus,\n  title={Auritus: An open-source optimization toolkit for training and development of human movement models and filters using earables},\n  author={Saha, Swapnil Sayan and Sandha, Sandeep Singh and Pei, Siyou and Jain, Vivek and Wang, Ziqi and Li, Yuchen and Sarker, Ankur and Srivastava, Mani},\n  journal={Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},\n  volume={6},\n  number={2},\n  pages={1--34},\n  year={2022},\n  publisher={ACM New York, NY, USA}\n}\n```\n```\n@article{saha2022tinyodom,\n  title={Tinyodom: Hardware-aware efficient neural inertial navigation},\n  author={Saha, Swapnil Sayan and Sandha, Sandeep Singh and Garcia, Luis Antonio and Srivastava, Mani},\n  journal={Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},\n  volume={6},\n  number={2},\n  pages={1--32},\n  year={2022},\n  publisher={ACM New York, NY, USA}\n}\n```\n```\n@article{saha2022thin,\n  title={THIN-Bayes: Platform-Aware Machine Learning for Low-End IoT Devices},\n  author={Saha, Swapnil Sayan and Sandha, Sandeep Singh and Aggarwal, Mohit and Srivastava, Mani},\n  year={2022}\n}\n```\n\n\n### Slides\n\nSlides explaining Mango abstractions and design choices are available. [Mango Slides-1](https://github.com/ARM-software/mango/blob/master/documents/Mango_github_slides.pdf), [Mango Slides-2](https://drive.google.com/file/d/1_sUOnbW-LkHMMcjq_WgzabN7IQ-wBRgn/view).\n\n### Contribute\n\nPlease take a look at [open issues](https://github.com/ARM-software/mango/issues) if you are looking for areas to contribute to.\n\n### Questions\n\nFor any questions feel free to reach out by creating an issue [here](https://github.com/ARM-software/mango/issues/new).\n',
    'author': 'Sandeep Singh Sandha',
    'author_email': 'sandha.iitr@gmail.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/ARM-software/mango',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.8',
}


setup(**setup_kwargs)
