# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['yamlett']

package_data = \
{'': ['*']}

install_requires = \
['cloudpickle>=1.6.0,<2.0.0',
 'pymongo>=3.11.1,<4.0.0',
 'python-box[all]>=5.2.0,<6.0.0']

setup_kwargs = {
    'name': 'yamlett',
    'version': '0.0.3',
    'description': 'Yet Another ML Experiment Tracking Tool',
    'long_description': '# yamlett - Yet Another Machine Learning Experiment Tracking Tool\n\n1.  [What is `yamlett`?](#what-is-yamlett)\n2.  [Installation](#installation)\n3.  [Getting started](#org7e0759d)\n4.  [Example](#example)\n    1.  [Set up the experiment](#set-up-experiment)\n    2.  [MLflow-like tracking](#mlflow-like-tracking)\n    3.  [`yamlett` tracking](#yamlett-like-tracking)\n\n![PyPI](https://img.shields.io/pypi/v/yamlett)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/yamlett)\n![PyPI - License](https://img.shields.io/pypi/l/yamlett)\n\n\n<a id="what-is-yamlett"></a>\n\n## What is `yamlett`?\n\n`yamlett` provides a simple but flexible way to track your ML experiments.\n\nIt has a simple interface with only two primitives: `Run` and `Experiment`.\n\n-   A `Run` is used to store information about one iteration of your `Experiment`. You can use it to record any ([BSON](http://bsonspec.org)-serializable) information you want such as model parameters, metrics, or pickled artifacts.\n-   An `Experiment` is a collection of `Run` objects. It has a `name` and it is a wrapper around a `pymongo.collection.Collection` object ([reference](https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection)), meaning that you can query it using `find` or `aggregate`. Think of it as a way to collect all the modeling iterations for a specific project.\n\nThe main difference with other tracking tools (e.g. MLflow) is that `yamlett` lets you save complex structured information using dictionaries or lists and filter on them later using MongoDB queries.\n\n`yamlett` is particularly useful if your experiments are configuration-driven. Once your configuration is loaded as a python object, storing it is as easy as `run.store("config", config)`.\n\n\n<a id="installation"></a>\n\n## Installation\n\n`yamlett` can be installed with `pip`:\n\n```sh\npip install yamlett\n```\n\nIt also requires a MongoDB instance that you can connect to. If you don&rsquo;t have one and just want to try out `yamlett`, we provide a [docker compose file](docker-compose.yaml) that starts a MongoDB instance available at `localhost:27017` (along with instances of [Presto](https://prestodb.io) and [Metabase](https://www.metabase.com)).\n\n\n<a id="org7e0759d"></a>\n\n## Getting started\n\nIn `yamlett`, `MongoClient` [connection parameters](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient) can be passed as keyword arguments in both `Run` and `Experiment` to specify what MongoDB instance you want to connect to. If you don&rsquo;t pass anything, the default arguments (`localhost:27017`) will be used. If you have a custom MongoDB instance, you can specify its `host` and `port` when creating a `Run` using `run = Run(host="mymongo.host.com", port=27017)`.\n\n\n<a id="example"></a>\n\n## Example\n\nIn this section, we compare the same model run but with two different tracking different approaches: MLflow-like vs yamlett.\n\n\n<a id="set-up-experiment"></a>\n\n### Set up the experiment\n\nFirst, let&rsquo;s load a dataset for a simple classification problem that ships with scikit-learn.\n\n```python\nfrom sklearn.datasets import load_iris\n\nX, y = load_iris(return_X_y=True)\n```\n\nThen, we create a logistic regression model and train that model on the iris dataset, increasing the number of iterations and changing the regularization strength.\n\n```python\nfrom sklearn.linear_model import LogisticRegression\n\nmodel = LogisticRegression(max_iter=200, C=0.1)\nmodel.fit(X, y)\n```\n\n\n<a id="mlflow-like-tracking"></a>\n\n### MLflow-like tracking\n\nWith `yamlett`, you are free to organize you tracking information so you could decide to store it using a &ldquo;flat&rdquo; approach similar to MLflow where each key has an associated value and there can be no nesting.\n\n```python\nfrom yamlett import Run\nfrom sklearn.metrics import f1_score\n\nrun = Run()\n\n# store some information about your trained model: its class and its parameters\nrun.store("params_model_class", model.__class__.__name__)\nfor param_name, param_value in model.get_params().items():\n    run.store(f"params_model_{param_name}", param_value)\n\n# store information about your data\nrun.store("data_n_features", X.shape[0])\nrun.store("data_n_observations", X.shape[1])\n\n# store the F1 score on the train data\nrun.store("metrics_train_f1_score", f1_score(y, model.predict(X), average="weighted"))\n\n# you could even store a pickled version of your model\n# run.store("model", pickle.dumps(model))\n```\n\nAfter running this code, we can retrieve the stored information by calling `run.data`:\n\n    {\'_id\': \'901c6823493d429cae4ddb84c91a7768\',\n     \'_yamlett\': {\'created_at\': datetime.datetime(2020, 12, 5, 21, 36, 14, 17000),\n                  \'last_modified_at\': datetime.datetime(2020, 12, 5, 21, 36, 14, 461000)},\n     \'data_n_features\': 150,\n     \'data_n_observations\': 4,\n     \'metrics_train_f1_score\': 0.9599839935974389,\n     \'params_model_C\': 0.1,\n     \'params_model_class\': \'LogisticRegression\',\n     \'params_model_class_weight\': None,\n     \'params_model_dual\': False,\n     \'params_model_fit_intercept\': True,\n     \'params_model_intercept_scaling\': 1,\n     \'params_model_l1_ratio\': None,\n     \'params_model_max_iter\': 200,\n     \'params_model_multi_class\': \'auto\',\n     \'params_model_n_jobs\': None,\n     \'params_model_penalty\': \'l2\',\n     \'params_model_random_state\': None,\n     \'params_model_solver\': \'lbfgs\',\n     \'params_model_tol\': 0.0001,\n     \'params_model_verbose\': 0,\n     \'params_model_warm_start\': False}\n\nThis approach is straightforward: one scalar for each key in the document. However, one downside is that you need to maintain your own namespace convention. For example here, we used underscores to separate the different levels of information (params, data, metrics, etc) but this can quickly get confusing if chosen incorrectly: is it `params/model/fit_intercept` or `params/model_fit/intercept`? It is also more work than needed when information already comes nicely organized (e.g. `model.get_params()`).\n\n\n<a id="yamlett-like-tracking"></a>\n\n### `yamlett` tracking\n\nThe method we propose in this package leverages Python dictionaries / NoSQL DB documents to automatically store your information in a structured way. Let&rsquo;s see what it looks like using the same run as above:\n\n```python\nfrom yamlett import Run\nfrom sklearn.metrics import f1_score\n\nrun = Run()\n\n# store your model information\nmodel_info = {\n    "class": model.__class__.__name__,\n    "params": model.get_params(),\n}\nrun.store(f"model", model_info)\n\n# store information about your data\nrun.store("data", {"n_features": X.shape[0], "n_observations": X.shape[1]})\n\n# store the F1 score on your train data\nrun.store("metrics.f1_score", f1_score(y, model.predict(X), average="weighted"))\n\n# you could even store a pickled version of your model\n# run.store("model.artifact", pickle.dumps(model))\n```\n\nOnce again, let&rsquo;s call `run.data` and see what information we stored:\n\n    {\'_id\': \'b7736c7b3cc3439ca379e3e6a2b6d9b8\',\n     \'_yamlett\': {\'created_at\': datetime.datetime(2020, 12, 5, 22, 43, 2, 446000),\n                  \'last_modified_at\': datetime.datetime(2020, 12, 5, 22, 43, 2, 529000)},\n     \'data\': {\'n_features\': 150, \'n_observations\': 4},\n     \'metrics\': {\'f1_score\': 0.9599839935974389},\n     \'model\': {\'class\': \'LogisticRegression\',\n               \'params\': {\'C\': 0.1,\n                          \'class_weight\': None,\n                          \'dual\': False,\n                          \'fit_intercept\': True,\n                          \'intercept_scaling\': 1,\n                          \'l1_ratio\': None,\n                          \'max_iter\': 200,\n                          \'multi_class\': \'auto\',\n                          \'n_jobs\': None,\n                          \'penalty\': \'l2\',\n                          \'random_state\': None,\n                          \'solver\': \'lbfgs\',\n                          \'tol\': 0.0001,\n                          \'verbose\': 0,\n                          \'warm_start\': False}}}\n\nThe run information is now stored in a document that can be easily parsed based on its structure. The top level keys of the document are `data`, `metrics`, and `model` and we argue this makes it easier to find information than with long keys in a flat dictionary. For instance, you may want to look at all the metrics for a given run using `run.data["metrics"]`.\n\n    {\'f1_score\': 0.9599839935974389}\n\nNote that `yamlett` does not impose the document hierarchy so you are free to organize your run data as you see fit. Additionally, because `yamlett` is a light abstraction layer on top of MongoDB, you can query runs in an `Experiment` using `find` or `aggregate`. For example, we can retrieve all runs in the default experiment for which:\n\n1.  the model was fit with a bias term\n2.  on a dataset with at least 3000 data points\n3.  that yielded an F1 score of at least 0.9\n\n```python\nfrom yamlett import Experiment\n\ne = Experiment()\n\ne.find(\n    {\n        "model.params.fit_intercept": True,\n        "data.n_observations": {"$gte": 3000},\n        "metrics.f1_score": {"$gte": 0.9},\n    }\n)\n```\n',
    'author': 'Virgile Landeiro',
    'author_email': 'virgile.landeiro@gmail.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/vlandeiro/yamlett',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)
