Metadata-Version: 2.1
Name: mlf-core
Version: 1.11.1
Summary: Reproducible machine learning pipelines using mlflow.
Home-page: https://github.com/mlf-core/mlf-core
Author: Lukas Heumos
Author-email: lukas.heumos@posteo.net
License: Apache2.0
Keywords: mlf-core
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Python: >=3.7
Requires-Dist: cookiecutter (==1.7.2)
Requires-Dist: click (==7.1.2)
Requires-Dist: ruamel.yaml (==0.17.4)
Requires-Dist: pygithub (==1.54.1)
Requires-Dist: GitPython (==3.1.14)
Requires-Dist: autopep8 (>=1.5.4)
Requires-Dist: cffi (==1.14.5)
Requires-Dist: cryptography (==3.4.7)
Requires-Dist: requests (>=2.24.0)
Requires-Dist: rich (==10.1.0)
Requires-Dist: packaging (==20.9)
Requires-Dist: appdirs (==1.4.4)
Requires-Dist: questionary (==1.9.0)
Requires-Dist: pynacl (==1.4.0)
Requires-Dist: mlflow

.. image:: https://user-images.githubusercontent.com/21954664/84388841-84b4cc80-abf5-11ea-83f3-b8ce8de36e25.png
    :target: https://mlf-core.com
    :alt: mlf-core logo

|

========
mlf-core
========

.. image:: https://github.com/mlf-core/mlf-core/workflows/Build%20mlf-core%20Package/badge.svg?branch=master
        :target: https://github.com/mlf-core/mlf-core/actions?query=workflow%3A%22Build+mlf-core+Package%22
        :alt: Github Workflow Build mlf-core Status

.. image:: https://github.com/mlf-core/mlf-core/workflows/Run%20mlf-core%20Tox%20Test%20Suite/badge.svg?branch=master
        :target: https://github.com/mlf-core/mlf-core/actions?query=workflow%3A%22Run+mlf-core+Tox+Test+Suite%22
        :alt: Github Workflow Tests Status

.. image:: https://img.shields.io/pypi/v/mlf-core.svg
        :target: https://pypi.python.org/pypi/mlf-core
        :alt: PyPI Status

.. image:: https://static.pepy.tech/personalized-badge/mlf-core?units=international_system&left_color=grey&right_color=green&left_text=Downloads
        :target: https://pepy.tech/project/mlf-core
        :alt: Pepy Downloads

.. image:: https://img.shields.io/discord/742367395196305489?color=passing
        :target: https://discord.gg/Mv8sAcq
        :alt: Discord

.. image:: https://readthedocs.org/projects/mlf-core/badge/?version=latest
        :target: https://mlf-core.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status

.. image:: https://flat.badgen.net/dependabot/thepracticaldev/dev.to?icon=dependabot
        :target: https://flat.badgen.net/dependabot/thepracticaldev/dev.to?icon=dependabot
        :alt: Dependabot Enabled


Overview
--------

.. figure:: https://user-images.githubusercontent.com/31141763/110704981-02921c80-81f6-11eb-8775-bd73f565568c.png
   :alt: mlf-core soverview

   mlf-core provides CPU and GPU deterministic machine learning templates based on MLflow, Conda, Docker and a strong Github integration.
   Templates are available for PyTorch, TensorFlow and XGBoost.
   A custom linter ensures that projects stay deterministic in all phases of development and deployment.

Installing
---------------

Start your journey with mlf-core by installing it via ``$ pip install mlf-core``.

See `Installation  <https://mlf_core.readthedocs.io/en/latest/readme.html#installing>`_.

config
------
Configure mlf-core to get started.

.. figure:: https://user-images.githubusercontent.com/31141763/102669098-f6199d00-418d-11eb-9ae6-26c12d9c1231.gif

See `Configuring mlf-core <https://mlf_core.readthedocs.io/en/latest/config.html>`_

list
----
List all available mlf-core templates.

.. figure:: https://user-images.githubusercontent.com/31141763/102668939-8d322500-418d-11eb-8b2c-acd895fc50e3.gif

See `Listing all templates <https://mlf_core.readthedocs.io/en/latest/list_info.html#list>`_.

info
----
Get detailed information on a mlf-core template.

.. figure:: https://user-images.githubusercontent.com/31141763/102669191-324cfd80-418e-11eb-9542-d2995b7318a9.gif

See `Get detailed template information <https://mlf_core.readthedocs.io/en/latest/list_info.html#info>`_.

create
------
Kickstart your deterministic machine laerning project with one of mlf-core's templates in no time.

.. figure:: https://user-images.githubusercontent.com/31141763/102669143-1184a800-418e-11eb-853b-0deb0387efc6.gif

See `Create a project <https://mlf_core.readthedocs.io/en/latest/create.html>`_.

lint
----
Use advanced linting to ensure your project always adheres to mlf-core's standards and stays deterministic.

.. image:: https://user-images.githubusercontent.com/31141763/102668893-696edf00-418d-11eb-888e-822244a6f5dc.gif

See `Linting your project <https://mlf_core.readthedocs.io/en/latest/lint.html>`_

bump-version
------------
Bump your project version across several files.

.. figure:: https://user-images.githubusercontent.com/31141763/102668987-aaff8a00-418d-11eb-9292-dc512f77f09b.gif

See `Bumping the version of an existing project  <https://mlf_core.readthedocs.io/en/latest/bump_version.html>`_.

sync
------
Sync your project with the latest mlf-core release to get the latest template features.

.. figure:: https://user-images.githubusercontent.com/31141763/102669065-de421900-418d-11eb-9e1b-a76487d02b2a.gif

See `Syncing a project <https://mlf_core.readthedocs.io/en/latest/sync.html>`_.

upgrade
-------
Check whether you are using the latest mlf-core version and update automatically to benefit from the latest features.

See `<https://mlf_core.readthedocs.io/en/latest/upgrade.html>`_.


Credits
-------

Primary idea and main development by `Lukas Heumos <https://github.com/zethson/>`_. mlf-core is inspired by nf-core_.
This package was created with cookietemple_ based on a modified `audreyr/cookiecutter-pypackage`_ project template using cookiecutter_.

.. _MLflow: https://mlflow.org
.. _cookietemple: https://cookietemple.com
.. _cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
.. _nf-core: https://nf-co.re


.. _changelog_f:

==========
Changelog
==========

This project adheres to `Semantic Versioning <https://semver.org/>`_.

1.11.1 (2021-04-16)
-------------------

**Added**

**Fixed**

* XGBoost cudatoolkit channel

**Dependencies**

**Deprecated**


1.11.0 (2021-03-30)
-------------------

**Added**

**Fixed**

* [MLFLOW-PYTORCH] MNIST data loading (#298)
* Improved console output info in case the repos sync secret could not be set automatically and added it to FAQ docs

**Dependencies**

**Deprecated**

* Removed mlflow-xgboost_dask. It was never deterministic to begin with.
  If you are interested in using mlf-core and XGBoost with multiple GPUs please refer to the old releases.


1.10.0 (2021-03-11)
-------------------

**Added**

* Tutorial accessible at: mlf-core.com/tutorial (#159)
* [ALL TEMPLATES] Removed AUTHORS.rst and moved content into docs/authors.rst (#267)
* [ALL TEMPLATES] Added boto3 to environment to allow for S3 accessibility (#278)
* [ALL TEMPLATES] Workflows, that were triggered when pusing/pr to default master branch do now accept main as default branch

**Fixed**

* Bandit complaining about insecure md5
* mlf-core pytorch template creation and updated its dependencies versions accordingly (#298)
* Pytorch template hparam logging with tensorboard now working
* blacklisted files for sync defined by the user are now correctly picked up
* blacklisted sync files, that are newly introduced by a PR are now included in the sync PR, they were introduced but not in the following ones

**Dependencies**

**Deprecated**


1.9.1 (2021-02-16)
------------------

**Added**

**Fixed**

* [ALL TEMPLATES] lint badge in README.rst

**Dependencies**

**Deprecated**


1.9.0 (2021-02-16)
------------------

**Added**

* Possibility to log input files in all template with the MLFCore object.
* [ALL TEMPLATES] Using new mlf-core/base:1.2.0 container, which is based on CUDA 11.2.1 and cudnn 8.1
* [PYTORCH] Upgraded Pytorch to 1.7.1
* [PYTORCH] Added set_deterministic
* [ALL TEMPLATES] Using new mlflow autolog
* [ALL TEMPLATES] Changed mlflow autolog for loss to every 1 iteration

**Fixed**

* mlf-core fix-artifact-paths does now operate as expected.
* [ALL TEMPLATES] fixed a path error that causes the general template linter to fail searching for
  subprocess.call([\'conda\', \'env\', \'export\', \'--name\', \'<<project_name>>\'], stdout=conda_env_filehandler) and
  mlflow.log_artifact(f\'{{reports_output_dir}}/<<project_name>>_conda_environment.yml\', artifact_path=\'reports\') in the project's mlf_core.py file

**Dependencies**

**Deprecated**


1.8.0 (2021-02-01)
------------------

**Added**

* fixed sync command: blacklisted files now fully working
* updated sync code to the latest version and added logging
* added a output-directory parameter option to create command, allowing users to specifify the
  directory, where the project should be created
* added a user information before the user is prompted to create a GitHub repo with mlf-core
* added flake8 linting for each create template workflow
* GitHub workflow badges in README now link to the corresponding workflow
* [ALL TEMPLATES] added vscode files to .gitignore for templates
* [ALL TEMPLATES] now feature a mlf-core lint workflow with colored linting result output
* [ALL TEMPLATES] renamed train_cpu workflows to use project_slug (was project_slug_no_hyphen)
* [MLFLOW TEMPLATES] refactored the common files for all mlflow templates into a common files directory
* config default values are now preconfigured values (if any)
* added logging to all commands mlf-core offers
* refactored sync, bump-version and create code (added type hints, fixed some variable scoping)
* [ALL TEMPLATES] replaced click with argparse
* [ALL TEMPLATES] renamed some parameters to harmonize between templates
* [PyTorch Template] New template with autologging via pytorch-lightning and mlflow 1.13.1

**Fixed**

* fixed publish_docs WF for all templates working on main or master branch

* fixed gh_pages setup for default branch main

* fixed a bug causing the check upgrade version function to fail if local version is a SNAPSHOT version

**Dependencies**

**Deprecated**


1.7.8 (2020-12-04)
------------------

**Added**

* Instructions to make Docker container public

**Fixed**

**Dependencies**

**Deprecated**


1.7.7 (2020-11-29)
------------------

**Added**

* Support for deploying the documentation on Github Pages. By default the Documentation is pushed to the gh-pages branch.
  Simply enable Github pages (repository settings) with the gh-pages branch and your documentation will build on ``https://username.github.io/repositoryname``

**Fixed**

* Workflows are now also triggered on PR

**Dependencies**

**Deprecated**


1.7.6 (2020-11-22)
------------------

**Added**

**Fixed**

* Github project creation support due to Github's new main branch

**Dependencies**

**Deprecated**

1.7.5 (2020-11-18)
------------------

**Added**

**Fixed**

sync workflow set-env

**Dependencies**

**Deprecated**


1.7.4 (2020-11-11)
------------------

**Added**

**Fixed**

* Sync now compares against the development branch and not the master branch.

**Dependencies**

**Deprecated**


1.7.3 (2020-11-09)
------------------

**Added**

**Fixed**

* Added CHANGELOG.rst to blacklisted files

**Dependencies**

**Deprecated**


1.7.2 (2020-11-07)
------------------

**Added**

**Fixed**

* Removed redundant print in xgboost

**Dependencies**

**Deprecated**


1.7.1 (2020-11-07)
------------------

**Added**

**Fixed**

* mlf-core sync does now correctly find attributes

**Dependencies**

**Deprecated**


1.7.0 (2020-11-06)
------------------

**Added**

* fix-artifact-paths which replaces the artifact paths with the paths of the current system
* More structured documentation

**Fixed**

* Now using GPUs by default only when GPUs are available for XGBoost templates

**Dependencies**

**Deprecated**


1.6.1 (2020-11-06)
------------------

**Added**

* Workflows for package-prediction
* Documentation for package-prediction

**Fixed**

**Dependencies**

**Deprecated**


1.6.0 (2020-11-02)
------------------

**Added**

* New package templates (package-prediction) for Pytorch, Tensorflow and XGBoost

**Fixed**

**Dependencies**

**Deprecated**


1.5.0 (2020-10-29)
------------------

**Added**

* Check for non-deterministic functions for mlflow-tensorflow linter
* Check for all_reduce for mlflow-xgboost templates
* Check for OS for system-intelligence runs. If not Linux -> don't run system-intelligence
* .gitattributes to templates, which ignores mlruns
* Documentation on creating releases

**Fixed**

* Sync now operates correctly with the correct PR URL

**Dependencies**

**Deprecated**


1.4.4 (2020-10-22)
------------------

**Added**

**Fixed**

* Conda report generation

**Dependencies**

**Deprecated**


1.4.3 (2020-09-17)
------------------

**Added**

**Fixed**

* Internal Github workflows
* Docker documentation

**Dependencies**

**Deprecated**

1.4.2 (2020-09-11)
------------------

**Added**

**Fixed**

* Accidentally left a - in the train_cpu.yml of mlflow-pytorch
* mlflow-pytorch and mlflow-tensorflow now only train for 2 epochs on train_cpu.yml

**Dependencies**

**Deprecated**


1.4.1 (2020-09-10)
------------------

**Added**

**Fixed**

* Github username must now always be lowercase, since Docker does not like uppercase letters
* Fixed train_cpu workflows to use the correct containers

**Dependencies**

**Deprecated**

1.4.0 (2020-08-28)
------------------

**Added**

* model.rst documentation for all templates
* added support for verbose output

**Fixed**

* Publish Docker workflows now use the new Github registry
* Default Docker container names are now   ```image: ghcr.io/{{ cookiecutter.github_username }}/{{ cookiecutter.project_slug_no_hyphen }}:{{ cookiecutter.version }}```

**Dependencies**

**Deprecated**


1.3.0 (2020-08-27)
------------------

**Added**

* automatically mounting /data now in all mlflow templates (#56)
* mlflow-xgboost xgboost from 1.1.1 to 1.2.0

**Fixed**

* mlf_core.py now uses project_slug; adapted linter accordingly (#55)
* Removed dask-cuda from mlflow-xgboost

**Dependencies**

**Deprecated**


1.2.2 (2020-08-21)
------------------

**Added**

**Fixed**

* A couple of parameters were not with hyphen -> now default behavior

**Dependencies**

**Deprecated**


1.2.1 (2020-08-21)
------------------

**Added**

**Fixed**

* flake8 for mlflow-pytorch

**Dependencies**

**Deprecated**


1.2.0 (2020-08-21)
------------------

**Added**

* Option --view to config to view the current configuration
* Option --set_token to sync to set the sync token again

**Fixed**

* #41 https://github.com/mlf-core/mlf-core/issues/41 -> mlflow-pytorch multi GPU Support

**Dependencies**

**Deprecated**


1.1.0 (2020-08-19)
------------------

**Added**

* Publish Docker workflow. Publishes to Github Packages per default, but can be configured.
* Linting function, which checks mlflow-pytorch for any used atomic_add functions.
* system-intelligence 1.2.2 -> 1.2.3
* Support for both, MLF-CORE TODO: and TODO MLF-CORE: statements

**Fixed**

* Default project version from 0.1.0 to 0.1.0-SNAPSHOT.
* Outdated screenshots
* Nightly versions now warn instead of wrongly complaining about outdated versions.
* Sync actor, but not yet completely for organizations
* A LOT of documentation
* Now using project_slug_no_hyphen to facilitate the creation of repositories with - characters.
* Removed boston dataset from XGBoost and XGBoost_dask
* Renamed all parameters to use hyphens instead of underscores

**Dependencies**

**Deprecated**


1.0.1 (2020-08-11)
------------------

**Added**

**Fixed**

* Sync workflow now uses the correct secret

**Dependencies**

**Deprecated**


1.0.0 (2020-08-11)
------------------

**Added**

* Created the project using cookietemple
* Added all major commands: create, list, info, lint, sync, bump-version, config, upgrade
* Added mlflow-pytorch, mlflow-tensorflow, mlflow-xgboost, mlflow-xgboost_dask templates

**Fixed**

**Dependencies**

**Deprecated**


