Metadata-Version: 2.1
Name: metricx
Version: 0.0.2
Summary: A library for managing, exploring, and analyzing benchmark data.
Home-page: https://github.com/k15z/metricx
Author: Kevin Alex Zhang
Author-email: kevz@mit.edu
License: MIT license
Keywords: metricx
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: bokeh (>=1.4.0)
Requires-Dist: matplotlib (>=3.1.0)
Requires-Dist: numpy (>=1.19.0)
Requires-Dist: pandas (>=1.1.0)
Requires-Dist: scipy (>=1.2.0)
Requires-Dist: statsmodels (>=0.12.0)
Provides-Extra: all
Requires-Dist: bokeh (>=1.4.0) ; extra == 'all'
Requires-Dist: matplotlib (>=3.1.0) ; extra == 'all'
Requires-Dist: numpy (>=1.19.0) ; extra == 'all'
Requires-Dist: pandas (>=1.1.0) ; extra == 'all'
Requires-Dist: scipy (>=1.2.0) ; extra == 'all'
Requires-Dist: statsmodels (>=0.12.0) ; extra == 'all'
Requires-Dist: pytest-runner (>=5.2) ; extra == 'all'
Requires-Dist: black (>=19.10b0) ; extra == 'all'
Requires-Dist: codecov (>=2.1.4) ; extra == 'all'
Requires-Dist: flake8 (>=3.8.3) ; extra == 'all'
Requires-Dist: flake8-debugger (>=3.2.1) ; extra == 'all'
Requires-Dist: pytest (>=5.4.3) ; extra == 'all'
Requires-Dist: pytest-cov (>=2.9.0) ; extra == 'all'
Requires-Dist: pytest-raises (>=0.11) ; extra == 'all'
Requires-Dist: bumpversion (>=0.6.0) ; extra == 'all'
Requires-Dist: coverage (>=5.1) ; extra == 'all'
Requires-Dist: ipython (>=7.15.0) ; extra == 'all'
Requires-Dist: m2r (>=0.2.1) ; extra == 'all'
Requires-Dist: nbsphinx (>=0.6.1) ; extra == 'all'
Requires-Dist: Sphinx (<3,>=2.0.0b1) ; extra == 'all'
Requires-Dist: sphinx-rtd-theme (>=0.4.3) ; extra == 'all'
Requires-Dist: tox (>=3.15.2) ; extra == 'all'
Requires-Dist: twine (>=3.1.1) ; extra == 'all'
Requires-Dist: wheel (>=0.34.2) ; extra == 'all'
Provides-Extra: dev
Requires-Dist: pytest-runner (>=5.2) ; extra == 'dev'
Requires-Dist: black (>=19.10b0) ; extra == 'dev'
Requires-Dist: codecov (>=2.1.4) ; extra == 'dev'
Requires-Dist: flake8 (>=3.8.3) ; extra == 'dev'
Requires-Dist: flake8-debugger (>=3.2.1) ; extra == 'dev'
Requires-Dist: pytest (>=5.4.3) ; extra == 'dev'
Requires-Dist: pytest-cov (>=2.9.0) ; extra == 'dev'
Requires-Dist: pytest-raises (>=0.11) ; extra == 'dev'
Requires-Dist: bumpversion (>=0.6.0) ; extra == 'dev'
Requires-Dist: coverage (>=5.1) ; extra == 'dev'
Requires-Dist: ipython (>=7.15.0) ; extra == 'dev'
Requires-Dist: m2r (>=0.2.1) ; extra == 'dev'
Requires-Dist: nbsphinx (>=0.6.1) ; extra == 'dev'
Requires-Dist: Sphinx (<3,>=2.0.0b1) ; extra == 'dev'
Requires-Dist: sphinx-rtd-theme (>=0.4.3) ; extra == 'dev'
Requires-Dist: tox (>=3.15.2) ; extra == 'dev'
Requires-Dist: twine (>=3.1.1) ; extra == 'dev'
Requires-Dist: wheel (>=0.34.2) ; extra == 'dev'
Provides-Extra: setup
Requires-Dist: pytest-runner (>=5.2) ; extra == 'setup'
Provides-Extra: test
Requires-Dist: black (>=19.10b0) ; extra == 'test'
Requires-Dist: codecov (>=2.1.4) ; extra == 'test'
Requires-Dist: flake8 (>=3.8.3) ; extra == 'test'
Requires-Dist: flake8-debugger (>=3.2.1) ; extra == 'test'
Requires-Dist: pytest (>=5.4.3) ; extra == 'test'
Requires-Dist: pytest-cov (>=2.9.0) ; extra == 'test'
Requires-Dist: pytest-raises (>=0.11) ; extra == 'test'

# MetricX

[![Build Status](https://github.com/k15z/MetricX/workflows/Build%20Main/badge.svg)](https://github.com/k15z/MetricX/actions)
[![Documentation](https://github.com/k15z/MetricX/workflows/Documentation/badge.svg)](https://k15z.github.io/MetricX)
[![Code Coverage](https://codecov.io/gh/k15z/MetricX/branch/main/graph/badge.svg)](https://codecov.io/gh/k15z/MetricX)
[![PyPI version](https://badge.fury.io/py/metricx.svg)](https://badge.fury.io/py/metricx)

A library for managing, exploring, and analyzing benchmark data. Given a set of tasks
and a set of models which can be evaluated on the tasks, `MetricX` provides a suite 
of features including:

 - Monitoring and logging of modeling results.
 - Export to CSV, matplotlib, bokeh, and more!
 - Smart selection of the next model to evaluate.
 - Interactive visualization in Jupyter notebooks.
 - Interactive HTML reports (deployable via Github Pages).

---

## Quick Start

### Install MetricX
You can install the latest stable release

```bash
pip install metricx
```
Or you can install the development head

```bash
pip install git+https://github.com/k15z/MetricX.git
```

### Define your task
First, you'll want to define your task(s). Every task has a name and a
set of metrics.

```python
from metricx import Metric, Task

task = Task(
    name="mnist-digits",
    metrics=[
        Metric(name="accuracy", is_higher_better=True),
        Metric(name="fit-time", is_higher_better=False),
        Metric(name="predict-time", is_higher_better=False),
    ],
)
```

### Report your results
Then, you can report your results by providing (1) the name of the model
being evaluated and (2) a results dictionary containing a value for each of
the metrics specified earlier.

```python
task.report("convnet", {
  "accuracy": 1.0, 
  "fit-time": 100.0,
  "predict-time": 3.0,
})
```

```python
task.report("logistic-regression", {
  "accuracy": 0.6, 
  "fit-time": 10.0,
  "predict-time": 1.0,
})
```

### Generate plots and rankings
The `Task` object provides numerous functionality from generating plots 
to ranking models.

```python
task.to_bokeh() # generate a Bokeh plot
task.to_figure() # generate a matplotlib Figure
task.rank() # return a ranking of models
```

### Combine multiple tasks
If you have multiple tasks, as is typical in a benchmarking scenario, you 
can use the `TaskGrid` to wrap them together and generated combined 
visualizations and more.

```python
from metricx import TaskGrid

grid = TaskGrid([task])
grid.to_html("benchmark.html")
```

This will create a standalone HTML file which allows you to interactively
explore the benchmark results.


