# cimetrics

[![Build Status](https://dev.azure.com/jumaffre/metrics-devops/_apis/build/status/jumaffre.metrics-devops?branchName=master)](https://dev.azure.com/jumaffre/metrics-devops/_build/latest?definitionId=1&branchName=master)

`cimetrics` lets you track crucial metrics to avoid unwanted regressions. It is easy to integrate with your existing projects and automatically provides quick feedback in your GitHub Pull Requests. See it in action [here](https://github.com/jumaffre/cimetrics/pull/28#issuecomment-519552043). 

<p align="center"> <img src="https://raw.githubusercontent.com/jumaffre/cimetrics/master/cimetrics_demo.PNG" width="600"></p>


## Development

Install as editable/dev package:
```sh
pip install -e .
```

Submit sample metrics and create plots:
```sh
export METRICS_MONGO_CONNECTION=...
python app/main.py
python -m cimetrics.plot
ls _cimetrics
```

## Using cimetrics in your own project

### Setup storage

Metrics data are stored by in any MongoDB-compatible database.

An easy way to get storage set up is to spin up a [Cosmos DB](https://docs.microsoft.com/en-us/azure/cosmos-db/introduction) instance in Azure. The connection string should be stored as the `METRICS_MONGO_CONNECTION` secret variable in your CI system.

### Pushing metrics from your tests

You can use the simple python API to push your metrics to your storage. First, make sure to install the `cimetrics` python module (for example, by running `pip install cimetrics` or adding `cimetrics` to your `requirements.txt`). Once this is done, you will be able to write:

```python
import cimetrics.upload

metrics = cimetrics.upload.Metrics()

# Run some tests and collect some data

metrics.put("metric1 name (unit)", metric_1)
metrics.put("metric2 name (unit)", metric_2)

metrics.publish()
```

Note that `metric_1` and `metric_2` should _not_ be string but integer or float.

### Setup the CI

Your CI is responsible for rendering the metrics report and posting them to your Pull Requests in GitHub. For this, you should create a [personal authentication token](https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line) with Write access to the repository for the account you want to post on behalf of `cimetrics`. Then, you should set up the token as the `GITHUB_TOKEN` secret variable in your CI system. Don't forget to add that user as a personal contributor (Write access) to your Github repository as well.

Then, you should add the following steps to your CI configuration file, e.g. for Azure Pipelines:

```yaml
# Plot the metrics on the branch against the target branch (e.g. master)
- script: python -m cimetrics.plot
  env:
    METRICS_MONGO_CONNECTION: $(METRICS_MONGO_CONNECTION)
  displayName: 'Plot cimetrics'

# Publish the rendered plot on the Github PR
# (only run for builds triggered by Pull Requests)
- script: python -m cimetrics.github_pr
  env:
    GITHUB_TOKEN: $(GITHUB_TOKEN)
  displayName: 'Post cimetrics graphs as PR comment'
  condition: eq(variables['Build.Reason'], 'PullRequest')

# Publish plot as a build artifact (optional)
- task: PublishBuildArtifacts@1
  inputs:
    pathtoPublish: _cimetrics
    artifactName: cimetrics
  displayName: 'Publish cimetrics graphs as build artifact'
```

See [azure-pipelines.yml](https://github.com/jumaffre/cimetrics/blob/master/azure-pipelines.yml) for a full working example.

### Create the `metrics.yml` file

The last step is to create a new `metrics.yml` configuration file at the root of your repository. The file should specify the name of the database and collection used for MongoDB. For example:

```yaml
db: 'metrics'
collection: 'metrics_performance'
```

That's it! The next time you create a Pull Request, your CI will automatically store your metrics and publish a graph comparing your metrics against the same metrics on the branch you are merging to.

## Caveats

- If the CI has never run on the target branch (e.g. `master` - likely to happen when you first set up `cimetrics`), the report will show the metrics results of the branch to merge against itself. It will be displayed fine once the CI has run on the target branch.
- The rendered images are currently hosted in the target GitHub repository itself, under the `cimetrics` branch, in the `cimetrics` directory.

## to-do

- [x] Publish to PyPi (https://pypi.org/project/cimetrics/)
- [ ] Use a GitHub bot instead of tokens
- [ ] Improve plotting (standard deviation, etc.)
- [ ] Add personalised PR message

## Supported CI pipelines

CI Metrics currently supports [Azure Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/), but it should be very easy to add support for other build pipelines by [subclassing GitEnv](https://github.com/jumaffre/cimetrics/blob/master/cimetrics/env.py#L72) and providing the right attributes.

