### Challenge

Reproduce and automate [TinyMLPerf benchmarks](https://github.com/mlcommons/tiny).

### Organizers

* [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
* [cTuning foundation](https://cTuning.org)
* [cKnowledge Ltd](https://cKnowledge.org)

### Status

We have successfully reproduced [TinyMLPerf v1.0 submission with microTVM on the STMicroelectronics NUCLEO-L4R5ZI board](https://github.com/mlcommons/tiny_results_v1.0/tree/main/closed/OctoML),
automated it with the latest version of the [MLCommons CM automation language](https://github.com/mlcommons/ck/blob/master/docs/README.md),
submit reproduce results to the TinyMLperf v1.1 round,
and added all past TinyMLPerf results to the [MLCommons CK playground](https://access.cknowledge.org/playground/?action=experiments&tags=mlperf-tiny)
for further collaborative analysis and improvement.

Pleases check our tutorial and reproducibility report:
* [Automate TinyMLPerf benchmark](https://github.com/ctuning/mlcommons-ck/blob/master/docs/tutorials/automate-mlperf-tiny.md) - useful for all SW/HW stacks and submission rounds.
* [Reproduce TinyMLPerf v1.0 submission](https://github.com/ctuning/mlcommons-ck/blob/master/docs/tutorials/reproduce-mlperf-tiny.md).

TinyMLPerf v1.1 results will be published at te [MLCommons CK playground](https://access.cknowledge.org/playground/?action=experiments&tags=mlperf-tiny)
in mid June 2023.

### Related discussions for the future

* https://github.com/mlcommons/ck/pull/693
* https://github.com/mlcommons/ck/pull/700
* https://github.com/mlcommons/ck/pull/701
* https://github.com/mlcommons/ck/issues/606

### Results

All results will be available in [this GitHub repo](https://github.com/ctuning/cm4mlperf-results)
and can be visualized and compared using the [MLCommons Collective Knowledge Playground](https://access.cknowledge.org/playground/?action=experiments&tags=mlperf-tiny).
