# CZ Benchmarks

### What is cz-benchmarks?
cz-benchmarks is a package for standardized evaluation and comparison of machine learning models for biological applications (first, in the single-cell transcriptomics domain, with future plans to expand to additional domains). The package provides a toolkit for running containerized models, executing biologically-relevant tasks, and computing performance metrics. We see this tool as a step towards ensuring that large-scale ML Models can be harnessed to deliver genuine biological insights -- by building trust, accelerating development, and bridging the gap between ML and biology communities.

### Why benchmarking? Why now?
Last year, CZI hosted a workshop focused on benchmarking and evaluation of ML Models in biology, and the [insights gained](https://virtualcellmodels.cziscience.com/micro-pub/benchmarking-workshop) have reinforced our commitment to supporting the development of a robust benchmarking infrastructure, which we see as critical to achieving our Virtual Cell vision.

### 💬 Community Feedback & Contributions
We're working to get the alpha version of cz-benchmarks stable to build with the community. In the meantime, for issues you may identify, feel free to open an issue on GitHub or reach out to us at [virtualcellmodels@chanzuckerberg.com](mailto:virtualcellmodels@chanzuckerberg.com).


## Table of contents

### Getting Started
- [Quick Start Guide](docs/source/quick_start.md)

### How-To Guides
- [Add a Custom Dataset](docs/source/how_to_guides/add_custom_dataset.md)
- [Add a New Metric](docs/source/how_to_guides/add_new_metric.md)
- [Add a New Task](docs/source/how_to_guides/add_new_task.md)
- [Interactive Mode](docs/source/how_to_guides/interactive_mode.md)
- [Visualize Results](docs/source/how_to_guides/visualize_results.md)

### Developer Guides
- [Datasets](docs/source/developer_guides/datasets.md)
- [Metrics](docs/source/developer_guides/metrics.md)
- [Tasks](docs/source/developer_guides/tasks.md)
- [Debugging](docs/source/developer_guides/debugging.md)
- [Release Process](docs/source/developer_guides/release_process.md)
- [Design Overview](docs/source/developer_guides/design_overview.rst)

### Assets
- [Assets](docs/source/assets.md)

### Policies
- [Definitions](docs/source/policy/definitions.md)
- [Code of Conduct](CODE_OF_CONDUCT.md)

### Additional Resources
- [Changelog & Release Notes](CHANGELOG.md)
- [Roadmap](docs/source/roadmap.md)
- [Legal](LICENSE.md)
- [Security](SECURITY.md)

---

## 📚 Documentation

Full documentation: [cz-benchmarks website](https://chanzuckerberg.github.io/cz-benchmarks/)

---

## 📦 PyPI

Find the package on [PyPI](https://pypi.org/project/cz-benchmarks/)