# ModelBenchX: ML Inference Benchmarking Toolkit

![PyPI](https://img.shields.io/pypi/v/modelbenchx?cacheSeconds=60)
![License](https://img.shields.io/pypi/l/modelbenchx?cacheSeconds=60)
![Python](https://img.shields.io/pypi/pyversions/modelbenchx.svg)

A lightweight, extensible toolkit for benchmarking ML model inference
performance.

ModelBenchX is designed for ML engineers who need quick and reliable
measurements of:

-   Inference latency
-   Throughput (requests/sec)
-   Memory usage
-   Model performance comparison
-   Automated Markdown report generation

------------------------------------------------------------------------

## 🚀 Installation

``` bash
pip install modelbenchx
```

------------------------------------------------------------------------

## 🧩 Requirements

  Component    Version
  ------------ -----------------------
  Python       3.9 -- 3.11
  OS           Linux, macOS, Windows
  Dependency   psutil

ModelBenchX is continuously tested against:

-   Python 3.9\
-   Python 3.10\
-   Python 3.11

------------------------------------------------------------------------

## ⚡ Quick Example

``` python
from modelbench import benchmark_model
import numpy as np

def model(x):
    return x * 2

input_data = np.array([1, 2, 3])

results = benchmark_model(model, input_data, runs=200)

print(results)
```

------------------------------------------------------------------------

## 🔍 Comparing Multiple Models

``` python
from modelbench import compare_models
import numpy as np
import time

def fast_model(x):
    return x

def slow_model(x):
    time.sleep(0.01)
    return x

models = {
    "FastModel": fast_model,
    "SlowModel": slow_model
}

input_data = np.array([1])

results = compare_models(models, input_data, runs=100)

for name, metrics in results.items():
    print(name, metrics)
```

------------------------------------------------------------------------

## 📄 Generate Benchmark Report

``` python
from modelbench.report import generate_markdown_report

generate_markdown_report(results)
```

This creates:

benchmark_report.md

------------------------------------------------------------------------

## 🛠 CLI Usage

Benchmark pickled models directly from the command line:

``` bash
modelbench --model model.pkl --input input.pkl --runs 200
```

------------------------------------------------------------------------

## 📐 Benchmarking Methodology

ModelBenchX follows a consistent benchmarking approach:

-   Warm-up runs to reduce cold-start bias
-   High-resolution timing using `time.perf_counter()`
-   Memory tracking via process RSS measurement
-   Throughput calculated as total runs / total execution time

⚠ Microsecond-level latency measurements may include Python loop
overhead.\
For production-critical benchmarking, batch benchmarking is recommended
(planned in v1.1).

------------------------------------------------------------------------

## 📦 Versioning & Stability

ModelBenchX follows Semantic Versioning:

-   MAJOR → Breaking API changes
-   MINOR → New features
-   PATCH → Bug fixes & improvements

The public API (`benchmark_model`, `compare_models`) is considered
stable as of v1.0.x.

------------------------------------------------------------------------

## 🧪 Testing & CI

ModelBenchX includes automated unit tests covering:

-   Core benchmarking engine\
-   Model comparison logic\
-   Metric structure validation

Continuous Integration runs on:

-   Python 3.9\
-   Python 3.10\
-   Python 3.11

Run tests locally:

``` bash
pip install -e .
pip install pytest
pytest
```

------------------------------------------------------------------------

## 🎯 Ideal Use Cases

-   Model optimization workflows
-   Latency-sensitive systems
-   Edge deployment benchmarking
-   ML infrastructure experimentation
-   Performance regression detection

------------------------------------------------------------------------

## 🗺 Roadmap

### v1.1 (Planned)

-   Batch size benchmarking
-   Peak memory tracking
-   JSON + HTML reporting support
-   Extended framework adapters (PyTorch / TensorFlow)

### Future

-   GPU benchmarking support
-   Visualization utilities
-   Async benchmarking
-   Dockerized benchmarking mode

------------------------------------------------------------------------

## 🤝 Contributing

Contributions are welcome.

Please read CONTRIBUTING.md before submitting pull requests.

------------------------------------------------------------------------

## 📜 License

This project is licensed under the MIT License.
