# Benchmarks

`turboagents` currently ships lightweight synthetic benchmark surfaces that are
safe to run on a development laptop. They are intended to validate the package
surface, compare bit-width trends, and catch regressions before heavier
benchmarking on larger hardware.

## Commands

```bash
turboagents bench kv
turboagents bench kv --format json
turboagents bench rag --format markdown
turboagents bench paper
```

## Current Built-in Datasets

- `tiny-kv`: small deterministic vector batch for KV-style reconstruction tests
- `tiny-rag`: small deterministic retrieval set for TurboRAG sanity checks
- `medium-rag`: reserved for a larger synthetic retrieval pass
- `paper-sim`: synthetic paper-style MSE comparison set

These are generated by `turboagents/bench/datasets.py`.

## What The Metrics Mean

- `mean_payload_bytes`: average serialized payload size for the compressed vector
- `mse`: mean squared reconstruction error after quantize/dequantize
- `mean_cosine_similarity`: cosine agreement between original and reconstructed vectors
- `ip_mae`: mean absolute error for inner products against a held-out query
- `recall_at_1`, `recall_at_10`: retrieval agreement against exact dot-product ranking

## What These Benchmarks Are Not

- They are not LongBench, Needle, BEIR, or MTEB.
- They are not end-to-end serving numbers for real models.
- They are not the final paper reproduction harness.

Those larger evaluations should run on hardware with the required memory,
runtimes, and datasets installed.

## Full Benchmark Workflow

The repository includes a reproducible benchmark harness for fuller runs:

```bash
uv sync --extra rag --extra mlx
uv run python scripts/run_benchmark_matrix.py --output-dir benchmark-results/$(date +%Y%m%d-%H%M%S)
```

Optional real MLX run:

```bash
uv run python scripts/run_benchmark_matrix.py \
  --mlx-model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --output-dir benchmark-results/$(date +%Y%m%d-%H%M%S)
```

Optional live pgvector validation:

```bash
uv run python scripts/run_benchmark_matrix.py \
  --pgvector-dsn postgresql://localhost/turboagents \
  --output-dir benchmark-results/$(date +%Y%m%d-%H%M%S)
```

Supporting scripts:

- `scripts/run_benchmark_matrix.py`
- `scripts/benchmark_mlx.py`
- `scripts/benchmark_rag_adapters.py`
- `scripts/benchmark_needle.py`
- `scripts/summarize_benchmark_results.py`
- `benchmarks/README.md`

## Latest Benchmark Run

Latest artifact set:

- `benchmark-results/20260326-128gb-run/summary.md`
- `benchmark-results/20260326-128gb-run/mlx-benchmark.json`
- `benchmark-results/20260326-128gb-run/rag-adapters.json`
- `benchmark-results/20260327-chroma-local/rag-adapters.json`

### MLX Sweep

Model:

- `mlx-community/Llama-3.2-3B-Instruct-4bit`

Observed behavior:

- `2.5` bits was the fastest run at about `152 tok/s`
- `3.5` and `4.0` bits were close behind at about `139 tok/s`
- `2.0` and `2.5` produced visibly worse completions than `3.0+`
- `3.5` is the best quality/performance operating point from this run

| Bits | Elapsed Seconds | Completion Tokens | Tokens / Second |
| --- | --- | --- | --- |
| 2.0 | 1.3136 | 129 | 98.2015 |
| 2.5 | 0.8479 | 129 | 152.1374 |
| 3.0 | 0.4926 | 64 | 129.9294 |
| 3.5 | 0.4606 | 64 | 138.9408 |
| 4.0 | 0.4630 | 64 | 138.2228 |

### RAG Adapter Matrix

Dataset:

- `medium-rag`
- `1024` base vectors
- `32` queries
- `128` dimensions

FAISS results:

- `recall@1 = 1.0` across all tested bit-widths
- `recall@10 = 1.0` across all tested bit-widths
- query time stayed around `0.0015s` to `0.0023s`

| Adapter | Bits | Build Seconds | Query Seconds | Recall@1 | Recall@10 |
| --- | --- | --- | --- | --- | --- |
| FAISS | 2.0 | 1.6098 | 0.002290 | 1.0 | 1.0 |
| FAISS | 2.5 | 2.0668 | 0.001462 | 1.0 | 1.0 |
| FAISS | 3.0 | 2.7302 | 0.001497 | 1.0 | 1.0 |
| FAISS | 3.5 | 4.0106 | 0.001527 | 1.0 | 1.0 |
| FAISS | 4.0 | 4.7658 | 0.001499 | 1.0 | 1.0 |

Chroma results:

- `recall@1 = 1.0` across the tested bit-width sweep
- `recall@10 = 1.0` across the tested bit-width sweep
- query time was slower than FAISS but much faster than LanceDB and pgvector in this benchmark path

| Adapter | Bits | Build Seconds | Query Seconds | Recall@1 | Recall@10 |
| --- | --- | --- | --- | --- | --- |
| Chroma | 2.0 | 1.823308 | 0.024900 | 1.0 | 1.0 |
| Chroma | 2.5 | 2.179364 | 0.017113 | 1.0 | 1.0 |
| Chroma | 3.0 | 2.708539 | 0.015125 | 1.0 | 1.0 |
| Chroma | 3.5 | 3.806835 | 0.016054 | 1.0 | 1.0 |
| Chroma | 4.0 | 4.611422 | 0.015874 | 1.0 | 1.0 |

LanceDB results:

- `recall@1 = 1.0` across all tested bit-widths
- `recall@10` was materially weaker than FAISS, ranging from `0.70` to `0.75`
- query time stayed around `0.063s`

| Adapter | Bits | Build Seconds | Query Seconds | Recall@1 | Recall@10 |
| --- | --- | --- | --- | --- | --- |
| LanceDB | 2.0 | 8.0862 | 0.065101 | 1.0 | 0.750000 |
| LanceDB | 2.5 | 2.1825 | 0.063661 | 1.0 | 0.743750 |
| LanceDB | 3.0 | 2.8432 | 0.063563 | 1.0 | 0.700000 |
| LanceDB | 3.5 | 4.1354 | 0.063443 | 1.0 | 0.721875 |
| LanceDB | 4.0 | 4.9028 | 0.063786 | 1.0 | 0.721875 |

### Synthetic KV Trend Snapshot

The synthetic KV report still shows the expected monotonic trend:

- higher bit-widths reduce `mse`
- higher bit-widths improve cosine similarity
- higher bit-widths reduce inner-product error

From this run:

- `b2.0_mse = 0.192804`
- `b3.5_mse = 0.029482`
- `b4.0_mse = 0.014735`

## Live pgvector Validation

Latest artifact set:

- `benchmark-results/20260326-pgvector-adapters-fixed.json`

Validation notes:

- PostgreSQL `17.9` was used locally with the `vector` extension enabled
- the Python `pgvector` client dependency is now included in `turboagents[rag]`
- the adapter benchmark was fixed to use isolated tables and normalize database row ids correctly

Observed behavior on `medium-rag`:

- `pgvector` reached `recall@1 = 1.0` across the tested bit-width sweep
- `recall@10` improved monotonically with higher bit-widths
- `pgvector` was slower than both FAISS and LanceDB in this current path, at about `0.87s` for the 32-query batch
- `pgvector` beat LanceDB on `recall@10` at `3.0+` bits in this run

| Adapter | Bits | Build Seconds | Query Seconds | Recall@1 | Recall@10 |
| --- | --- | --- | --- | --- | --- |
| pgvector | 2.0 | 1.7564 | 0.865659 | 1.0 | 0.656250 |
| pgvector | 2.5 | 2.1937 | 0.866529 | 1.0 | 0.718750 |
| pgvector | 3.0 | 2.8430 | 0.874390 | 1.0 | 0.796875 |
| pgvector | 3.5 | 4.1119 | 0.873350 | 1.0 | 0.837500 |
| pgvector | 4.0 | 4.8137 | 0.872895 | 1.0 | 0.896875 |

## Minimal Long-Context Eval

The repository now includes a minimal Needle-style long-context harness for MLX:

```bash
uv run python scripts/benchmark_needle.py \
  --model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --context-tokens 2048 4096 8192 \
  --output benchmark-results/needle-$(date +%Y%m%d-%H%M%S).json
```

Current scope:

- deterministic exact-match retrieval of a single secret string
- configurable context lengths
- configurable insertion positions
- bit-width sweep across the MLX path

Still out of scope:

- LongBench
- full Needle-in-a-Haystack reproductions
- multi-document or multi-hop tasks
- serving latency matrix integration

Latest artifact:

- `benchmark-results/needle-20260326-20260326-234159.json`

Observed behavior on `mlx-community/Llama-3.2-3B-Instruct-4bit`:

- exact-match retrieval succeeded at insertion fraction `0.1` across `2048`, `8192`, and `16384` token prompts for all tested bit-widths
- exact-match retrieval failed consistently at insertion fractions `0.5` and `0.9`
- `contains_needle` stayed positive for some `2048` and `4096` cases at insertion fraction `0.5`, but this is weaker than the exact-match criterion
- the current minimal harness therefore shows early-position recall, not robust long-context retrieval
- elapsed time scaled materially with context length, reaching about `15.6s` to `21.5s` per case at `16384` tokens

| Bits | Context Tokens | Insertion Fraction | Elapsed Seconds | Exact Match | Contains Needle |
| --- | --- | --- | --- | --- | --- |
| 3.0 | 2048 | 0.1 | 1.1014 | true | true |
| 3.0 | 2048 | 0.5 | 1.1488 | false | true |
| 3.0 | 2048 | 0.9 | 1.1465 | false | false |
| 3.0 | 4096 | 0.1 | 2.5599 | false | true |
| 3.0 | 4096 | 0.5 | 2.6351 | false | true |
| 3.0 | 4096 | 0.9 | 2.7681 | false | false |
| 3.0 | 8192 | 0.1 | 6.1731 | true | true |
| 3.0 | 8192 | 0.5 | 6.7962 | false | false |
| 3.0 | 8192 | 0.9 | 6.3600 | false | false |
| 3.0 | 16384 | 0.1 | 15.6448 | true | true |
| 3.0 | 16384 | 0.5 | 16.1179 | false | false |
| 3.0 | 16384 | 0.9 | 15.6592 | false | false |
| 3.5 | 2048 | 0.1 | 1.2985 | true | true |
| 3.5 | 2048 | 0.5 | 1.4403 | false | true |
| 3.5 | 2048 | 0.9 | 1.4370 | false | false |
| 3.5 | 4096 | 0.1 | 2.9586 | false | true |
| 3.5 | 4096 | 0.5 | 2.9688 | false | true |
| 3.5 | 4096 | 0.9 | 2.9501 | false | false |
| 3.5 | 8192 | 0.1 | 6.3312 | true | true |
| 3.5 | 8192 | 0.5 | 6.5699 | false | false |
| 3.5 | 8192 | 0.9 | 6.5293 | false | false |
| 3.5 | 16384 | 0.1 | 18.4821 | true | true |
| 3.5 | 16384 | 0.5 | 18.9856 | false | false |
| 3.5 | 16384 | 0.9 | 19.2857 | false | false |
| 4.0 | 2048 | 0.1 | 1.5542 | true | true |
| 4.0 | 2048 | 0.5 | 1.6880 | false | true |
| 4.0 | 2048 | 0.9 | 1.6921 | false | false |
| 4.0 | 4096 | 0.1 | 3.5982 | false | true |
| 4.0 | 4096 | 0.5 | 3.6594 | false | true |
| 4.0 | 4096 | 0.9 | 3.8123 | false | false |
| 4.0 | 8192 | 0.1 | 7.8715 | true | true |
| 4.0 | 8192 | 0.5 | 8.3909 | false | false |
| 4.0 | 8192 | 0.9 | 8.4198 | false | false |
| 4.0 | 16384 | 0.1 | 20.2147 | true | true |
| 4.0 | 16384 | 0.5 | 21.5121 | false | false |
| 4.0 | 16384 | 0.9 | 20.6090 | false | false |
