# rs_bpe Benchmark Tool

This benchmark tool compares the performance of `rs_bpe` against other popular tokenization libraries:
- `tiktoken` - OpenAI's BPE tokenizer
- `tokenizers` - HuggingFace's tokenizers library

## Features

- **Multiple Text Sizes**: Tests with small, medium, and large text samples
- **Multiple Benchmarks**: Measures encoding, decoding, and roundtrip times
- **Detailed Metrics**: Provides raw timings, speedup ratios, and token counts
- **Visualization**: Generates charts to visualize performance differences
- **Fail-safe Design**: Gracefully handles missing dependencies

## Requirements

- Python 3.8+
- Required packages:
  - `tiktoken`
  - `tokenizers`
  - `matplotlib`
  - `pandas`
  - `rs_bpe` (optional - will run comparisons without it if not available)

## Usage

1. Install dependencies:
   ```bash
   uv pip install tiktoken tokenizers matplotlib pandas
   ```

2. Build the rs_bpe module if you want to include it in the benchmark:
   ```bash
   maturin develop
   ```

3. Run the benchmark:
   ```bash
   python benchmark.py
   ```

## What Is Measured

The benchmark script measures several important metrics:

1. **Encoding Speed**: Time taken to convert text into token IDs
2. **Decoding Speed**: Time taken to convert token IDs back to text
3. **Roundtrip Speed**: Time for the complete encode-decode cycle
4. **Token Count**: Number of tokens generated for each text size

## Interpreting Results

The benchmark outputs several types of metrics:

- **Raw Timing Results**: Absolute time in seconds for each operation
- **Speedup Ratios**: Relative performance compared to the baseline (default: tiktoken)
- **Token Counts**: Number of tokens generated by each tokenizer

A performance visualization is saved as `tokenizer_benchmark_results.svg`.

## Adding New Tokenizers

To add a new tokenizer to the benchmark:

1. Create a new adapter class that inherits from `TokenizerAdapter`
2. Implement the required `encode` and `decode` methods
3. Add an instance of your adapter to the list in the `run_benchmarks` function

## Sample Results

When rs_bpe is properly built and the benchmark is run against all tokenizers, you'll see output similar to:

```
===== BENCHMARK RESULTS =====

ENCODING TIMES (seconds):
SMALL TEXT:
  tiktoken: 0.000384s
  tokenizers: 0.000273s
  rs_bpe: 0.000182s
...

===== SPEEDUP RESULTS =====
(Higher values indicate faster performance relative to baseline)

ENCODE SPEEDUP:
SMALL TEXT:
  tiktoken: 1.00x
  tokenizers: 1.41x
  rs_bpe: 2.11x
...
```

## Troubleshooting

If the rs_bpe module is not available or fails to initialize, the benchmark will continue to run with the other tokenizers. 