Metadata-Version: 2.4
Name: speculators
Version: 0.1.0
Summary: A unified library for creating, representing, and storing speculative decoding algorithms for LLM serving such as in vLLM.
Author: Red Hat
License-Expression: Apache-2.0
Project-URL: homepage, https://github.com/neuralmagic/speculators
Project-URL: source, https://github.com/neuralmagic/speculators
Project-URL: issues, https://github.com/neuralmagic/speculators/issues
Keywords: speculative decoding,transformers,llm,inference,vllm,machine learning,deep learning,nlp,language models,serving,standard format,productization,decoding algorithms,inference server
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: accelerate
Requires-Dist: click
Requires-Dist: datasets
Requires-Dist: deepspeed
Requires-Dist: httpx[http2]
Requires-Dist: huggingface-hub
Requires-Dist: loguru
Requires-Dist: numpy
Requires-Dist: protobuf
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: safetensors
Requires-Dist: torch
Requires-Dist: transformers<4.54
Requires-Dist: typer-slim>=0.12.0
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: setuptools>=61.0; extra == "dev"
Requires-Dist: setuptools-git-versioning<3,>=2.0; extra == "dev"
Requires-Dist: pre-commit~=3.5.0; extra == "dev"
Requires-Dist: sphinx~=7.1.2; extra == "dev"
Requires-Dist: tox~=4.16.0; extra == "dev"
Requires-Dist: pytest~=8.2.2; extra == "dev"
Requires-Dist: pytest-asyncio~=0.23.8; extra == "dev"
Requires-Dist: pytest-cov~=5.0.0; extra == "dev"
Requires-Dist: pytest-mock~=3.14.0; extra == "dev"
Requires-Dist: pytest-rerunfailures~=14.0; extra == "dev"
Requires-Dist: respx~=0.22.0; extra == "dev"
Requires-Dist: mypy~=1.15.0; extra == "dev"
Requires-Dist: ruff~=0.11.7; extra == "dev"
Requires-Dist: mdformat~=0.7.17; extra == "dev"
Requires-Dist: mdformat-footnote~=0.1.1; extra == "dev"
Requires-Dist: mdformat-frontmatter~=2.0.8; extra == "dev"
Requires-Dist: mdformat-gfm~=0.3.6; extra == "dev"
Requires-Dist: types-click~=7.1.8; extra == "dev"
Requires-Dist: types-PyYAML~=6.0.1; extra == "dev"
Requires-Dist: types-requests~=2.32.0; extra == "dev"
Requires-Dist: types-toml; extra == "dev"
Requires-Dist: mkdocs-linkcheck~=1.0.6; extra == "dev"
Dynamic: license-file

<div align="center">

<picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/branding/speculators-logo-white.svg" />
    <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/branding/speculators-logo-black.svg" />
    <img alt="Speculators logo" src="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/branding/speculators-logo-black.svg" height="64" />
  </picture>

[![License](https://img.shields.io/github/license/neuralmagic/speculators.svg)](https://github.com/neuralmagic/speculators/blob/main/LICENSE) [![Python Versions](https://img.shields.io/badge/Python-3.9--3.13-orange)](https://pypi.python.org/pypi/speculators)

</div>

## Overview

**Speculators** is a unified library for building, evaluating, and storing speculative decoding algorithms for large language model (LLM) inference, including in frameworks like vLLM. Speculative decoding is a lossless technique that speeds up LLM inference by using a smaller, faster speculator model to propose tokens, which are then verified by the larger base model, reducing latency without compromising output quality. Speculators standardizes this process with reusable formats and tools, enabling easier integration and deployment of speculative decoding in production-grade inference servers.

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/branding/speculators-user-flow-dark.svg" />
    <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/branding/speculators-user-flow-light.svg" />
    <img alt="Speculators user flow diagram" src="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/branding/speculators-user-flow-light.svg" />
  </picture>
</p>

### Key Features

- **Unified Speculative Decoding Toolkit:** Simplifies the development, evaluation, and representation of speculative decoding algorithms, supporting both research and production use cases for LLMs.
- **Standardized, Extensible Format:** Provides a Hugging Face-compatible format for defining speculative models, with tools to convert from external research repositories for easy adoption.
- **Seamless vLLM Integration:** Built for direct deployment into vLLM, enabling low-latency, production-grade inference with minimal overhead.

## Getting Started

### Installation

Before installing, ensure you have the following prerequisites:

- OS: Linux or MacOS
- Python: 3.9 or higher

Install Speculators directly from source using pip::

```bash
pip install git+https://github.com/neuralmagic/speculators.git
```

## Resources

Here you can find links to our research implementations. These provide prototype code for immediate enablement and experimentation, with plans for productization into the main package soon.

- [eagle3](https://github.com/neuralmagic/speculators/tree/main/research/eagle3): This implementation trains models similar to the EAGLE 3 architecture, specifically utilizing the Train Time Test method.

- [hass](https://github.com/neuralmagic/speculators/tree/main/research/hass): This implementation trains models that are a variation on the EAGLE 1 architecture using the [HASS](https://github.com/HArmonizedSS/HASS) method.

## vLLM Inference

Once in the speculators format, you can serve the speculator using vLLM:

```bash
VLLM_USE_V1=1 vllm serve RedHatAI/Qwen3-8B-speculator.eagle3
```

Served models can then be benchmarked using [GuideLLM](https://github.com/vllm-project/guidellm). Below, we show sample benchmark results where we compare our speculator with its dense counterpart. We also additionally compare [quantization](https://github.com/vllm-project/llm-compressor) to explore additional performance improvements by swapping the dense verifier, `Qwen/Qwen3-8B` with the quantized FP8 model, [RedHatAI/Qwen3-8B-FP8-dynamic](https://huggingface.co/RedHatAI/Qwen3-8B-FP8-dynamic) in the `speculator_config`.

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/qwen_quant_benchmark.png">
    <img alt="GuideLLM Logo" src="https://raw.githubusercontent.com/neuralmagic/speculators/main/docs/assets/qwen_quant_benchmark.png" width=180%>
  </picture>
</p>

### License

Speculators is licensed under the [Apache License 2.0](https://github.com/neuralmagic/speculators/blob/main/LICENSE).

### Cite

If you find Speculators helpful in your research or projects, please consider citing it:

```bibtex
@misc{speculators2025,
  title={Speculators: A Unified Library for Speculative Decoding Algorithms in LLM Serving},
  author={Red Hat},
  year={2025},
  howpublished={\url{https://github.com/neuralmagic/speculators}},
}
```
