Metadata-Version: 2.1
Name: llmcompressor
Version: 0.1.0
Summary: A library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post training techniques. The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.
Home-page: https://github.com/neuralmagic/llm-compressor
Author: Neuralmagic, Inc.
Author-email: support@neuralmagic.com
License: Apache
Keywords: llmcompressor,llms,large language models,transformers,pytorch,huggingface,compressors,compression,quantization,pruning,sparsity,optimization,model optimization,model compression,
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8.0,<3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: loguru
Requires-Dist: pyyaml >=5.0.0
Requires-Dist: numpy <2.0,>=1.17.0
Requires-Dist: requests >=2.0.0
Requires-Dist: tqdm >=4.0.0
Requires-Dist: click !=8.0.0,>=7.1.2
Requires-Dist: torch >=1.7.0
Requires-Dist: transformers <5.0,>4.0
Requires-Dist: datasets
Requires-Dist: accelerate >=0.20.3
Requires-Dist: compressed-tensors
Provides-Extra: dev
Requires-Dist: pytest >=6.0.0 ; extra == 'dev'
Requires-Dist: pytest-mock >=3.6.0 ; extra == 'dev'
Requires-Dist: pytest-rerunfailures >=13.0 ; extra == 'dev'
Requires-Dist: parameterized ; extra == 'dev'
Requires-Dist: black ~=24.4.2 ; extra == 'dev'
Requires-Dist: isort ~=5.13.2 ; extra == 'dev'
Requires-Dist: mypy ~=1.10.0 ; extra == 'dev'
Requires-Dist: ruff ~=0.4.8 ; extra == 'dev'
Requires-Dist: flake8 ~=7.0.0 ; extra == 'dev'
Requires-Dist: pre-commit ; extra == 'dev'

# LLM Compressor
`llm-compressor` is an easy-to-use library for optimizing models for deployment with `vllm`, including:

* Comprehensive set of quantization algorithms including weight-only and activation quantization
* Seemless integration Hugging Face models and repositories
* `safetensors`-based file format compatible with `vllm`

<p align="center">
   <img alt="LLM Compressor Flow" src="docs/images/architecture.png" width="75%" />
</p>


### Supported Formats
* Mixed Precision: W4A16, W8A16
* Activation Quantization: W8A8 (int8 and fp8)
* 2:4 Semi-structured Sparsity
* Unstructured Sparsity

### Supported Algorithms
* PTQ (Post Training Quantization)
* GPTQ
* SmoothQuant
* SparseGPT


## Installation

`llm-compressor` can be installed from the source code via a git clone and local pip install.

```bash
git clone https://github.com/vllm-project/llm-compressor.git
pip install -e llm-compressor
```

## Quick Tour
The following snippet is a minimal example with 4-bit weight-only quantization via GPTQ and inference of a `TinyLlama/TinyLlama-1.1B-Chat-v1.0`. Note that the model can be swapped for a local or remote HF-compatible checkpoint and the `recipe` may be changed to target different quantization algorithms or formats.

### Compression
Compression is easily applied by selecting an algorithm (GPTQ) and calling the `oneshot` API.

```python
from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization.gptq import GPTQModifier

# Sets parameters for the GPTQ algorithms - target Linear layer weights at 4 bits
recipe = GPTQModifier(scheme="W4A16", targets="Linear", ignore=["lm_head"])

# Apply GPTQ algorithm using open_platypus dataset for calibration.
oneshot(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    dataset="open_platypus",
    recipe=recipe,
    save_compressed=True,
    output_dir="llama-compressed-quickstart",
    overwrite_output_dir=True,
    max_seq_length=2048,
    num_calibration_samples=512,
)
```

### Inference with vLLM
The checkpoint is ready to run with vLLM (after install `pip install vllm`).

```python
from vllm import LLM

model = LLM("llama-compressed-quickstart")
output = model.generate("I love 4 bit models because")
```

## End-to-End Examples
The `llm-compressor` library provides a rich feature-set for model compression. Below are examples
and documentation of a few key flows:
* [`Meta-Llama-3-8B-Instruct` W4A16 With GPTQ](examples/quantization_w4a16)
* [`Meta-Llama-3-8B-Instruct` W8A8-Int8 With GPTQ and SmoothQuant](examples/quantization_w8a8_int8)
* [`Meta-Llama-3-8B-Instruct` W8A8-Fp8 With PTQ](examples/quantization_w8a8_fp8)

If you have any questions or requests open an [issue](https://github.com/vllm-project/llm-compressor/issues) and we will add an example or documentation.

## Contribute
We appreciate contributions to the code, examples, integrations, and documentation as well as bug reports and feature requests!
[Learn how here](CONTRIBUTING.md).
