Metadata-Version: 2.1
Name: llmformat
Version: 0.0.2.1
Summary: Format LLM language by using LALR(1) grammar. Supports JSON, XML, etc.
Author: Qiuling Xu
Author-email: Qiuling Xu <xennaughtyboy@gmail.com>
Project-URL: Homepage, https://github.com/qiulingxu/llmformat
Project-URL: Issues, https://github.com/qiulingxu/llmformat/issues
Keywords: python,llm,format,json,regex
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: lark >=1.1.8
Requires-Dist: vllm >=0.2.6

#  LLM Format

![](./assets/llmlogo.jpg)

## Introduction

LLM Format is a Large-language model formatter that constrain the outputs of language model to follow certain rules. Our design is to make this tool flexible and efficient to adapt to different use cases.

We currently supports **vllm** and any **LALR(1) grammar** including **JSON** format.

Different from other packages including lm-format-enforcer, jsonformer and guidance. This package ensures the generated text from LLM to be sound and complete. 


## Tutorial 

### Installation

`pip install llmformat`

### Usage

To enforce a new type of grammar, a grammar file written in EBNF is needed. We provide the JSON example as [here](https://github.com/qiulingxu/llmformat/blob/main/llmformat/grammar_files/json_min.bnf).

Once it is written, we can only need one-line code change to enforce the generation.

In vllm, add this option to sampling_param.

sampling_param.logits_processors=[llmformat.llminterface.build_vllm_logits_processor(model, "/root/llmformat/llmformat/json_min.bnf")]

### Example
The example of working on Llama2 and vllm can be found [here](https://github.com/qiulingxu/llmformat/blob/main/examples/vllm_llama2.ipynb). Note that you may want to change the location of grammar file.


- Add support for customized JSON
- Add more integrations

## Known Issues

We use cache to accelerate grammar parsing. The speed will becomes faster as it runns longer.
