GGUF
=======

Neural Speed also supports GGUF models generated by [llama.cpp](https://github.com/ggerganov/llama.cpp), you need to download the model and use llama.cpp to create it. 

Validated models: [llama2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), [falcon-7b](https://huggingface.co/tiiuae/falcon-7b), [falcon-40b](https://huggingface.co/tiiuae/falcon-40b), [mpt-7b](https://huggingface.co/mosaicml/mpt-7b), [mpt-40b](https://huggingface.co/mosaicml/mpt-40b) and [bloom-7b1](https://huggingface.co/bigscience/bloomz-7b1). 

Please check more validated GGUF models from HuggingFace in [list](./docs/supported_models.md).

## Examples

How to create the GGUF file in Neural Speed:
```python
# Example:
# please provide the local model path as the arg, 
# which means you need to `git clone https://huggingface.co/meta-llama/Llama-2-7b-chat-hf` first.
python neural_speed/convert/convert-hf-to-gguf.py /model_path/Llama-2-7b-chat-hf/

```

How to load the GGUF bin file in Neural Speed:

```python
    prompt = "Once upon a time"
    tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True)
    inputs = tokenizer(prompt, return_tensors="pt").input_ids
    streamer = TextStreamer(tokenizer)

    model = Model()
    model.init_from_bin(args.model_name, gguf_path)
    outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300, do_sample=True)

# Please check this script for more details and input parameters.
# python scripts/python_api_example_for_gguf.py --model_name falcon --model_path /home/model/falcon-7b -m /home/model/falcon-7b/ggml-model-f32.gguf
```

Note: These GGUF models can be accelerated by [Neural Speed BestTLA](https://github.com/intel/neural-speed/blob/c0312283f528d4a9ffebc283cd0f15a7a8eabf1a/bestla/README.md#L1).

How to accelerate GGUF by BTLA:
```python
# quantization and then re-run the above step python_api_example_for_gguf.py
./build/bin/quant_falcon --model_file /home/model/falcon-7b/ggml-model-f32.gguf --out_file ne-falcon-q4_j.bin --weight_dtype int4 --compute_dtype int8

python scripts/python_api_example_for_gguf.py --model_name falcon --model_path /home/model/falcon-7b -m ne-falcon-q4_j.bin
```

How to load the GGUF bin file in [intel-extension-for-transformers](https://github.com/intel/intel-extension-for-transformers/pull/1151):
```python
from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM, WeightOnlyQuantConfig

# Specify the GGUF repo on the Hugginface
model_name = "TheBloke/Llama-2-7B-Chat-GGUF"
# Download the the specific gguf model file from the above repo
model_file = "llama-2-7b-chat.Q4_0.gguf"
# make sure you are granted to access this model on the Huggingface.
tokenizer_name = "meta-llama/Llama-2-7b-chat-hf"

prompt = "Once upon a time"
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)
model = AutoModelForCausalLM.from_pretrained(model_name, model_file = model_file)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)
```
