# Language Pipes (Beta)

**Distribute language models across multiple systems**  

[![GitHub license][License-Image]](License-Url)
[![Release][Release-Image]][Release-Url] 

[License-Image]: https://img.shields.io/badge/license-MIT-blue.svg
[License-Url]: https://github.com/erinclemmer/language-pipes/blob/main/LICENSE

[Release-Url]: https://github.com/erinclemmer/language-pipes/releases/latest
[Release-Image]: https://img.shields.io/github/v/release/erinclemmer/language-pipes

[PyPiVersion-Url]: https://img.shields.io/pypi/v/language-pipes
[PythonVersion-Url]: https://img.shields.io/pypi/pyversions/language-pipes

Language pipes is a FOSS distributed network application designed to increase access to local language models.  

---  

**Disclaimer:** This software is currently in Beta. Please be patient and if you encounter an error, please [fill out a github issue](https://github.com/erinclemmer/language-pipes/issues/new)!   

Over the past few years open source language models have become much more powerful yet the most powerful models are still out of reach of the general population because of the extreme amounts of RAM that is needed to host these models. Language Pipes allows multiple computer systems to host the same model and move computation data between them so that no one computer has to hold all of the data for the model.
- Quick Setup
- Peer to peer network
- OpenAI compatible API
- Download and use models by HuggingFace ID
- Encrypted communication between nodes

### What Does it do?
In a basic sense, language models work by passing information through many layers. At each layer, several matrix multiplicatitons between the layer weights and the system state are performed and the data is moved to the next layer. Language pipes works by hosting different layers on different machines to split up the RAM cost across the system.

### Installation
Ensure that you have Python 3.10.18 (or any 3.10 version) installed. For an easy to use Python version manager use [pyenv](https://github.com/pyenv/pyenv). This specific version is necessary for the [transformers](https://github.com/huggingface/transformers) library to work properly.  
  
If you need gpu support, first make sure you have the correct pytorch version installed for your GPU's Cuda compatibility using this link:  
https://pytorch.org/get-started/locally/

To download the models from Huggingface, ensure that you have [git](https://git-scm.com/) and [git lfs](https://git-lfs.com/) installed.  

To start using the application, install the latest version of the package from PyPi.

Using Pip:
```bash
pip install language-pipes
```

### Quick Start

The easiest way to get started is with the interactive setup wizard:

```bash
language-pipes start
```

This guides you through network key generation, configuration, and starts the server.

---

# Two Node Example
The following example will show how to create a small network. Firstly, create a network key for the network on the first computer:
```bash
language-pipes keygen network.key
```

Also create a `config.toml` file to tell the program how to operate:

```toml
node_id="node-1"
oai_port=6000 # Hosts an OpenAI compatible server on port 6000

[[hosted_models]]
id="Qwen/Qwen3-1.7B"
device="cpu"
max_memory=1
```

**Note:** Go to the [configuration documentation](/documentation/configuration.md) for more information about how the config properties work.

Once the configuration has been created you can start the server:
```bash
language-pipes serve --config config.toml
```

This tells language pipes to download with the ID "Qwen/Qwen3-1.7B" from [huggingface.co](huggingface.co) and host it using 1GB of ram. This will load part of the model but not all of it.

Next, install the package on a separate computer on your home network and create a `config.toml` file like this:

```toml
node_id="node-2"
bootstrap_address="192.168.0.10" # Local ip address of node-1

[[hosted_models]]
id="Qwen/Qwen3-1.7B"
device="cpu"
max_memory=3
```

Copy the `network.key` file to the same directory that the config is in using a usb drive or sftp. 

Run the same command again on the computer two:
```bash
language-pipes serve --config config.toml
```

Node-2 will connect to node-1 and load the remaining parts of the model. The model is ready for inference using a [standard OpenAI chat API interface](https://platform.openai.com/docs/api-reference/chat/create). An example using the [OpenAI Python library](https://github.com/openai/openai-python):

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:6000/v1",  # node-1 IP address
    api_key="not-needed"  # API key not required for Language Pipes
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    max_completion_tokens=100,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about distributed systems."}
    ]
)

print(response.choices[0].message.content)
```

Install the OpenAI library with: `pip install openai`

### Models Supported
* Llama 2 & Llama 3.X  
* Qwen3
* More to come!

### Dependencies
- [pytorch](pytorch.org)
- [transformers](https://huggingface.co/docs/transformers) 

### Documentation
* [CLI Reference](./documentation/cli.md)
* [Configuration](./documentation/configuration.md)
* [Architecture](./documentation/architecture.md)
* [OpenAI API](./documentation/oai.md)
