# exo -- Distributed P2P LLM Inference

## META
- repo: https://github.com/exo-explore/exo (Apache 2.0)
- stars: 41,000+ (as of Feb 2026)
- latest: v1.0.67 (Jan 28, 2026)
- language: Python 61.7%, Svelte 21%, Swift 7.3%, TypeScript 4.7%, Rust 3.3%
- relevance: PRIMARY -- Tier 1 compute fabric for local cluster auto-discovery and distributed inference
- last_verified: 2026-02-13

## SUMMARY

exo enables running large LLMs across multiple consumer devices on a home network using peer-to-peer architecture. Devices auto-discover each other, form a logical cluster, and dynamically partition models across available hardware. Provides ChatGPT-compatible API as drop-in replacement for cloud APIs.

## CORE ARCHITECTURE

### P2P Auto-Discovery
- Zero-config device discovery via mDNS (`_exo._tcp.local`)
- libp2p for P2P communication
- No master-worker bottleneck -- any device can participate
- Automatic topology analysis (resources + network latency/bandwidth per link)
- Fault tolerant: nodes can join/leave dynamically

### Node Orchestration
- **Election and Failover**: Automatic master election, failover on master loss
- **Event Sourcing**: State management via event log for consistency
- **Master/Worker Roles**: Master coordinates inference, workers execute shards

### Model Partitioning (Pipeline Parallel)
- Splits model into contiguous layer "shards" across devices
- Each device holds and processes assigned layers sequentially
- Activations flow device-to-device through the pipeline
- Example: 9-layer model → 3 shards across 3 devices (3 layers each)

### Placement Engine
- Topology-aware auto-parallel: determines optimal distribution
- Considers: device GPU/CPU/memory, network latency, bandwidth
- PACK strategy (minimize cross-device communication) or SPREAD strategy

### Communication Backends
- MLX distributed for Apple Silicon
- RDMA over Thunderbolt 5 (99% latency reduction)
- Standard TCP/IP for general networking

## KEY FEATURES

### Hardware Support
- Apple Silicon (M1/M2/M3/M4)
- NVIDIA GPUs
- AMD GPUs (ROCm)
- Intel (CPU inference)
- Raspberry Pi
- iPhones/iPads
- Any device with Python support

### Model Support (as of Jan 2026)
- DeepSeek v3.1 671B
- Qwen3-235B
- Llama 3.x family
- Mixtral models
- Gemma models
- And many more via HuggingFace integration

### Performance
- Tensor parallelism: up to 1.8x speedup on 2 devices, 3.2x on 4 devices
- Day-0 RDMA over Thunderbolt 5 support
- Benchmarking tool: exo_bench for transparent performance measurement

### API Layer
- OpenAI-compatible REST API on port 52415
- Endpoints: /v1/chat/completions, /v1/completions, etc.
- Web dashboard with cluster status, model management, inference interface
- Streaming support

## SYSTEM ARCHITECTURE DETAIL

### Component Flow
```
User Request → API Server (port 52415)
    → Master Node (elected)
        → Placement Engine (shard assignment)
            → Worker Nodes (execute shards)
                → Forward activations through pipeline
            → Aggregate results
        → Return response
    → User
```

### Model Management
- Model cards with metadata from HuggingFace
- Coordinated download across cluster nodes
- Each node downloads only its assigned shards
- Automatic shard distribution based on placement engine decisions

### Dashboard (Svelte Web UI)
- Cluster status: connected peers, topology view
- Model selection and download progress
- Inference interface: chat completion requests
- Performance metrics: tokens/sec, memory usage, latency
- Instance management: running model control

## INSTALLATION & SETUP

### From Source (Linux/Mac)
```bash
git clone https://github.com/exo-explore/exo.git
cd exo
pip install -e .
# or with uv:
uv run exo
```

### macOS Application
- DMG download: https://assets.exolabs.net/EXO-latest.dmg
- System tray integration, background service
- Requires macOS Tahoe 26.2+
- Auto-start, health monitoring, crash recovery

### Runtime Paths
- Model cache: ~/.exo/models/
- Config: ~/.exo/config.json
- Logs: ~/Library/Logs/EXO/ (macOS)

## RELEVANCE TO LLMHOST

### Direct Integration Points
1. **Auto-Discovery**: Use exo's mDNS discovery as foundation for LLMHosts compute discovery
2. **P2P Inference**: Tier 1 local cluster uses exo's pipeline parallel inference
3. **OpenAI API Compatibility**: exo's API layer aligns with LLMHosts's API standard
4. **Dashboard**: Reference for compute fleet visualization

### Customization Needed
1. Extend discovery to support remote VPS nodes (beyond local network)
2. Add security overlay (Headscale/WireGuard) for encrypted communication
3. Integrate with LiteLLM for unified routing across local exo cluster + external APIs
4. Add resource reporting hooks for Conductor's routing decisions
5. Extend model management to support Conductor-directed model pulling

### Limitations to Address
- Currently focused on local network (mDNS scope)
- No built-in encryption for inter-node communication
- Master election could be improved for LLMHosts's use case
- No native integration with external API providers
