# Concurrent Inference Design — LLMHosts Core (GAP-03)

**Date:** 2026-03-10
**Status:** Design phase — implementation blocks Wave 2 (continuous batching)
**Issue:** #485
**Advisory:** GAP-03 severity 8/10 — current `Mutex<Option<model>>` prevents concurrent requests

---

## Problem

`llmhosts_core/src/lib.rs` holds the model behind `Mutex<Option<LLMHostsModel>>`:

```rust
pub struct LLMHostsEngine {
    model: Mutex<Option<model::transformer::LLMHostsModel>>,
}
```

`generate_tokens()` acquires a write lock, blocking all other requests for the entire
duration of generation. On a 70B model, this is 30-120 seconds per request.

**Consequence:** Continuous batching (Vision 2026 Wave 2 core promise) is architecturally
impossible on top of this design. Each request must wait for the previous to complete.

Note: `py.detach()` (GAP-04, fixed in commit d206299) unblocks Python threads during
inference, but Rust-level serialization remains. The Mutex fix is still required for true
concurrency.

---

## Why Concurrent Inference Is Hard for LLMs

LLM inference is inherently stateful:
- **Model weights** — immutable, ~4-40GB, shared across all requests (safe to read concurrently)
- **KV cache** — mutable, per-request, grows during generation (cannot be shared)
- **Sampling state** — per-request RNG, temperature/top-p state (cannot be shared)

The current design bundles all three into `LLMHostsModel`, making it impossible to share.

---

## Target Architecture: Weights/Cache Separation

### Phase 1 — Shared immutable weights (Wave 2)

Separate model into two parts:

```rust
pub struct LLMHostsWeights {
    // Immutable after load — safe for Arc<> sharing across threads
    layers: Vec<TransformerLayer>,
    config: LLMHostsModelConfig,
}

pub struct InferenceSession {
    // Per-request mutable state — one per concurrent request
    kv_cache: KVCache,
    rng: SmallRng,
}

pub struct LLMHostsEngine {
    weights: Arc<RwLock<Option<LLMHostsWeights>>>,
    //       ^-- write lock only during load/unload (rare)
    //           read lock during inference (many concurrent readers)
}
```

**generate_tokens()** flow:
1. Acquire **read** lock on `weights` (non-blocking if no load/unload in progress)
2. Create per-request `InferenceSession` (stack-allocated, cheap)
3. Run generation with shared weights + per-session state
4. Drop read lock (releases after generation, no GIL interaction)

**Concurrency model:** N concurrent requests run truly in parallel, each with their own
KVCache. Memory cost: `N × kv_cache_size` (e.g., 10 concurrent × 512MB = 5GB).

### Phase 2 — Continuous batching (Wave 2)

With weights separated, implement PagedAttention KV cache management:
- Fixed pool of KV cache blocks, allocated per-token
- New requests can be admitted mid-generation (no full sequence alignment required)
- Scheduler runs in a dedicated tokio task

```
                    ┌─────────────────────────────┐
                    │  Request Queue (mpsc channel) │
                    └──────────────┬──────────────┘
                                   │
                    ┌──────────────▼──────────────┐
                    │  Batch Scheduler             │
                    │  - Groups compatible sequences│
                    │  - Allocates KV cache blocks │
                    └──────────────┬──────────────┘
                                   │
                    ┌──────────────▼──────────────┐
                    │  Inference Worker (Rust)     │
                    │  - Shared weights (Arc<>)    │
                    │  - Per-batch KV cache        │
                    │  - CUDA/Metal kernels        │
                    └──────────────────────────────┘
```

---

## Migration Plan

| Step | Change | Breaking? |
|------|--------|-----------|
| 1 | Split `LLMHostsModel` into `LLMHostsWeights` + `InferenceSession` | Internal only |
| 2 | Replace `Mutex<Option<>>` with `Arc<RwLock<Option<>>>` | No API change |
| 3 | Update `generate_tokens()` to create session per-call | No API change |
| 4 | Add concurrent test: N threads calling generate_tokens() simultaneously | New test |
| 5 | Implement `PagedKVCache` and `BatchScheduler` | New API (Wave 2) |

Steps 1-4 can be done in Wave 1 without user-visible changes.
Step 5 requires new Python API for batch generation.

---

## Memory Budget

| Concurrent Requests | Model (Q4_K_M) | KV Cache (2048 ctx) | Total VRAM |
|--------------------|----------------|---------------------|------------|
| 1 | 4.9 GB (8B) | 0.5 GB | 5.4 GB |
| 4 | 4.9 GB (8B) | 2.0 GB | 6.9 GB |
| 8 | 4.9 GB (8B) | 4.0 GB | 8.9 GB |
| 1 | 42.0 GB (70B) | 2.0 GB | 44.0 GB |
| 2 | 42.0 GB (70B) | 4.0 GB | 46.0 GB |

KV cache grows linearly with concurrent requests. Default concurrency cap:
- RTX 4090 (24GB): 4 concurrent 8B requests or 1 concurrent 70B
- RTX 3090 (24GB): same
- M4 Max (128GB): 8 concurrent 70B requests

---

## Acceptance Criteria

- [ ] 4 concurrent `generate_tokens()` calls complete without serialization
- [ ] No VRAM increase beyond `N × kv_cache_size`
- [ ] load_model() still uses write lock (exclusive, rare)
- [ ] unload() still uses write lock (exclusive)
- [ ] Existing single-request tests unchanged
- [ ] New concurrent stress test: 8 threads × 100 token generation
