# Collective Intelligence & Privacy-Preserving Learning for LLMHosts
## Research Synthesis — March 10, 2026

---

## Executive Summary

This document synthesizes 50+ research threads across federated learning, differential privacy, LLM routing optimization, per-customer fine-tuning, and regulatory compliance. It produces four concrete deliverables:

1. **LLMHosts Collective Intelligence System** — platform-wide learning from anonymized usage signals
2. **Per-Customer Continuous Improvement** — individual LoRA adapters and prompt optimization
3. **Privacy Compliance Framework** — GDPR/CCPA opt-in architecture, data retention, audit trails
4. **Implementation Roadmap** — phased build plan from immediate to long-term

---

## Part 1: Research Findings

### 1.1 Federated Learning State of the Art (March 2026)

**Flower (flwr) — v1.26.x, Production-Ready**

Flower is the most mature open-source federated learning framework as of March 2026. Key capabilities relevant to LLMHosts:

- Native support for PyTorch, TensorFlow, JAX, scikit-learn, XGBoost
- Built-in differential privacy and secure aggregation protocols
- Production deployment via Docker and Kubernetes (Helm)
- TLS + authentication for secure cross-node communication
- **FlowerLLM**: World's first 1.3B parameter LLM trained via federated learning
- **FlowerTune**: Federated fine-tuning with active benchmarking leaderboard
- **Flower SuperGrid**: Next-generation platform for scalable, production-grade federated AI
- ISO 27001 compliance certified; used by healthcare (hospitals, BloodCounts!) and space (Starcloud)

**Algorithms**: FedAvg (baseline), FedAdam (adaptive), FedProx (heterogeneous clients), plus custom strategy API. FedAvg reduces communication by 10-100x vs naive distributed SGD.

**Key Architecture Insight**: Flower's Federated Averaging allows a coordinating server to compute the average of gradients from N clients without any individual gradient being inspectable — mathematically equivalent to training on the union of all client data, without data ever leaving the client.

**TensorFlow Federated (TFF)**: Google's production framework; powers Gboard on Android (200M+ devices). Strong for mobile/edge scenarios. Less suitable for LLMHosts's server-node architecture than Flower.

**PySyft / OpenMined**: Privacy-first framework for remote data science (study data without moving it). Status: undergoing significant refactor. Less production-ready than Flower for LLMHosts's use case.

---

### 1.2 Differential Privacy for ML Training

**Mathematical Foundation**

Differential privacy (DP) provides a mathematical guarantee: an algorithm M satisfies (ε, δ)-DP if for any two datasets D and D' differing by one individual's record, and any output set S:

```
P[M(D) ∈ S] ≤ e^ε × P[M(D') ∈ S] + δ
```

- **ε (epsilon)**: Privacy budget. Lower = stronger privacy. Production systems typically use ε ∈ [1, 10] for utility.
- **δ (delta)**: Probability of catastrophic privacy failure. Typically set to 1/(n²) where n = dataset size.

**Key Mechanisms**

- **Laplace Mechanism**: Adds Laplace noise calibrated to sensitivity/ε. Best for count queries.
- **Gaussian Mechanism**: Adds Gaussian noise. Better for high-dimensional data (gradients).
- **DP-SGD**: Clips per-example gradients to bound sensitivity, then adds Gaussian noise before aggregation. Standard approach for training ML models with DP.

**OpenDP** (Harvard-backed, v0.8 as of Feb 2026): Production-ready library for DP computations. Provides rigorous, verified implementations of DP mechanisms.

**Practical Production Epsilon Values**:
- Apple (local DP for usage analytics): ε ≈ 8-14 per day
- Google (Chrome/Gboard): ε ≈ 1-10
- Academic strong privacy: ε < 1
- Practical utility-preserving: ε ∈ [3, 10]

**DP for LLM Fine-tuning (Key Finding)**:
Research on fine-tuning LLMs with DP shows that even sentence-level DP with well-calibrated epsilon still leaks ~3% of PII sequences from training data. This validates the need for DP even when data appears anonymized.

**Privacy Auditing**: NeurIPS 2023 introduced efficient methods for auditing actual DP guarantees of algorithms — important for compliance verification.

---

### 1.3 LLM Routing Optimization — The State of the Art

**RouteLLM (2024) — Best-in-Class Academic Work**

The RouteLLM paper (arXiv:2406.18665) establishes the gold standard for learned LLM routing:

- Trains router models on human preference data + data augmentation
- Routes between a "strong" model (expensive) and "weak" model (cheap)
- Achieves 2x+ cost reduction with no quality degradation
- **Critical finding**: Routers exhibit strong transfer learning — they maintain performance even when the underlying models change at test time
- Routing signal: learned classifier trained on preference labels (which model output was preferred)

**FrugalGPT (arXiv:2305.05176) — Cascade Approach**

- Three strategies: prompt adaptation, LLM approximation, LLM cascade
- Cascade: learns which model combinations to use for different query types
- Results: matches GPT-4 quality with up to 98% cost reduction, or 4% quality improvement at same cost
- Key mechanism: cascade scoring — try cheap model first, escalate to expensive if confidence is low

**NotDiamond (Powers OpenRouter's Auto Router)**

- OpenRouter uses NotDiamond technology for automatic model selection
- Considers: provider availability, rate limits, request refusals, cost, performance
- Model variants: `:nitro` (speed), `:extended` (context), `:thinking` (reasoning), `:online` (web access)
- Does not publicly document training signal details, but observes query patterns across millions of requests

**OpenRouter Routing Signals** (from docs analysis):
- Provider health and uptime
- Model variant selection (`:free`, `:nitro`, `:extended`, `:thinking`)
- User-specified fallback chains
- Automatic failover on provider failures
- Usage-based routing optimization (proprietary)

---

### 1.4 Task Classification for Privacy-Preserving Routing

**Key Insight**: You can classify task type from structural features of requests without reading content.

**Content-free routing signals** (privacy-safe):
- **Token count of prompt**: Longer prompts → reasoning/analysis tasks; short prompts → casual chat
- **Token velocity** (tokens/second in streaming response): fast = retrieval/recall; slow = reasoning
- **Request latency pattern**: First-token latency vs total latency ratio distinguishes task types
- **Session structure**: Turn count, turn length variation, retry rate within session
- **Code-presence markers**: Presence of code fences (```), special characters (`[`, `{`, `(`), without reading code content
- **Response length**: Short responses → factual Q&A; long responses → creative/analytical
- **Time of day + day of week**: Strong correlation with task type (business hours → coding/analysis; evenings → creative)
- **Model capability required**: Measured by complexity of prior successful requests in session

**Task taxonomy** (derivable from structural signals):
1. **Coding** — code fence markers, short prompt, long response, technical tokens
2. **Reasoning/Analysis** — long prompt, long response, slow generation
3. **Creative Writing** — medium prompt, long response, fast generation
4. **Factual Q&A** — short prompt, short response, very fast generation
5. **Summarization** — very long prompt, short response
6. **Conversation** — short prompt, short response, multi-turn

**Mixture-of-Agents (MoA)**: Rather than selecting one model, recent work shows layered architectures where multiple LLMs each process the query and a final LLM synthesizes. MoA scored 65.1% vs GPT-4o's 57.5% on AlpacaEval 2.0 using only open-source models. Relevant for high-stakes routing decisions.

---

### 1.5 Bandit Algorithms for Model Selection

**Thompson Sampling** (Bayesian bandit):
- Maintains Beta(α, β) distribution per model × task-type pair
- On each request: sample θ_i from each model's distribution, select argmax θ_i
- On feedback: update α (successes) or β (failures) based on implicit signal
- Advantages: naturally handles exploration vs exploitation; converges faster than UCB in practice
- Key property: automatically increases exploration for models with high uncertainty

**UCB (Upper Confidence Bound)**:
- Select model i = argmax[μ_i + c × √(log(t)/n_i)]
- Deterministic, simple, proven regret bounds
- Less adaptive to non-stationary environments (model quality changes over time)

**Contextual Bandits** (best for LLMHosts):
- LinUCB or Neural Contextual Bandits extend basic bandits with context features
- Context = {task_type, token_count, hardware_config, time_of_day, session_history}
- Reward = implicit quality signal (see §1.6)
- This is what powers modern recommendation systems (LinkedIn, YouTube) at scale

**Key Insight for LLMHosts**: Use Thompson sampling with a contextual feature vector. The "context" includes content-free request features (§1.4). The "reward" is an implicit quality signal (§1.6). This creates a self-improving routing system with no privacy exposure.

---

### 1.6 Implicit Feedback Signals for AI Quality

**Content-free quality signals** (privacy-safe — do not require reading responses):

| Signal | Interpretation | Weight |
|--------|----------------|--------|
| Retry within 30 seconds | Response was bad/incomplete | Strong negative |
| Conversation continues (next turn < 2 min) | Response was good | Moderate positive |
| Session ends immediately after response | Response was bad or task complete | Ambiguous |
| Response length vs prompt complexity | Length appropriateness | Moderate |
| Streaming completion rate (no abort) | User read the full response | Positive |
| Time-to-first-token (TTFT) | Latency quality | Latency signal |
| Total generation time | Throughput quality | Latency signal |
| Follow-up question similarity | Needed clarification | Mild negative |
| Same request re-sent (exact match) | Response unsatisfactory | Strong negative |
| Response abort (stream cancelled) | User didn't want this response | Negative |
| Copy/paste action on response | Response was useful | Strong positive (if tracked) |

**Session-level signals**:
- Session depth (total turns): longer sessions = more engaged/successful
- Task completion indicators: session ends with longer gaps between turns (task done)
- Retry patterns: repeated topic attempts = model failing on this task type
- Cross-session return rate: user comes back = platform is working

**What NOT to collect** (privacy risk):
- Any portion of prompt or response content
- User identifiers linked to content
- IP addresses linked to content
- Timestamps at sub-minute resolution linked to specific requests

---

### 1.7 LoRA and QLoRA for Per-Customer Fine-tuning

**LoRA (Low-Rank Adaptation)** — Production Ready:
- Freezes pretrained weights; injects trainable rank-decomposition matrices into each transformer layer
- Trainable parameters: ~0.1-0.5% of total model parameters
- 10,000x fewer trainable parameters than full fine-tuning for GPT-3 scale
- 3x GPU memory reduction; higher training throughput
- **Critical**: No additional inference latency (adapters merge into weights at inference time)
- Multi-tenant architecture: 1 base model (40GB) + N×19MB adapters → vastly cheaper than N full models

**QLoRA** — Consumer-GPU Fine-tuning:
- 4-bit NormalFloat (NF4) quantization for base model weights
- Double quantization (quantize the quantization constants)
- Paged optimizers for memory spike management
- Result: 65B model fine-tunable on a single 48GB GPU
- 99.3% of ChatGPT quality after 24 hours on single GPU
- Enables per-customer fine-tuning on the LLMHosts node itself

**FFA-LoRA (ICLR 2024)** — Federated DP-LoRA:
- Specifically designed for federated learning + differential privacy combination
- Fixes the randomly initialized non-zero matrix A; only fine-tunes zero-initialized matrix B
- Eliminates instability from heterogeneous data + multi-step local updates
- 50% communication cost reduction vs vanilla FedLoRA
- Differential privacy noise added on updating gradients
- **This is the right approach for LLMHosts's federated collective learning**

**Practical Multi-tenant LoRA Architecture**:
```
Base Model (frozen, shared): llama-3.1-8B / mistral-7b [on device]
Customer A adapter: 19MB LoRA weights (rank=8)
Customer B adapter: 23MB LoRA weights (rank=16)
Platform collective adapter: 45MB LoRA weights (trained on anonymized aggregate)
```

---

### 1.8 Continual Learning for LLMs

**Key Problem**: Standard fine-tuning on new data causes catastrophic forgetting — the model loses prior capabilities as it learns new ones.

**Solutions** (from continual learning survey, arXiv:2402.01364):

1. **Continual Pretraining**: Extend pretraining on new domain data with learning rate warmup and curriculum scheduling
2. **Instruction Tuning**: Replay a mix of old and new instruction examples during fine-tuning
3. **Elastic Weight Consolidation (EWC)**: Penalizes changes to weights important for previous tasks (Fisher information matrix)
4. **Progressive Neural Networks**: Add new capacity for new tasks, freeze old capacity
5. **LoRA-based Continual Learning**: New LoRA adapters per task; compose at inference time → naturally avoids forgetting

**Recommended for LLMHosts**: LoRA-based continual learning. Each customer's adapter evolves over time by training new LoRA deltas on recent anonymized usage patterns. The base model never changes, so catastrophic forgetting is impossible by construction.

---

### 1.9 Apple's Privacy-Preserving On-Device Learning

**Apple's Local Differential Privacy (LDP) Architecture**:

1. **Device-side privatization**: Data is randomized locally before transmission. Server never sees raw data.
2. **IP/timestamp stripping**: Privatized records stripped of identifiers before aggregation.
3. **Daily transmission limits**: Per-user limits prevent excessive leakage across submissions.
4. **Event-level DP**: Each data point independently receives noise injection.

**Apple's Algorithms**:
- **Count Mean Sketch (CMS)**: Hash functions + bit-flipping to estimate frequencies privately
- **Hadamard CMS (HCMS)**: Single-bit transmission per event (extreme bandwidth efficiency)
- **Sequence Fragment Puzzle (SFP)**: Discovers popular patterns without seeing individual sequences

**Key Lesson for LLMHosts**: Apple's model is "collect structural signals, privatize on-device, aggregate privately on server." LLMHosts can adopt the same pattern: collect usage metadata locally on the GPU node, apply local DP before any signal leaves the node, aggregate centrally.

---

### 1.10 Google's Production Federated Learning

**Key Facts** (from Google's FL blog):
- Deployed on Gboard (200M+ Android devices)
- Federated Averaging reduces communication 10-100x vs naive distributed SGD
- Additional compression: random rotations + quantization → further 100x reduction
- On-device training: miniature TensorFlow runs only when device is idle + plugged in + on free WiFi
- **Secure Aggregation**: Cryptographic protocol where server can only decrypt the *average* if 100s-1000s of users participated — no individual update can be inspected

**Key Takeaway**: Google's secure aggregation protocol is exactly what LLMHosts needs for aggregating routing-quality signals across customer nodes. The math guarantees that even if the aggregation server is compromised, individual customer data cannot be recovered.

---

### 1.11 Homomorphic Encryption and ZK Proofs — Current State

**Homomorphic Encryption (HE)**:
- Allows computation on encrypted data without decryption
- Fully Homomorphic Encryption (FHE): supports arbitrary computation, but remains 1000-10000x slower than plaintext computation in 2026
- Partially Homomorphic: fast, but limited to one operation type (add OR multiply, not both)
- **Verdict for LLMHosts**: Not practical for model training. Useful only for specific, narrow aggregation operations (count queries, sums). Use DP + secure aggregation instead.

**Zero-Knowledge Proofs (ZKP)**:
- Prove a statement is true without revealing the underlying data
- Use case: prove that a model update was computed correctly on real data without revealing the data
- Current state (2026): zkML (ZK for ML) is active research area; proving inference for small models (<10M params) is feasible but proving training is still extremely slow
- **Verdict for LLMHosts**: Long-term research track. Phase 2 or later. Not ready for production at model scale.

---

### 1.12 Synthetic Data Generation for Privacy

**Select-Measure-Generate Paradigm** (from differentialprivacy.org):
1. **Select**: Choose which statistics to measure (marginals, correlations)
2. **Measure**: Measure those statistics with differential privacy
3. **Generate**: Produce synthetic data matching the privatized statistics

**Relevance for LLMHosts**:
- Instead of sharing raw usage logs, generate synthetic usage traces that preserve statistical properties (task type distribution, token length distribution, quality signal distributions) but are provably disconnected from any individual's queries.
- Synthetic usage data can be used to train collective routing improvements and published to all nodes.

**Key Use Case**: If LLMHosts wants to share "what task types worked well on Llama-3.1-70B vs Mistral-7B this week," it generates synthetic data that reflects the aggregate distribution without revealing any individual session.

---

### 1.13 GDPR and CCPA Compliance Requirements

**GDPR (EU)** — Article 6 Legal Bases:
- **Consent** (Art. 6(1)(a)): Explicit, freely given, specific, informed, and unambiguous. Required for AI training on personal data.
- **Legitimate Interests** (Art. 6(1)(f)): Processing for reasonable business purposes unless overridden by data subject rights. Arguable for purely structural (non-content) signals, but risky.
- **Data Minimization Principle**: Collect only what's necessary for the stated purpose.
- **Purpose Limitation**: Data collected for one purpose cannot be repurposed (e.g., usage data for billing cannot be silently repurposed for training).

**Key GDPR Requirements for AI Training**:
- Explicit opt-in consent required when personal data is used for training
- Right to withdraw consent → model must be retractable ("machine unlearning" or exclude from future training)
- Right to explanation for automated decisions
- Data retention limits: cannot retain training data indefinitely
- Privacy Impact Assessment (DPIA) required for large-scale automated processing

**CCPA (California)**:
- **Right to Know**: Disclose what personal data is collected and how it's used
- **Right to Delete**: Honor deletion requests (conflicts with gradient retention)
- **Right to Opt-Out**: Stop selling/sharing personal data (applies to data shared with third parties)
- **Right to Limit**: Restrict use of sensitive personal information

**Emerging Trend (UK ICO, 2025 Data Use and Access Act)**:
- AI and data protection guidance under active review as of 2025-2026
- Key principle: statistical accuracy vs individual accuracy distinction — aggregate statistics do not identify individuals and face lower burden

**Practical Compliance for LLMHosts** (see §3 for full framework):
- Structural metadata (token counts, latency, task classification from structure) = lowest risk, likely legitimate interests basis
- Opt-in collective learning program = safe for any data type under consent basis
- Must provide clear data deletion path (retraining exclusion)
- Must not link any signal to identifiable user without explicit consent

---

### 1.14 Model Performance Prediction from Hardware Configuration

**Hardware signals that predict model performance** (VRAM, architecture, bandwidth):
- VRAM capacity → which model sizes can run fully in VRAM without swapping
- GPU architecture generation (Ampere vs Hopper vs Ada Lovelace) → throughput per parameter
- Memory bandwidth (GB/s) → tokens/second for autoregressive generation (bandwidth-bound)
- TDP and sustained clock speeds → consistency of performance under load
- PCIE vs NVLink → multi-GPU scaling efficiency

**Model selection based on hardware** (existing in llmhosts HIM — Hardware Intelligence Manager):
- Already implemented: VRAM-based model tier selection
- Can be extended with: per-GPU-architecture throughput coefficients learned from collective data
- Example: "RTX 4090 achieves 85 tok/s on Llama-3.1-8B-Q4; 3090 achieves 62 tok/s" — shareable without exposing customer identity

---

## Part 2: LLMHosts Collective Intelligence System Design

### 2.1 System Overview

The **LLMHosts Collective Intelligence System (CIS)** enables platform-wide learning from anonymized usage signals. It answers three questions better over time:

1. **Which model should handle this request?** (routing optimization)
2. **Which model works best on this hardware?** (hardware-aware dispatch)
3. **Which models are improving/degrading?** (quality drift detection)

No individual customer's queries, prompts, or responses are ever shared or inspectable.

### 2.2 Data Collection Architecture

**Layer 0: Raw signals (on-node only, never transmitted)**
```
- Full prompt text
- Full response text
- User identity
- IP address
- Timestamps at sub-second resolution
```

**Layer 1: Structural metadata (collected, anonymized before transmission)**
```python
class RequestSignal:
    # Timing (no identity)
    request_id: str          # UUID, rotated every 24h
    session_hash: str        # SHA-256(customer_id + date), not reversible
    hour_of_week: int        # 0-167, rounded to nearest hour

    # Task structure (no content)
    prompt_token_count: int  # Bucketed: [1-50, 51-200, 201-1000, 1001+]
    has_code_markers: bool   # Presence of ```, <code>, etc. (not content)
    is_multi_turn: bool      # Is this turn 2+?
    turn_number: int         # Capped at 10

    # Response characteristics (no content)
    response_token_count: int  # Bucketed same as prompt
    time_to_first_token_ms: int  # Rounded to nearest 50ms
    total_generation_ms: int    # Rounded to nearest 100ms
    stream_completed: bool      # Did user receive full response?

    # Model & hardware (no identity)
    model_id: str            # e.g., "llama-3.1-8b-q4_k_m"
    gpu_arch: str            # e.g., "ampere", "ada_lovelace", "rdna3"
    vram_tier: str           # "8gb", "12gb", "16gb", "24gb", "48gb+"

    # Quality signals (implicit, no content)
    retry_within_60s: bool   # Same session, same approximate task
    session_continued: bool  # Follow-up within 5 minutes
    stream_aborted: bool     # User cancelled mid-generation
```

**Layer 2: Local DP anonymization (applied on-node before any transmission)**

```python
def anonymize_signal(signal: RequestSignal, epsilon: float = 5.0) -> AnonSignal:
    """Apply local differential privacy before any data leaves the node."""
    return AnonSignal(
        # Randomized response mechanism for boolean fields
        has_code_markers=randomized_response(signal.has_code_markers, epsilon),
        retry_within_60s=randomized_response(signal.retry_within_60s, epsilon),
        stream_completed=randomized_response(signal.stream_completed, epsilon),
        session_continued=randomized_response(signal.session_continued, epsilon),

        # Laplace noise for counts (already bucketed)
        prompt_token_bucket=add_laplace_noise(signal.prompt_token_count, sensitivity=1, epsilon=epsilon),
        response_token_bucket=add_laplace_noise(signal.response_token_count, sensitivity=1, epsilon=epsilon),

        # Non-sensitive fields passed through
        hour_of_week=signal.hour_of_week,
        model_id=signal.model_id,
        gpu_arch=signal.gpu_arch,
        vram_tier=signal.vram_tier,
    )
    # Note: session_hash and request_id are dropped entirely in Layer 2
```

**Layer 3: Secure aggregation (server-side)**
- Signals batched in 24-hour windows per participating node
- Server receives only aggregated distributions, never individual signals
- Minimum threshold: ≥50 signals per node per day to submit (protects small-volume customers)
- Google-style secure aggregation: server sees only sum/average across N nodes, not individual node contributions

### 2.3 What the Collective Intelligence System Learns

**Module A: Task Classifier (privacy-safe)**
```
Input: structural features (token counts, code markers, turn number, etc.)
Output: task_type ∈ {coding, reasoning, creative, factual_qa, summarization, conversation}
Training: supervised on synthetic labeled data; updated monthly from aggregate signals
Privacy: no content features; structural features only
```

**Module B: Per-Model-Per-Task Quality Matrix**
```
Maintains: Q[model_id][task_type][gpu_arch] → quality_score ∈ [0, 1]
Update rule: Thompson sampling with quality signal as reward
Signal: session_continued=True → +reward; retry_within_60s=True → -reward; stream_aborted=True → -reward
Update: daily batched update from anonymized collective signals
```

**Module C: Hardware Performance Profile**
```
Maintains: P[model_id][gpu_arch][vram_tier] → {avg_ttft_ms, avg_tps, p95_tps}
Update rule: median of reported performance statistics per (model, gpu, vram) cell
Minimum cell size: 10 reports before publishing (k-anonymity protection)
Distribution: published as public lookup table to all nodes; no sensitive data
```

**Module D: Drift Detector**
```
Monitors: quality_score[model_id][task_type] time series
Alert: if rolling 7-day average drops >15% vs 30-day baseline → model quality drift detected
Action: deprioritize model in routing; human review trigger
Privacy: computed on aggregate statistics only
```

### 2.4 Routing Policy Update Loop

```
Every 24 hours:
1. Each opt-in node submits locally-DP-anonymized signal batch
2. Platform server applies secure aggregation (FedAvg-style)
3. Collective Q matrix updated via Thompson sampling
4. Hardware performance profiles updated
5. Drift detector runs
6. Updated routing policy published to all nodes (small JSON file)
7. Nodes update their local routing model from published policy

Local routing on each node:
- Task classification from structural features (Module A)
- Model selection from Q matrix lookup (Module B)
- Hardware-aware filtering via performance profiles (Module C)
- Result: best available local model for this task on this hardware
```

---

## Part 3: Per-Customer Continuous Improvement System

### 3.1 Individual LoRA Adaptation

**Architecture**:
```
Base model (shared, frozen): e.g., Llama-3.1-8B-Q4
Customer adapter: LoRA rank=8, ~19MB
Training trigger: every N requests (N = 500 by default, configurable)
Training data: customer's own anonymized interaction history (prompt structure + feedback signals)
Privacy: adapter stored locally on customer's node; never transmitted to platform
```

**Training signal for individual adapter**:
- **Session continuation** as positive reinforcement: if user continued the session after this response, this (prompt_structure, model_response) pair was good
- **Retry signal** as negative reinforcement: retry within 60s = this response pattern failed
- **Only prompt structure is used, not content**: The adapter learns "on coding requests of this length, this model generates better responses" — not "the user was asking about X"

**Implementation approach using QLoRA + continual LoRA**:
```python
class CustomerAdapter:
    """Per-customer LoRA adapter that improves over time."""

    def __init__(self, base_model_id: str, customer_id: str):
        self.lora_rank = 8
        self.lora_alpha = 16
        self.adapter_path = f"~/.llmhosts/adapters/{customer_id}/{base_model_id}/"

    def update_from_session(self, session_signals: list[RequestSignal]):
        """Retrain adapter delta from recent session signals."""
        # Filter: only use signals with clear quality indication
        positive = [s for s in session_signals if s.session_continued and not s.retry_within_60s]
        negative = [s for s in session_signals if s.retry_within_60s or s.stream_aborted]

        if len(positive) + len(negative) < 20:
            return  # Not enough signal yet

        # Generate synthetic prompts matching the structural pattern
        # (length, code presence, turn number) without any content
        synthetic_training_data = self._generate_structural_synthetic(positive, negative)

        # QLoRA fine-tuning on synthetic structural data
        self._qlora_finetune(synthetic_training_data)
```

**Catastrophic Forgetting Prevention**:
- New LoRA delta is computed on recent signals only
- Adapter composition: `total_adapter = 0.7 * previous_adapter + 0.3 * new_delta`
- This exponential moving average ensures old learning persists
- Alternative: maintain a small replay buffer of "representative" historical patterns

### 3.2 System Prompt Optimization

**Goal**: Automatically improve the system prompt prepended to each request to get better results from the chosen model.

**What gets optimized**:
- Format instructions ("respond in markdown", "be concise")
- Persona calibration ("you are an expert software engineer")
- Context setting appropriate for the detected task type

**How it learns** (privacy-safe):
1. A/B test different system prompt templates per task type
2. Measure quality signal (session continuation rate) per template
3. Promote templates with higher quality signal
4. Never A/B test on prompt content; only vary system prompt text

**Implementation**:
```
prompt_template_library = {
    "coding": [
        "You are an expert software engineer. Respond with working code.",
        "You are a senior developer. Always explain your reasoning.",
        "You are a coding assistant. Be precise and complete."
    ],
    "reasoning": [...],
    "creative": [...],
}

# Thompson sampling over templates per task type
# Reward: session_continued=True; Penalty: retry_within_60s=True
```

### 3.3 Model Warm Recommendation

**Goal**: As a customer uses LLMHosts over time, proactively recommend model upgrades or changes.

**Triggers** (no content inspection):
- Quality score for current preferred model drops below threshold
- New model added to platform that benchmarks better for this customer's task distribution
- Customer's hardware upgraded (higher VRAM available)
- Collective intelligence signals suggest a better model for this task mix

**Recommendation mechanism**:
```python
def recommend_model_upgrade(customer_task_distribution: dict,
                            current_model: str,
                            available_models: list[str]) -> Optional[str]:
    """Recommend a better model based on collective quality matrix."""
    current_score = collective_Q[current_model]  # aggregate score

    for candidate in available_models:
        candidate_score = collective_Q[candidate]
        task_weighted_improvement = sum(
            customer_task_distribution[task] *
            (candidate_score[task] - current_score[task])
            for task in customer_task_distribution
        )
        if task_weighted_improvement > 0.05:  # 5% threshold
            return candidate
    return None
```

---

## Part 4: Privacy Compliance Framework

### 4.1 Data Classification Tiers

| Tier | Data Type | Retention | Sharing | Legal Basis |
|------|-----------|-----------|---------|-------------|
| T0 — Forbidden | Prompt/response content, PII | Never collected | Never | N/A |
| T1 — Private | Customer ID linked to any signal | Session only | Never | N/A |
| T2 — Sensitive | Raw structural signals (pre-DP) | 24 hours on-node | Never | Legitimate interests |
| T3 — Anonymized | DP-anonymized aggregate signals | 90 days | Platform only | Consent (opt-in) |
| T4 — Public | Hardware performance profiles (k-anon) | Indefinite | Public | Legitimate interests |
| T5 — Published | Collective routing policy | Indefinite | All nodes | Legitimate interests |

### 4.2 Opt-In Architecture

**Three-level consent model**:

```
Level 0 (Default, No Consent Required):
- Local routing optimization using the customer's own private data
- Individual LoRA adapter trained locally, never transmitted
- Hardware performance monitoring for local use
- No data leaves the customer's node

Level 1 (Opt-In: Collective Learning):
- DP-anonymized structural signals contributed to platform aggregate
- Node receives updated collective routing policy
- Contributes to hardware performance profiles
- 24-hour DP signals, no content, no identity

Level 2 (Opt-In: Enhanced Learning):
- Level 1 plus: participating in federated LoRA training for base models
- FFA-LoRA with ε=5 DP guarantee
- Node receives base model improvements from collective
- Requires explicit additional consent
```

**Consent UI Requirements**:
```
✓ Plain-language explanation of what is collected and what it learns
✓ Explicit opt-in action (not pre-checked box)
✓ Separate consent for Level 1 vs Level 2
✓ Easy opt-out path (one command: `llmhosts privacy opt-out`)
✓ Consent recorded with timestamp and version
✓ Consent renewal required on privacy policy changes
```

**Implementation in CLI**:
```bash
llmhosts privacy status       # Show current consent level and what's collected
llmhosts privacy opt-in       # Guided consent flow
llmhosts privacy opt-out      # Immediate opt-out, stops all signal transmission
llmhosts privacy export       # Export all collected signals (GDPR Right to Access)
llmhosts privacy delete       # Request deletion from aggregate models (GDPR Right to Erasure)
```

### 4.3 Data Retention Policy

| Data Type | Retention | Deletion Trigger |
|-----------|-----------|------------------|
| Raw structural signals (pre-DP) | 24 hours on-node | Automatic rotation |
| DP-anonymized batch submissions | 90 days on platform | Automatic purge |
| Individual LoRA adapters | Until user deletes | `llmhosts adapter delete` |
| Collective routing policy | Indefinite (no PII) | N/A |
| Hardware performance tables | Indefinite (no PII) | N/A |
| Consent records | Lifetime of account + 3 years | Account deletion |

**Machine Unlearning Protocol** (GDPR Right to Erasure):
- For DP-aggregated data: mathematical argument that contribution is indistinguishable (ε-DP guarantee) — legally sufficient in most EU jurisdictions
- For future training: mark customer ID as excluded; do not include in next training run
- For individual adapters: delete adapter files immediately upon request
- Response SLA: 72 hours for erasure confirmation

### 4.4 Audit Trail Architecture

```python
class PrivacyAuditLog:
    """Immutable audit log for all privacy-relevant events."""

    event_types = [
        "consent_granted",
        "consent_revoked",
        "signal_batch_submitted",  # logs: batch_id, epsilon_applied, node_id_hash
        "adapter_created",
        "adapter_updated",
        "adapter_deleted",
        "erasure_requested",
        "erasure_completed",
        "data_export_requested",
        "data_export_delivered",
        "policy_updated",           # logs: old_version, new_version, change_summary
    ]

    # Stored: customer_id_hash (not plaintext), event_type, timestamp, metadata
    # Never stored: signal content, prompt/response fragments
```

**Data Processing Records** (GDPR Article 30 requirement):
- Maintained as machine-readable records
- Covers: what data is processed, for what purpose, who has access, retention periods
- Updated whenever processing changes
- Available to regulators on request

### 4.5 Security Controls for DP Data Pipeline

**On-node (before transmission)**:
- DP noise applied locally (local DP — stronger than central DP)
- All network communication encrypted (TLS 1.3)
- Signal batch signed with node's key pair (integrity verification)
- IP address stripped at first hop before signals reach platform servers

**On-platform (aggregation server)**:
- Secure aggregation protocol: server computes sum of N batches but cannot inspect individual batches
- Minimum participation threshold: N≥50 nodes per aggregation window
- Aggregated model updates stored in HSM-backed storage
- Access control: only the automated pipeline service can read raw aggregate; no human access

**Published artifacts (routing policy, hardware profiles)**:
- Published as signed, versioned artifacts (similar to software package signing)
- Nodes verify signature before applying policy update
- Full public audit log of policy changes

---

## Part 5: Implementation Roadmap

### Phase 1: Foundation (Weeks 1-8) — Build First, Privacy First

**Week 1-2: Structural Signal Collection**
- Implement `RequestSignal` data class on-node (see §2.2)
- Collect signals locally in SQLite (never transmitted at this stage)
- Add `llmhosts privacy status` command showing what would be collected
- No transmission, no privacy risk — data never leaves node

**Week 3-4: Task Classifier (Local)**
- Train offline on synthetic labeled dataset (use Claude/GPT to generate labeled structural examples)
- Deploy locally on each node: classifies task type from structural features only
- Feeds into routing decisions immediately
- Zero privacy impact: runs entirely locally

**Week 5-6: Local Thompson Sampling Router**
- Implement per-node bandit router
- Context: {task_type, vram_tier, hour_of_week}
- Reward: session_continued, retry signal
- Starts with uniform priors; adapts after ~100 requests
- No data transmission; pure local improvement

**Week 7-8: Individual LoRA Infrastructure**
- LoRA adapter storage: `~/.llmhosts/adapters/{customer_id}/{model_id}/`
- QLoRA training pipeline for local adapter updates
- Trigger: every 500 requests with sufficient quality signal
- Adapter hot-swap at inference time (no downtime)
- `llmhosts adapter status` / `llmhosts adapter train` commands

**Phase 1 Deliverable**: Every customer's node self-improves based on their own private usage. Zero data shared. Zero privacy risk. Measurable quality improvement.

---

### Phase 2: Collective (Weeks 9-20) — Opt-In Aggregation

**Week 9-11: Privacy Pipeline**
- Implement local DP anonymization (randomized response + Laplace noise)
- Implement batch signing and transmission
- Implement `llmhosts privacy opt-in` consent flow (Level 1)
- Platform-side: receive and validate signed batches

**Week 12-14: Secure Aggregation Server**
- Deploy aggregation server with minimum-N threshold (50 nodes)
- Implement FedAvg-style aggregation over DP signal batches
- Compute collective Q matrix (per-model, per-task, per-gpu-arch quality scores)
- Publish routing policy as signed JSON artifact

**Week 15-17: Hardware Performance Profiles**
- Collect performance metrics (TTFT, TPS) with k-anonymity (min 10 per cell)
- Publish hardware performance lookup table
- Nodes use table for hardware-aware model selection

**Week 18-20: Drift Detection + Alerts**
- Implement quality drift detector on collective Q matrix time series
- Alert mechanism: email + `llmhosts models status` flag for degraded models
- Human review workflow for flagged models

**Phase 2 Deliverable**: Platform learns collectively. Routing gets better for everyone. Hardware selection improves. Quality regressions detected before users notice. All privacy-preserving.

---

### Phase 3: Advanced (Months 6-12) — Federated LoRA

**Month 6-7: FFA-LoRA Implementation**
- Implement FFA-LoRA training protocol (ICLR 2024 paper)
- DP-federated LoRA fine-tuning for base models
- `llmhosts privacy opt-in --level=2` consent flow
- Distribute improved base models to participating nodes

**Month 8-9: Prompt Optimization System**
- System prompt template library per task type
- A/B testing framework for prompt templates (Thompson sampling)
- Measure quality signal per template; promote winners
- Customer-specific prompt optimization separate from collective

**Month 10-11: Model Warm Recommendations**
- Track customer task distribution over time (locally)
- Match against collective Q matrix to recommend model upgrades
- Proactive notifications: "New model X benchmarks 23% better for your coding workload"

**Month 12: Compliance Automation**
- Automated GDPR erasure workflow
- Data export pipeline (Right to Access)
- Consent renewal system (triggered by policy changes)
- Annual privacy audit support tooling

---

### Phase 4: Research (Year 2+) — Long-Term Bets

**Synthetic Data Generation**:
- From anonymized structural patterns, generate synthetic usage traces
- Train models on synthetic data → public artifact, no privacy risk
- Enables sharing more detailed training improvements

**ZK-ML for Model Verification**:
- Zero-knowledge proofs that a model update was computed correctly
- Allows nodes to verify collective model improvements without trusting the server
- Technology readiness: 2-3 years from production-readiness

**Personalized Prefix Tuning**:
- Soft prompt tuning (prefix tokens) as alternative to LoRA for per-customer adaptation
- Potentially faster to update than LoRA; smaller storage footprint
- Evaluate vs LoRA based on quality signal

**On-Device Federated Pretraining**:
- Using Flower SuperGrid, enable collective pretraining on non-sensitive domain data
- Allows building specialized models for common customer domains (biomedical, legal, financial)
- Requires legal agreements and explicit per-domain consent

---

## Part 6: Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                     CUSTOMER NODE (GPU)                          │
│                                                                   │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              LAYER 0: Raw Inference                      │    │
│  │  prompt → [model] → response                            │    │
│  │  (NEVER leaves node; NEVER collected)                    │    │
│  └─────────────────────┬───────────────────────────────────┘    │
│                         │ structural signals only                 │
│  ┌─────────────────────▼───────────────────────────────────┐    │
│  │              LAYER 1: Local Intelligence                 │    │
│  │  • Task Classifier (structural features → task_type)    │    │
│  │  • Thompson Sampling Router                             │    │
│  │  • Individual LoRA Adapter (locally trained)            │    │
│  │  • Prompt Template Optimizer                            │    │
│  │  • Signal Collector (SQLite, 24h rotation)              │    │
│  └─────────────────────┬───────────────────────────────────┘    │
│                         │ if opted in (Level 1+)                  │
│  ┌─────────────────────▼───────────────────────────────────┐    │
│  │           LAYER 2: Local DP Anonymization               │    │
│  │  • Randomized response on booleans (ε=5)                │    │
│  │  • Laplace noise on counts                              │    │
│  │  • Drop all identifiers                                 │    │
│  │  • Sign batch with node keypair                         │    │
│  └─────────────────────┬───────────────────────────────────┘    │
└────────────────────────┼────────────────────────────────────────┘
                         │ encrypted, signed batch (no IP)
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                   LLMHOSTS PLATFORM (Cloud)                      │
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │           Secure Aggregation Server                       │   │
│  │  • Verify batch signatures                               │   │
│  │  • Enforce N≥50 minimum threshold                        │   │
│  │  • FedAvg aggregation of DP signals                      │   │
│  │  • No individual batch inspectable                       │   │
│  └──────────────┬──────────────────────────────────────────┘   │
│                  │                                                │
│  ┌───────────────▼───────────────────────────────────────────┐  │
│  │           Collective Intelligence Modules                   │  │
│  │  A: Quality Matrix Q[model][task][gpu]                    │  │
│  │  B: Hardware Performance P[model][gpu][vram]              │  │
│  │  C: Drift Detector                                        │  │
│  │  D: Synthetic Data Generator                              │  │
│  └───────────────┬───────────────────────────────────────────┘  │
│                  │                                                │
│  ┌───────────────▼───────────────────────────────────────────┐  │
│  │           Published Artifacts (signed, versioned)           │  │
│  │  • routing_policy.json (Q matrix → routing rules)         │  │
│  │  • hardware_profiles.json (perf lookup table)             │  │
│  │  • model_quality_alerts.json (drift notifications)        │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                         │ all nodes receive
                         ▼
              All LLMHosts Nodes (opt-in and non-opt-in)
              receive routing policy improvements
```

---

## Part 7: Key Design Decisions and Rationale

### Decision 1: Local DP over Central DP
**Choice**: Apply DP noise on-node before transmission (local DP).
**Why**: Stronger privacy guarantee. Even if the platform server is compromised, individual signals cannot be recovered. Central DP (noise added at server) requires trusting the server.
**Tradeoff**: Requires more noise for same epsilon → slightly lower utility. Acceptable at scale with N>50 nodes.

### Decision 2: FFA-LoRA for Federated Fine-tuning
**Choice**: Use FFA-LoRA (ICLR 2024) instead of vanilla FedLoRA.
**Why**: FFA-LoRA fixes the instability problem in heterogeneous federated settings. 50% communication cost reduction. Designed for DP integration. Production-validated.

### Decision 3: Thompson Sampling over UCB
**Choice**: Thompson sampling for routing bandit.
**Why**: Better empirical performance in non-stationary environments (model quality changes over time). Naturally handles exploration. Converges faster. Easy to implement correctly.

### Decision 4: Structural Signals Only (No Content)
**Choice**: Zero content features in any signal, even optional.
**Why**: Content signals create irreversible privacy risk. Structural signals provide 80% of the routing signal value with 0% of the privacy risk. GDPR compliance is dramatically simpler.

### Decision 5: Three-Level Consent (0, 1, 2)
**Choice**: Default=0 (no sharing), two explicit opt-in levels.
**Why**: Level 0 provides value immediately (local improvement). Opt-in creates trust. Separating federated model training (Level 2) from aggregate signal collection (Level 1) allows users to choose their comfort level. Aligns with GDPR consent requirements.

### Decision 6: k-Anonymity on Hardware Profiles
**Choice**: Only publish hardware performance cells with ≥10 reports.
**Why**: A cell with 1 report reveals the individual node's performance. k-anonymity (k=10) ensures any published metric represents at least 10 different nodes, making identification impossible.

---

## Part 8: References and Source Material

**Federated Learning**:
- Flower Framework v1.26 (flower.ai) — Production FL framework with LLM support, FlowerTune leaderboard
- Google FL Blog — FedAvg, Gboard deployment, Secure Aggregation protocol
- FFA-LoRA (ICLR 2024) — DP-federated LoRA for LLMs

**LLM Routing**:
- RouteLLM (arXiv:2406.18665) — Learned routers from preference data; 2x cost reduction
- FrugalGPT (arXiv:2305.05176) — LLM cascade; 98% cost reduction matching GPT-4
- OpenRouter docs — Production routing with NotDiamond; provider health, fallback chains

**Differential Privacy**:
- OpenDP v0.8 (Harvard, Feb 2026) — Production DP library; DP Wizard
- Apple Local DP (machinelearning.apple.com) — CMS, HCMS, SFP algorithms
- DifferentialPrivacy.org — NeurIPS 2023 auditing, synthetic data paradigms

**LoRA/Fine-tuning**:
- LoRA (arXiv:2106.09685) — 10,000x parameter reduction; no inference latency
- QLoRA (arXiv:2305.14314) — 65B fine-tuning on single GPU; NF4 quantization
- PEFT library (HuggingFace) — Production multi-tenant LoRA architecture
- Continual Learning survey (arXiv:2402.01364) — LoRA-based continual learning to prevent forgetting

**Privacy Compliance**:
- GDPR Article 6 (gdpr.eu) — Legal bases for processing; consent requirements
- CCPA (oag.ca.gov) — Rights to know, delete, opt-out, correct, limit
- UK ICO AI Guidance (ico.org.uk) — Under review post Data Use and Access Act 2025

**Privacy Attacks (motivating strong DP)**:
- PII leakage in LLMs (arXiv:2302.00539) — Even sentence-level DP leaks ~3% PII
- Diffusion model training data extraction — Models memorize and reproduce training data
