# RFC-002: Lattice — Bicameral Memory Architecture

**Status:** Draft  
**Created:** 2026-02-10  
**Last Updated:** 2026-02-17  
**Supersedes:** RFC-001

---

## 1. Problem Statement

### 1.1 The "Goldfish Agent" Problem
Current AI agents suffer from **Session Amnesia**. Every session starts from zero. Users repeatedly re-explain preferences, conventions, and decisions — whether the agent is writing code, conducting research, managing tasks, or assisting with any other domain.

### 1.2 Why Existing Approaches Fail
*   **Vector DB (RAG)**: Passive. Waits for query. Cannot enforce instincts. Retrieval quality is inconsistent.
*   **Static Config (CLAUDE.md, custom instructions)**: Users rarely update it. Insights from sessions are lost.
*   **Filesystem (Unix grep)**: Finds artifacts, but cannot find "Why we decided X last Tuesday."

### 1.3 The "Bicameral Mind" Solution
We split cognition into two independent systems:

| System | Role | Storage | Latency | Content |
| :--- | :--- | :--- | :--- | :--- |
| **System 1 (Instinct)** | Always-on rules | Markdown files | **0ms** (pre-loaded) | Preferences, conventions, guardrails |
| **System 2 (Memory)** | Deep recall | SQLite DB | **~100ms** (on-demand) | Full logs, decisions, context snapshots |

**Core Innovation**: The **Compiler** — an LLM process that reads System 2 (raw logs) and synthesizes System 1 (compact rules).

---

## 2. Core Concepts

### 2.1 The Compiler (Instinct Evolution)
Lattice does not just store logs. It **compiles** them into instincts.
*   **Input**: Episodic logs from `store.db` (indexed session history).
*   **Process**: LLM analyzes logs → identifies patterns → generates rule proposals.
*   **Output**: A Markdown proposal in `.lattice/drift/proposals/`.
*   **Constraint**: System 1 files should remain compact and high-signal. The Compiler must **rewrite and prune**, not just append. There is no hard token budget — instead, `lattice status` reports current token count with configurable advisory thresholds (defaults: warn at 3000, alert at 5000). Every `evolve` run includes a mandatory `<review>` phase that evaluates existing rules for staleness, redundancy, and contradiction.

### 2.2 The Store (Episodic Library)
A high-capacity, searchable archive of raw experiences.
*   **Input**: Real-time stream of User/Agent interactions (via passive Plugin capture, MCP `log_turn` tool, or SDK call).
*   **Process**: Indexed on ingest. Search uses BM25 (FTS5) by default; when embedding is configured, adds vector search + RRF fusion for hybrid recall.
*   **Output**: Search results via MCP `lattice_search` tool, SDK `client.search()`, or `lattice search` CLI.

### 2.3 The Overlay (Global vs. Project)
Memory is hierarchical at both System 1 and System 2 layers. **Project overrides Global.**

**System 1 (Rules)**:
*   **Global (`~/.config/lattice/rules/`)**: User preferences across all projects (e.g., "prefer concise responses", "always list pros/cons before deciding", "prefer TDD").
*   **Project (`./.lattice/rules/`)**: Project-specific conventions (e.g., "use Hexagonal Architecture", "snake_case", "summarize research findings in bullet points").
*   **Merge Strategy**: Concatenate both layers into a single prompt. When a project rule conflicts with a global rule on the same topic, the project rule takes precedence. Conflict detection is delegated to the consuming LLM — no automated topic-level matching. Compiler is encouraged (via prompt) to use section headings (`## Testing`, `## Style`) to help the LLM identify same-topic rules.

**System 2 (Store)**:
*   **Project (`store.db`)**: Raw episodic logs from sessions within this project. This is the primary write target for all data capture paths.
*   **Global (`global.db`)**: Cross-project evidence summaries, written exclusively by the Global Compiler. Not a copy of raw logs — a curated index of patterns and supporting evidence extracted during global evolution.

**Search Scope**:
*   Default: project `store.db` only.
*   `lattice search --global` or `lattice_search(query, scope="global")`: queries `global.db` for cross-project recall.

### 2.4 Promotion (Global Compiler)
Promotion is not a separate mechanism — it is the **Global Compiler**. The same Structured CoT process used at the project level operates at the global level, with different input scope and convergence granularity.

| | Project Compiler | Global Compiler |
| :--- | :--- | :--- |
| **Trigger** | `lattice evolve` | `lattice evolve --global` |
| **Reads Store** | Current project `store.db` | All projects' `rules/*.md` + `drift/traces/` + targeted evidence from project stores |
| **Reads Rules** | Project rules + global rules (read-only context) | Global rules |
| **Writes Rules** | Project `rules/*.md` | Global `rules/*.md` + marks `<!-- lattice:promoted -->` in source project files |
| **Writes Store** | — | `global.db` (evidence summaries) |
| **Convergent Evidence** | Same pattern across multiple **sessions** | Same pattern across multiple **projects** (≥3) |

**Pattern similarity detection**: When embedding is configured, embedding similarity pre-screening (cosine > 0.8, configurable) clusters candidate patterns across projects before LLM analysis. When embedding is not configured, the `<triage>` and `<cross_ref>` phases handle pattern matching via LLM judgment alone. In both cases, the `<cross_ref>` phase confirms genuine convergence (vs. superficial similarity).

**Post-promotion behavior**: Promoted rules are marked in their source project files with a `<!-- lattice:promoted -->` metadata tag. Marked rules are skipped during instinct assembly (the global version is used instead) and do not count against the project's token budget. Projects can override a promoted global rule with `<!-- lattice:override -->` — this uses the existing "Project overrides Global" semantics.

---

## 3. Directory Structure

### 3.1 Global Profile (`~/.config/lattice/`)

```text
~/.config/lattice/
├── config.toml             # Global settings (LLM provider, thresholds)
├── projects.toml           # Registry of all lattice-enabled projects
├── rules/
│   ├── identity.md         # "You are a helpful coding assistant..."
│   └── preferences.md      # "User prefers TDD. Never use 'any'."
├── store/
│   └── global.db           # Cross-project evidence summaries (written by Global Compiler)
├── drift/
│   ├── proposals/          # Pending global rule proposals
│   │   └── 20260217_prefer_explicit_imports.md
│   └── traces/             # Global Compiler CoT audit logs
│       └── 20260217_run.trace.md
└── backups/                # Safety snapshots for global rules
    └── preferences.md.20260217-1030.bak
```

**`config.toml`**: User-managed global settings. Created with sensible defaults on first use (formerly by `lattice init`, now auto-initialized).

```toml
[compiler]
model = "openrouter/google/gemini-3-flash-preview"  # litellm format, OR "cli:claude" / "cli:opencode"
api_key_env = "OPENROUTER_API_KEY"                  # env var name (not needed for cli: mode)
batch_cap = 30                                      # max sessions per incremental evolve run
# base_url = "https://custom-endpoint.com"          # optional: override provider endpoint
# model = "cli:claude"                              # zero-config: use claude CLI (user's subscription)
# model = "cli:opencode"                            # zero-config: use opencode CLI
# model = "cli:opencode:anthropic/claude-sonnet"    # opencode CLI with specific model

# [embedding]                                        # OPTIONAL — omit for FTS5-only search (zero-config)
# model = "ollama/nomic-embed-text"                  # litellm model format: "provider/model"
# dimensions = 768                                   # must match vec0 schema (default: 768)
# api_key_env = "OLLAMA_API_KEY"                     # optional: for cloud embedding providers
# base_url = "http://localhost:11434"                # optional: override provider endpoint

[capture]
mode = "passive"                             # "passive" (hook/plugin captures) | "active" (MCP log_turn)
# passive: data captured by framework hook (OpenCode plugin, Claude Code hook) via `lattice ingest`
#          MCP server does NOT register log_turn tool — agent is unaware of data capture
# active:  no hook available — MCP server registers log_turn tool for agent to call (fallback)

[thresholds]
warn_tokens = 3000                       # advisory: lattice status shows warning
alert_tokens = 5000                      # advisory: Compiler <review> consolidates aggressively
per_session_tokens = 4000                # max tokens per session fed to Compiler (keeping most recent)
                                         # increase for larger models (e.g., 16000 for claude-3.5-sonnet)

[safety]
auto_apply = true                        # auto-apply proposals after evolve (see §4.3 design note)
backup_keep = 10                         # number of timestamped backups to retain
secret_patterns = [                      # regex patterns for secret sanitization (§7.2)
    "(?i)(api[_-]?key|secret|token|password)\\s*[:=]\\s*\\S+",
    "(?i)bearer\\s+\\S+",
    "sk-[a-zA-Z0-9]{20,}",
]
```

**`projects.toml`**: Maintained automatically. Source of truth for which projects the Global Compiler scans.

```toml
[[projects]]
path = "/Users/tefx/Projects/lattice"
name = "lattice"
registered_at = "2026-02-10T10:00:00"

[[projects]]
path = "/Users/tefx/Projects/my-app"
name = "my-app"
registered_at = "2026-02-17T09:00:00"
exclude_from_global = false   # optional, default false
```

### 3.2 Project Memory (`./.lattice/`)

```text
./.lattice/
├── rules/                  # System 1: The Brain
│   ├── architecture.md     # Coding: "We use Hexagonal Architecture."
│   │                        # Research: "Summarize findings in bullet points."
│   └── conventions.md      # Coding: "snake_case for Python, camelCase for JS."
│                            # General: "Always cite sources. Prefer concise answers."
├── store.db                # System 2: SQLite (FTS5 + sqlite-vec)
├── drift/
│   ├── proposals/          # Pending evolution proposals
│   │   └── 20260211_no_any.md
│   └── traces/             # Compiler CoT audit logs
│       └── 20260211_no_any.trace.md
└── backups/                # Safety snapshots (timestamped)
    └── conventions.md.20260211-1030.bak
```

**Versioning Strategy**: Markdown-Native.
*   Lattice operates on plain files. It does **not** depend on Git.
*   If the project has Git, `rules/*.md` changes will show up in `git status` naturally. Users commit them at their own discretion.
*   If there is no Git, Lattice maintains timestamped backups in `backups/` for safe revert.

---

## 4. Functional Architecture

Lattice uses a **three-frontend architecture**: an MCP server for agent integration, a lightweight IDE plugin for passive data capture, and a CLI for human operations. All three share the same `lattice.core` Python library.

```
+----------------------------------------------------------+
|              LATTICE MCP SERVER (长驻进程)                  |
|                                                          |
|  server_instructions:  动态生成的 System 1 instincts      |
|                                                          |
|  tools:                                                  |
|    - lattice_search(query, limit)    # System 2 查询     |
|    - log_turn(role, content, meta)   # 数据摄入 (fallback)|
|    - get_instincts()                 # 手动刷新 System 1  |
|    - propose_rule(pattern, obs, ev)  # 结构化提案         |
|    - get_status()                    # 查看 proposals     |
|                                                          |
|  resources:                                              |
|    - lattice://rules/{file}          # 可浏览的规则文件    |
|    - lattice://proposals/{file}      # 可浏览的提案       |
|                                                          |
|  [SQLite engine]  [Embedding cache]  [Rules cache]       |
+----------------------------^-----------------------------+
                             | stdio / SSE (MCP transport)
                             |
          +------------------+------------------+
          |                  |                  |
    [Claude Code]     [OpenCode]          [Cursor]
    (MCP client)      (MCP client)        (MCP client)

+----------------------------------------------------------+
|         LATTICE CLI (人类操作, 共享 lattice.core)           |
|  lattice ingest / evolve / status / apply / revert / search |
+----------------------------------------------------------+
```

### 4.1 Write Path (Data Capture)
How interactions become memory. All captured data flows to the **current project's** `store.db`. The global `global.db` is never written to by the data capture path — it is populated exclusively by the Global Compiler (see §4.4).

**MCP-capable Agents (Primary)**:
*   Agent calls `log_turn` MCP tool to record interactions. This is the fallback path — works with any MCP client, but requires agent awareness.

**OpenCode (Plugin — Passive Capture)**:
*   A lightweight plugin (~15 lines) hooks into `chat.message` to capture interactions passively. The agent is unaware — data flows silently to `store.db`. This is an **optional enhancement**, not a required component.

**Self-Developed Agents (SDK)**:
1.  Agent calls `client.log_turn(user=..., assistant=..., tools_called=[...])`.
2.  SDK writes directly to `store.db` via Python API.

*Design Note: Passive capture (plugin) produces more complete data than active capture (MCP tool) because the agent doesn't need to "remember" to call it. Where available, prefer plugin-based capture. The `[capture] mode` config controls this: when `mode = "passive"`, the MCP server does not register the `log_turn` tool — the agent never sees it, eliminating duplicate writes and wasted tokens. When `mode = "active"` (no hook available), the server registers `log_turn` as a fallback.*

*Design Note: Lattice does NOT address within-session context management (compaction). Compaction — compressing conversation history when the context window fills — is the agent framework's responsibility (OpenCode, Claude Code, etc.). Lattice's value is cross-session learning: extracting durable patterns from past sessions and injecting them into future sessions. These are orthogonal concerns.*

**Session Lifecycle**:

`session_id` is the fundamental grouping unit for evolution — convergent evidence is measured across sessions, batch caps count sessions, and incremental queries are ordered by session.

*   **Generation**: UUIDv4, assigned at session start.
*   **Boundary heuristic**: A session maps to a single continuous user↔agent conversation. How the boundary is determined depends on the capture path:
    *   **MCP server**: One session per MCP connection lifetime. The server generates a `session_id` on client connect; all `log_turn` calls within that connection share it. Reconnection = new session.
    *   **Plugin (passive)**: The plugin receives the agent framework's native session ID (e.g., OpenCode's `ses_*` identifier) via message metadata. If no native session ID is available, the plugin uses an inactivity heuristic: gap > 30 minutes between messages = new session.
    *   **SDK**: The caller provides `session_id` explicitly. If omitted, the SDK generates one per `Client()` instance lifetime.
*   **Storage**: `session_id` is indexed in `logs` for efficient range queries during evolution.

### 4.2 Read Path (Instinct Injection)
How memory becomes context.

**System 1 (Always-On)** — Instincts are injected via **MCP server instructions**.

When an MCP client connects, the Lattice server dynamically generates its `server_instructions` from the current `rules/*.md` files (Global + Project, with overlay resolution). MCP-capable agents automatically include server instructions in their system prompt — this is the "always-on" injection mechanism.

**Assembled server_instructions structure**:

```text
## Lattice — Active Instincts
Follow these rules in all responses. They are synthesized from project history.

### Global
{{ ~/.config/lattice/rules/*.md }}

### Project
{{ ./.lattice/rules/*.md }}

### Memory Tools
- Use `lattice_search` when you need to recall past decisions or context.
- In long sessions (30+ min), call `get_instincts()` to refresh rules.
{{ if capture.mode == "active" }}
- Call `log_turn` after each interaction to record it for future learning.
{{ endif }}
```

**System 2 (Deep Recall)** is **on-demand**, not auto-injected. The agent uses the `lattice_search` MCP tool or SDK `client.search()` when it needs to recall past context. This avoids wasting tokens on irrelevant memory in every prompt.

**Hot-Swap Limitation**: Unlike the previous `system.transform` design, MCP server instructions are typically loaded once at connection time. If `lattice evolve` updates rules mid-session, the agent sees stale instincts until it calls `get_instincts()` or reconnects. This is an acceptable trade-off — evolution is infrequent, and the `get_instincts()` tool provides a manual refresh path.

**Self-Developed Agents (SDK)**: Call `client.get_instincts()` before each LLM call for the same merged rules output.

### 4.3 Project Evolution (Project Compiler)
How raw logs within a single project become project-level instincts.

**Trigger**: Manual `lattice evolve` (CLI only). SDK cannot trigger evolution — evolution is infrastructure, not application logic.

> **Future**: Automated evolution trigger (data-threshold on ingest path, or idle-gated via Plugin `session.idle` hook) planned for post-launch iteration, once Compiler output quality is validated through real usage.

**Scope**: Reads the current project's `store.db` + project `rules/*.md` + global `rules/*.md` (read-only context, to avoid duplicating global rules). Writes only to project `rules/*.md` and project `drift/`.

**Incremental Mode (default)**: The Compiler operates incrementally — each `evolve` run processes only sessions recorded since the last successful evolution. Current rules already encode all historical patterns, so re-reading old sessions is unnecessary.

*   `store.db` maintains a `last_evolved_at` timestamp (in a `metadata` table). On each successful `evolve`, this timestamp is updated.
*   Query: `SELECT * FROM logs WHERE timestamp > last_evolved_at ORDER BY session_id, timestamp`.
*   If no `last_evolved_at` exists (first run), all sessions are processed (equivalent to full mode).
*   **Batch cap**: If accumulated sessions exceed a configurable limit (default: 30 sessions), a single `evolve` processes the oldest batch up to the cap. Subsequent `evolve` calls consume remaining sessions. `lattice status` reports "X sessions pending evolution" to prompt the user.
*   **Full recompile**: `lattice evolve --full` ignores `last_evolved_at` and processes all sessions in `store.db`. Useful after major project pivots or when rules have drifted significantly. Expensive but available as an escape hatch.
*   **Context budget**: Individual sessions are truncated to the `per_session_tokens` config value (default: 4000, keeping the most recent messages) before being assembled into the Compiler's input. With default settings: batch cap of 30 sessions × 4000 tokens + current rules (~3K) = ~123K tokens worst case. Users with larger context models (e.g., claude-3.5-sonnet with 200K context) can increase `per_session_tokens` to 16000 for better early-decision coverage.
*   **Zero-proposal runs**: If the Compiler finds no convergent patterns and the `<review>` phase confirms all existing rules are still valid, `last_evolved_at` is still updated. This is correct behavior — the sessions were processed, they just didn't yield changes. `drift/traces/` records the reasoning for auditability.

The Compiler uses **Structured Chain-of-Thought** — a single LLM call with forced phased reasoning. This preserves full information (no lossy intermediate extraction) while keeping the reasoning process auditable.

1.  **Invoke**: LLM receives new sessions (since `last_evolved_at`) from `store.db` + current `rules/*.md` in a single context window. The prompt enforces four reasoning phases:
    *   **`<triage>`**: Scan each session's logs. Output key signals as a structured list and noise markers (what to skip, with rationale). This is the denoising step — but it happens *inside* the Compiler's reasoning, not as a separate pipeline stage, so no information is irreversibly discarded.
    *   **`<cross_ref>`**: Compare signals across sessions. Identify convergent patterns — the same pattern must appear across multiple sessions before it qualifies. Single-session observations are flagged but never promoted directly.
    *   **`<synthesis>`**: Generate rule proposals from convergent patterns only. Identify contradictions with existing rules.
    *   **`<review>`**: Evaluate **all** existing rules against current evidence. For each rule, assess: (a) Is it still supported by recent session evidence? (b) Does it overlap or conflict with another rule or a new proposal? (c) Should it be merged, reworded, or removed? Output explicit kept/merged/removed decisions with rationale. When `lattice status` reports token count above the alert threshold, the review phase is prompted to be more aggressive in consolidation.
2.  **Log**: The full CoT output (all four phases) is saved to `drift/traces/` as an audit trail. This allows debugging *why* the Compiler generated, kept, or removed a specific rule.
3.  **Propose**: Rule proposals (additions, merges, removals) are extracted from `<synthesis>` and `<review>` and written to `drift/proposals/`.
4.  **Apply**: Behavior depends on `auto_apply` config:
    *   `auto_apply = true` (default): **Backup** current rule file to `backups/`, then **rewrite** `rules/*.md`. Git PR flow serves as the team review layer.
    *   `auto_apply = false`: Proposal waits in `drift/proposals/` for explicit `lattice apply`.
    *   *Design Note: `auto_apply = true` is safe as a default because `lattice evolve` is currently manual — the user explicitly chose to run the Compiler. If automated evolution triggers are implemented in the future, this default MUST be re-evaluated, as it would mean rules change without user initiation.*

*Design Note: A two-stage pipeline (separate Crystallizer → Compiler) was considered and rejected. The Crystallizer's filtering decisions would be an irreversible information bottleneck — it cannot anticipate what cross-session patterns the Compiler will need. Structured CoT achieves the same denoising within a single context window, with zero information loss and full auditability.*

*Design Note: A hard token budget (e.g., 2000 tokens) was considered and rejected. Fixed budgets force the Compiler into multi-objective optimization (add new insights AND compress to fit), which degrades output quality. Instead, the `<review>` phase provides semantic pressure to keep rules compact, and `lattice status` provides user-visible advisory thresholds. If rules grow beyond the alert threshold, the Compiler's `<review>` phase is prompted to consolidate more aggressively. Deleted rules are not lost — the underlying evidence remains in `store.db` and can be re-synthesized if patterns re-emerge.*

**Prompt Template Reference**: The production prompt template is at `data/compiler_prompt.md`. Key interface points for implementation:

| Interface | Value |
|-----------|-------|
| Template variables | `{current_rules}`, `{sessions}`, `{current_token_count}`, `{alert_tokens}`, `{token_warning}` |
| Phase XML tags | `<triage>`, `<cross_ref>`, `<synthesis>`, `<review>` |
| Proposal format | `## PROPOSAL: [Action] - [Topic]` with fields: Action, Target, Content, Evidence, Rationale |
| Review format | `## REVIEW: [Rule Title]` with fields: Decision (KEEP/MERGE/REMOVE/REWORD), Rationale |
| Action types | ADD, MERGE, REMOVE, REWORD |

**Safety**:
*   `lattice revert`: Restores the most recent backup.
*   `drift/traces/` provides full reasoning audit for every evolution run.
*   See §7.4 for the three-layer defense against accidental drift and prompt injection.

### 4.4 Global Evolution (Global Compiler)
How patterns across projects become global instincts.

**Trigger**: Manual `lattice evolve --global` (CLI only).

**Scope**: Reads all registered projects (from `projects.toml`). Writes to global `rules/*.md`, global `drift/`, and `global.db`.

The Global Compiler uses the same Structured CoT framework as the Project Compiler, but operates in **two phases** to manage context size:

**Phase 1 — Rule-Level Scan (lightweight)**:

Input:
*   All registered projects' `rules/*.md`
*   All registered projects' `drift/traces/` (most recent trace per project)
*   Current global `rules/*.md`

The prompt enforces:
*   **`<triage>`**: Scan each project's rules and traces. Flag cross-project patterns that appear similar. Pay special attention to patterns in traces that were **considered but not adopted** at the project level — these may be globally convergent even if they didn't reach the project-level threshold.
*   **`<candidate_patterns>`**: Output a list of candidate global patterns, each with source projects.

**Phase 2 — Evidence Verification (targeted deep-dive)**:

For each candidate pattern from Phase 1, perform targeted queries against the relevant projects' `store.db` to retrieve supporting evidence.

Input:
*   Candidate patterns from Phase 1
*   Targeted evidence snippets from relevant project stores

The prompt enforces:
*   **`<cross_ref>`**: Verify that cross-project evidence genuinely describes the same pattern (not just superficially similar). Confirm convergence across ≥3 projects.
*   **`<synthesis>`**: Generate global rule proposals. Identify contradictions with existing global rules. Apply the same advisory token thresholds and `<review>`-phase consolidation as the Project Compiler (see §2.1).

**Post-synthesis**:
1.  **Log**: Full CoT output from both phases saved to `~/.config/lattice/drift/traces/`.
2.  **Write evidence**: Adopted evidence summaries written to `global.db` with source project references and session IDs for traceability.
3.  **Propose**: Global rule proposals written to `~/.config/lattice/drift/proposals/`.
4.  **Apply**: Same `auto_apply` behavior as Project Compiler.
5.  **Mark promoted**: In each source project, tag promoted rules with `<!-- lattice:promoted -->`. Marked rules are excluded from project instinct assembly (global version used instead) and freed from the project's token budget.

*Design Note: Phase 1 reads `drift/traces/` rather than raw project logs. Traces are structured CoT outputs from project-level Compiler runs — they have much higher information density than raw logs and capture patterns the project Compiler considered but didn't adopt. This keeps Phase 1 context manageable (est. ~50K tokens for 10 projects) while preserving access to sub-threshold signals. For large registries (30+ projects), Phase 1 may need to paginate — process projects in batches of 15, then merge candidate patterns. This is a straightforward extension of the existing two-phase design.*

**Concurrency**: `lattice evolve` and `lattice evolve --global` must not run simultaneously on overlapping project files. The Global Compiler's Step 5 (marking `<!-- lattice:promoted -->` in project rule files) writes to the same files that the Project Compiler reads and writes. Since both are manual CLI commands, concurrent execution is unlikely. As a safeguard, `evolve` acquires a file lock (`.lattice/evolve.lock` for project, `~/.config/lattice/evolve.lock` for global) and fails fast if another `evolve` is already running.

---

## 5. Storage Engine

### 5.1 Technology Stack
*   **Database**: SQLite v3.35+ (single file, zero config).
*   **Vector Search**: `sqlite-vec` extension.
*   **Full-Text Search**: `FTS5` (built into SQLite).
*   **Language**: Python 3.10+.
*   **No External Dependencies**: No Node, Bun, Java, Docker.

### 5.2 Search (BM25 default, optional Hybrid RRF)
Inspired by `qmd`'s pipeline. Strategy: **Clone Logic, Not Code**.

**Embedding is optional.** When no `[embedding]` section is configured, search uses FTS5 BM25 only — zero external dependencies, zero API keys. When embedding is configured, search upgrades to hybrid mode automatically.

**FTS5-only mode** (default, no embedding configured):
1.  **BM25 Query**: `SELECT rowid, rank FROM logs_fts WHERE logs_fts MATCH ? ORDER BY rank`
2.  **Output**: Top-N results joined back to `logs` via rowid. BM25 is highly effective for keyword-heavy technical text (coding logs, error messages, CLI output).

**Hybrid mode** (embedding configured):
1.  **Parallel Query**:
    *   BM25 (Keyword): `SELECT rowid, rank FROM logs_fts WHERE logs_fts MATCH ? ORDER BY rank`
    *   Vector (Semantic): `SELECT rowid, distance FROM logs_vec WHERE embedding MATCH ? ORDER BY distance`
2.  **Positional Ranking**: Each result set is sorted by its native score (BM25 `rank`, vector `distance`). Results are then assigned positional ranks (1st, 2nd, 3rd...) — the RRF formula operates on these positions, not raw scores.
3.  **Reciprocal Rank Fusion** (Python):
    *   `RRF Score = 1/(k + pos_bm25) + 1/(k + pos_vec)`, where `k = 60`.
4.  **Output**: Top-N results joined back to `logs` via rowid, with source citations.

*Design Note: FTS5 BM25 is the default search mode and is highly effective (85-95% recall vs hybrid) for keyword-heavy short text typical of coding agent logs. Zero-config, zero external dependencies. Users can add `[embedding]` later to upgrade to hybrid search for better semantic recall — no code changes needed, same interface.*

### 5.3 Schema

**Project Store (`store.db`)**:

```sql
-- Events (structured, compressed at capture time)
CREATE TABLE events (
    id INTEGER PRIMARY KEY,
    external_id TEXT UNIQUE NOT NULL,   -- UUIDv4
    session_id TEXT NOT NULL,
    timestamp TEXT NOT NULL,            -- ISO8601
    type TEXT NOT NULL,                 -- user | assistant | reasoning | tool
    content TEXT NOT NULL,              -- text content for user/assistant/reasoning; tool name for tool
    -- tool-specific fields (NULL for non-tool types)
    tool_input TEXT,                    -- compressed input: "src/auth.py", "npm test"
    tool_status TEXT,                   -- "success" | "error"
    tool_error TEXT,                    -- error message (max 500 chars)
);
CREATE INDEX idx_events_session ON events(session_id);
CREATE INDEX idx_events_timestamp ON events(timestamp);
CREATE INDEX idx_events_type ON events(type);

-- Evolution State
CREATE TABLE metadata (
    key TEXT PRIMARY KEY,
    value TEXT NOT NULL
);
-- Tracks: 'last_evolved_at' (ISO8601), 'evolve_count' (integer)

-- Full-Text Search Index
CREATE VIRTUAL TABLE events_fts USING fts5(content, content=events, content_rowid=id);

CREATE TRIGGER events_ai AFTER INSERT ON events BEGIN
    INSERT INTO events_fts(rowid, content) VALUES (new.id, new.content);
END;
CREATE TRIGGER events_ad AFTER DELETE ON events BEGIN
    INSERT INTO events_fts(events_fts, rowid, content) VALUES ('delete', old.id, old.content);
END;
```

**Design Note**: The schema change from `logs` to `events` reflects the new compression-aware capture model. See RFC-002-R1 for the rationale. Key changes:
- `role` → `type` (more accurate semantics)
- `metadata` JSON → typed columns (`tool_input`, `tool_status`, `tool_error`)
- No `tool_output` column (compressed at capture, not stored)

**Global Store (`global.db`)**:

```sql
-- Cross-project evidence summaries (written by Global Compiler only)
CREATE TABLE evidence (
    id INTEGER PRIMARY KEY,            -- rowid alias (for FTS5/vec0 join)
    source_project TEXT NOT NULL,       -- project path (from projects.toml)
    source_session_id TEXT NOT NULL,    -- original session ID (traceable to project store.db)
    source_rule_file TEXT,              -- project rule file this evidence supported (if any)
    pattern TEXT NOT NULL,              -- normalized pattern description (Compiler-generated)
    summary TEXT NOT NULL,              -- evidence summary (≤500 chars)
    original_snippet TEXT,              -- raw log snippet for verification (≤2000 chars, optional)
    extracted_at TEXT NOT NULL,         -- ISO8601
    compiler_trace TEXT,                -- path to drift/traces/ entry
    confidence REAL DEFAULT 0.0
);

-- Links global rules to their supporting evidence
CREATE TABLE rule_evidence (
    rule_file TEXT NOT NULL,            -- global rules/*.md filename
    evidence_id TEXT NOT NULL,
    PRIMARY KEY (rule_file, evidence_id)
);

-- Search indexes for global.db (same pattern as store.db: triggers + rowid alignment)
CREATE VIRTUAL TABLE evidence_fts USING fts5(
    pattern, summary, original_snippet,
    content=evidence, content_rowid=id
);

CREATE TRIGGER evidence_ai AFTER INSERT ON evidence BEGIN
    INSERT INTO evidence_fts(rowid, pattern, summary, original_snippet)
    VALUES (new.id, new.pattern, new.summary, new.original_snippet);
END;
CREATE TRIGGER evidence_ad AFTER DELETE ON evidence BEGIN
    INSERT INTO evidence_fts(evidence_fts, rowid, pattern, summary, original_snippet)
    VALUES ('delete', old.id, old.pattern, old.summary, old.original_snippet);
END;

CREATE VIRTUAL TABLE evidence_vec USING vec0(embedding float[768]);
-- Insert contract: embedding must be inserted with rowid = evidence.id in the same transaction.
```

---

## 6. Interface

### 6.1 CLI (Human-Facing)

```bash
# Project Setup (auto-initializes on first use)
lattice ingest                  # Ingest message from stdin and auto-init project
lattice serve                   # Start MCP server (usually auto-managed by agent framework)

# Project Evolution (System 1)
lattice evolve                  # Run Project Compiler (incremental: new sessions since last evolve)
lattice evolve --full           # Full recompile: process all sessions in store.db
lattice status                  # Show pending proposals, rule stats, token budget, sessions pending
lattice apply <proposal>        # Apply a proposal (with backup)
lattice revert                  # Revert last rule change from backup

# Global Evolution
lattice evolve --global         # Run Global Compiler: cross-project patterns → global rules
lattice status --global         # Show global proposals, token budget, promotion stats
lattice apply --global <prop>   # Apply a global proposal
lattice revert --global         # Revert last global rule change

# Project Registry
lattice projects                # List all registered projects
lattice projects add <path>     # Manually register a project (without running init)
lattice projects remove <name>  # Remove a project from registry
lattice projects verify         # Check all registered paths exist

# System 2 (Memory)
lattice ingest                  # Ingest message from stdin into project store.db (used by Plugin)
lattice search "query"          # Search project episodic memory (hybrid: BM25 + vector + RRF)
lattice search --global "query" # Search global evidence summaries
```

### 6.2 MCP Server (Agent-Facing)

The MCP server is the primary integration layer for all MCP-capable agents (Claude Code, OpenCode, Cursor, etc.). It is a long-running Python process managed by the agent framework — no user-facing process management required.

**Tools**:

| Tool | Purpose | Notes |
| :--- | :--- | :--- |
| `lattice_search(query, limit, scope)` | System 2 on-demand recall | `scope`: `"project"` (default) or `"global"`. Hybrid BM25 + Vec + RRF |
| `log(type, content, ...)` | Record event to `store.db` | Unified interface. See RFC-002-R1 for details |
| `get_instincts()` | Refresh System 1 rules mid-session | For long sessions (30+ min) |
| `propose_rule(pattern, observation, evidence, suggested_action)` | Submit structured rule proposal | **Always registered** regardless of `capture.mode`. Writes a pending proposal file; does not write to `store.db`. See §7.4 for schema constraints |
| `get_status()` | View pending proposals, token budget | Read-only diagnostic |

**Resources**:

| URI | Content |
| :--- | :--- |
| `lattice://rules/{filename}` | Individual rule files (browsable) |
| `lattice://proposals/{filename}` | Pending evolution proposals |

**Server Instructions** (dynamically generated):

System 1 instincts are embedded directly in the MCP server's `server_instructions` field. MCP clients automatically inject these into the agent's system prompt at connection time. See §4.2 for the assembled structure.

### 6.3 OpenCode Plugin (Passive Data Capture)

An optional, lightweight plugin that passively captures interactions. The agent is unaware — data flows silently to `store.db`.

**Implementation** (`index.ts`):
```typescript
import { definePlugin } from "@opencode-ai/plugin"
import { execFile } from "child_process"

export default definePlugin({
  name: "opencode-lattice",
  hooks: {
    "chat.message": async ({ message }) => {
      // Extract and compress events
      const events = (message.parts || []).map(part => {
        if (part.type === "text" || part.type === "reasoning") {
          return { type: part.type, content: part.text }
        }
        if (part.type === "tool") {
          return {
            type: "tool",
            content: part.tool,
            tool_input: summarize_input(part.tool, part.state?.input),
            tool_status: part.state?.status,
            tool_error: part.state?.status === "error" 
              ? part.state?.output?.slice(0, 500) 
              : undefined,
          }
        }
        return null
      }).filter(Boolean)
      
      const payload = JSON.stringify({
        session_id: message.sessionID,
        events,
      })
      execFile("lattice", ["ingest", "--stdin"], { input: payload }, () => {})
    },
  },
})
```

*Design Note: Data is compressed at capture time (Layer 0). Tool outputs are not stored — only status and errors. See RFC-002-R1 for compression rationale.*

### 6.4 Python SDK (For Self-Developed Agents)

```python
import lattice

client = lattice.Client()

# ---- Read (System 1) ----
prompt = client.get_instincts()              # Merged Global + Project rules as string

# ---- Read (System 2) ----
results = client.search("auth bug", limit=5) # Hybrid search (BM25 + Vec + RRF)

# ---- Write (Unified log interface) ----
client.log("user", "Fix the auth bug")
client.log("assistant", "I'll update the middleware...")
client.log("reasoning", "Need to check UserValidator first...")
client.log("tool", "read", input="src/auth.py", status="success")
client.log("tool", "edit", input="src/auth.py", status="error", error="syntax at line 45")

# ---- Propose (Structured Schema) ----
client.propose_rule(
    pattern="bug-pattern",                    # enum: convention | bug-pattern | preference | architecture
    observation="Expired JWT tokens caused auth failures",  # max 200 chars
    evidence=["ses_abc123", "ses_def456"],     # session IDs (verifiable against store.db)
    suggested_action="validate-jwt-expiry",    # max 100 chars, short slug
)
```

**Key Constraints**:
*   SDK is **Read-Only + Propose-Only** for System 1. Only the Compiler writes to `rules/*.md`.
*   **No `client.evolve()`**. Evolution is infrastructure, not application logic. Agents cannot trigger the Compiler — only humans via `lattice evolve` CLI (or future automated triggers).
*   `log()` uses a **unified interface** across SDK, MCP, and Plugin. See RFC-002-R1 for details.

---

## 7. Safety & Privacy

### 7.1 Local-First
*   All data in local files. No cloud sync by default.
*   Global Profile can be synced via dotfiles or private repo at user's discretion.

### 7.2 Secret Sanitization
*   Logs are scrubbed for API keys, passwords, tokens before writing to disk.
*   Rules files must **never** contain secrets.

### 7.3 Backup & Revert
*   Every `apply` creates a timestamped backup.
*   `lattice revert` restores the last known good state.
*   Backups are rotated (keep last 10 by default).
*   `auto_apply` is a single config flag — no solo/team distinction. Git PR flow is the natural team review layer.

### 7.4 Three-Layer Defense (Compiler Integrity)

The Compiler reads proposals and session logs (both untrusted data from agents) and writes rules (trusted, injected into every LLM call). This makes it a critical trust boundary.

**Recognized threats**:
*   **Accidental Drift**: Compiler learns from noisy human feedback, encoding cognitive biases (e.g., sycophancy) rather than genuine quality signals. Mitigation lives in Compiler prompt design — weight technical outcomes over sentiment.
*   **Prompt Injection via Proposals**: Agent-generated proposal content enters the Compiler's LLM context. Structured or not, this is a data→instruction channel.

**Three-layer defense**:

| Layer | Mechanism | Purpose |
| :--- | :--- | :--- |
| **1. Schema** | `propose_rule()` uses structured fields (enum `pattern`, short `observation`, session ID `evidence`, short `suggested_action`) with length limits | Reduces injection attack surface. Short, structured data is harder to inject through than free-form prose |
| **2. Compiler Prompt Hardening** | Compiler treats proposals as **untrusted data**, not instructions. Cross-references `evidence` session IDs against actual logs in `store.db` | Detects fabricated or misrepresented evidence |
| **3. Convergent Evidence** | A single proposal or single-session observation **never** becomes a rule. The same pattern must appear across multiple sessions before the Compiler synthesizes it | Strongest defense. Even a successful injection is a single data point — insufficient to alter rules |

---

## 8. Success Metrics

**Quantitative (instrumented)**:

| Metric | Target | Instrumentation |
| :--- | :--- | :--- |
| Instinct Latency | 0ms | By design: file read, no network call |
| Token Efficiency | Below alert threshold (default: 5000) | `lattice status` reports token count; tracked per `evolve` run in `metadata` table |
| Evolution Revert Rate | < 10% | `lattice revert` logs are counted against the preceding `evolve`. Low revert rate = high precision. Tracked automatically in `metadata` (`revert_count` / `evolve_count`) |

**Qualitative (user-assessed, periodic)**:

| Metric | Target | Assessment Method |
| :--- | :--- | :--- |
| Drift Rate | < 5% | Periodic manual audit: sample 5 rules, check each against current codebase. Binary pass/fail per rule |
| User Correction Rate | Decreasing trend | User self-reports: "Did the agent repeat a known mistake this session?" (future: instrument via `propose_rule` submissions that contradict existing rules) |
