# Lattice System Architecture Specification

**Version:** 1.0
**Date:** 2026-02-17
**Source:** RFC-002 Lattice Bicameral Memory Architecture
**Purpose:** Complete implementation specification for `@python-engineer`

---

## Table of Contents

1. [System Overview](#1-system-overview)
2. [Module Map](#2-module-map)
3. [Shared Types](#3-shared-types)
4. [Interface Contracts](#4-interface-contracts)
5. [Data Flow Diagrams](#5-data-flow-diagrams)
6. [Dependency Graph](#6-dependency-graph)
7. [Configuration Schema](#7-configuration-schema)
8. [Database Schema](#8-database-schema)
9. [File System Layout](#9-file-system-layout)
10. [Error Model](#10-error-model)
11. [Concurrency Model](#11-concurrency-model)

---

## 1. System Overview

Lattice is a local-first persistent memory system for AI agents using a "bicameral mind" architecture:

- **System 1 (Instinct)**: Always-on rules stored as Markdown files, injected via MCP server instructions. 0ms latency.
- **System 2 (Memory)**: Episodic logs stored in SQLite (FTS5 + sqlite-vec), queried on-demand via hybrid search. ~100ms latency.
- **Compiler**: LLM process that reads System 2 logs and synthesizes System 1 rules via Structured Chain-of-Thought.

Three frontends share the same `lattice.core` / `lattice.shell` library:
- **CLI** (`lattice` command) -- human operations
- **MCP Server** (`lattice serve`) -- agent integration via stdio/SSE
- **Python SDK** (`lattice.Client`) -- direct library usage for self-developed agents

### Zone Discipline

| Zone | Location | Rules |
|------|----------|-------|
| **Core** | `src/lattice/core/` | Pure logic. `@pre`/`@post` contracts + doctests. NO I/O imports (`os`, `sys`, `pathlib`, `socket`, `requests`, etc.). Receives data as strings/dicts/dataclasses. |
| **Shell** | `src/lattice/shell/` | I/O operations. Returns `Result[T, E]` from `returns` library. Handles files, network, SQLite, process management. |

---

## 2. Module Map

### 2.1 Core Modules

| File | Responsibility | NOT Responsible For |
|------|---------------|---------------------|
| `src/lattice/core/types.py` | Shared domain types (dataclasses). Imported by all modules. No logic. | Any computation or validation beyond type construction |
| `src/lattice/core/rules.py` | Rule merging, overlay resolution, promoted/override tag parsing, token counting, instinct assembly | Reading rule files from disk (that is Shell) |
| `src/lattice/core/search.py` | RRF fusion algorithm, positional rank assignment | Executing SQL queries, calling embedding APIs |
| `src/lattice/core/sanitizer.py` | Secret detection and redaction from text content | Reading config files, accessing filesystem |
| `src/lattice/core/compiler.py` | Compiler prompt assembly, CoT output parsing, proposal extraction, session truncation, rule diffing | LLM API calls, file I/O, database queries |

### 2.2 Shell Modules

| File | Responsibility | NOT Responsible For |
|------|---------------|---------------------|
| `src/lattice/shell/config.py` | Read/write config.toml and projects.toml from disk. Resolve paths. | Validating config structure (that is Core) |
| `src/lattice/shell/schema.py` | Database schema initialization (DDL execution) for store.db and global.db | Query logic, data insertion |
| `src/lattice/shell/store.py` | Store CRUD: insert logs, query logs, metadata get/set, FTS5 search, session management | RRF fusion (that is Core), embedding generation |
| `src/lattice/shell/embeddings.py` | Embedding via litellm.embedding(), vector insert/search | RRF fusion, FTS5 queries |
| `src/lattice/shell/hybrid_search.py` | Orchestrate FTS5 search + vector search + core RRF fusion into unified hybrid search | Individual search implementations (delegates to store.py and embeddings.py) |
| `src/lattice/shell/llm.py` | LLM completion via litellm.completion(), thin wrapper | Prompt construction (that is Core) |
| `src/lattice/shell/evolution.py` | Project evolution orchestration: session query, batching, LLM invocation, proposal writing, apply/backup/revert | Global evolution, prompt assembly (that is Core) |
| `src/lattice/shell/global_evolution.py` | Global evolution orchestration: two-phase compiler, promotion marking, global.db evidence writes | Project-level evolution |
| `src/lattice/shell/server.py` | MCP server: tools, resources, server_instructions, connection lifecycle | CLI commands, SDK API |
| `src/lattice/shell/cli.py` | Typer CLI application: all `lattice` subcommands | Direct business logic (delegates to other shell modules) |

### 2.3 Public API

| File | Responsibility |
|------|---------------|
| `src/lattice/__init__.py` | Public SDK surface: exports `Client`, core types |
| `src/lattice/client.py` | `lattice.Client` class: SDK entry point wrapping core+shell calls |

### 2.4 Data Files

| File | Responsibility |
|------|---------------|
| `data/compiler_prompt.md` | Production project compiler prompt template (Structured CoT with 4 phases) |
| `data/global_compiler_prompt_phase1.md` | Global compiler Phase 1 prompt template (triage + candidate_patterns) |
| `data/global_compiler_prompt_phase2.md` | Global compiler Phase 2 prompt template (cross_ref + synthesis + review) |

### 2.5 Test Structure

```
tests/
  core/
    test_rules.py
    test_search.py
    test_sanitizer.py
    test_compiler.py
  shell/
    test_store.py
    test_evolution.py
    test_server.py
    test_cli.py
    test_global_evolution.py
  test_sdk.py
  e2e/
    test_single_project.py
    test_multi_project.py
    test_adversarial.py
```

---

## 3. Shared Types

All types defined in `src/lattice/core/types.py`. Use `dataclasses` (not Pydantic -- keep dependencies minimal for core). All fields are immutable where practical (`frozen=True`).

### 3.1 Configuration Types

```python
@dataclass(frozen=True)
class CompilerConfig:
    model: str              # litellm format: "provider/model" (e.g. "openrouter/google/gemini-3-flash-preview")
    api_key_env: str        # env var name (never the key itself)
    batch_cap: int          # max sessions per evolve run (default: 30)
    base_url: str | None    # optional: override provider endpoint (e.g. "https://ollama.com")

@dataclass(frozen=True)
class EmbeddingConfig:
    model: str              # litellm format: "provider/model" (e.g. "ollama/nomic-embed-text")
    dimensions: int         # must match vec0 schema (default: 768)
    api_key_env: str        # optional: env var name for cloud embedding providers
    base_url: str | None    # optional: override provider endpoint

@dataclass(frozen=True)
class ThresholdsConfig:
    warn_tokens: int        # advisory warning threshold (default: 3000)
    alert_tokens: int       # aggressive consolidation threshold (default: 5000)

@dataclass(frozen=True)
class SafetyConfig:
    auto_apply: bool        # auto-apply proposals after evolve (default: True)
    backup_keep: int        # number of backups to retain (default: 10)
    secret_patterns: list[str]  # regex patterns for secret sanitization

@dataclass(frozen=True)
class CaptureConfig:
    mode: str               # "passive" (hook/plugin via `lattice ingest`) | "active" (MCP log_turn)
                            # passive: MCP server does NOT register log_turn — agent unaware of capture
                            # active:  MCP server registers log_turn as fallback (no hook available)

@dataclass(frozen=True)
class Config:
    compiler: CompilerConfig
    embedding: EmbeddingConfig | None  # None = FTS5-only search (zero-config default)
    capture: CaptureConfig             # Controls data capture path (passive hook vs active log_turn)
    thresholds: ThresholdsConfig
    safety: SafetyConfig

@dataclass(frozen=True)
class ProjectEntry:
    path: str               # absolute filesystem path
    name: str               # project short name
    registered_at: str      # ISO8601 timestamp
    exclude_from_global: bool  # default: False
```

### 3.2 Domain Types

```python
@dataclass(frozen=True)
class LogEntry:
    id: int                         # rowid (assigned by SQLite)
    external_id: str                # UUIDv4
    session_id: str                 # session grouping key
    timestamp: str                  # ISO8601
    role: str                       # "user" | "assistant" | "tool"
    content: str                    # message text
    metadata: str | None            # JSON string or None

@dataclass(frozen=True)
class Rule:
    file_name: str                  # e.g. "conventions.md"
    content: str                    # raw markdown content
    source: str                     # "global" | "project"
    is_promoted: bool               # has <!-- lattice:promoted --> tag
    is_override: bool               # has <!-- lattice:override --> tag

@dataclass(frozen=True)
class SearchResult:
    log_id: int                     # logs.id
    external_id: str                # logs.external_id
    session_id: str
    timestamp: str
    role: str
    content: str
    score: float                    # RRF score (higher = better)
    source: str                     # "bm25" | "vec" | "both"

@dataclass(frozen=True)
class VecResult:
    rowid: int                      # rowid from logs_vec or evidence_vec
    distance: float                 # cosine distance (lower = more similar)

@dataclass(frozen=True)
class FtsResult:
    rowid: int                      # rowid from logs_fts or evidence_fts
    rank: float                     # BM25 rank (lower = more relevant)

@dataclass(frozen=True)
class RankedItem:
    rowid: int
    position: int                   # 1-based positional rank
    source: str                     # "bm25" | "vec"

@dataclass(frozen=True)
class FusedResult:
    rowid: int
    rrf_score: float                # combined RRF score
    sources: list[str]              # which lists contributed ("bm25", "vec", or both)
```

### 3.3 Compiler Types

```python
@dataclass(frozen=True)
class Session:
    session_id: str
    entries: list[LogEntry]         # ordered by timestamp
    token_count: int                # pre-computed token count

@dataclass(frozen=True)
class ProposalAction:
    """Single action within a proposal."""
    action: str                     # "add" | "merge" | "remove" | "reword"
    target_file: str                # target rule file (e.g. "conventions.md")
    content: str                    # new/merged rule text (empty for "remove")
    evidence_sessions: list[str]    # session_ids supporting this action
    rationale: str                  # why this action was chosen

@dataclass(frozen=True)
class Proposal:
    id: str                         # timestamped identifier (e.g. "20260217_no_any")
    actions: list[ProposalAction]
    timestamp: str                  # ISO8601 creation time
    trace_path: str                 # relative path to CoT trace file

@dataclass(frozen=True)
class CotPhases:
    """Parsed Structured Chain-of-Thought output."""
    triage: str                     # raw text from <triage> phase
    cross_ref: str                  # raw text from <cross_ref> phase
    synthesis: str                  # raw text from <synthesis> phase
    review: str                     # raw text from <review> phase

@dataclass(frozen=True)
class CompilerOutput:
    phases: CotPhases
    proposals: list[Proposal]
    raw_output: str                 # full LLM response for trace logging

@dataclass(frozen=True)
class RuleDiff:
    """Diff between current rules and proposed changes."""
    added: list[str]                # new rule texts
    removed: list[str]             # removed rule texts
    merged: list[tuple[str, str]]  # (old_text, new_text) pairs
    unchanged: list[str]           # rules not affected
```

### 3.4 Global Evolution Types

```python
@dataclass(frozen=True)
class CandidatePattern:
    pattern: str                    # normalized pattern description
    source_projects: list[str]      # project names where pattern was found
    confidence: float               # 0.0-1.0
    evidence_refs: list[str]        # references to source material

@dataclass(frozen=True)
class GlobalCotPhases:
    """Parsed output from Global Compiler Phase 1."""
    triage: str
    candidate_patterns: str

@dataclass(frozen=True)
class GlobalPhase2Phases:
    """Parsed output from Global Compiler Phase 2."""
    cross_ref: str
    synthesis: str

@dataclass(frozen=True)
class Evidence:
    """Row in global.db evidence table."""
    id: int
    source_project: str
    source_session_id: str
    source_rule_file: str | None
    pattern: str
    summary: str                    # <= 500 chars
    original_snippet: str | None    # <= 2000 chars
    extracted_at: str               # ISO8601
    compiler_trace: str | None
    confidence: float

@dataclass(frozen=True)
class PromoteAction:
    """Tracks a promotion marking in a project rule file."""
    project_path: str
    rule_file: str
    rule_text: str                  # the text that was promoted
```

### 3.5 MCP Types

```python
@dataclass(frozen=True)
class ProposeRuleInput:
    """Structured schema for propose_rule MCP tool and SDK method."""
    pattern: str            # enum: "convention" | "bug-pattern" | "preference" | "architecture"
    observation: str        # max 200 chars
    evidence: list[str]     # session IDs
    suggested_action: str   # max 100 chars, short slug

@dataclass(frozen=True)
class StatusReport:
    """Output of get_status tool."""
    pending_proposals: list[str]    # proposal IDs
    rule_token_count: int           # current total tokens in rules
    warn_threshold: int
    alert_threshold: int
    sessions_pending_evolution: int
    evolve_count: int
    revert_count: int
```

### 3.6 Secret Detection Type

```python
@dataclass(frozen=True)
class SecretSpan:
    start: int              # start index in original string
    end: int                # end index in original string
    pattern_name: str       # which pattern matched (for diagnostics)
```

---

## 4. Interface Contracts

### 4.1 Core: `core/rules.py`

```python
from lattice.core.types import Rule, Config

def merge_rules(
    global_rules: list[Rule],
    project_rules: list[Rule],
) -> list[Rule]:
    """Merge global and project rules with overlay resolution.

    - Project rules override global rules (project takes precedence).
    - Rules with `is_promoted=True` are excluded from the project list
      (the global version is used instead).
    - Rules with `is_override=True` suppress the corresponding promoted
      global rule (project version is used instead).
    - Order: global rules first, then project rules.

    Returns: Merged list of Rule objects, deduplicated per overlay logic.
    """
    ...

def count_tokens(text: str) -> int:
    """Count tokens in text using tiktoken (cl100k_base encoding).

    Returns: Non-negative integer token count.
    """
    ...

def parse_tags(content: str) -> tuple[bool, bool]:
    """Parse lattice metadata tags from rule file content.

    Scans for:
    - <!-- lattice:promoted --> -> is_promoted = True
    - <!-- lattice:override --> -> is_override = True

    Returns: (is_promoted, is_override) tuple.
    """
    ...

def assemble_instincts(
    global_rules: list[Rule],
    project_rules: list[Rule],
) -> str:
    """Assemble merged rules into the server_instructions text format.

    Output format:
        ## Lattice -- Active Instincts
        Follow these rules in all responses. ...

        ### Global
        {global rule contents, excluding promoted duplicates}

        ### Project
        {project rule contents, excluding promoted rules}

        ### Memory Tools
        - Use `lattice_search` when you need to recall past decisions or context.
        - In long sessions (30+ min), call `get_instincts()` to refresh rules.

    Returns: Formatted string ready for MCP server_instructions.
    """
    ...
```

### 4.2 Core: `core/search.py`

```python
from lattice.core.types import FtsResult, VecResult, RankedItem

def assign_positional_ranks(
    results: list[FtsResult] | list[VecResult],
) -> dict[int, int]:
    """Assign positional ranks (1-based) to results based on sort order.

    For FTS results: lower rank (more negative) = higher position.
    For Vec results: lower distance = higher position.

    Returns: Dict mapping rowid -> positional rank (1-based).
    """
    ...

def fuse_results(
    bm25_results: list[FtsResult],
    vec_results: list[VecResult],
    k: int = 60,
) -> list[RankedItem]:
    """Fuse BM25 and vector search results using RRF.

    Handles positional ranking internally. Accepts raw FtsResult/VecResult.
    Formula: RRF(d) = 1/(k + pos_bm25(d)) + 1/(k + pos_vec(d))
    - If d appears in only one list, the missing term is 0.
    - Works with empty vec_results (FTS5-only mode).
    - Output sorted by rrf_score descending.
    - Ties broken by rowid ascending (deterministic).

    Returns: List of RankedItem sorted by RRF score descending.
    """
    ...
```

### 4.3 Core: `core/sanitizer.py`

```python
from lattice.core.types import SecretSpan

def detect_secrets(
    content: str,
    patterns: list[str],
) -> list[SecretSpan]:
    """Detect secret spans in content using regex patterns.

    Args:
        content: Text to scan.
        patterns: List of regex pattern strings.

    Returns: List of SecretSpan indicating match positions. May overlap.
    """
    ...

def sanitize(
    content: str,
    patterns: list[str],
    replacement: str = "[REDACTED]",
) -> str:
    """Remove secrets from content by replacing matched spans.

    Uses detect_secrets internally. Overlapping spans are merged.
    Output length <= input length (replacement may be shorter than match).

    Returns: Sanitized content string.
    """
    ...
```

### 4.4 Core: `core/compiler.py`

```python
from lattice.core.types import (
    Session, Rule, CotPhases, ProposalAction, Proposal,
    CompilerOutput, RuleDiff, Config,
)

def assemble_compiler_prompt(
    sessions: list[Session],
    current_rules: list[Rule],
    alert_tokens: int,
    current_token_count: int,
) -> str:
    """Assemble the full prompt for the Project Compiler LLM call.

    Loads the prompt template from data/compiler_prompt.md (passed as
    parameter or read from a known location -- implementation decides).

    The assembled prompt contains:
    - Current rules as context
    - Session logs (pre-truncated)
    - Four-phase CoT instructions (<triage>, <cross_ref>, <synthesis>, <review>)
    - If current_token_count > alert_tokens, adds aggressive consolidation instruction

    Returns: Complete prompt string containing all four phase tags.
    """
    ...

def parse_cot_output(raw_output: str) -> CotPhases:
    """Parse LLM output into structured CoT phases.

    Extracts text between XML-style tags:
    <triage>...</triage>, <cross_ref>...</cross_ref>,
    <synthesis>...</synthesis>, <review>...</review>

    Tolerates missing phases (returns empty string for missing phase).
    Tolerates extra whitespace and minor formatting variations.

    Returns: CotPhases with extracted phase content.
    """
    ...

def extract_proposals(phases: CotPhases) -> list[ProposalAction]:
    """Extract structured proposal actions from synthesis and review phases.

    Parses the <synthesis> and <review> phase content for:
    - New rule additions
    - Rule merges (two rules -> one)
    - Rule removals (with rationale)
    - Rule rewordings

    Each action must have: action type, target file, content, evidence, rationale.

    Returns: List of ProposalAction. May be empty (zero-proposal run).
    """
    ...

def truncate_session(
    session: Session,
    max_tokens: int = 4000,
) -> Session:
    """Truncate a session to fit within token budget.

    Keeps the most recent messages (from the end). Removes oldest messages
    first until total tokens <= max_tokens.

    Returns: New Session with truncated entries and updated token_count.
    """
    ...

def diff_rules(
    current_rules: list[Rule],
    proposals: list[ProposalAction],
) -> RuleDiff:
    """Compute the diff between current rules and proposed changes.

    Maps each ProposalAction to the affected current rules to produce
    a structured diff showing what would be added, removed, merged, unchanged.

    Returns: RuleDiff describing the full change set.
    """
    ...
```

### 4.5 Shell: `shell/config.py`

```python
from returns.result import Result
from lattice.core.types import Config, ProjectEntry

def read_config(config_path: str) -> Result[Config, str]:
    """Read and validate config.toml from the given path.

    Applies defaults for any missing keys. Delegates structural
    validation to core.

    Returns: Success(Config) or Failure(error_message).
    """
    ...

def read_projects(projects_path: str) -> Result[list[ProjectEntry], str]:
    """Read projects.toml and return list of registered projects.

    Returns: Success(list[ProjectEntry]) or Failure(error_message).
    """
    ...

def write_projects(
    projects_path: str,
    projects: list[ProjectEntry],
) -> Result[None, str]:
    """Write projects list to projects.toml.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def register_project(
    projects_path: str,
    project_path: str,
    project_name: str,
) -> Result[ProjectEntry, str]:
    """Register a new project in projects.toml.

    Reads existing entries, appends new entry, writes back.
    Fails if project path is already registered.

    Returns: Success(ProjectEntry) or Failure(error_message).
    """
    ...

def unregister_project(
    projects_path: str,
    project_name: str,
) -> Result[None, str]:
    """Remove a project from projects.toml by name.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def resolve_global_dir() -> str:
    """Return the global Lattice config directory path.

    Default: ~/.config/lattice/
    Respects XDG_CONFIG_HOME if set.

    Returns: Absolute path string.
    """
    ...

def resolve_project_dir() -> Result[str, str]:
    """Find the .lattice/ directory for the current project.

    Searches current directory and parents for .lattice/.

    Returns: Success(absolute_path) or Failure(error_message).
    """
    ...
```

### 4.6 Shell: `shell/schema.py`

```python
import sqlite3
from returns.result import Result

def create_store(db_path: str) -> Result[sqlite3.Connection, str]:
    """Initialize a project store.db with full schema.

    Creates tables: logs, metadata, logs_fts, logs_vec.
    Creates indexes: idx_logs_session, idx_logs_timestamp.
    Creates triggers: logs_ai, logs_ad (FTS5 sync).
    Loads sqlite-vec extension.

    Idempotent: safe to call on existing database (uses IF NOT EXISTS).

    Returns: Success(Connection) or Failure(error_message).
    """
    ...

def create_global_store(db_path: str) -> Result[sqlite3.Connection, str]:
    """Initialize a global.db with full schema.

    Creates tables: evidence, rule_evidence, evidence_fts, evidence_vec.
    Creates triggers: evidence_ai, evidence_ad (FTS5 sync).
    Loads sqlite-vec extension.

    Idempotent: safe to call on existing database.

    Returns: Success(Connection) or Failure(error_message).
    """
    ...
```

### 4.7 Shell: `shell/store.py`

```python
import sqlite3
from returns.result import Result
from lattice.core.types import LogEntry, FtsResult, Session

def insert_log(
    conn: sqlite3.Connection,
    session_id: str,
    role: str,
    content: str,
    metadata: str | None = None,
) -> Result[int, str]:
    """Insert a log entry into store.db.

    Generates external_id (UUIDv4) and timestamp (ISO8601 now).

    Returns: Success(log_id) or Failure(error_message).
    """
    ...

def query_logs_since(
    conn: sqlite3.Connection,
    since_timestamp: str,
) -> Result[list[LogEntry], str]:
    """Query all logs since a timestamp, ordered by session_id then timestamp.

    Used by evolution to get new sessions since last evolve.

    Returns: Success(list[LogEntry]) or Failure(error_message).
    """
    ...

def query_logs_by_session(
    conn: sqlite3.Connection,
    session_id: str,
) -> Result[list[LogEntry], str]:
    """Query all logs for a specific session, ordered by timestamp.

    Returns: Success(list[LogEntry]) or Failure(error_message).
    """
    ...

def get_metadata(
    conn: sqlite3.Connection,
    key: str,
) -> Result[str | None, str]:
    """Get a metadata value by key.

    Returns: Success(value_or_None) or Failure(error_message).
    """
    ...

def set_metadata(
    conn: sqlite3.Connection,
    key: str,
    value: str,
) -> Result[None, str]:
    """Set a metadata key-value pair (upsert).

    Returns: Success(None) or Failure(error_message).
    """
    ...

def search_fts(
    conn: sqlite3.Connection,
    query: str,
    limit: int = 20,
) -> Result[list[FtsResult], str]:
    """Execute BM25 full-text search on logs_fts.

    Returns: Success(list[FtsResult]) sorted by rank ascending.
    """
    ...

def group_logs_into_sessions(
    logs: list[LogEntry],
) -> list[Session]:
    """Group a flat list of LogEntry into Session objects.

    Groups by session_id, orders entries by timestamp within each session.
    Computes token_count per session using core.rules.count_tokens.

    Returns: List of Session objects.
    """
    ...

def count_pending_sessions(
    conn: sqlite3.Connection,
    last_evolved_at: str | None,
) -> Result[int, str]:
    """Count distinct sessions since last_evolved_at.

    If last_evolved_at is None, counts all sessions.

    Returns: Success(count) or Failure(error_message).
    """
    ...
```

### 4.8 Shell: `shell/embeddings.py`
### 4.8 Shell: `shell/embeddings.py`

```python
from typing import Protocol
from returns.result import Result
from lattice.core.types import VecResult, EmbeddingConfig
import sqlite3

def embed_text(config: EmbeddingConfig, text: str) -> Result[list[float], str]:
    """Generate embedding vector for text using litellm.embedding().

    Provider is encoded in config.model (e.g. "ollama/nomic-embed-text").
    Uses config.base_url and config.api_key_env if set.

    Returns: Success(embedding) of configured dimensions, or Failure.
    """
    ...

def insert_embedding(
    conn: sqlite3.Connection,
    log_id: int,
    embedding: list[float],
) -> Result[None, str]:
    """Insert embedding into logs_vec with rowid = log_id.

    MUST be called in the same transaction as the log insert.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def search_vec(
    conn: sqlite3.Connection,
    query_embedding: list[float],
    limit: int = 20,
) -> Result[list[VecResult], str]:
    """Search logs_vec for nearest neighbors.

    Returns: Success(list[VecResult]) sorted by distance ascending.
    """
    ...

def insert_evidence_embedding(
    conn: sqlite3.Connection,
    evidence_id: int,
    embedding: list[float],
) -> Result[None, str]:
    """Insert embedding into evidence_vec (global.db) with rowid = evidence_id.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def search_evidence_vec(
    conn: sqlite3.Connection,
    query_embedding: list[float],
    limit: int = 20,
) -> Result[list[VecResult], str]:
    """Search evidence_vec (global.db) for nearest neighbors.

    Returns: Success(list[VecResult]) sorted by distance ascending.
    """
    ...
```

> **See §10.4** for embedding fallback behavior when the embedding model is unavailable.
### 4.9 Shell: `shell/hybrid_search.py`

```python
from returns.result import Result
from lattice.core.types import SearchResult, EmbeddingConfig
import sqlite3

def hybrid_search(
    conn: sqlite3.Connection,
    query: str,
    query_embedding: list[float] | None = None,
    limit: int = 10,
) -> Result[list[SearchResult], str]:
    """Execute search: FTS5-only (default) or hybrid (FTS5 + vector + RRF).

    Default mode (query_embedding is None):
    1. Run FTS5 BM25 search
    2. Join back to logs table for full content

    Enhanced mode (query_embedding provided, embedding configured):
    1. Run FTS5 search via store.search_fts
    2. Run vector search via embeddings.search_vec
    3. Fuse via core.search.fuse_rrf
    4. Join back to logs table for full content

    Returns: Success(list[SearchResult]) sorted by relevance.
    """
    ...

def hybrid_search_global(
    conn: sqlite3.Connection,
    query: str,
    query_embedding: list[float] | None = None,
    limit: int = 10,
) -> Result[list[SearchResult], str]:
    """Execute search on global.db evidence table.

    Same pattern as hybrid_search: FTS5-only by default, upgrades
    to hybrid when query_embedding provided.

    Returns: Success(list[SearchResult]) sorted by relevance.
    """
    ...
```

### 4.10 Shell: `shell/llm.py`

```python
from returns.result import Result
from lattice.core.types import CompilerConfig

def llm_complete(config: CompilerConfig, prompt: str) -> Result[str, str]:
    """Send prompt and return completion text.

    Two modes based on config.model prefix:
    - "cli:<tool>" → shell out to CLI (e.g. "cli:claude", "cli:opencode",
      "cli:opencode:anthropic/claude-sonnet"). Zero API key needed.
    - Otherwise → litellm.completion() (e.g. "openrouter/google/gemini-3-flash").
      Reads API key from config.api_key_env. Uses config.base_url if set.

    Returns: Success(completion_text) or Failure(error_message).
    """
    ...
```

### 4.11 Shell: `shell/evolution.py`

```python
import sqlite3
from returns.result import Result
from lattice.core.types import Config, Proposal, CompilerOutput, Rule

def query_pending_sessions(
    conn: sqlite3.Connection,
    config: Config,
) -> Result[list["Session"], str]:
    """Query sessions since last_evolved_at, apply batch_cap, truncate.

    Steps:
    1. Read last_evolved_at from metadata
    2. Query logs since that timestamp
    3. Group into sessions
    4. Apply batch_cap (take oldest N sessions)
    5. Truncate each session to per-session token budget (4000 tokens)

    Returns: Success(sessions) or Failure(error_message).
    """
    ...

def run_evolution(
    conn: sqlite3.Connection,
    config: Config,
    project_rules: list[Rule],
    global_rules: list[Rule],
    llm_provider: "LLMProvider",
    full: bool = False,
) -> Result[CompilerOutput, str]:
    """Execute the full project evolution pipeline.

    Steps:
    1. Query pending sessions (or all if full=True)
    2. If zero sessions, return empty CompilerOutput (no LLM call)
    3. Assemble compiler prompt via core.compiler
    4. Call LLM
    5. Parse CoT output via core.compiler
    6. Extract proposals
    7. Update last_evolved_at and evolve_count in metadata

    Returns: Success(CompilerOutput) or Failure(error_message).
    """
    ...

def write_proposal(
    proposal: Proposal,
    proposals_dir: str,
    traces_dir: str,
    trace_content: str,
) -> Result[None, str]:
    """Write proposal to drift/proposals/ and trace to drift/traces/.

    File naming: {timestamp}_{slug}.md for proposals,
                 {timestamp}_{slug}.trace.md for traces.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def list_proposals(proposals_dir: str) -> Result[list[str], str]:
    """List pending proposal files in drift/proposals/.

    Returns: Success(list_of_filenames) or Failure(error_message).
    """
    ...

def read_proposal(proposal_path: str) -> Result[str, str]:
    """Read proposal content from file.

    Returns: Success(content_string) or Failure(error_message).
    """
    ...

def apply_proposal(
    proposal_path: str,
    rules_dir: str,
    backups_dir: str,
) -> Result[None, str]:
    """Apply a proposal: backup current rules, write new rules, delete proposal.

    Steps:
    1. Backup current rule files to backups/ (timestamped)
    2. Write new/modified rules from proposal
    3. Delete the proposal file

    Returns: Success(None) or Failure(error_message).
    """
    ...

def backup_rules(
    rules_dir: str,
    backups_dir: str,
) -> Result[list[str], str]:
    """Create timestamped backup of all rule files.

    Returns: Success(list_of_backup_paths) or Failure(error_message).
    """
    ...

def revert_rules(
    rules_dir: str,
    backups_dir: str,
    conn: sqlite3.Connection,
) -> Result[None, str]:
    """Restore most recent backup of each rule file. Increment revert_count.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def rotate_backups(
    backups_dir: str,
    keep: int = 10,
) -> Result[int, str]:
    """Delete oldest backups beyond the keep limit.

    Returns: Success(number_deleted) or Failure(error_message).
    """
    ...
```

### 4.12 Shell: `shell/global_evolution.py`

```python
import sqlite3
from returns.result import Result
from lattice.core.types import (
    Config, ProjectEntry, CandidatePattern,
    Evidence, PromoteAction, Rule,
)

def run_global_phase1(
    projects: list[ProjectEntry],
    global_rules: list[Rule],
    llm_provider: "LLMProvider",
    embedding_config: "EmbeddingConfig | None" = None,
    similarity_threshold: float = 0.8,
) -> Result[list[CandidatePattern], str]:
    """Execute Global Compiler Phase 1: Rule-Level Scan.

    Steps:
    1. Read rules/*.md from each registered project
    2. Read most recent drift/traces/ from each project
    3. If embedding configured: similarity pre-screening (cosine > threshold)
       If no embedding: skip pre-screening, rely on LLM triage
    4. Assemble Phase 1 prompt
    5. Call LLM
    6. Parse candidate patterns

    Returns: Success(candidates) or Failure(error_message).
    """
    ...

def run_global_phase2(
    candidates: list[CandidatePattern],
    projects: list[ProjectEntry],
    global_rules: list[Rule],
    llm_provider: "LLMProvider",
    embedding_config: "EmbeddingConfig | None" = None,
    min_projects: int = 3,
) -> Result["CompilerOutput", str]:
    """Execute Global Compiler Phase 2: Evidence Verification.

    Steps:
    1. For each candidate, query relevant project store.db for evidence
    2. Assemble Phase 2 prompt (cross_ref + synthesis)
    3. Call LLM
    4. Confirm convergence (>= min_projects)
    5. Generate global rule proposals

    Returns: Success(CompilerOutput) or Failure(error_message).
    """
    ...

def write_evidence(
    conn: sqlite3.Connection,
    evidence_items: list[Evidence],
    embedding_config: "EmbeddingConfig | None" = None,
) -> Result[None, str]:
    """Write evidence summaries to global.db.

    Inserts into evidence table. If embedding configured, also inserts
    into evidence_vec.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def write_rule_evidence_links(
    conn: sqlite3.Connection,
    rule_file: str,
    evidence_ids: list[int],
) -> Result[None, str]:
    """Write rule-evidence associations to global.db rule_evidence table.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def mark_promoted(
    promote_actions: list[PromoteAction],
) -> Result[None, str]:
    """Add <!-- lattice:promoted --> tag to source project rule files.

    For each action, reads the project rule file, finds the matching rule
    text, inserts the promotion tag.

    Returns: Success(None) or Failure(error_message).
    """
    ...

def run_global_evolution(
    config: Config,
    projects: list[ProjectEntry],
    global_rules: list[Rule],
    global_conn: sqlite3.Connection,
    llm_provider: "LLMProvider",
) -> Result["CompilerOutput", str]:
    """Full global evolution orchestration.

    Steps:
    1. Phase 1: Rule scan -> candidate patterns
    2. Phase 2: Evidence verification -> proposals
    3. Write trace to global drift/traces/
    4. Write proposals to global drift/proposals/
    5. If auto_apply: apply proposals, mark promoted, write evidence
    6. Update global metadata

    Returns: Success(CompilerOutput) or Failure(error_message).
    """
    ...
```

### 4.13 Shell: `shell/server.py`

```python
# MCP Server -- uses mcp Python SDK

def create_server(project_dir: str, config: "Config") -> "MCPServer":
    """Create and configure the MCP server instance.

    Registers:
    - Tools: lattice_search, log_turn, get_instincts, propose_rule, get_status
    - Resources: lattice://rules/{filename}, lattice://proposals/{filename}
    - Server instructions: dynamically assembled from current rules

    Connection lifecycle:
    - On connect: generate session_id (UUIDv4), load rules
    - On disconnect: no special cleanup needed

    Returns: Configured MCP server instance.
    """
    ...

# --- Tool Handlers ---

async def handle_lattice_search(
    query: str,
    limit: int = 10,
    scope: str = "project",
) -> list[dict]:
    """MCP tool: lattice_search

    scope: "project" (default) queries store.db, "global" queries global.db.
    Returns: List of search result dicts.
    """
    ...

async def handle_log_turn(
    role: str,
    content: str,
    metadata: dict | None = None,
) -> dict:
    """MCP tool: log_turn

    Sanitizes content before storage. Uses server's session_id.
    Returns: Confirmation dict with log_id.
    """
    ...

async def handle_get_instincts() -> str:
    """MCP tool: get_instincts

    Re-reads rules from disk and re-assembles instincts.
    Returns: Assembled instincts string.
    """
    ...

async def handle_propose_rule(
    pattern: str,
    observation: str,
    evidence: list[str],
    suggested_action: str,
) -> dict:
    """MCP tool: propose_rule

    Validates structured schema:
    - pattern must be one of: convention, bug-pattern, preference, architecture
    - observation max 200 chars
    - evidence must be list of session IDs
    - suggested_action max 100 chars

    Writes proposal to drift/proposals/.
    Returns: Confirmation dict.
    """
    ...

async def handle_get_status() -> dict:
    """MCP tool: get_status

    Returns: StatusReport as dict.
    """
    ...

# --- Resource Handlers ---

async def handle_rules_resource(filename: str) -> str:
    """MCP resource: lattice://rules/{filename}

    Returns: Rule file content string.
    """
    ...

async def handle_proposals_resource(filename: str) -> str:
    """MCP resource: lattice://proposals/{filename}

    Returns: Proposal file content string.
    """
    ...
```

### 4.14 Shell: `shell/cli.py`

```python
import typer

app = typer.Typer()

# --- Project Setup ---

@app.command()
def init() -> None:
    """Initialize .lattice/ in current directory and register in projects.toml.

    Creates: .lattice/rules/, .lattice/drift/proposals/, .lattice/drift/traces/,
             .lattice/backups/, .lattice/store.db
    Registers project in ~/.config/lattice/projects.toml.
    """
    ...

@app.command()
def serve() -> None:
    """Start MCP server (stdio transport).

    Blocks until stdin closes or process is killed.
    """
    ...

# --- Evolution ---

@app.command()
def evolve(
    full: bool = typer.Option(False, "--full", help="Full recompile"),
    global_: bool = typer.Option(False, "--global", help="Global evolution"),
) -> None:
    """Run Compiler to evolve rules from session logs.

    Default: incremental project evolution.
    --full: process all sessions (full recompile).
    --global: cross-project global evolution.
    """
    ...

@app.command()
def status(
    global_: bool = typer.Option(False, "--global", help="Global status"),
) -> None:
    """Show pending proposals, rule stats, token budget, pending sessions."""
    ...

@app.command()
def apply(
    proposal: str = typer.Argument(..., help="Proposal filename"),
    global_: bool = typer.Option(False, "--global", help="Apply global proposal"),
) -> None:
    """Apply a pending proposal (with backup)."""
    ...

@app.command()
def revert(
    global_: bool = typer.Option(False, "--global", help="Revert global rules"),
) -> None:
    """Revert last rule change from backup."""
    ...

# --- Projects ---

@app.command()
def projects(
    action: str = typer.Argument(None, help="add|remove|verify"),
    target: str = typer.Argument(None, help="path or name"),
) -> None:
    """Manage project registry."""
    ...

# --- Search & Ingest ---

@app.command()
def search(
    query: str = typer.Argument(..., help="Search query"),
    global_: bool = typer.Option(False, "--global", help="Search global.db"),
    limit: int = typer.Option(10, "--limit", "-n"),
) -> None:
    """Search episodic memory (hybrid: BM25 + vector + RRF)."""
    ...

@app.command()
def ingest() -> None:
    """Ingest message from stdin (JSON) into store.db.

    Expected JSON: {role, content, session_id?, parts?, metadata?}
    Sanitizes content before storage.
    """
    ...
```

### 4.15 SDK: `client.py`

```python
from lattice.core.types import SearchResult, ProposeRuleInput

class Client:
    """Lattice SDK client for self-developed agents.

    Usage:
        client = lattice.Client()
        instincts = client.get_instincts()
        results = client.search("auth bug", limit=5)
        client.log_turn(user="Fix auth", assistant="Updating middleware...")
    """

    def __init__(self, project_dir: str | None = None) -> None:
        """Initialize client.

        If project_dir is None, auto-discovers .lattice/ from cwd.
        Generates session_id (UUIDv4) per Client instance.
        Opens store.db connection.
        """
        ...

    def get_instincts(self) -> str:
        """Return merged Global + Project rules as formatted string.

        Reads rules from disk on each call (fresh read).

        Returns: Assembled instincts string.
        """
        ...

    def search(
        self,
        query: str,
        limit: int = 10,
        scope: str = "project",
    ) -> list[SearchResult]:
        """Hybrid search (BM25 + Vec + RRF).

        Args:
            query: Search text.
            limit: Max results.
            scope: "project" or "global".

        Returns: List of SearchResult sorted by relevance.
        """
        ...

    def log_turn(
        self,
        user: str,
        assistant: str,
        tools_called: list[str] | None = None,
    ) -> None:
        """Record an interaction turn to store.db.

        Logs two entries: one for user role, one for assistant role.
        Sanitizes content before storage.
        tools_called is stored in metadata JSON.

        Returns: None. Failures are logged but not raised (fire-and-forget).
        """
        ...

    def propose_rule(
        self,
        pattern: str,
        observation: str,
        evidence: list[str],
        suggested_action: str,
    ) -> None:
        """Submit a structured rule proposal.

        Validates against ProposeRuleInput schema constraints.
        Writes to drift/proposals/.

        Raises: ValueError if schema validation fails.
        """
        ...
```

### 4.16 Public API: `__init__.py`

```python
"""Lattice -- Local-first persistent memory for AI agents."""

from lattice.client import Client
from lattice.core.types import (
    SearchResult,
    Config,
    Rule,
    Proposal,
    LogEntry,
    Session,
    StatusReport,
)

__all__ = [
    "Client",
    "SearchResult",
    "Config",
    "Rule",
    "Proposal",
    "LogEntry",
    "Session",
    "StatusReport",
]
```

---

## 5. Data Flow Diagrams

### 5.1 Ingest (Session Data -> store.db)

Three entry paths, same destination:

```
Path A: MCP log_turn Tool (capture.mode = "active" only)
========================================================
  Agent calls log_turn(role, content, metadata)
  NOTE: Only available when capture.mode = "active". In passive mode, data
  capture is handled by Path B (hook/plugin) and this tool is not registered.
       |
       v
  server.py::handle_log_turn()
       |
       v
  core/sanitizer.py::sanitize(content, patterns)    [Core: pure]
       |
       v
  shell/store.py::insert_log(conn, session_id, role, sanitized, metadata)
       |                                              [Shell: I/O]
       +---> logs table INSERT
       |
       +---> FTS5 trigger fires automatically: logs_fts INSERT
       |
       +--- config.embedding is not None? ---+
       |         YES                         |         NO (FTS5-only)
       v                                     v
  shell/embeddings.py                   (skip — no embedding)
  ::embed_text(sanitized)
       |
       v
  shell/embeddings.py
  ::insert_embedding(conn, log_id, embedding)
       |
       v
  logs_vec INSERT (same transaction)

Path B: CLI ingest (Plugin)
===========================
  Plugin calls: lattice ingest --stdin < JSON
       |
       v
  cli.py::ingest()  -- reads JSON from stdin
       |
       v
  core/sanitizer.py::sanitize(content, patterns)
       |
       v
  shell/store.py::insert_log(...)
       |
       +---> (same as Path A from here)

Path C: SDK log_turn
=====================
  Agent calls: client.log_turn(user=..., assistant=...)
       |
       v
  client.py::log_turn()  -- creates 2 LogEntries (user + assistant)
       |
       v
  core/sanitizer.py::sanitize(...)
       |
       v
  shell/store.py::insert_log(...)  x2
       |
       +---> (same as Path A from here)
```

**Key invariant**: Content is ALWAYS sanitized before any write to store.db. The embedding is computed on the sanitized content (not the original).

### 5.2 Search (Query -> Results)

Two modes — same interface, caller does not branch:

```
  Query string arrives (via MCP tool / CLI / SDK)
       |
       v
  shell/hybrid_search.py::hybrid_search(conn, query, query_embedding, limit)
       |
       +--- query_embedding is None? ----+
       |       YES (default mode)        |       NO (enhanced mode)
       v                                 v
  FTS5 BM25 search only          FTS5 + Vector search
  store.search_fts(query)        store.search_fts(query)
       |                         embeddings.search_vec(embedding)
       v                                 |
  list[FtsResult]                        v
       |                         list[FtsResult] + list[VecResult]
       v                                 |
  fuse_rrf(bm25, [], k=60)              v
  (BM25 scores only)            fuse_rrf(bm25, vec, k=60)
       |                                 |
       +-------+-------------------------+
               |
               v
  JOIN back to logs table for full content
               |
               v
  list[SearchResult]
```

**Key**: The interface is identical. Callers pass `query_embedding` if they have it, `None` if not. No branching logic needed in callers.

**For global search**: Same flow but queries `evidence`/`evidence_fts`/`evidence_vec` tables in global.db via `hybrid_search_global`.

### 5.3 Evolve (Sessions -> Compiler -> Proposals -> Rules)

```
  User runs: lattice evolve [--full]
       |
       v
  cli.py::evolve()
       |
       v
  Acquire .lattice/evolve.lock (fail fast if locked)
       |
       v
  shell/config.py::read_config(...)  -> Config
  shell/config.py::resolve_project_dir() -> project_dir
       |
       v
  Read rule files from disk:
    global: ~/.config/lattice/rules/*.md  -> list[Rule]
    project: .lattice/rules/*.md          -> list[Rule]
       |
       v
  shell/evolution.py::query_pending_sessions(conn, config)
       |
       +---> store.py::get_metadata(conn, "last_evolved_at")
       +---> store.py::query_logs_since(conn, last_evolved_at)
       +---> store.py::group_logs_into_sessions(logs)
       +---> Apply batch_cap: take oldest N sessions
       +---> core/compiler.py::truncate_session(session, 4000) for each
       |
       v
  If zero sessions: update last_evolved_at, release lock, return
       |
       v
  shell/evolution.py::run_evolution(conn, config, project_rules, global_rules, llm)
       |
       +---> core/compiler.py::assemble_compiler_prompt(sessions, rules, alert, count)
       |         Loads data/compiler_prompt.md template
       |
       +---> shell/llm.py::provider.complete(prompt)   [LLM API call]
       |
       +---> core/compiler.py::parse_cot_output(raw_output)
       |         Returns CotPhases
       |
       +---> core/compiler.py::extract_proposals(phases)
       |         Returns list[ProposalAction]
       |
       +---> store.py::set_metadata(conn, "last_evolved_at", now)
       +---> store.py::set_metadata(conn, "evolve_count", count+1)
       |
       v
  shell/evolution.py::write_proposal(proposal, proposals_dir, traces_dir, raw_output)
       |
       v
  If auto_apply:
       +---> shell/evolution.py::apply_proposal(proposal, rules_dir, backups_dir)
       |         backup_rules -> write new rules -> delete proposal
       |
       v
  Release lock
```

### 5.4 Global Evolve (Project Rules -> Global Compiler -> Promoted Rules)

```
  User runs: lattice evolve --global
       |
       v
  cli.py::evolve(global_=True)
       |
       v
  Acquire ~/.config/lattice/evolve.lock (fail fast if locked)
       |
       v
  shell/config.py::read_config(...) -> Config
  shell/config.py::read_projects(...) -> list[ProjectEntry]
       |
       v
  Read global rules: ~/.config/lattice/rules/*.md -> list[Rule]
       |
       v
  ============ PHASE 1: Rule-Level Scan ============
       |
  shell/global_evolution.py::run_global_phase1(projects, global_rules, llm, emb_config)
       |
       +---> For each project:
       |       Read project rules/*.md
       |       Read most recent drift/traces/*
       |
       +---> If embedding configured:
       |       Embedding pre-screening: cluster by cosine similarity > 0.8
       |     Else:
       |       Skip pre-screening (LLM handles pattern detection in <triage>)
       |
       +---> Assemble Phase 1 prompt (data/global_compiler_prompt_phase1.md)
       +---> LLM call
       +---> Parse: extract <triage> + <candidate_patterns>
       |
       v
  list[CandidatePattern]
       |
       v
  ============ PHASE 2: Evidence Verification ============
       |
  shell/global_evolution.py::run_global_phase2(candidates, projects, global_rules, llm, emb)
       |
       +---> For each candidate:
       |       Query relevant project store.db for evidence
       |       (targeted queries, not full scan)
       |
       +---> Assemble Phase 2 prompt (data/global_compiler_prompt_phase2.md)
       +---> LLM call
       +---> Parse: extract <cross_ref> + <synthesis>
       +---> Filter: keep only patterns with >= 3 project convergence
       |
       v
  CompilerOutput (global proposals)
       |
       v
  ============ POST-PROCESSING ============
       |
  Write trace: ~/.config/lattice/drift/traces/
  Write proposals: ~/.config/lattice/drift/proposals/
       |
  If auto_apply:
       |
       +---> Backup global rules
       +---> Write new global rules
       +---> shell/global_evolution.py::mark_promoted(promote_actions)
       |       For each promoted rule: insert <!-- lattice:promoted --> in source project
       +---> shell/global_evolution.py::write_evidence(global_conn, evidence, emb_config)
       |       Insert evidence rows + embeddings into global.db
       +---> shell/global_evolution.py::write_rule_evidence_links(global_conn, ...)
       |
       v
  Release lock
```

### 5.5 Serve (MCP Server Lifecycle)

```
  lattice serve
       |
       v
  cli.py::serve()
       |
       v
  shell/server.py::create_server(project_dir, config)
       |
       +---> Open store.db connection
       +---> Create embedding provider
       +---> Load rules from disk (global + project)
       +---> Assemble server_instructions via core/rules.py::assemble_instincts
       +---> Register tools: lattice_search, get_instincts, propose_rule, get_status
       +---> IF config.capture.mode == "active": also register log_turn
       +---> Register resources: lattice://rules/*, lattice://proposals/*
       |
       v
  MCP Server starts (stdio transport)
       |
       v
  On client connect:
       +---> Generate session_id (UUIDv4) for this connection
       +---> Serve server_instructions
       |
       v
  [Event loop: handle tool calls and resource reads]
       |
  On client disconnect:
       +---> No special cleanup (connection closes naturally)
```

---

## 6. Dependency Graph

### 6.1 Module Import Dependencies

```
core/types.py           <- (no dependencies, leaf module)
      ^
      |
      +------- core/rules.py     (imports types)
      |              ^
      +------- core/search.py    (imports types)
      |              ^
      +------- core/sanitizer.py (imports types)
      |              ^
      +------- core/compiler.py  (imports types, imports rules for count_tokens)
      |              ^               ^
      |              |               |
      |        (reads data/compiler_prompt.md -- passed as string param)
      |
      +------- shell/config.py   (imports types)
      |              ^
      +------- shell/schema.py   (no core imports, just sqlite3)
      |              ^
      +------- shell/store.py    (imports types, imports core/rules for count_tokens)
      |              ^
      +------- shell/embeddings.py (imports types)
      |              ^
      +------- shell/llm.py      (imports types)
      |              ^
      +------- shell/hybrid_search.py (imports types, store, embeddings, core/search)
      |              ^
      +------- shell/evolution.py (imports types, core/compiler, store, llm, config)
      |              ^
      +------- shell/global_evolution.py (imports types, core/compiler, store,
      |              ^                    embeddings, llm, config, evolution)
      |              |
      +------- shell/server.py   (imports types, core/rules, core/sanitizer,
      |              ^            store, embeddings, hybrid_search, config)
      |              |
      +------- shell/cli.py      (imports config, schema, store, evolution,
      |                           global_evolution, server, hybrid_search, core/sanitizer)
      |
      +------- client.py         (imports types, core/rules, core/sanitizer,
                                  shell/config, shell/store, shell/embeddings,
                                  shell/hybrid_search)
```

### 6.2 Dependency Table

| Module | Depends On (Core) | Depends On (Shell) | Depended By |
|--------|------------------|--------------------|-------------|
| `core/types.py` | -- | -- | All modules |
| `core/rules.py` | types | -- | compiler, server, client, cli |
| `core/search.py` | types | -- | hybrid_search |
| `core/sanitizer.py` | types | -- | store (via callers), server, client, cli |
| `core/compiler.py` | types, rules | -- | evolution, global_evolution |
| `shell/config.py` | types | -- | cli, evolution, global_evolution, server, client |
| `shell/schema.py` | -- | -- | cli (init), server (startup) |
| `shell/store.py` | types, rules | -- | evolution, hybrid_search, server, client |
| `shell/embeddings.py` | types | -- | hybrid_search, global_evolution, server, client |
| `shell/llm.py` | types | -- | evolution, global_evolution |
| `shell/hybrid_search.py` | types, search | store, embeddings | server, client, cli |
| `shell/evolution.py` | types, compiler | store, llm, config | cli, global_evolution |
| `shell/global_evolution.py` | types, compiler | store, embeddings, llm, config, evolution | cli |
| `shell/server.py` | types, rules, sanitizer | store, embeddings, hybrid_search, config | cli |
| `shell/cli.py` | -- | config, schema, store, evolution, global_evolution, server, hybrid_search | -- (entry point) |
| `client.py` | types, rules, sanitizer | config, store, embeddings, hybrid_search | __init__.py |

### 6.3 Circular Dependency Check

**No circular dependencies exist.** The dependency graph is a DAG:
- Core modules depend only on `core/types.py` and (for compiler) `core/rules.py`
- Shell modules depend on core modules and other shell modules, but no shell module depends on cli.py
- cli.py and client.py are leaf consumers (nothing imports them except __init__.py for client)

---

## 7. Configuration Schema

### 7.1 `~/.config/lattice/config.toml`

```toml
# LLM provider for Compiler
[compiler]
provider = "openrouter"                   # REQUIRED: "openrouter" | "ollama" | "anthropic" | "openai"
model = "google/gemini-3-flash-preview"   # REQUIRED: model identifier for the provider
api_key_env = "OPENROUTER_API_KEY"        # REQUIRED: env var name (never store keys in config)
batch_cap = 30                            # OPTIONAL: max sessions per evolve run (default: 30)

# Embedding provider (OPTIONAL — omit for FTS5-only search)
# [embedding]
# provider = "ollama"                     # litellm format: "provider/model"
# model = "nomic-embed-text"             # embedding model identifier
# dimensions = 768                       # must match vec0 schema (default: 768)

# Advisory token thresholds for rules
[thresholds]
warn_tokens = 3000                        # OPTIONAL: default 3000
alert_tokens = 5000                       # OPTIONAL: default 5000

# Safety settings
[safety]
auto_apply = true                         # OPTIONAL: default true
backup_keep = 10                          # OPTIONAL: default 10
secret_patterns = [                       # OPTIONAL: has sensible defaults
    '(?i)(api[_-]?key|secret|token|password)\s*[:=]\s*\S+',
    '(?i)bearer\s+\S+',
    'sk-[a-zA-Z0-9]{20,}',
]
```

**Defaults**: If `config.toml` does not exist, it is auto-created on first use with the defaults shown above. The `compiler.model` and `compiler.api_key_env` fields have no meaningful defaults and must be set by the user before running `lattice evolve`. The `[embedding]` section is optional — when omitted, search uses FTS5 BM25 only (zero-config, zero external dependencies).

### 7.2 `~/.config/lattice/projects.toml`

```toml
[[projects]]
path = "/Users/tefx/Projects/lattice"     # REQUIRED: absolute path
name = "lattice"                           # REQUIRED: short identifier
registered_at = "2026-02-10T10:00:00"     # REQUIRED: ISO8601
exclude_from_global = false                # OPTIONAL: default false
```

Each `[[projects]]` entry is one registered project. Managed automatically and by `lattice projects add/remove`.

---

## 8. Database Schema

### 8.1 Project Store (`store.db`)

```sql
-- Core episodic log table
CREATE TABLE IF NOT EXISTS logs (
    id INTEGER PRIMARY KEY,            -- rowid alias (required for FTS5/vec0 join)
    external_id TEXT UNIQUE NOT NULL,   -- UUIDv4
    session_id TEXT NOT NULL,
    timestamp TEXT NOT NULL,            -- ISO8601
    role TEXT NOT NULL,                 -- 'user' | 'assistant' | 'tool'
    content TEXT NOT NULL,
    metadata TEXT                       -- JSON string or NULL
);

CREATE INDEX IF NOT EXISTS idx_logs_session ON logs(session_id);
CREATE INDEX IF NOT EXISTS idx_logs_timestamp ON logs(timestamp);

-- Evolution state key-value store
CREATE TABLE IF NOT EXISTS metadata (
    key TEXT PRIMARY KEY,
    value TEXT NOT NULL
);
-- Known keys:
--   'last_evolved_at'  : ISO8601 timestamp of last successful evolve
--   'evolve_count'     : integer (as string), total evolve runs
--   'revert_count'     : integer (as string), total revert operations

-- FTS5 full-text search index (external content mode)
CREATE VIRTUAL TABLE IF NOT EXISTS logs_fts USING fts5(
    content,
    content=logs,
    content_rowid=id
);

-- FTS5 sync triggers
CREATE TRIGGER IF NOT EXISTS logs_ai AFTER INSERT ON logs BEGIN
    INSERT INTO logs_fts(rowid, content) VALUES (new.id, new.content);
END;

CREATE TRIGGER IF NOT EXISTS logs_ad AFTER DELETE ON logs BEGIN
    INSERT INTO logs_fts(logs_fts, rowid, content)
    VALUES ('delete', old.id, old.content);
END;

CREATE TRIGGER IF NOT EXISTS logs_au AFTER UPDATE ON logs BEGIN
    INSERT INTO logs_fts(logs_fts, rowid, content)
    VALUES ('delete', old.id, old.content);
    INSERT INTO logs_fts(rowid, content) VALUES (new.id, new.content);
END;

-- Vector search index (rowid-aligned with logs.id)
CREATE VIRTUAL TABLE IF NOT EXISTS logs_vec USING vec0(
    embedding float[768]
);
-- Insert contract: embedding rowid MUST equal logs.id, in same transaction.
```

### 8.2 Global Store (`global.db`)

```sql
-- Cross-project evidence summaries
CREATE TABLE IF NOT EXISTS evidence (
    id INTEGER PRIMARY KEY,
    source_project TEXT NOT NULL,       -- project path from projects.toml
    source_session_id TEXT NOT NULL,    -- traceable to project store.db
    source_rule_file TEXT,              -- project rule file (if any)
    pattern TEXT NOT NULL,              -- normalized pattern description
    summary TEXT NOT NULL,              -- <= 500 chars
    original_snippet TEXT,              -- <= 2000 chars, optional
    extracted_at TEXT NOT NULL,         -- ISO8601
    compiler_trace TEXT,                -- path to trace file
    confidence REAL DEFAULT 0.0        -- 0.0 to 1.0
);

-- Links global rules to supporting evidence
CREATE TABLE IF NOT EXISTS rule_evidence (
    rule_file TEXT NOT NULL,            -- global rules/*.md filename
    evidence_id INTEGER NOT NULL,       -- FK to evidence.id
    PRIMARY KEY (rule_file, evidence_id)
);

-- FTS5 index for evidence table
CREATE VIRTUAL TABLE IF NOT EXISTS evidence_fts USING fts5(
    pattern, summary, original_snippet,
    content=evidence,
    content_rowid=id
);

CREATE TRIGGER IF NOT EXISTS evidence_ai AFTER INSERT ON evidence BEGIN
    INSERT INTO evidence_fts(rowid, pattern, summary, original_snippet)
    VALUES (new.id, new.pattern, new.summary, new.original_snippet);
END;

CREATE TRIGGER IF NOT EXISTS evidence_ad AFTER DELETE ON evidence BEGIN
    INSERT INTO evidence_fts(evidence_fts, rowid, pattern, summary, original_snippet)
    VALUES ('delete', old.id, old.pattern, old.summary, old.original_snippet);
END;

CREATE TRIGGER IF NOT EXISTS evidence_au AFTER UPDATE ON evidence BEGIN
    INSERT INTO evidence_fts(evidence_fts, rowid, pattern, summary, original_snippet)
    VALUES ('delete', old.id, old.pattern, old.summary, old.original_snippet);
    INSERT INTO evidence_fts(rowid, pattern, summary, original_snippet)
    VALUES (new.id, new.pattern, new.summary, new.original_snippet);
END;

-- Vector search index for evidence
CREATE VIRTUAL TABLE IF NOT EXISTS evidence_vec USING vec0(
    embedding float[768]
);
-- Insert contract: embedding rowid MUST equal evidence.id, in same transaction.
```

### 8.3 Schema Invariants

1. **Rowid alignment**: `logs_vec.rowid == logs.id` and `evidence_vec.rowid == evidence.id`. This is enforced at insert time, not by SQL constraints. Violation breaks search join logic.
2. **FTS5 sync**: Maintained by triggers. If triggers fail, FTS index becomes stale. Recovery: `INSERT INTO logs_fts(logs_fts) VALUES('rebuild')`.
3. **Transaction boundary**: When embedding is configured, log insert + embedding insert MUST be in the same transaction. When embedding is not configured, logs are searchable via FTS5 only — the logs_vec table remains empty.
4. **Embedding dimensions**: The `float[768]` in vec0 schema MUST match `config.embedding.dimensions`. Changing dimensions requires recreating the vec0 table.

---

## 9. File System Layout

### 9.1 Global Profile

```
~/.config/lattice/                     (or $XDG_CONFIG_HOME/lattice/)
  config.toml                          Global settings
  projects.toml                        Project registry
  rules/                               Global System 1 rules
    identity.md                        User identity / persona
    preferences.md                     Cross-project preferences
  store/
    global.db                          Cross-project evidence (System 2)
  drift/
    proposals/                         Pending global rule proposals
      {YYYYMMDD}_{slug}.md
    traces/                            Global Compiler CoT audit logs
      {YYYYMMDD}_{slug}.trace.md
  backups/                             Global rule backups
    {filename}.{YYYYMMDD}-{HHMM}.bak
  evolve.lock                          Global evolution lock file
```

### 9.2 Project Memory

```
./.lattice/
  rules/                               Project System 1 rules
    architecture.md
    conventions.md
  store.db                             Project System 2 (SQLite)
  drift/
    proposals/                         Pending project proposals
      {YYYYMMDD}_{slug}.md
    traces/                            Project Compiler CoT audit logs
      {YYYYMMDD}_{slug}.trace.md
  backups/                             Project rule backups
    {filename}.{YYYYMMDD}-{HHMM}.bak
  evolve.lock                          Project evolution lock file
```

### 9.3 Source Layout

```
src/lattice/
  __init__.py                          Public SDK API (Client + types)
  client.py                            SDK Client class
  core/
    __init__.py
    types.py                           Shared domain types
    rules.py                           Rule merging, overlay, token counting
    search.py                          RRF fusion algorithm
    sanitizer.py                       Secret detection and redaction
    compiler.py                        Prompt assembly, CoT parsing
  shell/
    __init__.py
    config.py                          Config I/O (read/write TOML)
    schema.py                          Database schema initialization
    store.py                           Store CRUD, FTS5 search
    embeddings.py                      Embedding provider abstraction
    hybrid_search.py                   Orchestrated hybrid search
    llm.py                             LLM provider abstraction
    evolution.py                       Project evolution orchestration
    global_evolution.py                Global evolution orchestration
    server.py                          MCP server
    cli.py                             Typer CLI application
data/
  compiler_prompt.md                   Project Compiler prompt template
  global_compiler_prompt_phase1.md     Global Compiler Phase 1 template
  global_compiler_prompt_phase2.md     Global Compiler Phase 2 template
```

### 9.4 Auto-Initialization Creates

When a command requires project initialization (e.g., `lattice ingest`, `lattice status`) and `.lattice/` doesn't exist:
1. `.lattice/` directory
2. `.lattice/rules/` directory (empty)
3. `.lattice/drift/proposals/` directory
4. `.lattice/drift/traces/` directory
5. `.lattice/backups/` directory
6. `.lattice/store.db` (initialized with full schema)
7. Entry in `~/.config/lattice/projects.toml`
8. `~/.config/lattice/` directory tree (if first init ever)
9. `~/.config/lattice/config.toml` (with defaults, if not present)
10. `~/.config/lattice/rules/` directory (if not present)
11. `~/.config/lattice/store/global.db` (initialized with schema, if not present)
12. `~/.config/lattice/drift/proposals/` and `drift/traces/` (if not present)
13. `~/.config/lattice/backups/` (if not present)

---

## 10. Error Model

### 10.1 Core Errors

Core modules use contracts (`@pre`/`@post`) and raise standard Python exceptions:
- `ValueError`: Invalid input that violates contracts
- Contract violations are caught by `deal` at runtime

Core modules do NOT define custom exceptions. They are pure functions that either return a value or raise.

### 10.2 Shell Errors

Shell modules return `Result[T, str]` where the error type is always `str` (human-readable error message). Categories of shell errors:

| Category | Example Error String | Modules |
|----------|---------------------|---------|
| **File I/O** | `"Config file not found: {path}"` | config, evolution |
| **Database** | `"SQLite error: {detail}"` | schema, store |
| **Network** | `"LLM API call failed: {detail}"` | llm, embeddings |
| **Lock** | `"Evolution already in progress (lock held)"` | evolution, global_evolution |
| **Validation** | `"Invalid proposal pattern: {value}"` | server (propose_rule) |
| **Not Found** | `"Project .lattice/ not found"` | config |

### 10.3 Error Propagation

```
Core function raises ValueError
       |
       v
Shell function catches -> returns Failure("description")
       |
       v
CLI function matches Result:
  Success -> print result, exit 0
  Failure -> typer.echo(error, err=True), raise typer.Exit(1)
```

```
Shell function returns Failure
       |
       v
MCP tool handler:
  Success -> return tool result
  Failure -> return MCP error response
```

```
Shell function returns Failure
       |
       v
SDK Client:
  For log_turn: swallow error (fire-and-forget, log warning)
  For search/get_instincts: raise RuntimeError(error_message)
  For propose_rule: raise ValueError(error_message)
```

---


### 10.4 Embedding Fallback Behavior

Lattice supports three modes of operation for search:

1. **Hybrid (FTS5 + Vector)**: When `config.embedding` is configured and embedding API is available
2. **FTS5-only**: When `config.embedding` is `None` (default) or embedding generation fails
3. **Degraded**: When embedding is configured but unavailable at runtime

#### Configuration Options

```toml
[embedding]
model = "ollama/nomic-embed-text"  # litellm format: provider/model
dimensions = 768                    # must match vec0 schema
api_key_env = "OPENAI_API_KEY"      # optional: env var for cloud providers
base_url = "https://api.example.com" # optional: override endpoint
```

- **Optional**: If `[embedding]` section is omitted, `config.embedding` is `None`
- **Zero-config default**: FTS5-only search requires no embedding configuration
- **Local models**: For Ollama, `api_key_env` can be empty (no authentication needed)

#### Runtime Behavior

| Scenario | Config | Behavior |
|----------|--------|----------|
| No embedding configured | `embedding: None` | FTS5-only (BM25) search |
| Embedding configured, API available | `embedding: EmbeddingConfig` | Hybrid search (FTS5 + Vector + RRF) |
| Embedding configured, API unavailable | `embedding: EmbeddingConfig` | Graceful degradation to FTS5-only |

#### Graceful Degradation

The system degrades gracefully when embedding fails:

```
Ingest (log_turn):
  if config.embedding:
      embed_result = embed_text(config.embedding, content)
      if isinstance(embed_result, Success):
          insert_embedding(conn, log_id, embedding)
      # Silent failure - log is stored, embedding skipped
  
Search (hybrid_search):
  query_embedding = None
  if config.embedding:
      embed_result = embed_text(config.embedding, query)
      if isinstance(embed_result, Success):
          query_embedding = embed_result.unwrap()
      # Falls back to FTS5-only if embedding fails
```

**Key invariant**: Logs are ALWAYS written to the database, even if embedding fails. Embedding is a non-blocking enhancement.

#### Error Categories

| Error | Failure Message | System Response |
|-------|-----------------|-----------------|
| `litellm` not installed | `"litellm not installed"` | Degrade to FTS5-only |
| Invalid response format | `"Unexpected response format: missing {key}"` | Degrade to FTS5-only |
| API error | `"Embedding error: {exception}"` | Degrade to FTS5-only |
| Missing model | Included in API error | Degrade to FTS5-only |

#### No Automatic Fallback to Alternative Models

Lattice does NOT automatically switch to alternative embedding models. Failures degrade to FTS5-only search. To use a different embedding model:

1. Update `config.toml` with new model configuration
2. Restart the MCP server
3. For existing logs, re-run ingestion with new model (truncates `logs_vec`)

#### Monitoring Embedding Health

Embedding failures are logged but do not raise exceptions to the caller. To monitor:

- Check logs for `"Embedding error"` messages
- Query `logs_vec` table: if `COUNT(*)` < `logs` count, embedding is failing
- Use `lattice search` with `--verbose` to see which search mode is active
## 11. Concurrency Model

### 11.1 File Locks

Two lock files prevent concurrent evolution:
- `.lattice/evolve.lock` -- project-level evolution
- `~/.config/lattice/evolve.lock` -- global-level evolution

Lock mechanism: Create file with `O_CREAT | O_EXCL` (atomic). On success, write PID. On failure, read PID and report. Release: delete file.

### 11.2 SQLite Concurrency

- Single-writer model (SQLite default). WAL mode is NOT required because Lattice is not multi-process for the same database.
- The MCP server holds a connection for its lifetime. The CLI opens/closes per command.
- If both MCP server and CLI try to write simultaneously, SQLite handles serialization via its internal locking (acceptable for Lattice's write frequency).

### 11.3 What Cannot Run Simultaneously

| Operation A | Operation B | Conflict? | Resolution |
|------------|------------|-----------|------------|
| `evolve` | `evolve` | YES | evolve.lock prevents |
| `evolve` | `evolve --global` | NO (different locks) | But global writes to project rule files, so logically should not overlap. RFC-002 notes this is "unlikely" since both are manual CLI. |
| `evolve` | `ingest` / `log_turn` | NO | Different tables/operations |
| `evolve --global` | `evolve --global` | YES | global evolve.lock prevents |
| MCP log_turn | CLI ingest | NO | SQLite serializes writes |

---

## Appendix A: pyproject.toml Updates Required

The current `pyproject.toml` needs these changes for P1-S1:

| Field | Current Value | Required Value |
|-------|--------------|----------------|
| `project.scripts.lattice` | `"lattice.cli.main:app"` | `"lattice.shell.cli:app"` |
| `project.dependencies` | `["typer>=0.9.0"]` | Add: `returns`, `deal`, `sqlite-vec`, `tiktoken`, `mcp`, `tomli` (for py<3.11), `httpx` (for LLM providers) |
| `tool.invar.guard.core_paths` | `["src/core"]` | `["src/lattice/core"]` |
| `tool.invar.guard.shell_paths` | `["src/shell"]` | `["src/lattice/shell"]` |

---

## Appendix B: Data File Templates

### B.1 Compiler Prompt Template (`data/compiler_prompt.md`)

The existing prototype template in `data/compiler_prompt.md` is Eidos-specific and must be replaced with a production version during P5-S1. The production template must:
- Accept current rules as input context (not hardcoded project name)
- Include all four phase tags: `<triage>`, `<cross_ref>`, `<synthesis>`, `<review>`
- Include instruction for aggressive `<review>` consolidation when above alert threshold
- Require explicit kept/merged/removed decisions with rationale in `<review>` phase
- Output structured proposal format parseable by `extract_proposals`

### B.2 Global Compiler Prompt Templates

Two new template files needed for P10-S1:
- `data/global_compiler_prompt_phase1.md`: Tags `<triage>` + `<candidate_patterns>`
- `data/global_compiler_prompt_phase2.md`: Tags `<cross_ref>` + `<synthesis>` (includes `<review>` sub-phase)

---

## Appendix C: Gap Analysis — Architecture Spec vs plan.yaml

### C.1 Module Coverage

| Spec Module | Zone | Plan Step(s) | Status |
|-------------|------|-------------|--------|
| `core/types.py` | Core | P1-S2 | COVERED |
| `core/rules.py` | Core | P2-S1, P2-S2, P2-S3 | COVERED (CONTRACT/TEST/IMPL gated) |
| `core/search.py` | Core | P3-S1, P3-S2, P3-S3 | COVERED (CONTRACT/TEST/IMPL gated) |
| `core/sanitizer.py` | Core | P4-S1, P4-S2, P4-S3 | COVERED (CONTRACT/TEST/IMPL gated) |
| `core/compiler.py` | Core | P5-S1, P5-S2, P5-S3 | COVERED (CONTRACT/TEST/IMPL gated) |
| `shell/config.py` | Shell | P1-S3 | COVERED |
| `shell/schema.py` | Shell | P1-S4 | COVERED |
| `shell/store.py` | Shell | P6-S1, P6-S2 | COVERED |
| `shell/embeddings.py` | Shell | P6-S3 | COVERED |
| `shell/hybrid_search.py` | Shell | P6-S4 | COVERED |
| `shell/llm.py` | Shell | P7-S1 | COVERED |
| `shell/evolution.py` | Shell | P7-S2, P7-S3, P7-S4, P7-S5 | COVERED |
| `shell/global_evolution.py` | Shell | P10-S1, P10-S2, P10-S3 | COVERED |
| `shell/server.py` | Shell | P9-S1, P9-S2, P9-S3, P9-S4 | COVERED |
| `shell/cli.py` | Shell | P8-S1, P8-S2, P8-S3, P8-S4, P10-S4 | COVERED |
| `client.py` | SDK | P11-S1 | COVERED |
| `__init__.py` | SDK | P11-S1 | COVERED (within SDK step) |
| `data/compiler_prompt.md` | Data | P5-S1 | COVERED |
| `data/global_compiler_prompt_phase1.md` | Data | P10-S1 | COVERED |
| `data/global_compiler_prompt_phase2.md` | Data | P10-S1 | COVERED |
| OpenCode plugin | External | P11-S2 | COVERED |

**Result: All spec modules have corresponding plan steps. No orphan modules.**

### C.2 Interface Coverage

| Spec Interface | Plan Step | Status |
|---------------|-----------|--------|
| `merge_rules` | P2-S1 contracts | COVERED |
| `count_tokens` | P2-S1 contracts | COVERED |
| `parse_tags` | P2-S1 contracts | COVERED |
| `assemble_instincts` | P2-S1 contracts | COVERED |
| `assign_positional_ranks` | P3-S1 contracts | COVERED |
| `fuse_rrf` | P3-S1 contracts | COVERED |
| `detect_secrets` | P4-S1 contracts | COVERED |
| `sanitize` | P4-S1 contracts | COVERED |
| `assemble_compiler_prompt` | P5-S1 contracts | COVERED |
| `parse_cot_output` | P5-S1 contracts | COVERED |
| `extract_proposals` | P5-S1 contracts | COVERED |
| `truncate_session` | P5-S1 contracts | COVERED |
| `diff_rules` | P5-S1 contracts | COVERED |
| `read_config` / `read_projects` / etc. | P1-S3 | COVERED |
| `create_store` / `create_global_store` | P1-S4 | COVERED |
| `insert_log` / `query_logs_*` / `search_fts` | P6-S1, P6-S2 | COVERED |
| `EmbeddingProvider` protocol | P6-S3 | COVERED |
| `hybrid_search` / `hybrid_search_global` | P6-S4 | COVERED |
| `LLMProvider` protocol | P7-S1 | COVERED |
| `query_pending_sessions` / `run_evolution` | P7-S2, P7-S3 | COVERED |
| `write_proposal` / `list_proposals` / `read_proposal` | P7-S4 | COVERED |
| `apply_proposal` / `backup_rules` / `revert_rules` / `rotate_backups` | P7-S5 | COVERED |
| `run_global_phase1` / `run_global_phase2` | P10-S1, P10-S2 | COVERED |
| `mark_promoted` / `write_evidence` / `write_rule_evidence_links` | P10-S3 | COVERED |
| MCP tools (5 tools) | P9-S2, P9-S3 | COVERED |
| MCP resources (2 URIs) | P9-S4 | COVERED |
| `Client` class (5 methods) | P11-S1 | COVERED |
| `resolve_global_dir` / `resolve_project_dir` | P1-S3 | COVERED |
| `group_logs_into_sessions` | P6-S1 | COVERED |
| `count_pending_sessions` | P6-S1 | COVERED |

**Result: All spec interfaces have corresponding plan steps.**

### C.3 Data Flow Coverage

| Data Flow | Plan Steps That Test It | Status |
|-----------|------------------------|--------|
| Ingest via MCP `log_turn` | P9-S2, P9-S5, P12-S1 | COVERED |
| Ingest via CLI `ingest` | P8-S4, P8-S5, P12-S1 | COVERED |
| Ingest via SDK `log_turn` | P11-S1, P11-S3 | COVERED |
| Sanitization in ingest pipeline | P4-S2, P12-S3 | COVERED |
| Search (hybrid: project) | P6-S4, P6-S5, P8-S5, P9-S5, P12-S1 | COVERED |
| Search (hybrid: global) | P8-S5, P10-S5, P12-S2 | COVERED |
| Evolve (incremental) | P7-S6, P8-S5, P12-S1 | COVERED |
| Evolve (full) | P7-S6 | COVERED |
| Evolve (zero sessions) | P7-S6, P12-S3 | COVERED |
| Global evolve (Phase 1+2) | P10-S1, P10-S2, P10-S5, P12-S2 | COVERED |
| Apply proposal | P7-S5, P7-S6, P8-S5 | COVERED |
| Revert rules | P7-S5, P7-S6, P8-S5 | COVERED |
| Backup rotation | P7-S5, P7-S6 | COVERED |
| Promotion marking | P10-S3, P10-S5, P12-S2 | COVERED |
| Override tag suppresses promoted | P10-S5, P12-S2 | COVERED |
| MCP server lifecycle | P9-S1, P9-S5, P12-S1 | COVERED |
| Server instructions assembly | P9-S1, P9-S5, P12-S1 | COVERED |
| `get_instincts()` mid-session refresh | P9-S3, P9-S5 | COVERED |
| Concurrency lock (project evolve) | P8-S2, P12-S3 | COVERED |
| Concurrency lock (global evolve) | P10-S4, P12-S3 | COVERED |
| Secret sanitization E2E | P12-S3 | COVERED |
| Fabricated evidence rejection | P12-S3 | COVERED |

**Result: All data flow paths have plan step coverage.**

### C.4 Identified Gaps

| # | Gap Description | Severity | Spec Section | Plan Location | Recommendation |
|---|----------------|----------|-------------|---------------|----------------|
| G1 | **`core/config.py` (validate_config) mentioned in P1-S3 but absent from spec Module Map.** P1-S3 describes a core config validation function `validate_config(raw: dict) -> Config`. The architecture spec places all config logic in `shell/config.py` with validation inline. Either the plan should drop `core/config.py` or the spec should add it. | IMPORTANT | Section 2.1 | P1-S3 | **Decision needed:** The spec omits `core/config.py` because config validation is thin enough to inline in `shell/config.py`. If you prefer strict core/shell separation for config validation, add `core/config.py` with a `validate_config(raw: dict) -> Config` pure function. The plan already references it. **Recommendation: Add `core/config.py` to the spec** to match the plan and maintain zone discipline. |
| G2 | **`hybrid_search.py` as separate module vs inline in `store.py`.** The plan (P6-S4) describes hybrid search as being added to `store.py`, but the spec creates a separate `shell/hybrid_search.py`. | SUGGESTION | Section 2.2, 4.9 | P6-S4 | The spec's separation is cleaner (store.py handles CRUD, hybrid_search.py orchestrates). P6-S4 refs `src/lattice/shell/store.py` and `src/lattice/core/search.py`. **Recommendation: Update P6-S4 refs to include `src/lattice/shell/hybrid_search.py`**, or consolidate into store.py if the implementer prefers. |
| G3 | **No plan step for `shell/config.py` path resolution functions.** `resolve_global_dir()` and `resolve_project_dir()` are in the spec (Section 4.5) but the plan P1-S3 description does not explicitly mention them. | SUGGESTION | Section 4.5 | P1-S3 | These are utility functions that naturally belong in P1-S3. The implementer will discover the need during P1-S3. No plan change strictly needed, but adding them to P1-S3 description would be cleaner. |
| G4 | **No explicit plan step for `tomli` / `tomllib` TOML parsing dependency.** The spec uses TOML for config. Python 3.11+ has `tomllib` built-in. For 3.10 support, `tomli` is needed. pyproject.toml currently only lists `typer`. | IMPORTANT | Appendix A | P1-S1 | P1-S1 already says "add all dependencies" but does not list `tomli`. **Recommendation: Ensure P1-S1 implementer knows to add `tomli` (or `tomli; python_version < "3.11"`) to dependencies.** Currently caught by the spec Appendix A, but the plan could be more explicit. |
| G5 | **`mcp` Python SDK not in pyproject.toml dependencies.** The MCP server (P9) needs the `mcp` SDK package. P1-S1 lists deps to add but does not include `mcp`. | IMPORTANT | Appendix A | P1-S1 | **Recommendation: Add `mcp` to the dependency list in P1-S1.** Alternatively, the implementer adds it when starting P9, but that causes a re-sync. Better to front-load. |
| G6 | **`httpx` dependency not in plan.** The LLM and embedding providers (P7-S1, P6-S3) will need an HTTP client. `httpx` is the standard choice. Not mentioned in P1-S1. | SUGGESTION | Appendix A | P1-S1 | **Recommendation: Add `httpx` to P1-S1 dependencies.** Minor -- implementer will naturally add it during P7-S1 or P6-S3. |
| G7 | **No plan step mentions `__init__.py` files for `core/` and `shell/` packages.** P1-S1 says "Create directory structure src/lattice/{core,shell,__init__.py}" which covers the top-level `__init__.py` but does not explicitly mention `core/__init__.py` and `shell/__init__.py`. | SUGGESTION | Section 9.3 | P1-S1 | Trivially covered by "create directory structure." No plan change needed. The implementer will create package `__init__.py` files naturally. |
| G8 | **Plan P1-S3 references `src/lattice/core/config.py` but spec has no such module.** See G1. This is the inverse view of the same gap. | IMPORTANT | Section 2 | P1-S3 refs | Resolve as part of G1. |
| G9 | **`FtsResult` and `VecResult` types not listed in P1-S2 description.** P1-S2 lists: Config, Rule, SearchResult, VecResult, Proposal, LogEntry, Session, CompilerOutput. Missing: `FtsResult`, `RankedItem`, `FusedResult`, `CotPhases`, `ProposalAction`, `RuleDiff`, `SecretSpan`, `ProposeRuleInput`, `StatusReport`, `CandidatePattern`, `GlobalCotPhases`, `GlobalPhase2Phases`, `Evidence`, `PromoteAction`. | IMPORTANT | Section 3 | P1-S2 | **Recommendation: Update P1-S2 description to list ALL types from the spec**, or add a note: "See ARCHITECTURE.md Section 3 for the complete type catalog." Many of these types will be discovered during their respective phases, but defining them all upfront in types.py prevents circular imports and establishes the shared vocabulary. |
| G10 | **No plan step for `data/global_compiler_prompt_phase2.md`.** P10-S1 mentions "global compiler prompt templates" but refs only `data/global_compiler_prompt.md` (singular). The spec requires two separate template files (phase1 and phase2). | SUGGESTION | Section 2.4, Appendix B.2 | P10-S1 | P10-S1 description already says "Phase 1 template with ... Phase 2 template with..." so the two files are implied. The refs entry should be updated to list both files. |
| G11 | **`group_logs_into_sessions` placement ambiguity.** The spec places it in `shell/store.py` (Section 4.7) but it calls `core.rules.count_tokens`, making it a hybrid function. It does no I/O itself (just transforms data). | SUGGESTION | Section 4.7 | P6-S1 | This function is pure transformation (no I/O). It could live in core. However, keeping it in shell/store.py alongside the query functions that produce its input is pragmatic. The plan covers it in P6-S1. No change needed unless you prefer strict zone purity, in which case move it to `core/compiler.py` or `core/rules.py`. |
| G12 | **Evidence-side FTS search function not explicitly specified.** The spec defines `search_fts` for `logs_fts` (Section 4.7) but the corresponding `search_evidence_fts` for `evidence_fts` in global.db is implied by `hybrid_search_global` but not explicitly defined as a standalone function. | SUGGESTION | Section 4.7, 4.9 | P10-S1 | **Recommendation: Add `search_evidence_fts` function signature to the spec**, or note that `hybrid_search_global` handles this internally. The implementer will need this during P10 or when implementing `hybrid_search_global`. |
| G13 | **Rule file reading functions not explicitly specified.** The data flow diagrams show "Read rule files from disk" but there is no explicit function signature for reading `.lattice/rules/*.md` into `list[Rule]`. This is used by evolution, server, client, and CLI. | IMPORTANT | Section 4.5 or 4.11 | P7-S2, P9-S1, P11-S1 | **Recommendation: Add `read_rules(rules_dir: str) -> Result[list[Rule], str]` to `shell/config.py` or create a `shell/rules_io.py` module.** This is a shared operation used by multiple shell modules. Without an explicit function, each module will reinvent it. |

### C.5 Summary

| Severity | Count | Description |
|----------|-------|-------------|
| BLOCKER | 0 | No blocking gaps found |
| IMPORTANT | 5 | G1/G8 (core/config.py), G4 (tomli dep), G5 (mcp dep), G9 (type list), G13 (read_rules) |
| SUGGESTION | 7 | G2, G3, G6, G7, G10, G11, G12 |

### C.6 Recommended Plan Amendments

**Priority 1 (address before P1 starts):**

1. **G1/G8 -- Decide on `core/config.py`**: Either add `validate_config(raw: dict) -> Config` as a core pure function (matching P1-S3 description), or remove it from P1-S3 description and inline validation in shell. **Recommendation: Keep it. It enforces zone discipline for config validation logic.**

2. **G9 -- Expand types list in P1-S2**: Either list all types from the spec, or add note "See ARCHITECTURE.md Section 3 for complete catalog." At minimum, implementer should create placeholder dataclasses for all types in the first pass, even if some fields are refined later.

3. **G13 -- Add `read_rules` function**: Add to `shell/config.py` spec (Section 4.5). This should be covered by P1-S3 or a small addition to P2's scope.

**Priority 2 (address during implementation):**

4. **G4/G5/G6 -- Dependency completeness**: Update P1-S1 to explicitly list: `returns`, `deal`, `sqlite-vec`, `tiktoken`, `mcp`, `tomli`, `httpx`, `toml` (for writing). These are all known at design time.

5. **G2 -- hybrid_search module**: Update P6-S4 refs to include `shell/hybrid_search.py`.

6. **G10 -- Global prompt template refs**: Update P10-S1 refs to list both `data/global_compiler_prompt_phase1.md` and `data/global_compiler_prompt_phase2.md`.
