# Write Pipeline

The write pipeline transforms natural language messages into structured, versioned knowledge. Every call to `memory.write()` runs four stages: **Extract → Resolve → Reconcile → Upsert**.

```mermaid
flowchart LR
    A["Message"] --> B["Extract"]
    B --> C["Resolve Entities"]
    C --> D["Reconcile"]
    D --> E["Upsert"]
    E --> F["WriteResult"]
```

## Overview

When you call `memory.write(user_id, message)`, the pipeline:

1. **Creates an immutable event** — The raw message is logged as a `MemoryEvent` with its embedding. This record is never modified or deleted — it's the audit trail.
2. **Extracts entities, facts, and relationships** — An LLM parses the message to identify structured information.
3. **Resolves entities to canonical records** — Deduplicates mentions ("Ana", "my wife Ana", "her") into a single entity.
4. **Reconciles against existing knowledge** — Decides whether each fact is new (ADD), supersedes an old fact (UPDATE), is already known (NOOP), or retracts something (DELETE).
5. **Upserts the results** — Executes the reconciliation decisions in the database.

Each stage is independently fail-safe: if extraction fails, the event is still logged. If reconciliation fails for one fact, the others proceed normally.

---

## Stage 1: Extraction

The extraction stage uses an LLM to parse natural language into structured data: entities, facts, and relationships.

### Two Modes

=== "Multi-pass (default)"

    Multi-pass extraction runs three sequential LLM calls for higher accuracy:

    1. **Entity scan** — Identify all entities mentioned in the message
    2. **Fact extraction** — For each batch of entities, extract facts (batched by `entity_batch_size`)
    3. **Relation extraction** — Identify relationships between entities

    Best for messages that mention multiple entities or contain complex information.

=== "Single-pass"

    Single-pass extraction runs a single LLM call that extracts entities, facts, and relationships together.

    Faster and cheaper, but may miss nuanced relationships in complex messages. Falls back automatically when entity count ≤ `multi_pass_entity_threshold`.

    Includes an optional **reflexion pass** — a second LLM call that reviews the extraction for quality.

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `extraction_model` | `"gpt-4o"` | LLM model for extraction |
| `extraction_mode` | `"multi_pass"` | `"multi_pass"` or `"single_pass"` |
| `extraction_timeout_sec` | `30.0` | Timeout per LLM call |
| `multi_pass_entity_threshold` | `3` | Entity count below which multi-pass falls back to single-pass |
| `entity_batch_size` | `5` | Entities per fact-extraction batch in multi-pass mode |

### What Gets Extracted

For each message, the extraction stage produces:

- **Entities** — Named things: people, organizations, places, concepts, etc.
- **Facts** — Statements about entities in natural language (e.g., "lives in São Paulo", "works as a backend engineer")
- **Relations** — Connections between entities (e.g., "Rafael" → `works_at` → "Acme Corp")

Each fact includes a **confidence level**:

| Level | Score | Example |
|-------|-------|---------|
| Explicit statement | 0.95 | "I live in São Paulo" |
| Strong inference | 0.80 | "We went to the São Paulo office" (implies location) |
| Weak inference | 0.60 | Contextual implication |
| Speculation | 0.40 | Uncertain information |

### Fail-safe Behavior

If an LLM call fails (timeout, invalid JSON, rate limit), the extraction returns an empty result rather than raising an exception. The event is still logged — no data is lost. The next message may capture the same information.

!!! info "Neuroscience parallel"
    Extraction mirrors **encoding** in human memory — the process of converting sensory input (a conversation) into a memory trace. Just as human encoding is selective (we don't remember every word), the LLM extracts only salient facts and entities.

---

## Stage 2: Entity Resolution

Entity resolution maps extracted entity names to canonical records. This prevents duplicates: "Ana", "my wife Ana", and "Aninha" all resolve to the same `person:ana` entity.

### Three-Phase Resolution

```mermaid
flowchart LR
    A["Entity name"] --> B{"Exact match?"}
    B -->|Yes| F["Resolved"]
    B -->|No| C{"Fuzzy match?"}
    C -->|"≥ 0.85"| F
    C -->|"0.50–0.85"| D{"LLM decides"}
    C -->|"< 0.50"| E["Create new entity"]
    D -->|Match| F
    D -->|No match| E
    E --> F
```

**Phase 1: Exact match**

Checks the alias cache, entity slugs, and display names. Instant, no LLM call.

Includes **prefix/diminutive matching** for person entities: "Carol" matches "Carolina" (minimum 3 characters).

**Phase 2: Fuzzy match**

Uses embedding cosine similarity (via pgvector) to find candidates:

- **≥ 0.85** — High confidence match, resolves directly
- **0.50–0.85** — Ambiguous, forwards top-3 candidates to Phase 3
- **< 0.50** — No match, creates a new entity

Falls back to `difflib.SequenceMatcher` when embeddings are unavailable.

**Phase 3: LLM fallback**

Sends ambiguous candidates to an LLM (`gpt-4o-mini`) for disambiguation. The LLM sees the entity name, the candidates, and decides which (if any) is a match.

### Special Cases

- **Self-references** — "I", "me", "eu", "myself" automatically resolve to `user:self`
- **Relationship terms** — "my girlfriend", "my brother", "meu amigo" resolve with person type inference
- **Relational hints** — `"Carol (Rafael's girlfriend)"` strips the hint and forces `type="person"`

### Alias Registration

When a new alias is discovered (e.g., "Aninha" resolves to `person:ana`), it's registered in `MemoryEntityAlias` with **first-write-wins** semantics — concurrent writes won't create conflicting aliases.

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `fuzzy_threshold` | `0.85` | Cosine similarity threshold for direct fuzzy match |
| `enable_llm_resolution` | `True` | Whether to use LLM for ambiguous cases |
| `entity_resolution_model` | `"gpt-4o-mini"` | Model for LLM resolution |

!!! info "Neuroscience parallel"
    Entity resolution mirrors **associative memory** — the brain's ability to link new stimuli to existing representations. Hearing "Carol" activates the neural pattern for "Carolina" through pattern completion, just as fuzzy matching activates candidate entities through embedding similarity.

---

## Stage 3: Reconciliation

Reconciliation compares new facts against existing knowledge and decides what to do with each one. This is how the memory stays accurate over time — contradictions are resolved, redundancies are skipped, and outdated information is superseded.

### Decision Logic

For each extracted fact, the reconciler:

1. **Fetches existing facts** for the same entity
2. **Computes similarity** between the new fact and each existing fact (via embeddings)
3. **Decides the action**:

| Action | When | Example |
|--------|------|---------|
| **ADD** | New information, no similar existing fact (similarity < 0.50) | "speaks French" when no language fact exists |
| **UPDATE** | Supersedes an existing fact (similarity 0.50–0.85+) | "lives in Rio" supersedes "lives in São Paulo" |
| **NOOP** | Already known (high similarity) | "works at Acme" when this fact already exists |
| **DELETE** | Explicitly retracts a fact | "I no longer work at Acme" |

- **Low similarity (< 0.50)**: Auto-ADD without LLM call (fast path)
- **Ambiguous similarity (0.50+)**: LLM decides the action with full context

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `reconciliation_model` | `"gpt-4o-mini"` | Model for reconciliation decisions |

### Fail-safe Behavior

If the reconciliation LLM call fails, the system defaults to **ADD** — it's better to have a near-duplicate than to lose information. The background consolidation jobs (clustering, deduplication) clean up duplicates later.

### Fact Versioning

Facts are versioned using temporal validity windows (`valid_from`, `valid_to`):

- **Active facts** have `valid_to = NULL`
- **Updated facts** get `valid_to` set to the current timestamp, and a new fact is created with `supersedes_fact_id` pointing to the old one
- **Deleted facts** get both `valid_to` and `invalidated_at` set

This enables time-travel queries: you can ask what the system knew at any point in time.

!!! info "Neuroscience parallel"
    Reconciliation mirrors **reconsolidation** — the process by which retrieved memories become labile and can be modified. When you recall a memory ("lives in São Paulo") and encounter new information ("just moved to Rio"), the original memory is updated. The brain doesn't simply overwrite — it creates a new trace linked to the original, just as UPDATE creates a new fact with `supersedes_fact_id`.

---

## Stage 4: Upsert

The upsert stage executes the reconciliation decisions in the database:

| Decision | Database action |
|----------|-----------------|
| ADD | Create new `MemoryFact` with embedding |
| UPDATE | Close old fact (`valid_to = now`), create new one with `supersedes_fact_id` |
| NOOP | Update `last_confirmed_at` on existing fact |
| DELETE | Close fact (`valid_to = now`, `invalidated_at = now`) |

### Relationship Tracking

During upsert, extracted relationships are also persisted:

- Creates/updates `MemoryEntityRelationship` records
- Resolves source and target entities via the entity map
- **Strength reinforcement**: repeated relationships increase `strength` (up to 1.0)
- Uses `ON CONFLICT DO UPDATE` for idempotent upserts

### Transaction Safety

The entire write pipeline runs inside a database transaction. Individual fact upserts use **savepoints** (`session.begin_nested()`) so that a failure in one fact doesn't abort the entire batch:

```python
# If this fact fails, only this savepoint rolls back
async with session.begin_nested():
    session.add(new_fact)
    await session.flush()
```

The event record is created and flushed first, so it survives even if all subsequent stages fail.

---

## WriteResult

After the pipeline completes, you get a `WriteResult` with full observability:

```python
result = await memory.write(user_id="user_123", message="...")

# What happened
print(result.facts_added)       # List of facts created
print(result.facts_updated)     # List of facts superseded
print(result.facts_unchanged)   # Count of NOOP decisions
print(result.facts_deleted)     # Count of DELETE decisions
print(result.entities_resolved) # List of resolved entities
print(result.duration_ms)       # Total pipeline time
```

---

## Pipeline Diagram (Complete)

```mermaid
flowchart TD
    MSG["User message"] --> EVT["Create MemoryEvent\n(immutable log + embedding)"]
    EVT --> EXT{"Extraction mode?"}
    EXT -->|multi-pass| MP["Entity Scan → Fact Batches → Relations"]
    EXT -->|single-pass| SP["Single LLM call\n(entities + facts + relations)"]
    MP --> RES["Entity Resolution\n(exact → fuzzy → LLM)"]
    SP --> RES
    RES --> REC["Reconciliation\n(ADD / UPDATE / NOOP / DELETE)"]
    REC --> UPS["Upsert\n(with savepoints)"]
    UPS --> REL["Relationship Tracking\n(strength reinforcement)"]
    REL --> WR["WriteResult"]
```
