# Write Pipeline

The write pipeline transforms natural language messages into structured, versioned knowledge. Every call to `memory.write()` runs four stages: **Extract → Resolve → Reconcile → Upsert**.

```mermaid
flowchart LR
    A["Message"] --> B["Extract"]
    B --> C["Resolve Entities"]
    C --> D["Reconcile"]
    D --> E["Upsert"]
    E --> F["WriteResult"]
```

## Overview

When you call `memory.write(user_id, message)`, the pipeline:

1. **Creates an immutable event** — The raw message is logged as a `MemoryEvent` with its embedding. This record is never modified or deleted — it's the audit trail.
2. **Extracts entities, facts, and relationships** — An LLM parses the message to identify structured information.
3. **Resolves entities to canonical records** — Deduplicates mentions ("Ana", "my wife Ana", "her") into a single entity.
4. **Reconciles against existing knowledge** — Decides whether each fact is new (ADD), supersedes an old fact (UPDATE), is already known (NOOP), or retracts something (DELETE).
5. **Upserts the results** — Executes the reconciliation decisions in the database.

Each stage is independently fail-safe: if extraction fails, the event is still logged. If reconciliation fails for one fact, the others proceed normally.

---

## Stage 1: Extraction

The extraction stage uses an LLM to parse natural language into structured data: entities, facts, and relationships.

### Two Modes

=== "Multi-pass (default)"

    Multi-pass extraction runs an entity scan first, then fact batches and relation extraction concurrently via `asyncio.gather`. The total number of LLM calls depends on entity count and `entity_batch_size`.

    1. **Entity scan** — Identify all entities mentioned in the message
    2. **Fact extraction + Relation extraction** — Run concurrently: fact extraction batches entities by `entity_batch_size`, while relation extraction identifies relationships between entities

    Best for messages that mention multiple entities or contain complex information.

    Automatically falls back to single-pass when entity count ≤ `multi_pass_entity_threshold`.

=== "Single-pass"

    Single-pass extraction runs a single LLM call that extracts entities, facts, and relationships together.

    Faster and cheaper, but may miss nuanced relationships in complex messages. Also used as automatic fallback when multi-pass finds few entities (≤ `multi_pass_entity_threshold`).

    Includes an automatic **reflexion pass** — a second LLM call that reviews the extraction for quality. Includes a built-in retry (1 attempt) on extraction failure before returning empty.

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `extraction_model` | `"gpt-4o"` | LLM model for extraction |
| `extraction_mode` | `"multi_pass"` | `"multi_pass"` or `"single_pass"` |
| `extraction_timeout_sec` | `30.0` | Timeout per LLM call |
| `multi_pass_entity_threshold` | `3` | Entity count below which multi-pass falls back to single-pass |
| `entity_batch_size` | `5` | Entities per fact-extraction batch in multi-pass mode |
| `reflexion_timeout_sec` | `10.0` | Timeout for the reflexion quality-review pass (single-pass only) |

### What Gets Extracted

For each message, the extraction stage produces:

- **Entities** — Named things: people, organizations, places, concepts, etc.
- **Facts** — Statements about entities in natural language (e.g., "lives in São Paulo", "works as a backend engineer")
- **Relations** — Connections between entities (e.g., "Rafael" → `works_at` → "Acme Corp")

Each fact includes a **confidence level**:

| Level | Score | Example |
|-------|-------|---------|
| Explicit statement | 0.95 | "I live in São Paulo" |
| Strong inference | 0.80 | "We went to the São Paulo office" (implies location) |
| Weak inference | 0.60 | Contextual implication |
| Speculation | 0.40 | Uncertain information |

### Alias Grouping & Subject Normalization

When the same entity is mentioned by multiple names in a single message (e.g., "my friend Guili (Guilherme Maturana)"), the extraction groups them into a **single entity with aliases** instead of creating duplicates.

The LLM is instructed to pick one canonical name (usually the most complete) and list the others as aliases:

```json
{
  "entities": [
    {"name": "Guilherme Maturana", "type": "person", "aliases": ["Guili"]}
  ]
}
```

After extraction, a **subject normalization** pass rewrites any fact or relation that references an alias to use the canonical name instead. Identity relations (e.g., `same_as` between an alias and its canonical name) are removed automatically since they become self-referencing after normalization.

This eliminates intra-message entity duplication at the source — before entity resolution even runs.

### Entity Type Validation

The reflexion pass (quality review) validates entity type classification. When messages mention many people alongside places, the LLM may misclassify places as `person`. The reflexion cross-references entity types with the associated facts — for example, if a fact says "Botafogo is a neighborhood in Rio de Janeiro", then "Rio de Janeiro" must be type `place`, not `person`.

In multi-pass mode (where reflexion doesn't run), the entity scan prompt includes explicit instructions to classify types carefully, especially distinguishing `person` vs `place`.

### Fail-safe Behavior

If an LLM call fails (timeout, invalid JSON, rate limit), the extraction returns an empty result rather than raising an exception. The event is still logged — no data is lost. The next message may capture the same information.

!!! info "Neuroscience parallel"
    Extraction mirrors **encoding** in human memory — the process of converting sensory input (a conversation) into a memory trace. Just as human encoding is selective (we don't remember every word), the LLM extracts only salient facts and entities.

---

## Stage 2: Entity Resolution

Entity resolution maps extracted entity names to canonical records. This prevents duplicates: "Ana", "my wife Ana", and "Aninha" all resolve to the same `person:ana` entity.

### Three-Phase Resolution

```mermaid
flowchart LR
    A["Entity name"] --> B{"Exact match?"}
    B -->|Yes| F["Resolved"]
    B -->|No| C{"Fuzzy match?"}
    C -->|"≥ 0.85"| F
    C -->|"0.50–0.85"| D{"LLM decides"}
    C -->|"< 0.50"| E["Create new entity"]
    D -->|Match| F
    D -->|No match| E
    E --> F
```

**Phase 1: Exact match**

Checks the alias cache, entity slugs, and display names. Instant, no LLM call.

Includes **prefix/diminutive matching** for person entities: "Carol" matches "Carolina" (minimum 3 characters).

**Phase 2: Fuzzy match**

Uses embedding cosine similarity (in-memory) to find candidates:

- **≥ 0.85** — High confidence match, resolves directly
- **0.50–0.85** — Ambiguous, forwards top-3 candidates to Phase 3
- **< 0.50** — No match, creates a new entity

Falls back to `difflib.SequenceMatcher` when embeddings are unavailable.

**Phase 3: LLM fallback**

Sends ambiguous candidates to the injected `LLMProvider` for disambiguation. The LLM sees the entity name, the candidates, and decides which (if any) is a match.

### Special Cases

- **Self-references** — "I", "me", "eu", "myself" automatically resolve to `user:self`
- **Relationship terms** — "my girlfriend", "my brother", "meu amigo" resolve to `user:self` (the relationship is about the user, not a separate entity)
- **Relational hints** — `"Carol (Rafael's girlfriend)"` strips the hint and forces `type="person"`

### Alias Registration

When a new alias is discovered (e.g., "Aninha" resolves to `person:ana`), it's registered in `MemoryEntityAlias` with **first-write-wins** semantics — concurrent writes won't create conflicting aliases.

**Extraction-provided aliases** are also registered automatically: when entity resolution creates a new entity that has aliases from extraction (e.g., "Guili" for "Guilherme Maturana"), all aliases are registered in `MemoryEntityAlias` and added to the in-memory alias cache. This means subsequent entities in the same batch can immediately resolve via Phase 1 exact match — no fuzzy or LLM calls needed.

This creates a two-line defense against duplicates:

1. **Intra-message** — Alias grouping in extraction prevents duplicates within a single message
2. **Cross-message** — Registered aliases enable exact match in future messages (e.g., if message 1 creates "Guilherme Maturana" with alias "Guili", message 2 mentioning "Guili" resolves instantly via Phase 1)

### Entity Persistence

After entity resolution completes, the pipeline ensures **every resolved entity** has a row in the `memory_entities` table — not just newly created ones. Entities resolved via exact match, fuzzy match, or LLM disambiguation are also upserted using `ON CONFLICT DO UPDATE` (idempotent).

This is critical because background jobs (importance scoring, summary refresh, spreading activation) read from `memory_entities`. Without a row, these jobs are blind to the entity and can't operate on it.

The entity upsert is fail-safe: if one entity fails to persist (e.g., constraint violation), the others proceed normally and the pipeline continues.

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `fuzzy_threshold` | `0.85` | Cosine similarity threshold for direct fuzzy match |
| `enable_llm_resolution` | `True` | Whether to use LLM for ambiguous cases. When `False`, ambiguous candidates create a new entity instead of calling LLM. |

!!! note "Model selection"
    The LLM model used for entity resolution is determined by the `LLMProvider` you inject into `MemoryClient`. To use a different model for resolution vs. extraction, inject different providers.

!!! info "Neuroscience parallel"
    Entity resolution mirrors **associative memory** — the brain's ability to link new stimuli to existing representations. Hearing "Carol" activates the neural pattern for "Carolina" through pattern completion, just as fuzzy matching activates candidate entities through embedding similarity.

---

## Stage 3: Reconciliation

Reconciliation compares new facts against existing knowledge and decides what to do with each one. This is how the memory stays accurate over time — contradictions are resolved, redundancies are skipped, and outdated information is superseded.

### Decision Logic

For each extracted fact, the reconciler:

1. **Fetches existing facts** for the same entity
2. **Computes similarity** between the new fact and each existing fact (via embeddings)
3. **Decides the action**:

| Action | When | Example |
|--------|------|---------|
| **ADD** | New information, no similar existing fact (similarity < 0.50) | "speaks French" when no language fact exists |
| **UPDATE** | Supersedes an existing fact (similarity 0.50–0.85+) | "lives in Rio" supersedes "lives in São Paulo" |
| **NOOP** | Already known (high similarity) | "works at Acme" when this fact already exists |
| **DELETE** | Explicitly retracts a fact | "I no longer work at Acme" |

- **Low similarity (< 0.50)**: Auto-ADD without LLM call (fast path)
- **Ambiguous similarity (0.50+)**: LLM decides the action with full context

### Fail-safe Behavior

If the reconciliation LLM call fails, the system defaults to **ADD** — it's better to have a near-duplicate than to lose information. The background consolidation jobs (clustering, deduplication) clean up duplicates later.

### Fact Versioning

Facts are versioned using temporal validity windows (`valid_from`, `valid_to`):

- **Active facts** have `valid_to = NULL`
- **Updated facts** get both `valid_to` and `invalidated_at` set, and a new fact is created with `supersedes_fact_id` pointing to the old one
- **Deleted facts** get both `valid_to` and `invalidated_at` set

This enables time-travel queries: you can ask what the system knew at any point in time.

!!! info "Neuroscience parallel"
    Reconciliation mirrors **reconsolidation** — the process by which retrieved memories become labile and can be modified. When you recall a memory ("lives in São Paulo") and encounter new information ("just moved to Rio"), the original memory is updated. The brain doesn't simply overwrite — it creates a new trace linked to the original, just as UPDATE creates a new fact with `supersedes_fact_id`.

---

## Stage 4: Upsert

The upsert stage executes the reconciliation decisions in the database:

| Decision | Database action |
|----------|-----------------|
| ADD | Create new `MemoryFact` with embedding |
| UPDATE | Close old fact (`valid_to = now`), create new one with `supersedes_fact_id` |
| NOOP | Update `last_confirmed_at` on existing fact |
| DELETE | Close fact (`valid_to = now`, `invalidated_at = now`) |

### Relationship Tracking

During upsert, extracted relationships are also persisted:

- Creates/updates `MemoryEntityRelationship` records
- Resolves source and target entities via the entity map
- **Strength reinforcement**: repeated relationships increase `strength` (up to 1.0)
- Uses `ON CONFLICT DO UPDATE` for idempotent upserts

#### Evidence Linkage & Cascade Invalidation

**The problem:** Without linkage between facts and relationships, contradictory edges accumulate. If a user says "I live in Curitiba" and later "I moved to São Paulo", the old relationship `user --[lives_in]--> curitiba` would remain active alongside the new one — polluting retrieval with stale context.

**The solution:** Each relationship is linked to the fact that supports it via `evidence_fact_id`. When that fact is superseded (UPDATE) or retracted (DELETE), the relationship is **automatically invalidated** — no manual cleanup needed.

How evidence linkage works:

1. After facts are persisted in the upsert stage, a heuristic match associates each relationship with a corresponding fact. For a relationship `(source, target)`, the matcher looks for facts whose `fact_text` mentions both entity names.
2. If multiple facts match, the one with the highest confidence is selected.
3. The matched fact's ID is stored as `evidence_fact_id` on the relationship.

When a fact is invalidated (via UPDATE or DELETE), **cascade invalidation** automatically sets `invalidated_at` and `valid_to` on all relationships that reference it. The [graph retrieval BFS](../advanced/read-api.md) already filters out invalidated relationships, so stale edges are immediately excluded from context.

```
User: "I live in Curitiba"
  → fact: "User lives in Curitiba" (fact_1)
  → rel:  user --[lives_in]--> curitiba (evidence_fact_id = fact_1)

User: "I moved to São Paulo"
  → reconciliation: UPDATE fact_1 → fact_2 "User lives in São Paulo"
  → cascade: rel lives_in→curitiba is INVALIDATED (evidence_fact_id = fact_1)
  → new rel: user --[lives_in]--> sao_paulo (evidence_fact_id = fact_2)
```

!!! note "Relationship types are dynamic"
    The `rel_type` field accepts any descriptive `snake_case` string — not just a fixed set. Common types include `works_at`, `lives_in`, `family_of`, but the LLM may also produce types like `mentored_by` or `inspired_by`. See [Dynamic Relationship Types](../advanced/data-types.md#dynamic-relationship-types) for details on normalization and aliases.

#### Mirror Facts

Sometimes the LLM infers a relationship from context without extracting a corresponding fact. For example, "I'm going to Curitiba to visit my mom" implies `mom --[lives_in]--> curitiba`, but the LLM may only extract a fact about the user's trip — not about where mom lives. Without a fact, the relationship can't participate in cascade invalidation and isn't findable via semantic search.

To solve this, a **mirror fact** is automatically created as a fallback when no heuristic match is found. The mirror fact is a simple natural-language sentence generated from the relationship: `"{source_name} {rel_type} {target_name}"` (e.g., `"Mom lives in Curitiba"`).

Mirror facts are marked with:

- `confidence = 0.60` (weak inference — lower priority in retrieval ranking)
- `source_context = "inferred_from_relation"` (allows filtering or downranking if needed)

The mirror fact's ID is used as the relationship's `evidence_fact_id`, so cascade invalidation works for inferred relationships too.

!!! tip "Reducing mirror facts"
    The extraction prompts instruct the LLM to extract implicit facts alongside relationships (e.g., "my mom lives in Curitiba" as a fact, not just a relation). As LLM extraction improves, fewer mirror facts are needed — they're the safety net, not the primary mechanism.

### Transaction Safety

The entire write pipeline runs inside a database transaction. Individual fact upserts use **savepoints** (`session.begin_nested()`) so that a failure in one fact doesn't abort the entire batch:

```python
# If this fact fails, only this savepoint rolls back
async with session.begin_nested():
    session.add(new_fact)
    await session.flush()
```

The event record is created and flushed first, so it survives even if all subsequent stages fail.

---

## WriteResult

After the pipeline completes, you get a `WriteResult` with full observability:

```python
result = await memory.write(user_id="user_123", message="...")

# What happened
print(result.facts_added)       # List of facts created
print(result.facts_updated)     # List of facts superseded
print(result.facts_unchanged)   # List of confirmed facts (NOOP decisions)
print(result.facts_deleted)     # List of retracted facts (DELETE decisions)
print(result.entities_resolved) # List of resolved entities
print(result.duration_ms)       # Total pipeline time
print(result.event_id)          # Unique event ID for this write
print(result.pipeline)          # PipelineTrace (when verbose=True)
print(result.success)           # True if pipeline completed without errors
print(result.error)             # Error message if pipeline failed (None otherwise)
```

---

## Pipeline Diagram (Complete)

```mermaid
flowchart TD
    MSG["User message"] --> EVT["Create MemoryEvent\n(immutable log + embedding)"]
    EVT --> EXT{"Extraction mode?"}
    EXT -->|multi-pass| MP["Entity Scan → Fact Batches → Relations"]
    EXT -->|single-pass| SP["Single LLM call\n(entities + facts + relations)"]
    MP --> RES["Entity Resolution\n(exact → fuzzy → LLM)"]
    SP --> RES
    RES --> REC["Reconciliation\n(ADD / UPDATE / NOOP / DELETE)"]
    REC --> UPS["Upsert\n(with savepoints)"]
    UPS --> REL["Relationship Tracking\n(strength reinforcement)"]
    REL --> WR["WriteResult"]
```
