# Background Jobs

Background jobs improve memory quality over time. They run **separately** from `write()` and `retrieve()` — you schedule them yourself (every few hours, via cron, APScheduler, or a simple loop).

### Do I need them?

**For getting started: no.** The `write()` and `retrieve()` pipelines work without background jobs. Your agent will still extract facts, resolve entities, and return relevant context.

**For production: yes.** Without them, importance scores stay flat (0.5 for everything), entity summaries are never generated, patterns and contradictions go undetected, and retrieval quality degrades over time as the memory grows.

```mermaid
flowchart LR
    A["Scheduler\n(periodic)"] --> B["Clustering"]
    A --> C["Consolidation"]
    A --> D["Importance\nScoring"]
    A --> E["Summary\nRefresh"]
```

## Overview

`arandu` provides four categories of background jobs:

| Job | Purpose | Uses LLM? | Frequency |
|-----|---------|-----------|-----------|
| **Clustering** | Group related facts semantically | Yes (summaries) | Every 4-8 hours |
| **Consolidation** | Detect patterns, contradictions, trends | Yes | Every 4-8 hours |
| **Memify** | Convert episodic facts to procedural/semantic knowledge | Yes | Daily |
| **Sleep-time compute** | Score importance, refresh summaries, detect communities | Partially | Every 4-8 hours |

All jobs are exposed as async functions you can call directly or schedule with your preferred task runner (APScheduler, Celery, cron, etc.).

!!! info "Neuroscience parallel"
    Background jobs mirror **sleep-time processing** in the brain. During sleep, the brain consolidates memories, transfers information from hippocampus (short-term) to neocortex (long-term), prunes irrelevant connections, and strengthens important ones. These jobs perform the same operations on your agent's memory.

---

## Clustering

**In plain English:** Groups related facts together. Facts about someone's job, colleagues, and projects end up in one cluster. This makes retrieval more contextual — when you ask about someone's work, the system knows which facts are related.

### Fact Clustering

```python
from arandu import cluster_user_facts, ClusteringResult

result: ClusteringResult = await cluster_user_facts(
    session=db_session,
    agent_id="user_123",
    embedding_provider=embedding_provider,
    llm_provider=llm_provider,
    config=memory_config,
)
```

**How it works:**

1. Groups facts by `(entity_type, entity_key)` — facts about the same entity stay together
2. Generates a 2-3 sentence summary per cluster using an LLM
3. Computes and stores cluster embeddings for later community detection
4. Idempotent — updates existing clusters rather than creating duplicates

### Community Detection

```python
from arandu import detect_communities, CommunityDetectionResult

result: CommunityDetectionResult = await detect_communities(
    session=db_session,
    agent_id="user_123",
    embedding_provider=embedding_provider,
    llm_provider=llm_provider,
    config=memory_config,
)
```

**How it works:**

1. Compares cluster embeddings using cosine similarity
2. Groups clusters above `community_similarity_threshold` (default 0.75)
3. Creates `MemoryMetaObservation` records with type `"community_theme"`
4. Example: a "work" community might include clusters about colleagues, projects, and company facts

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `cluster_max_age_days` | `90` | Maximum age of facts to include in clustering |
| `community_similarity_threshold` | `0.75` | Cosine similarity threshold for grouping clusters |

??? example "Walkthrough: before and after clustering"
    **Before clustering** — 12 facts about a user, ungrouped:
    ```
    person:fernanda → "Fernanda works at Orion Tech"
    person:fernanda → "Fernanda graduated from Unicamp"
    person:fernanda → "Fernanda built the data pipeline"
    person:bruno → "Bruno works at Orion Tech"
    person:bruno → "Bruno developed the ML model"
    person:bruno → "Bruno runs marathons"
    person:marcos → "Marcos lives in Porto Alegre"
    person:marcos → "Marcos is a product manager"
    ...
    ```

    **After clustering:**
    ```
    Cluster 1: "Orion Tech engineering team"
      → Fernanda works at Orion Tech
      → Bruno works at Orion Tech
      → Fernanda built the data pipeline
      → Bruno developed the ML model
      Summary: "The Orion Tech data/ML team includes Fernanda (pipeline) and Bruno (ML model)"

    Cluster 2: "Marcos's personal life"
      → Marcos lives in Porto Alegre
      → Marcos is married to Carolina
      Summary: "Marcos lives in Porto Alegre with his wife Carolina"
    ```

    **Impact on retrieval:** When someone asks "Tell me about the Orion Tech team", the cluster summary and its facts score higher because they're grouped together — the system understands they're related.

---

## Consolidation

**In plain English:** Looks across all recent facts and events to find bigger patterns: "This person mentions running every Monday" (pattern), "They said they live in SP but also in RJ" (contradiction), "Their mood has been improving lately" (trend). Stores these as meta-observations that enrich retrieval.

### Periodic Consolidation (L2)

```python
from arandu import run_consolidation, ConsolidationResult

result: ConsolidationResult = await run_consolidation(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
    config=memory_config,
)
```

**How it works:**

1. Analyzes events and facts over a lookback window (`consolidation_lookback_days`)
2. Detects patterns across facts:
   - **Insights** — Emergent understanding from multiple facts
   - **Patterns** — Repeated behaviors or preferences
   - **Contradictions** — Conflicting facts that need resolution
   - **Trends** — Changes over time
3. Generates `MemoryMetaObservation` records
4. Tags events with emotions (emotion, intensity, energy level)

### Profile Consolidation (L3)

```python
from arandu import run_profile_consolidation

await run_profile_consolidation(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
)
```

**How it works:**

1. Refreshes entity summaries via LLM — a higher-level view of each entity
2. Updates the overall profile overview
3. Triggered periodically (less frequently than L2)

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `consolidation_min_events` | `3` | Minimum events before running consolidation |
| `consolidation_lookback_days` | `7` | How far back to look for patterns |

!!! info "Neuroscience parallel"
    Consolidation mirrors the brain's **memory consolidation during sleep**. The hippocampus replays recent experiences, the neocortex detects patterns and integrates them into existing knowledge structures, and contradictions are flagged for resolution. L2 consolidation is analogous to slow-wave sleep (SWS) replay, while L3 profile consolidation is analogous to REM sleep's role in integrating memories into semantic knowledge.

---

## Memify

**In plain English:** Over time, specific details ("went to a Python meetup on March 5") become general knowledge ("regularly attends tech meetups"). Memify distills episodic facts into higher-level knowledge and prunes stale facts that haven't been mentioned in a while.

### Run Memify

```python
from arandu import run_memify, MemifyResult

result: MemifyResult = await run_memify(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
    embedding_provider=embedding_provider,
    config=memory_config,
)
```

**How it works:**

1. Groups related facts by entity and topic
2. Generates distilled summaries (procedural/semantic knowledge)
3. Checks vitality — facts mentioned recently are kept; stale facts may be deprecated
4. Merges similar procedures to prevent knowledge fragmentation

### Vitality Scoring

```python
from arandu import compute_vitality

# Synchronous, per-fact function — NOT async
score = compute_vitality(fact)          # uses datetime.now(UTC)
score = compute_vitality(fact, now=ts)  # custom timestamp
```

Vitality measures how "alive" a fact is based on:

- **Retrieval** (0.30) — Log-scale of times_retrieved
- **Recency** (0.25) — Exponential decay from last_retrieved_at (30-day half-life)
- **Confidence** (0.20) — Raw confidence value
- **Reinforcement** (0.15) — Bounded reinforcement count (up to 5)

Correction penalty: `0.8 ^ user_correction_count`. Superseded facts (valid_to is not None) return 0.0.

!!! info "Neuroscience parallel"
    Memify mirrors the **forgetting curve** described by Hermann Ebbinghaus (1885). Memories decay exponentially over time unless reinforced through retrieval practice. Facts with high vitality (frequently accessed) resist decay, while low-vitality facts gradually fade — just as the brain prunes synaptic connections for unused information.

---

## Sleep-Time Compute

**In plain English:** Three maintenance jobs that keep retrieval sharp: (1) score which entities matter most, (2) refresh entity summaries for the important ones, (3) detect communities of related entities. The first one is pure SQL (cheap), the other two use LLM calls.

### Job 1: Entity Importance Scoring

```python
from arandu import compute_entity_importance, EntityImportanceResult

result: EntityImportanceResult = await compute_entity_importance(
    session=db_session,
    agent_id="user_123",
    config=memory_config,
)
```

Pure SQL computation (no LLM calls). Scores each entity from 0.0 to 1.0 using four normalized signals:

| Signal | Weight | Description |
|--------|--------|-------------|
| Fact density | 0.30 | Number of facts linked to the entity (via `MemoryFactEntityLink`). Includes facts where the entity is primary subject AND facts that merely mention it. |
| Recency | 0.25 | Exponential decay (30-day half-life) |
| Retrieval frequency | 0.25 | How often facts about this entity are retrieved |
| Relationship degree | 0.20 | Number of incoming + outgoing relationships |

The importance score is used as a signal in retrieval scoring and as a priority factor for summary refresh.

??? example "Walkthrough: how importance scoring changes retrieval"
    **Before importance scoring** — all entities have default importance = 0.5:
    ```
    person:fernanda  → importance: 0.50 (default)
    person:marcos    → importance: 0.50 (default)
    organization:vertix → importance: 0.50 (default)
    product:xgboost  → importance: 0.50 (default)
    ```

    **After importance scoring:**
    ```
    person:fernanda  → importance: 0.85 (12 facts, recent, high retrieval freq)
    organization:vertix → importance: 0.78 (8 facts, many relationships)
    person:marcos    → importance: 0.55 (4 facts, moderate activity)
    product:xgboost  → importance: 0.25 (1 fact, mentioned once, no relationships)
    ```

    **Impact on retrieval:** When two facts have similar semantic scores, the one about Fernanda (importance=0.85) ranks above the one about XGBoost (importance=0.25). This reflects the reality that Fernanda is a central entity in this user's memory while XGBoost is a peripheral detail.

    **Impact on summary refresh:** Fernanda and Vertix get their summaries refreshed first (higher priority). XGBoost may never get a summary — it's too low priority.

### Job 2: Entity Summary Refresh

```python
from arandu import refresh_entity_summaries, SummaryRefreshResult

result: SummaryRefreshResult = await refresh_entity_summaries(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
    config=memory_config,
)
```

Refreshes stale entity summaries:

- **Stale condition**: `summary_text IS NULL` or last refresh > 7 days ago
- **Priority**: entities with higher `importance_score` refreshed first
- **Limit**: 10 entities per run (prevents timeout)
- Generates 2-3 sentence summaries from the entity's facts using an LLM

### Job 3: Entity Community Detection

```python
from arandu import detect_entity_communities

result = await detect_entity_communities(
    session=db_session,
    agent_id="user_123",
    config=memory_config,
)
```

Connected-component clustering on the entity relationship graph (no LLM calls):

1. Loads active entities and edges (strength >= 0.3)
2. Runs Union-Find (with path compression + union by rank) to find connected components
3. Groups entities into communities (components with >= 2 members)
4. Returns `{"communities_found": int, "total_entities_assigned": int}`

!!! info "Neuroscience parallel"
    Sleep-time compute mirrors **offline processing during sleep**. The brain doesn't just passively store memories during sleep — it actively reorganizes them. Importance scoring is analogous to the brain's process of **synaptic homeostasis** (Tononi & Cirelli), where strongly activated synapses are maintained while weakly activated ones are pruned. Summary refresh mirrors the formation of **gist memories** — compressed representations that capture the essence of detailed episodes.

---

## Scheduling

`arandu` doesn't include a scheduler — you bring your own. All background functions are simple async callables that can be integrated with any scheduling system.

### Example: APScheduler

```python
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from arandu import (
    cluster_user_facts,
    run_consolidation,
    compute_entity_importance,
    refresh_entity_summaries,
)

scheduler = AsyncIOScheduler()

async def maintenance_cycle():
    async with get_session() as session:
        for agent_id in await get_active_users(session):
            await compute_entity_importance(session, agent_id, config=config)
            await refresh_entity_summaries(session, agent_id, llm_provider=llm, config=config)
            await cluster_user_facts(session, agent_id, embeddings, llm, config)
            await run_consolidation(session, agent_id, llm_provider=llm, config=config)

scheduler.add_job(maintenance_cycle, "interval", hours=4)
scheduler.start()
```

### Example: Simple Loop

```python
import asyncio

async def background_loop():
    while True:
        await maintenance_cycle()
        await asyncio.sleep(4 * 3600)  # every 4 hours
```

### Recommended Cadence

| Job | Frequency | Cost |
|-----|-----------|------|
| Entity importance | Every 4h | Cheap (SQL only) |
| Summary refresh | Every 4h | Moderate (LLM, limited to 10/run) |
| Clustering | Every 4-8h | Moderate (LLM for summaries) |
| Consolidation | Every 4-8h | Moderate (LLM for pattern detection) |
| Memify | Daily | Moderate (LLM for distillation) |
| Community detection | Daily | Moderate (LLM + embeddings) |

Run importance scoring first — its output is used by summary refresh to prioritize entities.
