# Background Jobs

Background jobs maintain and optimize memory over time — clustering facts, consolidating patterns, scoring importance, and refreshing summaries. They run periodically (not during message handling) and improve retrieval quality without impacting response latency.

```mermaid
flowchart LR
    A["Scheduler\n(periodic)"] --> B["Clustering"]
    A --> C["Consolidation"]
    A --> D["Importance\nScoring"]
    A --> E["Summary\nRefresh"]
```

## Overview

`arandu` provides four categories of background jobs:

| Job | Purpose | Uses LLM? | Frequency |
|-----|---------|-----------|-----------|
| **Clustering** | Group related facts semantically | Yes (summaries) | Every 4-8 hours |
| **Consolidation** | Detect patterns, contradictions, trends | Yes | Every 4-8 hours |
| **Memify** | Convert episodic facts to procedural/semantic knowledge | Yes | Daily |
| **Sleep-time compute** | Score importance, refresh summaries, detect communities | Partially | Every 4-8 hours |

All jobs are exposed as async functions you can call directly or schedule with your preferred task runner (APScheduler, Celery, cron, etc.).

!!! info "Neuroscience parallel"
    Background jobs mirror **sleep-time processing** in the brain. During sleep, the brain consolidates memories, transfers information from hippocampus (short-term) to neocortex (long-term), prunes irrelevant connections, and strengthens important ones. These jobs perform the same operations on your agent's memory.

---

## Clustering

Clustering groups related facts into semantic clusters, making retrieval more contextual and enabling community detection.

### Fact Clustering

```python
from arandu import cluster_user_facts, ClusteringResult

result: ClusteringResult = await cluster_user_facts(
    session=db_session,
    user_id="user_123",
    llm=llm_provider,
    embeddings=embedding_provider,
    config=memory_config,
)
```

**How it works:**

1. Groups facts by `(entity_type, entity_key)` — facts about the same entity stay together
2. Generates a 2-3 sentence summary per cluster using an LLM
3. Computes and stores cluster embeddings for later community detection
4. Idempotent — updates existing clusters rather than creating duplicates

### Community Detection

```python
from arandu import detect_communities, CommunityDetectionResult

result: CommunityDetectionResult = await detect_communities(
    session=db_session,
    user_id="user_123",
    llm=llm_provider,
    embeddings=embedding_provider,
    config=memory_config,
)
```

**How it works:**

1. Compares cluster embeddings using cosine similarity
2. Groups clusters above `community_similarity_threshold` (default 0.75)
3. Creates `MemoryMetaObservation` records with type `"entity_community"`
4. Example: a "work" community might include clusters about colleagues, projects, and company facts

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `cluster_max_age_days` | `90` | Maximum age of facts to include in clustering |
| `community_similarity_threshold` | `0.75` | Cosine similarity threshold for grouping clusters |

---

## Consolidation

Consolidation analyzes recent events and facts to detect higher-order patterns — insights, contradictions, trends, and behavioral preferences.

### Periodic Consolidation (L2)

```python
from arandu import run_consolidation, ConsolidationResult

result: ConsolidationResult = await run_consolidation(
    session=db_session,
    user_id="user_123",
    llm=llm_provider,
    config=memory_config,
)
```

**How it works:**

1. Analyzes events and facts over a lookback window (`consolidation_lookback_days`)
2. Detects patterns across facts:
   - **Insights** — Emergent understanding from multiple facts
   - **Patterns** — Repeated behaviors or preferences
   - **Contradictions** — Conflicting facts that need resolution
   - **Trends** — Changes over time
3. Generates `MemoryMetaObservation` records
4. Tags events with emotions (emotion, intensity, energy level)

### Profile Consolidation (L3)

```python
from arandu import run_profile_consolidation

await run_profile_consolidation(
    session=db_session,
    user_id="user_123",
    llm=llm_provider,
    config=memory_config,
)
```

**How it works:**

1. Refreshes entity summaries via LLM — a higher-level view of each entity
2. Updates the overall profile overview
3. Triggered periodically (less frequently than L2)

### Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `consolidation_min_events` | `3` | Minimum events before running consolidation |
| `consolidation_lookback_days` | `7` | How far back to look for patterns |

!!! info "Neuroscience parallel"
    Consolidation mirrors the brain's **memory consolidation during sleep**. The hippocampus replays recent experiences, the neocortex detects patterns and integrates them into existing knowledge structures, and contradictions are flagged for resolution. L2 consolidation is analogous to slow-wave sleep (SWS) replay, while L3 profile consolidation is analogous to REM sleep's role in integrating memories into semantic knowledge.

---

## Memify

Memify converts episodic facts (specific events) into procedural and semantic knowledge (general knowledge). It also manages fact vitality — how "alive" a fact is based on recent mentions.

### Run Memify

```python
from arandu import run_memify, MemifyResult

result: MemifyResult = await run_memify(
    session=db_session,
    user_id="user_123",
    llm=llm_provider,
    config=memory_config,
)
```

**How it works:**

1. Groups related facts by entity and topic
2. Generates distilled summaries (procedural/semantic knowledge)
3. Checks vitality — facts mentioned recently are kept; stale facts may be deprecated
4. Merges similar procedures to prevent knowledge fragmentation

### Vitality Scoring

```python
from arandu import compute_vitality

vitality_scores = await compute_vitality(
    session=db_session,
    user_id="user_123",
    config=memory_config,
)
```

Vitality measures how "alive" a fact is based on:

- **Recency** — When was the fact last confirmed or mentioned?
- **Reinforcement** — How many times has this fact been confirmed (NOOP decisions)?
- **Importance** — How relevant is this fact to the user's profile?

!!! info "Neuroscience parallel"
    Memify mirrors the **forgetting curve** described by Hermann Ebbinghaus (1885). Memories decay exponentially over time unless reinforced through retrieval practice. Facts with high vitality (frequently accessed) resist decay, while low-vitality facts gradually fade — just as the brain prunes synaptic connections for unused information.

---

## Sleep-Time Compute

Sleep-time compute runs three targeted maintenance jobs that improve retrieval quality:

### Job 1: Entity Importance Scoring

```python
from arandu import compute_entity_importance, EntityImportanceResult

result: EntityImportanceResult = await compute_entity_importance(
    session=db_session,
    user_id="user_123",
    config=memory_config,
)
```

Pure SQL computation (no LLM calls). Scores each entity from 0.0 to 1.0 using four normalized signals:

| Signal | Weight | Description |
|--------|--------|-------------|
| Fact density | 0.30 | Number of facts relative to other entities |
| Recency | 0.25 | Exponential decay (30-day half-life) |
| Retrieval frequency | 0.25 | How often facts about this entity are retrieved |
| Relationship degree | 0.20 | Number of incoming + outgoing relationships |

The importance score is used as a signal in retrieval scoring and as a priority factor for summary refresh.

### Job 2: Entity Summary Refresh

```python
from arandu import refresh_entity_summaries, SummaryRefreshResult

result: SummaryRefreshResult = await refresh_entity_summaries(
    session=db_session,
    user_id="user_123",
    llm=llm_provider,
    config=memory_config,
)
```

Refreshes stale entity summaries:

- **Stale condition**: `summary_text IS NULL` or last refresh > 7 days ago
- **Priority**: entities with higher `importance_score` refreshed first
- **Limit**: 10 entities per run (prevents timeout)
- Generates 2-3 sentence summaries from the entity's facts using an LLM

### Job 3: Entity Community Detection

```python
from arandu import detect_entity_communities

result = await detect_entity_communities(
    session=db_session,
    user_id="user_123",
    llm=llm_provider,
    embeddings=embedding_provider,
    config=memory_config,
)
```

Finds groups of related entities using the relationship graph:

1. Loads active entities and edges (strength ≥ 0.3)
2. Runs Union-Find (with path compression + union by rank) to find connected components
3. Filters by minimum entity threshold
4. Generates LLM summary and embedding for each community
5. Deduplicates against existing communities (Jaccard member overlap)
6. Stores as `MemoryMetaObservation` records

!!! info "Neuroscience parallel"
    Sleep-time compute mirrors **offline processing during sleep**. The brain doesn't just passively store memories during sleep — it actively reorganizes them. Importance scoring is analogous to the brain's process of **synaptic homeostasis** (Tononi & Cirelli), where strongly activated synapses are maintained while weakly activated ones are pruned. Summary refresh mirrors the formation of **gist memories** — compressed representations that capture the essence of detailed episodes.

---

## Scheduling

`arandu` doesn't include a scheduler — you bring your own. All background functions are simple async callables that can be integrated with any scheduling system.

### Example: APScheduler

```python
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from arandu import (
    cluster_user_facts,
    run_consolidation,
    compute_entity_importance,
    refresh_entity_summaries,
)

scheduler = AsyncIOScheduler()

async def maintenance_cycle():
    async with get_session() as session:
        for user_id in await get_active_users(session):
            await compute_entity_importance(session, user_id, config)
            await refresh_entity_summaries(session, user_id, llm, config)
            await cluster_user_facts(session, user_id, llm, embeddings, config)
            await run_consolidation(session, user_id, llm, config)

scheduler.add_job(maintenance_cycle, "interval", hours=4)
scheduler.start()
```

### Example: Simple Loop

```python
import asyncio

async def background_loop():
    while True:
        await maintenance_cycle()
        await asyncio.sleep(4 * 3600)  # every 4 hours
```

### Recommended Cadence

| Job | Frequency | Cost |
|-----|-----------|------|
| Entity importance | Every 4h | Cheap (SQL only) |
| Summary refresh | Every 4h | Moderate (LLM, limited to 10/run) |
| Clustering | Every 4-8h | Moderate (LLM for summaries) |
| Consolidation | Every 4-8h | Moderate (LLM for pattern detection) |
| Memify | Daily | Moderate (LLM for distillation) |
| Community detection | Daily | Moderate (LLM + embeddings) |

Run importance scoring first — its output is used by summary refresh to prioritize entities.
