# Dreaming Services

REM memory indexing and insight extraction services.

## Overview

The dreaming services module provides modular, composable services for building the REM knowledge graph through:

- **User Model Updates** (`user_model_service.py`): Extract and update user profiles from activity
- **Moment Construction** (`moment_service.py`): Identify temporal narratives from resources
- **Resource Affinity** (`affinity_service.py`): Build semantic relationships between resources
- **Ontology Extraction** (`ontology_service.py`): Extract domain-specific structured knowledge from files (stub)

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    DreamingWorker                           │
│                    (Orchestrator)                           │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐  │
│  │ User Model    │  │    Moment     │  │   Resource    │  │
│  │   Service     │  │   Service     │  │   Affinity    │  │
│  └───────┬───────┘  └───────┬───────┘  └───────┬───────┘  │
│          │                  │                  │          │
│          └──────────────────┼──────────────────┘          │
│                            │                              │
│                    ┌───────▼───────┐                      │
│                    │  PostgreSQL   │                      │
│                    │  Repository   │                      │
│                    │  REM Service  │                      │
│                    └───────────────┘                      │
└─────────────────────────────────────────────────────────────┘
```

## Services

### User Model Service

**File**: `user_model_service.py`

**Function**: `update_user_model(user_id, db, default_model, time_window_days, max_sessions, max_moments, max_resources)`

**Purpose**: Analyzes recent sessions, moments, and resources to generate comprehensive user profile summaries using LLM analysis.

**Process**:
1. Query PostgreSQL for recent sessions, moments, resources for this user
2. Load UserProfileBuilder agent schema
3. Generate user profile using LLM
4. Update User entity with profile data and metadata
5. Add graph edges to key resources and moments

**Returns**: Statistics about user model update (sessions analyzed, moments included, resources included, graph edges added, etc.)

### Moment Service

**File**: `moment_service.py`

**Function**: `construct_moments(user_id, db, default_model, lookback_hours, limit)`

**Purpose**: Analyzes recent resources and sessions to identify temporal narratives (meetings, coding sessions, conversations) and creates Moment entities.

**Process**:
1. Query PostgreSQL for recent resources and sessions for this user
2. Load MomentBuilder agent schema from filesystem
3. Run agent to extract moments from data
4. Create Moment entities via Repository
5. Link moments to source resources via graph edges
6. Embeddings auto-generated by embedding worker

**Returns**: Statistics about moment construction (resources queried, sessions queried, moments created, graph edges added, analysis summary)

### Affinity Service

**File**: `affinity_service.py`

**Function**: `build_affinity(user_id, db, mode, default_model, lookback_hours, limit, similarity_threshold, top_k)`

**Purpose**: Creates semantic relationships between resources using either vector similarity (fast) or LLM analysis (intelligent).

**Modes**:
- **SEMANTIC**: Fast vector similarity search via REM SEARCH query (cheap, fast)
- **LLM**: Intelligent LLM-based relationship assessment (expensive, slow)

**Process**:
1. Query PostgreSQL for recent resources for this user
2. For each resource:
   - Semantic: Query similar resources by vector using REM SEARCH
   - LLM: Assess relationships using ResourceAffinityAssessor agent
3. Create graph edges with deduplication (keep highest weight)
4. Update resource entities with affinity edges

**Returns**: Statistics about affinity construction (resources processed, edges created, LLM calls made)

### Ontology Service

**File**: `ontology_service.py`

**Function**: `extract_ontologies(user_id, lookback_hours, limit)`

**Purpose**: Extract domain-specific knowledge from files using custom agents (stub - not yet implemented).

**Returns**: Statistics about ontology extraction (files queried, configs matched, ontologies created, embeddings generated, agent calls made)

## Utilities

**File**: `utils.py`

**Function**: `merge_graph_edges(existing_edges, new_edges)`

**Purpose**: Merge graph edges with deduplication. Keeps highest weight edge for each (dst, rel_type) pair.

## Usage

### Standalone Service Usage

```python
from rem.services.dreaming import (
    update_user_model,
    construct_moments,
    build_affinity,
    extract_ontologies,
    AffinityMode,
)
from rem.services.postgres import get_postgres_service

# Initialize database connection
db = get_postgres_service()
await db.connect()

try:
    # Update user model from recent activity
    result = await update_user_model(
        user_id="user-123",
        db=db,
        default_model="gpt-4o",
        time_window_days=30,
    )
    print(f"User model updated: {result}")

    # Extract moments from resources
    result = await construct_moments(
        user_id="user-123",
        db=db,
        default_model="gpt-4o",
        lookback_hours=24,
    )
    print(f"Moments created: {result['moments_created']}")

    # Build resource affinity (semantic mode)
    result = await build_affinity(
        user_id="user-123",
        db=db,
        mode=AffinityMode.SEMANTIC,
        default_model="gpt-4o",
        lookback_hours=168,
    )
    print(f"Edges created: {result['edges_created']}")

finally:
    await db.disconnect()
```

### Orchestrated Usage via DreamingWorker

```python
from rem.workers.dreaming import DreamingWorker

worker = DreamingWorker(
    rem_api_url="http://rem-api:8000",
    default_model="gpt-4o",
    lookback_hours=24,
)

try:
    # Run complete dreaming workflow
    results = await worker.process_full(
        user_id="user-123",
        use_llm_affinity=False,
        lookback_hours=24,
    )
    print(results)
finally:
    await worker.close()
```

## Design Principles

1. **Modularity**: Each service is independent and can be used standalone
2. **Composability**: Services can be composed together in custom workflows
3. **DRY**: Shared utilities extracted to utils.py
4. **Delegation**: DreamingWorker delegates to services, acting as thin orchestrator
5. **Database Connection Management**: Caller manages database connection lifecycle
6. **Error Handling**: Services return statistics dicts with status field
7. **User-ID First**: All operations scoped by user_id (primary identifier)

## File Structure

```
rem/src/rem/services/dreaming/
├── __init__.py                # Public API facade
├── README.md                  # This file
├── user_model_service.py      # User profile updates (260 lines)
├── moment_service.py          # Temporal narrative extraction (260 lines)
├── affinity_service.py        # Resource relationship building (320 lines)
├── ontology_service.py        # Domain knowledge extraction (45 lines, stub)
└── utils.py                   # Shared utilities (graph edge merging)
```

## Refactoring Benefits

**Before**: Single 1,297-line monolithic `workers/dreaming.py` file

**After**:
- 5 focused service modules (~900 lines total)
- 1 thin orchestrator (~400 lines)
- Improved testability (each service can be tested independently)
- Better separation of concerns
- Easier to extend (add new services without modifying orchestrator)
- Reusable services (can be used in other workflows)

## Future Enhancements

1. **Implement Ontology Service**: Complete the stub implementation
2. **Add Service-Level Caching**: Cache agent schemas and LLM responses
3. **Batch Operations**: Optimize database operations with batching
4. **Parallelization**: Run independent services concurrently
5. **Metrics and Tracing**: Add OpenTelemetry instrumentation
6. **Service Configuration**: Extract hardcoded values to configuration
