Metadata-Version: 2.4
Name: remdb
Version: 0.2.1
Summary: Resources Entities Moments - Bio-inspired memory system for agentic AI workloads
Project-URL: Homepage, https://github.com/mr-saoirse/remstack
Project-URL: Documentation, https://github.com/mr-saoirse/remstack/blob/main/README.md
Project-URL: Repository, https://github.com/mr-saoirse/remstack
Project-URL: Issues, https://github.com/mr-saoirse/remstack/issues
Author-email: mr-saoirse <amartey@gmail.com>
License: MIT
Keywords: agents,ai,mcp,memory,postgresql,vector-search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: aioboto3>=13.0.0
Requires-Dist: arize-phoenix>=5.0.0
Requires-Dist: asyncpg>=0.30.0
Requires-Dist: boto3>=1.35.0
Requires-Dist: click>=8.1.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: fastmcp>=0.5.0
Requires-Dist: gitpython>=3.1.45
Requires-Dist: gmft
Requires-Dist: hypercorn>=0.17.0
Requires-Dist: itsdangerous>=2.0.0
Requires-Dist: json-schema-to-pydantic>=0.2.0
Requires-Dist: kreuzberg[gmft]>=3.21.0
Requires-Dist: loguru>=0.7.0
Requires-Dist: openinference-instrumentation-pydantic-ai>=0.1.0
Requires-Dist: opentelemetry-api>=1.28.0
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.28.0
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.28.0
Requires-Dist: opentelemetry-exporter-otlp>=1.28.0
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.49b0
Requires-Dist: opentelemetry-instrumentation>=0.49b0
Requires-Dist: opentelemetry-sdk>=1.28.0
Requires-Dist: psycopg[binary]>=3.2.0
Requires-Dist: pydantic-ai>=0.0.14
Requires-Dist: pydantic-settings>=2.6.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: pydub>=0.25.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: requests>=2.32.0
Requires-Dist: semchunk>=2.2.0
Requires-Dist: tenacity>=9.0.0
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: uvicorn[standard]>=0.32.0
Provides-Extra: all
Requires-Dist: ipdb>=0.13.0; extra == 'all'
Requires-Dist: ipython>=8.29.0; extra == 'all'
Requires-Dist: json-schema-to-pydantic>=0.2.0; extra == 'all'
Requires-Dist: mypy>=1.13.0; extra == 'all'
Requires-Dist: pandas-stubs>=2.0.0; extra == 'all'
Requires-Dist: pillow>=10.0.0; extra == 'all'
Requires-Dist: polars>=1.0.0; extra == 'all'
Requires-Dist: pydub>=0.25.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'all'
Requires-Dist: pytest-cov>=6.0.0; extra == 'all'
Requires-Dist: pytest-mock>=3.14.0; extra == 'all'
Requires-Dist: pytest>=8.0.0; extra == 'all'
Requires-Dist: ruff>=0.8.0; extra == 'all'
Requires-Dist: types-pyyaml>=6.0.0; extra == 'all'
Provides-Extra: audio
Requires-Dist: pydub>=0.25.0; extra == 'audio'
Provides-Extra: dev
Requires-Dist: ipdb>=0.13.0; extra == 'dev'
Requires-Dist: ipython>=8.29.0; extra == 'dev'
Requires-Dist: mypy>=1.13.0; extra == 'dev'
Requires-Dist: pandas-stubs>=2.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest-cov>=6.0.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.14.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.8.0; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0.0; extra == 'dev'
Provides-Extra: fs
Requires-Dist: pillow>=10.0.0; extra == 'fs'
Requires-Dist: polars>=1.0.0; extra == 'fs'
Requires-Dist: pydub>=0.25.0; extra == 'fs'
Provides-Extra: schema
Requires-Dist: json-schema-to-pydantic>=0.2.0; extra == 'schema'
Description-Content-Type: text/markdown

# REM - Resources Entities Moments

Cloud-native unified memory infrastructure for agentic AI systems built with Pydantic AI, FastAPI, and FastMCP.

## Architecture Overview

```mermaid
graph TD
    API[FastAPI<br/>Chat + MCP] --> AGENTS[JSON Schema<br/>Agents]
    AGENTS --> TOOLS[MCP Tools<br/>5 Tools]

    TOOLS --> QUERY[REM Query<br/>Dialect]
    QUERY --> DB[(PostgreSQL<br/>+pgvector)]

    FILES[File Processor] --> DREAM[Dreaming<br/>Workers]
    DREAM --> DB

    AGENTS --> OTEL[OpenTelemetry]
    OTEL --> PHOENIX[Arize<br/>Phoenix]

    EVAL[Evaluation<br/>Framework] --> PHOENIX

    classDef api fill:#4A90E2,stroke:#2E5C8A,color:#fff
    classDef agent fill:#7B68EE,stroke:#483D8B,color:#fff
    classDef db fill:#50C878,stroke:#2E7D4E,color:#fff
    classDef obs fill:#9B59B6,stroke:#6C3483,color:#fff

    class API,TOOLS api
    class AGENTS agent
    class DB,QUERY db
    class OTEL,PHOENIX,EVAL obs
```

**Key Components:**

- **API Layer**: OpenAI-compatible chat completions + MCP server (not separate deployments)
- **Agentic Framework**: JSON Schema-based agents with no-code configuration
- **Database Layer**: PostgreSQL 18 with pgvector for multi-index memory (KV + Vector + Graph)
- **REM Query Dialect**: Custom query language with O(1) lookups, semantic search, graph traversal
- **Ingestion & Dreaming**: Background workers for content extraction and progressive index enrichment (0% → 100% answerable)
- **Observability & Evals**: OpenTelemetry tracing + Arize Phoenix + LLM-as-a-Judge evaluation framework

## Features

| Feature | Description | Benefits |
|---------|-------------|----------|
| **OpenAI-Compatible Chat API** | Drop-in replacement for OpenAI chat completions API with streaming support | Use with existing OpenAI clients, switch models across providers (OpenAI, Anthropic, etc.) |
| **Built-in MCP Server** | FastMCP server with 5 tools + 3 resources for memory operations | Export memory to Claude Desktop, Cursor, or any MCP-compatible host |
| **REM Query Engine** | Multi-index query system (LOOKUP, FUZZY, SEARCH, SQL, TRAVERSE) with custom dialect | O(1) lookups, semantic search, graph traversal - all tenant-isolated |
| **Dreaming Workers** | Background workers for entity extraction, moment generation, and affinity matching | Automatic knowledge graph construction from resources (0% → 100% query answerable) |
| **PostgreSQL + pgvector** | CloudNativePG with PostgreSQL 18, pgvector extension, streaming replication | Production-ready vector search, no external vector DB needed |
| **AWS EKS Recipe** | Complete infrastructure-as-code with Pulumi, Karpenter, ArgoCD | Deploy to production EKS in minutes with auto-scaling and GitOps |
| **JSON Schema Agents** | Dynamic agent creation from YAML schemas via Pydantic AI factory | Define agents declaratively, version control schemas, load dynamically |
| **Content Providers** | Audio transcription (Whisper), vision (GPT-4V, Claude), PDFs, DOCX, images | Multimodal ingestion out of the box with format detection |
| **Configurable Embeddings** | Provider-agnostic embedding system (OpenAI, Cohere, Jina) | Switch embedding providers via env vars, no code changes |
| **Multi-Tenancy** | Tenant isolation at database level with automatic scoping | SaaS-ready with complete data separation per tenant |
| **Streaming Everything** | SSE for chat, background workers for embeddings, async throughout | Real-time responses, non-blocking operations, scalable |
| **Zero Vendor Lock-in** | Raw HTTP clients (no OpenAI SDK), swappable providers, open standards | Not tied to any vendor, easy to migrate, full control |

## Quick Start

Choose your path:

- **Option 1: Package Users** (Recommended for non-developers) - PyPI package + dockerized database
- **Option 2: Developers** - Clone repo, local development with uv

---

## Option 1: Package Users (Recommended)

**Best for**: Using REM as a service (API + CLI) without modifying code.

### Step 1: Start Database and API with Docker Compose

```bash
# Create a project directory
mkdir my-rem-project && cd my-rem-project

# Download prebuilt docker-compose file (uses Docker Hub image)
curl -O https://raw.githubusercontent.com/mr-saoirse/remstack/main/rem/docker-compose.prebuilt.yml

# IMPORTANT: Export API keys BEFORE running docker compose
# Docker Compose reads env vars at startup - exporting them after won't work!

# Required: OpenAI for embeddings (text-embedding-3-small)
export OPENAI_API_KEY="sk-..."

# Recommended: At least one chat completion provider
export ANTHROPIC_API_KEY="sk-ant-..."           # Claude Sonnet 4.5 (high quality)
export CEREBRAS_API_KEY="csk-..."               # Cerebras (fast, cheap inference)
export OPENROUTER_API_KEY="sk-or-v1-..."       # OpenRouter (multiple models)

# Start PostgreSQL + API
docker compose -f docker-compose.prebuilt.yml up -d

# Verify services are running
curl http://localhost:8000/health
```

This starts:
- **PostgreSQL** with pgvector on port **5051** (connection: `postgresql://rem:rem@localhost:5051/rem`)
- **REM API** on port **8000** with OpenAI-compatible chat completions + MCP server
- Uses pre-built Docker image from Docker Hub (no local build required)

### Step 2: Install and Configure CLI

```bash
# Install remdb package from PyPI
pip install remdb[all]

# Configure REM (defaults to port 5051 for package users)
rem configure --install --claude-desktop
```

The wizard will:
1. **Configure PostgreSQL**: Defaults to `localhost:5051` (prebuilt docker-compose)
   - Just press Enter to accept defaults for package users
   - Developers using `docker-compose.yml`: Change port to 5050
   - Custom database: Enter your own host/port/credentials
2. **Configure LLM providers**: Enter your OpenAI/Anthropic API keys
3. **Install database tables**: Creates schema, functions, indexes
4. **Register with Claude Desktop**: Adds REM MCP server to Claude

Configuration saved to `~/.rem/config.yaml` (can edit with `rem configure --edit`)

**Port Guide:**
- **5051**: Package users with `docker-compose.prebuilt.yml` (pre-built image)
- **5050**: Developers with `docker-compose.yml` (local build)
- **Custom**: Your own PostgreSQL database

### Step 3: Load Sample Data (Optional)

```bash
# Download sample data
curl -o test-data.yaml https://raw.githubusercontent.com/mr-saoirse/remstack/main/rem/tests/data/seed/test-user-data.yaml

# Load via Docker exec (use rem-api-test for prebuilt compose)
docker exec rem-api-test python3 << 'PYTHON'
import asyncio
import yaml
from pathlib import Path
from rem.services.postgres import get_postgres_service
from rem.services.postgres.repository import Repository
from rem.models.entities import User, Resource, Moment

async def load_data():
    db = get_postgres_service()
    await db.connect()

    data = yaml.safe_load(Path("/app/tests/data/seed/test-user-data.yaml").read_text())

    user_repo = Repository(User, "users", db)
    resource_repo = Repository(Resource, "resources", db)
    moment_repo = Repository(Moment, "moments", db)

    for user_data in data.get("users", []):
        await user_repo.create(User(**user_data))
    for resource_data in data.get("resources", []):
        await resource_repo.create(Resource(**resource_data))
    for moment_data in data.get("moments", []):
        await moment_repo.create(Moment(**moment_data))

    await db.disconnect()
    print("✅ Sample data loaded")

asyncio.run(load_data())
PYTHON
```

### Step 4: Test the Stack

```bash
# Test API
curl -X POST http://localhost:8000/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-User-Id: test-user" \
  -d '{
    "model": "anthropic:claude-sonnet-4-5-20250929",
    "messages": [{"role": "user", "content": "What resources do we have?"}],
    "stream": false
  }'

# Test CLI (if installed locally)
rem ask "What resources do we have?" --user-id test-user

# Or use CLI via Docker (no local install needed)
docker exec rem-api-test rem ask "What resources do we have?" --user-id test-user
```

---

## Option 2: Developers

**Best for**: Contributing to REM or customizing the codebase.

### Step 1: Clone Repository

```bash
git clone https://github.com/mr-saoirse/remstack.git
cd remstack/rem
```

### Step 2: Start PostgreSQL Only

```bash
# Start only PostgreSQL (port 5050 for developers, doesn't conflict with package users on 5051)
docker compose up postgres -d

# Verify connection
psql -h localhost -p 5050 -U rem -d rem -c "SELECT version();"
```

### Step 3: Set Up Development Environment

```bash
# Create virtual environment with uv
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install in editable mode with all dependencies
uv pip install -e ".[all]"

# Set LLM API keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export POSTGRES__CONNECTION_STRING="postgresql://rem:rem@localhost:5050/rem"

# Verify CLI
rem --version
```

### Step 4: Initialize Database

```bash
# Apply migrations
rem db migrate

# Verify tables
psql -h localhost -p 5050 -U rem -d rem -c "\dt"
```

### Step 5: Load Sample Data

```bash
# Load test data
python3 << 'PYTHON'
import asyncio
import yaml
from pathlib import Path
from rem.services.postgres import get_postgres_service
from rem.services.postgres.repository import Repository
from rem.models.entities import User, Resource, Moment

async def load_data():
    db = get_postgres_service()
    await db.connect()

    data = yaml.safe_load(Path("tests/data/seed/test-user-data.yaml").read_text())

    user_repo = Repository(User, "users", db)
    resource_repo = Repository(Resource, "resources", db)
    moment_repo = Repository(Moment, "moments", db)

    for user_data in data.get("users", []):
        await user_repo.create(User(**user_data))
    for resource_data in data.get("resources", []):
        await resource_repo.create(Resource(**resource_data))
    for moment_data in data.get("moments", []):
        await moment_repo.create(Moment(**moment_data))

    await db.disconnect()
    print("✅ Sample data loaded")

asyncio.run(load_data())
PYTHON
```

### Step 6: Run API Server (Optional)

```bash
# Start API server with hot reload
uv run python -m rem.api.main

# API runs on http://localhost:8000
```

### Step 7: Run Tests

```bash
# Run non-LLM tests (fast, no API costs)
uv run pytest tests/integration/ -m "not llm" -v

# Run all tests (uses API credits)
uv run pytest tests/integration/ -v

# Type check
../scripts/run_mypy.sh
```

---

## See Also

- [REM Query Dialect](#rem-query-dialect) - LOOKUP, SEARCH, TRAVERSE, SQL query types
- [API Endpoints](#api-endpoints) - OpenAI-compatible chat completions, MCP server
- [CLI Reference](#cli-reference) - Complete command-line interface documentation
- [Production Deployment](#production-deployment) - AWS EKS with Kubernetes

**Sample Data**: Test data with users, resources, and moments is at `tests/data/seed/test-user-data.yaml`

---

## REM Query Dialect

REM provides a custom query language for flexible memory retrieval with performance guarantees.

### Query Types

#### `LOOKUP` - O(1) Exact Label Lookup

Fast exact match on entity labels (natural language identifiers, not UUIDs).

```sql
LOOKUP "sarah-chen" FROM resources
LOOKUP "api-design-v2" FROM resources WHERE category = "projects"
```

**Performance**: O(1) - indexed on `label` column
**Returns**: Single entity or null
**Use case**: Fetch specific known entities by human-readable name

#### `FUZZY` - Fuzzy Text Search

Fuzzy matching for partial names or misspellings using PostgreSQL trigram similarity.

```sql
FUZZY "sara" FROM resources LIMIT 10
FUZZY "api desgin" FROM resources THRESHOLD 0.3 LIMIT 5
```

**Performance**: O(n) with pg_trgm GIN index (fast for small-medium datasets)
**Returns**: Ranked list by similarity score
**Use case**: Handle typos, partial names, or when exact label is unknown

#### `SEARCH` - Semantic Vector Search

Semantic search using pgvector embeddings with cosine similarity.

```sql
SEARCH "machine learning architecture" FROM resources LIMIT 10
SEARCH "contract disputes" FROM resources WHERE tags @> ARRAY['legal'] LIMIT 5
```

**Performance**: O(log n) with HNSW index
**Returns**: Ranked list of semantically similar entities
**Use case**: Find conceptually related content without exact keyword matches

#### `TRAVERSE` - Recursive Graph Traversal

Follow `graph_edges` relationships across the knowledge graph.

```sql
TRAVERSE FROM "sarah-chen" TYPE "authored_by" DEPTH 2
TRAVERSE FROM "api-design-v2" TYPE "references,depends_on" DEPTH 3
```

**Features**:
- **Polymorphic**: Seamlessly traverses `resources`, `moments`, `users` via `all_graph_edges` view
- **Filtering**: Filter by one or multiple edge types (comma-separated)
- **Depth Control**: Configurable recursion depth (default: 2)
- **Data Model**: Requires `InlineEdge` JSON structure in `graph_edges` column

**Returns**: Graph of connected entities with edge metadata
**Use case**: Explore relationships, find connected entities, build context

#### `SQL` - Direct SQL Queries

Raw SQL for complex temporal, aggregation, or custom queries.

```sql
SQL SELECT * FROM resources WHERE created_at > NOW() - INTERVAL '7 days' ORDER BY created_at DESC LIMIT 20
SQL SELECT category, COUNT(*) as count FROM resources GROUP BY category
```

**Performance**: Depends on query and indexes
**Returns**: Raw query results
**Use case**: Complex filtering, aggregations, temporal queries

### Graph Edge Format

Edges stored inline using `InlineEdge` pattern with human-readable destination labels.

```json
{
  "dst": "sarah-chen",
  "rel_type": "authored_by",
  "weight": 1.0,
  "properties": {
    "dst_entity_type": "users:engineers/sarah-chen",
    "created_at": "2025-01-15T10:30:00Z"
  }
}
```

**Destination Entity Type Convention** (`properties.dst_entity_type`):

Format: `<table_schema>:<category>/<key>`

Examples:
- `"resources:managers/bob"` → Look up bob in resources table with category="managers"
- `"users:engineers/sarah-chen"` → Look up sarah-chen in users table
- `"moments:meetings/standup-2024-01"` → Look up in moments table
- `"resources/api-design-v2"` → Look up in resources table (no category)
- `"bob"` → Defaults to resources table, no category

**Edge Type Format** (`rel_type`):
- Use snake_case: `"authored_by"`, `"depends_on"`, `"references"`
- Be specific but consistent
- Use passive voice for bidirectional clarity

### Multi-Turn Iterated Retrieval

REM enables agents to conduct multi-turn database conversations:

1. **Initial Query**: Agent runs SEARCH to find candidates
2. **Refinement**: Agent analyzes results, runs LOOKUP on specific entities
3. **Context Expansion**: Agent runs TRAVERSE to find related entities
4. **Temporal Filter**: Agent runs SQL to filter by time range
5. **Final Answer**: Agent synthesizes knowledge from all queries

**Plan Memos**: Agents track query plans in scratchpad for iterative refinement.

### Query Performance Contracts

| Query Type | Complexity | Index | Use When |
|------------|-----------|-------|----------|
| `LOOKUP` | O(1) | B-tree on `label` | You know exact entity name |
| `FUZZY` | O(n) | GIN on `label` (pg_trgm) | Handling typos/partial matches |
| `SEARCH` | O(log n) | HNSW on `embedding` | Semantic similarity needed |
| `TRAVERSE` | O(depth × edges) | B-tree on `graph_edges` | Exploring relationships |
| `SQL` | Variable | Custom indexes | Complex filtering/aggregation |

### Example: Multi-Query Session

```python
# Query 1: Find relevant documents
SEARCH "API migration planning" FROM resources LIMIT 5

# Query 2: Get specific document
LOOKUP "tidb-migration-spec" FROM resources

# Query 3: Find related people
TRAVERSE FROM "tidb-migration-spec" TYPE "authored_by,reviewed_by" DEPTH 1

# Query 4: Recent activity
SQL SELECT * FROM moments WHERE
    'tidb-migration' = ANY(topic_tags) AND
    start_time > NOW() - INTERVAL '30 days'
```

### Tenant Isolation

All queries automatically scoped by `user_id` for complete data isolation:

```sql
-- Automatically filtered to user's data
SEARCH "contracts" FROM resources LIMIT 10

-- No cross-user data leakage
TRAVERSE FROM "project-x" TYPE "references" DEPTH 3
```

## Settings

All settings via environment variables with `__` delimiter:

```bash
# LLM
LLM__DEFAULT_MODEL=anthropic:claude-sonnet-4-5-20250929
LLM__DEFAULT_TEMPERATURE=0.5

# Auth (disabled by default)
AUTH__ENABLED=false
AUTH__OIDC_ISSUER_URL=https://accounts.google.com

# OTEL (disabled by default for local dev)
OTEL__ENABLED=false
OTEL__SERVICE_NAME=rem-api

# Postgres
POSTGRES__CONNECTION_STRING=postgresql://rem:rem@localhost:5432/rem

# S3
S3__BUCKET_NAME=rem-storage
S3__REGION=us-east-1
```

## API Endpoints

### Chat Completions (OpenAI-compatible)

```bash
POST /api/v1/chat/completions
```

**Headers**:
- `X-Tenant-Id`: Tenant identifier (required for REM)
- `X-User-Id`: User identifier
- `X-Session-Id`: Session/conversation identifier
- `X-Agent-Schema`: Agent schema URI to use

**Body**:
```json
{
  "model": "anthropic:claude-sonnet-4-5-20250929",
  "messages": [
    {"role": "user", "content": "Find all documents Sarah authored"}
  ],
  "stream": true,
  "response_format": {"type": "text"}
}
```

**Streaming Response** (SSE):
```
data: {"id": "chatcmpl-123", "choices": [{"delta": {"role": "assistant", "content": ""}}]}

data: {"id": "chatcmpl-123", "choices": [{"delta": {"content": "[Calling: search_rem]"}}]}

data: {"id": "chatcmpl-123", "choices": [{"delta": {"content": "Found 3 documents..."}}]}

data: {"id": "chatcmpl-123", "choices": [{"delta": {}, "finish_reason": "stop"}]}

data: [DONE]
```

### MCP Endpoint

```bash
# MCP HTTP transport
POST /api/v1/mcp
```

Tools and resources for REM query execution, resource management, file operations.

### Health Check

```bash
GET /health
# {"status": "healthy", "version": "0.1.0"}
```

## CLI Reference

REM provides a comprehensive command-line interface for all operations.

### Configuration & Server

#### `rem configure` - Interactive Setup Wizard

Set up REM with PostgreSQL, LLM providers, and S3 storage. **Defaults to port 5051 (package users).**

```bash
# Complete setup (recommended for package users)
rem configure --install --claude-desktop

# This runs:
# 1. Interactive wizard (creates ~/.rem/config.yaml)
# 2. Installs database tables (rem db migrate)
# 3. Registers REM MCP server with Claude Desktop

# Other options:
rem configure                  # Just run wizard
rem configure --install        # Wizard + database install
rem configure --show           # Show current configuration
rem configure --edit           # Edit configuration in $EDITOR
```

**Default Configuration:**
- **Package users**: `localhost:5051` (docker-compose.prebuilt.yml with Docker Hub image)
- **Developers**: Change to `localhost:5050` during wizard (docker-compose.yml with local build)
- **Custom database**: Enter your own host/port/credentials

**Configuration File:** `~/.rem/config.yaml`

```yaml
postgres:
  # Package users (prebuilt)
  connection_string: postgresql://rem:rem@localhost:5051/rem
  # OR Developers (local build)
  # connection_string: postgresql://rem:rem@localhost:5050/rem
  pool_min_size: 5
  pool_max_size: 20

llm:
  default_model: anthropic:claude-sonnet-4-5-20250929
  openai_api_key: sk-...
  anthropic_api_key: sk-ant-...

s3:
  bucket_name: rem-storage
  region: us-east-1
```

**Precedence:** Environment variables > Config file > Defaults

**Port Guide:**
- **5051**: Package users with `docker-compose.prebuilt.yml` (recommended)
- **5050**: Developers with `docker-compose.yml` (local development)
- **Custom**: Your own PostgreSQL instance

#### `rem mcp` - Run MCP Server

Run the FastMCP server for Claude Desktop integration.

```bash
# Stdio mode (for Claude Desktop)
rem mcp

# HTTP mode (for testing)
rem mcp --http --port 8001
```

#### `rem serve` - Start API Server

Start the FastAPI server with uvicorn.

```bash
# Use settings from config
rem serve

# Development mode (auto-reload)
rem serve --reload

# Production mode (4 workers)
rem serve --workers 4

# Bind to all interfaces
rem serve --host 0.0.0.0 --port 8080

# Override log level
rem serve --log-level debug
```

### Database Management

#### `rem db migrate` - Run Migrations

Apply database migrations (install.sql and install_models.sql).

```bash
# Apply all migrations
rem db migrate

# Core infrastructure only (extensions, functions)
rem db migrate --install

# Entity tables only (Resource, Message, etc.)
rem db migrate --models

# Background indexes (HNSW for vectors)
rem db migrate --background-indexes

# Custom connection string
rem db migrate --connection "postgresql://user:pass@host:5432/db"

# Custom SQL directory
rem db migrate --sql-dir /path/to/sql
```

#### `rem db status` - Migration Status

Show applied migrations and execution times.

```bash
rem db status
```

#### `rem db rebuild-cache` - Rebuild KV Cache

Rebuild KV_STORE cache from entity tables (after database restart or bulk imports).

```bash
rem db rebuild-cache
```

### Schema Management

#### `rem db schema generate` - Generate SQL Schema

Generate database schema from Pydantic models.

```bash
# Generate install_models.sql from entity models
rem db schema generate \
  --models src/rem/models/entities \
  --output rem/src/rem/sql/install_models.sql

# Generate migration file
rem db schema generate \
  --models src/rem/models/entities \
  --output rem/src/rem/sql/migrations/003_add_fields.sql
```

#### `rem db schema indexes` - Generate Background Indexes

Generate SQL for background index creation (HNSW for vectors).

```bash
# Generate background_indexes.sql
rem db schema indexes \
  --models src/rem/models/entities \
  --output rem/src/rem/sql/background_indexes.sql
```

#### `rem db schema validate` - Validate Models

Validate Pydantic models for schema generation.

```bash
rem db schema validate --models src/rem/models/entities
```

### File Processing

#### `rem process files` - Process Files

Process files with optional custom extractor (ontology extraction).

```bash
# Process all completed files for tenant
rem process files \
  --tenant-id acme-corp \
  --status completed \
  --limit 10

# Process with custom extractor
rem process files \
  --tenant-id acme-corp \
  --extractor cv-parser-v1 \
  --limit 50

# Process files from the last 7 days
rem process files \
  --tenant-id acme-corp \
  --lookback-hours 168
```

#### `rem process uri` - Process Single URI

Process a single file URI and extract content.

```bash
# Process local file
rem process uri \
  --uri file:///path/to/document.pdf \
  --user-id user-123 \
  --tenant-id acme-corp

# Process S3 file
rem process uri \
  --uri s3://bucket/key.docx \
  --user-id user-123 \
  --tenant-id acme-corp
```

### Memory & Knowledge Extraction (Dreaming)

#### `rem dreaming full` - Complete Workflow

Run full dreaming workflow: extractors → moments → affinity → user model.

```bash
# Full workflow for user
rem dreaming full \
  --user-id user-123 \
  --tenant-id acme-corp

# Skip ontology extractors
rem dreaming full \
  --user-id user-123 \
  --tenant-id acme-corp \
  --skip-extractors

# Process last 24 hours only
rem dreaming full \
  --user-id user-123 \
  --tenant-id acme-corp \
  --lookback-hours 24

# Limit resources processed
rem dreaming full \
  --user-id user-123 \
  --tenant-id acme-corp \
  --limit 100
```

#### `rem dreaming custom` - Custom Extractor

Run specific ontology extractor on user's data.

```bash
# Run CV parser on user's files
rem dreaming custom \
  --user-id user-123 \
  --tenant-id acme-corp \
  --extractor cv-parser-v1

# Process last week's files
rem dreaming custom \
  --user-id user-123 \
  --tenant-id acme-corp \
  --extractor contract-analyzer-v1 \
  --lookback-hours 168 \
  --limit 50
```

#### `rem dreaming moments` - Extract Moments

Extract temporal narratives from resources.

```bash
# Generate moments for user
rem dreaming moments \
  --user-id user-123 \
  --tenant-id acme-corp \
  --limit 50

# Process last 7 days
rem dreaming moments \
  --user-id user-123 \
  --tenant-id acme-corp \
  --lookback-hours 168
```

#### `rem dreaming affinity` - Build Relationships

Build semantic relationships between resources using embeddings.

```bash
# Build affinity graph for user
rem dreaming affinity \
  --user-id user-123 \
  --tenant-id acme-corp \
  --limit 100

# Process recent resources only
rem dreaming affinity \
  --user-id user-123 \
  --tenant-id acme-corp \
  --lookback-hours 24
```

#### `rem dreaming user-model` - Update User Model

Update user model from recent activity (preferences, interests, patterns).

```bash
# Update user model
rem dreaming user-model \
  --user-id user-123 \
  --tenant-id acme-corp
```

### Evaluation & Experiments

#### `rem experiments` - Experiment Management

Manage evaluation experiments with datasets, prompts, and traces.

```bash
# Create experiment configuration
rem experiments create my-evaluation \
  --agent ask_rem \
  --evaluator rem-lookup-correctness \
  --description "Baseline evaluation"

# Run experiment
rem experiments run my-evaluation

# List experiments
rem experiments list
rem experiments show my-evaluation
```

#### `rem experiments dataset` - Dataset Management

```bash
# Create dataset from CSV
rem experiments dataset create rem-lookup-golden \
  --from-csv golden.csv \
  --input-keys query \
  --output-keys expected_label,expected_type

# Add more examples
rem experiments dataset add rem-lookup-golden \
  --from-csv more-data.csv \
  --input-keys query \
  --output-keys expected_label,expected_type

# List datasets
rem experiments dataset list
```

#### `rem experiments prompt` - Prompt Management

```bash
# Create agent prompt
rem experiments prompt create hello-world \
  --system-prompt "You are a helpful assistant." \
  --model-name gpt-4o

# List prompts
rem experiments prompt list
```

#### `rem experiments trace` - Trace Retrieval

```bash
# List recent traces
rem experiments trace list --project rem-agents --days 7 --limit 50
```

#### `rem experiments` - Experiment Config

Manage experiment configurations (A/B testing, parameter sweeps).

```bash
# Create experiment config
rem experiments create \
  --name cv-parser-test \
  --description "Test CV parser with different models"

# List experiments
rem experiments list

# Show experiment details
rem experiments show cv-parser-test

# Run experiment
rem experiments run cv-parser-test
```

### Interactive Agent

#### `rem ask` - Test Agent

Test Pydantic AI agent with natural language queries.

```bash
# Ask a question
rem ask "What documents did Sarah Chen author?"

# With context headers
rem ask "Find all resources about API design" \
  --user-id user-123 \
  --tenant-id acme-corp

# Use specific agent schema
rem ask "Analyze this contract" \
  --agent-schema contract-analyzer-v1
```

### Global Options

All commands support:

```bash
# Verbose logging
rem --verbose <command>
rem -v <command>

# Version
rem --version

# Help
rem --help
rem <command> --help
rem <command> <subcommand> --help
```

### Environment Variables

Override any setting via environment variables:

```bash
# Database
export POSTGRES__CONNECTION_STRING=postgresql://rem:rem@localhost:5432/rem
export POSTGRES__POOL_MIN_SIZE=5

# LLM
export LLM__DEFAULT_MODEL=openai:gpt-4o
export LLM__OPENAI_API_KEY=sk-...
export LLM__ANTHROPIC_API_KEY=sk-ant-...

# S3
export S3__BUCKET_NAME=rem-storage
export S3__REGION=us-east-1

# Server
export API__HOST=0.0.0.0
export API__PORT=8000
export API__RELOAD=true

# Run command with overrides
rem serve
```

## Development

### Docker Compose

```bash
# Start all services with hot reload
docker compose up

# Rebuild after dependency changes
docker compose up --build

# View logs
docker compose logs -f api
docker compose logs -f postgres

# Stop all services
docker compose down

# Stop and remove volumes (deletes database data)
docker compose down -v

# Connect to PostgreSQL
psql -h localhost -p 5050 -U rem -d rem
```

### Database Migrations

Migrations are automatically applied on container startup from `src/rem/sql/migrations/*.sql`:
- `001_init_schema.sql` - Core schema with all entity tables and indexes

To add new migrations:
1. Create `src/rem/sql/migrations/002_your_migration.sql`
2. Restart PostgreSQL container: `docker compose restart postgres`

### Test Data

Sample seed data is available in `tests/data/seed/001_sample_data.yaml` but **not** automatically loaded.

The YAML file contains structured test data for all entity types (users, resources, moments, messages, files, schemas).

To load test data, use Python:
```python
import yaml
from pathlib import Path
from rem.models.entities import User, Resource, Moment, Message, File, Schema

# Load YAML
data = yaml.safe_load(Path("tests/data/seed/001_sample_data.yaml").read_text())

# Create entities and insert into database
# See tests/data/seed/README.md for complete example
```

See `tests/data/seed/README.md` for complete loading instructions and pytest fixtures.

### Local Development

```bash
# Install dependencies
uv sync

# Run with auto-reload (requires PostgreSQL running)
uv run python -m rem.api.main

# Run tests
uv run pytest tests/integration/

# Type check (saves report to .mypy/ folder)
../scripts/run_mypy.sh

# Or run mypy directly (no report saved)
uv run mypy src/rem
```

Type checking reports are saved to `.mypy/report_YYYYMMDD_HHMMSS.txt` (gitignored).
Current status: 222 errors in 55 files (as of 2025-11-23).

## Production Deployment

For production deployment to AWS EKS with Kubernetes, see the main repository README:
- **Infrastructure**: [../../manifests/infra/pulumi/eks-yaml/README.md](../../manifests/infra/pulumi/eks-yaml/README.md)
- **Platform**: [../../manifests/platform/README.md](../../manifests/platform/README.md)
- **Application**: [../../manifests/application/README.md](../../manifests/application/README.md)


## REM Query Dialect (AST)

REM queries follow a structured dialect with formal grammar specification.

### Grammar

```
Query ::= LookupQuery | FuzzyQuery | SearchQuery | SqlQuery | TraverseQuery

LookupQuery ::= LOOKUP <key:string|list[string]>
  key         : Single entity name or list of entity names (natural language labels)
  performance : O(1) per key
  available   : Stage 1+
  examples    :
    - LOOKUP "Sarah"
    - LOOKUP ["Sarah", "Mike", "Emily"]
    - LOOKUP "Project Alpha"

FuzzyQuery ::= FUZZY <text:string> [THRESHOLD <t:float>] [LIMIT <n:int>]
  text        : Search text (partial/misspelled)
  threshold   : Similarity score 0.0-1.0 (default: 0.5)
  limit       : Max results (default: 5)
  performance : Indexed (pg_trgm)
  available   : Stage 1+
  example     : FUZZY "sara" THRESHOLD 0.5 LIMIT 10

SearchQuery ::= SEARCH <text:string> [TABLE <table:string>] [WHERE <clause:string>] [LIMIT <n:int>]
  text        : Semantic query text
  table       : Target table (default: "resources")
  clause      : Optional PostgreSQL WHERE clause for hybrid filtering (combines vector + structured)
  limit       : Max results (default: 10)
  performance : Indexed (pgvector)
  available   : Stage 3+
  examples    :
    - SEARCH "database migration" TABLE resources LIMIT 10
    - SEARCH "team discussion" TABLE moments WHERE "moment_type='meeting'" LIMIT 5
    - SEARCH "project updates" WHERE "created_at >= '2024-01-01'" LIMIT 20
    - SEARCH "AI research" WHERE "tags @> ARRAY['machine-learning']" LIMIT 10

  Hybrid Query Support: SEARCH combines semantic vector similarity with structured filtering.
  Use WHERE clause to filter on system fields or entity-specific fields.

SqlQuery ::= SQL <table:string> [WHERE <clause:string>] [ORDER BY <order:string>] [LIMIT <n:int>]
  table       : Table name ("resources", "moments", etc.)
  clause      : PostgreSQL WHERE conditions (any valid PostgreSQL syntax)
  order       : ORDER BY clause
  limit       : Max results
  performance : O(n) with indexes
  available   : Stage 1+
  dialect     : PostgreSQL (supports all PostgreSQL features: JSONB operators, array operators, etc.)
  examples    :
    - SQL moments WHERE "moment_type='meeting'" ORDER BY starts_timestamp DESC LIMIT 10
    - SQL resources WHERE "metadata->>'status' = 'published'" LIMIT 20
    - SQL moments WHERE "tags && ARRAY['urgent', 'bug']" ORDER BY created_at DESC

  PostgreSQL Dialect: SQL queries use PostgreSQL syntax with full support for:
  - JSONB operators (->>, ->, @>, etc.)
  - Array operators (&&, @>, <@, etc.)
  - Advanced filtering and aggregations

TraverseQuery ::= TRAVERSE [<edge_types:list>] WITH <initial_query:Query> [DEPTH <d:int>] [ORDER BY <order:string>] [LIMIT <n:int>]
  edge_types    : Relationship types to follow (e.g., ["manages", "reports-to"], default: all)
  initial_query : Starting query (typically LOOKUP)
  depth         : Number of hops (0=PLAN mode, 1=single hop, N=multi-hop, default: 1)
  order         : Order results (default: "edge.created_at DESC")
  limit         : Max nodes (default: 9)
  performance   : O(k) where k = visited nodes
  available     : Stage 3+
  examples      :
    - TRAVERSE manages WITH LOOKUP "Sally" DEPTH 1
    - TRAVERSE WITH LOOKUP "Sally" DEPTH 0  (PLAN mode: edge analysis only)
    - TRAVERSE manages,reports-to WITH LOOKUP "Sarah" DEPTH 2 LIMIT 5
```

### Query Availability by Evolution Stage

| Query Type | Stage 0 | Stage 1 | Stage 2 | Stage 3 | Stage 4 |
|------------|---------|---------|---------|---------|---------|
| LOOKUP     | ✗       | ✓       | ✓       | ✓       | ✓       |
| FUZZY      | ✗       | ✓       | ✓       | ✓       | ✓       |
| SEARCH     | ✗       | ✗       | ✗       | ✓       | ✓       |
| SQL        | ✗       | ✓       | ✓       | ✓       | ✓       |
| TRAVERSE   | ✗       | ✗       | ✗       | ✓       | ✓       |

**Stage 0**: No data, all queries fail.

**Stage 1** (20% answerable): Resources seeded with entity extraction. LOOKUP and FUZZY work for finding entities. SQL works for basic filtering.

**Stage 2** (50% answerable): Moments extracted. SQL temporal queries work. LOOKUP includes moment entities.

**Stage 3** (80% answerable): Affinity graph built. SEARCH and TRAVERSE become available. Multi-hop graph queries work.

**Stage 4** (100% answerable): Mature graph with rich historical data. All query types fully functional with high-quality results.

## License

MIT
