 HIEROMEM 🧠

**Hierarchical Episodic Memory System** - A memory framework for AI agents with graph-based semantic storage and multi-tenant support.

[![v1.0-alpha](https://img.shields.io/badge/Status-v1.0--alpha%3A%20Stabilization-blue?style=for-the-badge)](https://github.com/Rishikiran98/HIEROMEM/releases)
[![CI/CD](https://github.com/Rishikiran98/HIEROMEM/actions/workflows/ci.yml/badge.svg)](https://github.com/Rishikiran98/HIEROMEM/actions)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Compliance](https://img.shields.io/badge/Compliance-GDPR%20%7C%20CCPA%20%7C%20SOC2-informational.svg)](docs/compliance)

## 🚀 Features

### Core Memory System
- **Hierarchical Episodes**: Organize memories in parent-child relationships with automatic summarization
- **Graph-Based Storage**: Neo4j-powered semantic graph for complex relationship queries
- **Vector Search**: FAISS-accelerated similarity search with sentence transformers
- **Multi-Tenant**: Isolated namespaces with per-tenant rate limiting and quotas

### Production-Ready Infrastructure
- **🔒 Security**: TLS/HTTPS via Nginx, API key management with expiration/revocation
- **💾 Data Persistence**: Automated daily backups (SQLite, Neo4j, FAISS) with disaster recovery
- **📊 Observability**: Prometheus metrics, Loki logging, Grafana dashboards
- **⚡ Scalability**: Redis-backed distributed rate limiting, resource limits, health checks
- **🐳 Containerized**: Full Docker Compose stack with health checks and graceful shutdown

## 🚀 HIEROMEM Engine v1.0 (Stabilization)

> [!IMPORTANT]
> **Active Development**: The v1.0 engine is in stabilization phase. Test suite and documentation are being aligned with implementation.

This version contains:

- ✅ **Episode Management** – Hierarchical memory organization with parent-child relationships
- ✅ **Dual Retrieval** – Hybrid vector + symbolic graph search for comprehensive recall
- ✅ **Trust Decay + Forgetting** – Automatic memory aging and pruning based on access patterns
- ✅ **Graph Merge + Relation Extraction** – Semantic relationship discovery and consolidation
- ✅ **Encoder Selection** – Support for HuggingFace, Sentence Transformers, CLIP, Wav2Vec2
- ✅ **Governance Safety Region** – Configurable λ (trust threshold), τ (decay rate), θ (similarity), forbidden relations
- ✅ **Narrative Summarization** – Automatic episode summarization for hierarchical compression
- ✅ **Event Log + Metrics** – Prometheus instrumentation and structured logging
- ✅ **Multi-Tenant Namespaces** – Isolated memory spaces with per-tenant quotas and governance
- ✅ **OAuth2/OIDC Authentication** – Enterprise SSO support (Auth0, Okta, Google)
- ✅ **Role-Based Access Control** – Fine-grained permissions for memory operations
- ✅ **Cross-Realm Isolation** – Secure memory boundaries between agent realms with validation
- ✅ **AWS Neptune Backend** – Production-scale graph storage with Gremlin query support
- ✅ **LLM Client Abstraction** – Unified interface for OpenAI and local HuggingFace models
- ✅ **Benchmark CI Integration** – Automated retrieval quality, temporal QA, and latency benchmarks

**This is the stable API surface for Cloud deployment.** Additional features will be added in v1.1+ after Cloud launch.

## 📋 Table of Contents

- [Quick Start](#-quick-start)
- [Developer Guide](docs/DEVELOPER_GUIDE.md) - **Complete integration guide**
- [Architecture](#-architecture)
- [API Documentation](#-api-documentation)
- [Configuration](#-configuration)
- [Deployment](#-deployment)
- [Monitoring](#-monitoring)
- [Development](#-development)
- [Contributing](#-contributing)
- [SDK Quickstarts](#-sdk-quickstarts)
- [Compliance](#-compliance)

## 🎯 Quick Start

### Option A: Zero-Config (No Docker)

Get started in 3 commands - no Docker, Neo4j, or Redis required:

```bash
pip install hieromem
cd examples
python quickstart.py
```

```python
from hieromem_core.memory import Memory
from hieromem_core.encoder import Encoder, EncoderConfig

# Initialize
encoder = Encoder(EncoderConfig())
memory = Memory(Nmax=100, encoder=encoder)

# Store
vector = encoder.encode_single("Alice works at Acme Corp")
memory.insert(vector=vector, meta={"text": "Alice works at Acme Corp"})

# Retrieve (vector + graph)
results = memory.dual_retrieve(query_text="Who works at Acme?", k=3)
print(results.results[0].text)  # "Alice works at Acme Corp"
```

See [`examples/quickstart.py`](examples/quickstart.py) for a complete working example.

### Option B: Full Stack (Docker)

For production deployments with Neo4j, Redis, and monitoring:

#### Prerequisites

- Docker & Docker Compose
- Python 3.11+ (for local development)
- 8GB RAM minimum

#### 1. Clone the Repository

```bash
git clone https://github.com/Rishikiran98/HIEROMEM.git
cd HIEROMEM
```

#### 2. Configure Environment

```bash
cp .env.example .env
# Edit .env with your configuration
```

#### 3. Start the Stack

```bash
docker compose up -d
```

### 4. Access Services

- **API**: http://localhost:8000
- **Visualizer**: http://localhost:5173
- **Grafana**: http://localhost:3000 (admin/admin)
- **Prometheus**: http://localhost:9090
- **Neo4j Browser**: http://localhost:7474

### 5. Create an API Key

```bash
docker compose exec hieromem-api python -m hieromem_api.key_management create my-tenant --days 90
```

### 6. Test the API

```bash
curl -X POST http://localhost:8000/memory/remember \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "default",
    "event_type": "user_action",
    "content": "User completed onboarding tutorial",
    "metadata": {"user_id": "123", "duration_seconds": 45}
  }'
```

## 📦 SDK Quickstarts

### Python

```bash
pip install sdks/python
```

```python
from hieromem_client import HieromemClient

client = HieromemClient(base_url="http://localhost:8000", api_key="YOUR_KEY")
client.remember("hello world", namespace="demo")
print(client.recall("hello"))
```

### JavaScript

```bash
npm install ./sdks/javascript
```

```javascript
import { HieromemClient } from "@hieromem/sdk";

const client = new HieromemClient({ baseUrl: "http://localhost:8000", apiKey: "YOUR_KEY" });
await client.remember("hello world", { namespace: "demo" });
console.log(await client.recall("hello"));
```

### Go

```bash
go get github.com/Rishikiran98/hieromem/sdks/go
```

```go
import (
        "context"
        "fmt"

        hieromem "github.com/Rishikiran98/hieromem/sdks/go"
)

client := hieromem.NewClient("http://localhost:8000", "YOUR_KEY")
resp, _ := client.Recall(context.Background(), hieromem.RecallRequest{Query: "hello"})
fmt.Println(resp)
```

## 🤖 Agent SDK

The HIEROMEM Agent SDK provides high-level abstractions for building memory-aware AI agents.

### Installation

```bash
pip install -e ./packages/hieromem_agent
```

### LLM Client

The `LLMClient` supports both OpenAI API and local HuggingFace models:

```python
from hieromem_agent import LLMClient, LLMConfig

# OpenAI (default)
client = LLMClient(LLMConfig(model="gpt-4o-mini"))
response = client.complete("What is 2+2?")

# Local HuggingFace model
import os
os.environ["HIEROMEM_LLM_PROVIDER"] = "local_huggingface"
client = LLMClient(LLMConfig(model="Qwen/Qwen2.5-1.5B-Instruct"))
response = client.complete("What is 2+2?")
```

### Memory Agent

```python
from hieromem_agent import MemoryAgent
from hieromem_core.memory import Memory

memory = Memory(Nmax=100, encoder=encoder)
agent = MemoryAgent(memory=memory, llm_client=client)

# Agent with persistent memory
response = agent.chat("Remember that my favorite color is blue")
response = agent.chat("What is my favorite color?")  # Uses memory retrieval
```

## 🔐 Cross-Realm Isolation (Multi-Agent)

HIEROMEM supports secure memory isolation between agents with cross-realm access controls:

```python
from hieromem_core.realms import (
    MemoryRealm,
    check_permission,
    validate_cross_realm_access,
    get_accessible_realms,
    CrossRealmAccessError
)

# Create isolated realms
private_realm = MemoryRealm(name="agent-private", type="private")
shared_realm = MemoryRealm(name="team-shared", type="shared")

# Check permissions before access
accessible = get_accessible_realms(realms, agent_id="agent-1", required="read")

# Validate cross-realm references
if validate_cross_realm_access(source_realm, target_realm, agent_id):
    # Safe to create cross-realm reference
    pass
```

## 📊 Benchmarks

HIEROMEM includes automated benchmarks for measuring system performance:

| Benchmark | Description | Metrics |
|-----------|-------------|---------|
| **Retrieval Quality** | Measures recall@k, precision, MRR | `benchmarks/retrieval_quality.py` |
| **Temporal QA** | Tests temporal reasoning with memory conflicts | `benchmarks/temporal_qa.py` |
| **Latency** | Measures insert/retrieve latency at scale | `benchmarks/latency_benchmark.py` |

### Running Benchmarks

```bash
cd benchmarks
python retrieval_quality.py --output results/retrieval.json
python temporal_qa.py --output results/temporal.json
python latency_benchmark.py --output results/latency.json
```

Benchmarks run automatically on push to `main` via GitHub Actions.

## 🏢 Enterprise Features

Enterprise features are disabled by default. Enable them with:

```bash
export HIEROMEM_ENTERPRISE=1
```

| Feature | Description | Requires |
|---------|-------------|----------|
| **Billing** | Stripe integration for usage-based billing | `pip install hieromem[saas]` |
| **Dashboard** | Tenant management and analytics UI | `HIEROMEM_ENTERPRISE=1` |
| **OAuth2/OIDC** | Enterprise SSO (Auth0, Okta, Google) | `OIDC_ENABLED=true` |
| **Advanced Multi-Tenancy** | Per-tenant quotas and rate limits | `HIEROMEM_ENTERPRISE=1` |

See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full feature classification.

## 🏗️ Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    Nginx (TLS/HTTPS)                    │
│                    :80, :443                            │
└────────────────┬────────────────────────────────────────┘
                 │
        ┌────────┴────────┐
        │                 │
┌───────▼────────┐  ┌────▼──────────┐
│  API :8000     │  │  Visualizer   │
│  (FastAPI)     │  │  (React)      │
└───────┬────────┘  └───────────────┘
        │
        ├──────────┬──────────┬──────────┬──────────┐
        │          │          │          │          │
┌───────▼──────┐ ┌─▼──────┐ ┌─▼──────┐ ┌─▼──────┐ ┌─▼──────────┐
│ Neo4j :7687  │ │ Redis  │ │ SQLite │ │ FAISS  │ │ Prometheus │
│ (Graph DB)   │ │ (Cache)│ │ (Meta) │ │(Vector)│ │ (Metrics)  │
└──────────────┘ └────────┘ └────────┘ └────────┘ └─────┬──────┘
                                                          │
                                                    ┌─────▼─────┐
                                                    │  Grafana  │
                                                    └───────────┘
```

### Components

| Component | Purpose | Technology |
|-----------|---------|------------|
| **API** | REST API for memory operations | FastAPI, Python 3.11 |
| **Visualizer** | Web UI for exploring memories | React, TypeScript, Vite |
| **Neo4j** | Graph database for semantic relationships | Neo4j 5.15 |
| **Redis** | Distributed rate limiting & caching | Redis 7.2 |
| **SQLite** | Metadata and episode storage | SQLite (WAL mode) |
| **FAISS** | Vector similarity search | Facebook AI Similarity Search |
| **Prometheus** | Metrics collection and alerting | Prometheus + Alertmanager |
| **Loki** | Centralized log aggregation | Grafana Loki |
| **Grafana** | Visualization dashboards | Grafana |

## 🔗 Graph Capabilities

HIEROMEM supports **graph-based semantic storage** with multiple backend options for development, testing, and production environments.

### Graph Backends

| Backend | Use Case | Dependencies |
|---------|----------|--------------|
| `in_memory_graph` | Development, Testing, CI | None (NetworkX) |
| `neo4j` | Production | Neo4j 5.15+ |
| `neptune` | AWS Production | AWS Neptune, Gremlin |

### Configuration

#### In-Memory Graph (Development/Testing)

```yaml
# configs/default.yaml
stores:
  symbolic: in_memory_graph

memory:
  Nmax: 20
  embedding_dim: 768
```

Or via environment variable:
```bash
export HIEROMEM_MEMORY_BACKEND=in_memory_graph
```

#### Neo4j (Production)

```yaml
# configs/default.yaml
stores:
  symbolic: neo4j

neo4j:
  uri: ${NEO4J_URI}
  user: ${NEO4J_USER}
  password: ${NEO4J_PASSWORD}
  enable_multi_tenancy: true
```

Set credentials via environment variables:
```bash
export NEO4J_URI=bolt://localhost:7687
export NEO4J_USER=neo4j
export NEO4J_PASSWORD=<your-password>
```

### Graph-Enhanced Features

- **Multi-hop Reasoning**: Traverse relationships to discover connected episodes
- **Dual Retrieval**: Combines vector similarity with graph structure
- **Entity Indexing**: Fast lookup of episodes containing specific entities
- **Graph Statistics**: Monitor node/edge counts via health endpoints

### Graph Search API

```http
POST /memory/graph-search
X-API-Key: your-api-key

{
  "episode_id": "abc123",
  "depth": 2,
  "limit": 50,
  "namespace": "default"
}
```

**Response**:
```json
{
  "results": [
    {"source": "Alice", "target": "Bob", "relations": ["knows"], "trust": 0.9},
    {"source": "Bob", "target": "Charlie", "relations": ["knows"], "trust": 0.8}
  ]
}
```

### Checking Graph Status

```bash
# Via health endpoint
curl http://localhost:8000/health/detailed | jq '.memory.graph_store'

# Expected output when enabled:
{
  "backend": "networkx",
  "type": "InMemoryGraphStore",
  "node_count": 42,
  "edge_count": 78
}
```

## 📚 API Documentation

### Core Endpoints

#### Remember an Event
```http
POST /memory/remember
Content-Type: application/json
X-API-Key: your-api-key

{
  "namespace": "default",
  "event_type": "user_action",
  "content": "Event description",
  "metadata": {"key": "value"},
  "scene_graph": {
    "nodes": [{"id": "user1", "type": "User", "properties": {}}],
    "edges": [{"source": "user1", "target": "action1", "relation": "performed"}]
  }
}
```

#### Recall Memories
```http
POST /memory/recall
Content-Type: application/json
X-API-Key: your-api-key

{
  "namespace": "default",
  "query": "What did the user do?",
  "top_k": 5,
  "min_similarity": 0.7
}
```

#### Get Episode Tree
```http
GET /memory/tree?namespace=default
X-API-Key: your-api-key
```

#### Health Check
```http
GET /system/health
```

### Full API Documentation

Interactive API docs available at:
- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc

## ⚙️ Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `NEO4J_URI` | Neo4j connection URI | `bolt://neo4j:7687` |
| `NEO4J_PASSWORD` | Neo4j password | `password` |
| `REDIS_URL` | Redis connection URL | `redis://redis:6379/0` |
| `SQLITE_PATH` | SQLite database path | `/data/memory.db` |
| `HIEROMEM_API_KEY` | Default API key | `dev-key` |
| `GRAFANA_PASSWORD` | Grafana admin password | `admin` |

### Resource Limits

Default limits per service (configurable in `docker-compose.yml`):

- **API**: 2 CPU, 4GB RAM
- **Neo4j**: 2 CPU, 2GB RAM
- **Redis**: 1 CPU, 512MB RAM

## 🚢 Deployment

### Production Deployment Checklist

- [ ] Copy `.env.example` to `.env` and set production values
- [ ] Generate strong passwords for Neo4j and Grafana
- [ ] Replace self-signed SSL certificates with Let's Encrypt
- [ ] Configure offsite backup sync (S3, GCS, Azure Blob)
- [ ] Set up Alertmanager for Prometheus alerts
- [ ] Create production API keys with expiration
- [ ] Configure Grafana dashboards and data sources
- [ ] Adjust resource limits based on workload
- [ ] Set up DNS and domain routing
- [ ] Configure firewall rules

### Backup & Recovery

**Automated Backups**: Daily at 02:00 UTC, 7-day retention

```bash
# Manual backup
docker compose exec backup /scripts/backup.sh

# Restore from backup
docker compose run --rm backup /scripts/restore.sh <TIMESTAMP>
```

See [docs/disaster_recovery.md](docs/disaster_recovery.md) for full recovery procedures.

### Database Migrations

```bash
# Create a new migration
cd packages/hieromem_core
alembic revision -m "description"

# Apply migrations
alembic upgrade head

# Rollback
alembic downgrade -1
```

## 📊 Monitoring

### Prometheus Metrics

Available at http://localhost:9090

- Request rate and latency
- Memory usage and episode counts
- Rate limiting statistics
- Health check status

### Grafana Dashboards

Access at http://localhost:3000 (default: admin/admin)

Pre-configured dashboards:
- System Overview
- API Performance
- Resource Usage
- Log Analysis (via Loki)

### Alerts

Configured in `infra/prometheus/alerts.yml`:
- High memory usage (>90%)
- High CPU usage (>80%)
- Service down
- High error rate (>5%)
- Slow response time (p95 >2s)

## 🛠️ Development

### Local Setup

```bash
# Install dependencies
pip install -r requirements.txt

# Install packages in editable mode
pip install -e packages/hieromem_core
pip install -e packages/hieromem_api

# Run tests
pytest

# Run API locally
cd packages/hieromem_api
uvicorn hieromem_api.main:app --reload
```

### Running Tests

```bash
# All tests
pytest

# Specific test file
pytest tests/test_api_memory.py

# With coverage
pytest --cov=hieromem_core --cov=hieromem_api
```

### Code Quality

```bash
# Format code
black packages/

# Lint
flake8 packages/

# Type checking
mypy packages/
```

## 🛡️ Compliance

- [GDPR Readiness](docs/compliance/GDPR.md) – data processing agreements, erasure, portability, consent.
- [CCPA Controls](docs/compliance/CCPA.md) – consumer rights, Do Not Sell handling, deletion pipeline.
- [SOC 2 Preparation](docs/compliance/SOC2_preparation.md) – security, availability, and confidentiality controls.

## 🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

### Development Guidelines

- Write tests for new features
- Follow PEP 8 style guide
- Update documentation
- Ensure CI/CD passes

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Built with [FastAPI](https://fastapi.tiangolo.com/)
- Graph storage powered by [Neo4j](https://neo4j.com/)
- Vector search using [FAISS](https://github.com/facebookresearch/faiss)
- Embeddings from [Sentence Transformers](https://www.sbert.net/)

## 📞 Support

- **Issues**: [GitHub Issues](https://github.com/Rishikiran98/HIEROMEM/issues)
- **Discussions**: [GitHub Discussions](https://github.com/Rishikiran98/HIEROMEM/discussions)
- **Documentation**: [Wiki](https://github.com/Rishikiran98/HIEROMEM/wiki)

---

**Built with ❤️ for AI agents that never forget**
