# Engram SDK Integration Plan

## Overview

Multi-language SDK ecosystem for the Engram neural memory format.

## SDKs

| SDK | Repository | Package | Status |
|-----|------------|---------|--------|
| TypeScript | `engram` | `@terronex/engram` (npm) | v1.1.0 Released |
| Python | `engram-py` | `engram` (PyPI) | v1.0.0 Ready |
| Rust | `engram-rs` | `engram` (crates.io) | Planned |
| Go | `engram-go` | `github.com/Terronex-dev/engram-go` | Planned |

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Engram Format Spec v1.0                   │
│  (Magic bytes, MessagePack encoding, SHA-256 integrity)     │
└─────────────────────────────────────────────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│  TypeScript   │     │    Python     │     │   Rust / Go   │
│  (Reference)  │     │               │     │               │
├───────────────┤     ├───────────────┤     ├───────────────┤
│ @terronex/    │     │ engram        │     │ engram-rs     │
│ engram        │     │               │     │ engram-go     │
├───────────────┤     ├───────────────┤     ├───────────────┤
│ - MemoryTree  │     │ - MemoryTree  │     │ - MemoryTree  │
│ - read/write  │     │ - read/write  │     │ - read/write  │
│ - HNSW search │     │ - numpy vecs  │     │ - HNSW native │
└───────────────┘     └───────────────┘     └───────────────┘
        │                     │                     │
        ▼                     ▼                     ▼
┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│  Allo CLI     │     │  LangChain    │     │  Native Apps  │
│  Allo Teach   │     │  Allo Research│     │  Embedded     │
└───────────────┘     └───────────────┘     └───────────────┘
```

## Shared Components

### Format Specification (v1.0)

All SDKs must implement:

1. **File Structure**
   - Magic: `ENGRAM` (6 bytes)
   - Version: uint8 major, uint8 minor
   - Header length: uint32 LE
   - Header: MessagePack-encoded EngramHeader
   - Payload: MessagePack-encoded {nodes, entities, links}

2. **Integrity**
   - SHA-256 hash of payload bytes
   - Stored in header.security.integrity
   - Verified on read (optional bypass)

3. **Core Types**
   - MemoryNode (id, content, hierarchy, temporal, quality, metadata, embedding)
   - Entity (id, type, name, mentions, relationships)
   - MemoryLink (source, target, type, confidence)

### Compatibility Test Suite

Location: `engram-test-suite/` (separate repo)

```
fixtures/
  ├── basic.engram           # Simple text nodes
  ├── hierarchy.engram       # Parent-child relationships
  ├── embeddings.engram      # Nodes with vectors
  ├── multimodal.engram      # Text + images
  ├── large.engram           # 10k+ nodes
  └── openstax-sample.engram # Real-world sample
tests/
  ├── read_basic.json        # Expected output for basic.engram
  ├── read_hierarchy.json
  └── ...
```

Each SDK runs the same test fixtures and validates output matches.

## SDK Requirements

### Tier 1 (Required)

- [ ] Read .engram files with integrity verification
- [ ] Write .engram files with correct format
- [ ] MemoryNode CRUD operations
- [ ] Tag-based filtering
- [ ] Text search
- [ ] Hierarchy navigation (parent, children, ancestors, descendants)
- [ ] Pass compatibility test suite

### Tier 2 (Recommended)

- [ ] Embedding storage and retrieval
- [ ] Cosine similarity search
- [ ] Decay tier management
- [ ] Entity extraction
- [ ] Link management

### Tier 3 (Optional)

- [ ] HNSW index for fast vector search
- [ ] Encryption/decryption
- [ ] Streaming delta operations
- [ ] Cloud sync support

## Implementation Order

### Phase 1: Core SDKs (Current)

1. **TypeScript** — Reference implementation (DONE)
   - Full feature set
   - HNSW via hnswlib-node
   - Published to npm

2. **Python** — LangChain ecosystem (IN PROGRESS)
   - Core I/O complete
   - LangChain integration complete
   - Needs: PyPI publish, CI/CD, embedding models

### Phase 2: Systems Languages (Q2 2026)

3. **Rust** — Performance-critical applications
   - Native HNSW (instant-distance or hnswlib-rs)
   - WASM compilation target
   - FFI bindings for other languages

4. **Go** — Cloud services
   - Simple dependency tree
   - Good for microservices
   - gRPC integration potential

### Phase 3: Ecosystem (Q3 2026)

5. **C/C++ bindings** — Via Rust FFI
6. **Java/Kotlin** — Android SDK
7. **Swift** — iOS SDK

## CI/CD Pipeline

Each SDK repository includes:

```yaml
# .github/workflows/ci.yml
jobs:
  test:
    - Run unit tests
    - Run compatibility test suite
    - Lint and type check
  
  publish:
    - Version check
    - Build package
    - Publish to registry
    - Tag release
```

## Versioning Strategy

- **Format version**: Engram v1.0 (binary spec)
- **SDK versions**: Independent semver per language
- **Compatibility matrix**: Documented in each SDK

| Format | TypeScript | Python | Rust | Go |
|--------|------------|--------|------|-----|
| v1.0   | 1.0.x      | 1.0.x  | -    | -   |
| v1.0   | 1.1.x      | 1.0.x  | -    | -   |

## Documentation

### Per-SDK Docs

- README.md — Quick start
- API reference — Generated from docstrings/comments
- Examples — Common use cases
- CHANGELOG.md — Version history

### Central Documentation

Location: docs.terronex.dev/engram

- Format specification
- Cross-SDK compatibility
- Migration guides
- Best practices

## Next Steps

### Immediate (This Week)

1. [x] Python SDK core implementation
2. [x] Python LangChain integration
3. [ ] Python SDK documentation (README, CONTRIBUTING, etc.)
4. [ ] Create compatibility test fixtures
5. [ ] Python CI/CD pipeline
6. [ ] PyPI publication

### Short-term (Next 2 Weeks)

7. [ ] engram-test-suite repository
8. [ ] TypeScript SDK passes test suite
9. [ ] Python SDK passes test suite
10. [ ] Rust SDK skeleton

### Medium-term (Next Month)

11. [ ] Rust SDK core implementation
12. [ ] Go SDK core implementation
13. [ ] Central documentation site
14. [ ] Version 1.1 format planning
