# Piano: Neo Cortex v4

## Obiettivo
Aggiungere SQLite come index strutturato accanto a ChromaDB, estrarre osservazioni strutturate, progressive disclosure, fallback search. Backward compatible al 100%.

## Stato attuale (v3)
- 9 file sorgente, 1197 LOC
- 6 file test, 65 test GREEN
- Storage: ChromaDB only (`conversations_v2`, 272 memories)
- Classifier: Groq → project/topic/activity (3 campi)
- Search: Jina embedding → ChromaDB cosine similarity
- Problema: `get_all_metadata()` carica TUTTO in RAM per timeline/stats/dream
- Problema: nessun fallback se Jina down
- Problema: nessun filtro strutturato (solo ChromaDB $where, limitato)

## Architettura v4

```
ChromaDB (invariato)          SQLite (NUOVO)
├── embeddings                ├── memories (metadata + strutturate)
├── documents                 ├── memories_fts (FTS5 full-text)
└── metadatas                 └── (indici su timestamp, project, activity)

cortex.ingest() → embed → classify → store ENTRAMBI
cortex.recall() → try Jina+ChromaDB, fallback FTS5
cortex.search() → NUOVO: SQLite FTS5 + filtri → compact index
cortex.get()    → NUOVO: SQLite by ID → full detail
```

## Dipendenze tra fasi

```
Fase 1 (MemoryIndex) ← fondazione
  ↓
Fase 2 (Classifier upgrade) ← richiede nuovi campi
  ↓
Fase 3 (Dual-write) ← richiede Fase 1 + 2
  ↓
Fase 4 (Backfill) ← richiede Fase 3 (schema pronto)
  ↓
Fase 5 (Fallback search) ← richiede Fase 1
  ↓
Fase 6 (Progressive disclosure) ← richiede Fase 1
  ↓
Fase 7 (MCP tools) ← richiede Fase 6
```

---

## Fase 1: MemoryIndex (SQLite structured store)

### File: `src/neo_cortex/memory_index.py` (NUOVO, ~180 LOC)

**Schema:**
```sql
CREATE TABLE memories (
    id TEXT PRIMARY KEY,
    session_id TEXT NOT NULL,
    timestamp REAL NOT NULL,
    project TEXT DEFAULT 'general',
    topic TEXT DEFAULT 'unknown',
    activity TEXT DEFAULT 'discussion',
    memory_type TEXT DEFAULT 'episodic',
    energy REAL DEFAULT 1.0,
    model TEXT DEFAULT 'opus',
    source TEXT DEFAULT 'neo-cortex',
    -- Campi strutturati (NUOVI)
    title TEXT,
    summary TEXT,
    facts TEXT,           -- JSON array
    concepts TEXT,        -- JSON array
    files_touched TEXT,   -- JSON array
    -- Campi originali
    question TEXT,
    answer_preview TEXT
);

-- FTS5 per keyword search veloce
CREATE VIRTUAL TABLE memories_fts USING fts5(
    title, summary, facts, question, answer_preview,
    content=memories, content_rowid=rowid
);

-- Indici per filtri
CREATE INDEX idx_mem_timestamp ON memories(timestamp DESC);
CREATE INDEX idx_mem_project ON memories(project);
CREATE INDEX idx_mem_activity ON memories(activity);
CREATE INDEX idx_mem_session ON memories(session_id);
```

**Classe `MemoryIndex`:**
- `__init__(db_path)` — crea DB + tabelle + FTS5
- `insert(record: MemoryRecord, structured: StructuredFields)` — INSERT + FTS trigger
- `search_fts(query, project, activity, n)` → list[CompactMemory]
- `get_by_id(id)` → MemoryRecord | None
- `get_by_ids(ids)` → list[MemoryRecord]
- `timeline(n, project)` → list[CompactMemory] (ORDER BY timestamp DESC)
- `stats()` → dict (COUNT, sessions, avg energy — via SQL aggregates, NON caricando tutto)
- `update_energy(id, energy)` — UPDATE singolo
- `count()` → int

**Modello `CompactMemory`** (NUOVO in models.py):
```python
class CompactMemory(BaseModel):
    id: str
    title: str | None = None
    project: str = "general"
    activity: str = "discussion"
    timestamp: float
    energy: float = 1.0
    # NO question, NO answer — quelli costano token
```

**Modello `StructuredFields`** (NUOVO in models.py):
```python
class StructuredFields(BaseModel):
    title: str | None = None
    summary: str | None = None
    facts: list[str] = []
    concepts: list[str] = []
    files_touched: list[str] = []
```

### Test: `tests/test_memory_index.py` (~15 test)
1. create_table_on_init
2. insert_and_get_by_id
3. insert_and_get_by_ids
4. search_fts_basic
5. search_fts_with_project_filter
6. search_fts_with_activity_filter
7. search_fts_empty_query_returns_timeline
8. search_fts_no_results
9. timeline_returns_newest_first
10. timeline_filter_by_project
11. stats_returns_aggregates
12. stats_empty_db
13. update_energy
14. count
15. insert_duplicate_id_updates (upsert)

---

## Fase 2: Classifier upgrade (structured extraction)

### File: `src/neo_cortex/classifier.py` (MODIFICA)

**Nuovo prompt CLASSIFY_STRUCTURED_PROMPT:**
```
Classify and extract structured info from this conversation turn.
Return ONLY valid JSON:

{
  "project": "project name or 'general'",
  "topic": "1-3 word topic",
  "activity": "bugfix|feature|debug|config|deploy|research|discussion|learning|identity|behavior|growth",
  "title": "concise 5-10 word title describing what happened",
  "summary": "one sentence summary",
  "facts": ["fact 1", "fact 2"],
  "concepts": ["concept1", "concept2"],
  "files_touched": ["path/to/file.py"]
}

Extract facts = concrete things learned/done.
Extract concepts = abstract topics (e.g. "testing", "deployment", "authentication").
Extract files only if explicitly mentioned.
If uncertain about a field, omit it.
```

**Modifiche a `classify()`:**
- Usa il nuovo prompt
- Ritorna `ClassificationResult` esteso (backward compat: vecchi campi invariati)
- Nuovi campi opzionali: title, summary, facts, concepts, files_touched
- Se parsing fallisce, ritorna solo project/topic/activity come prima (graceful)

**Modifica `ClassificationResult`** (in models.py):
```python
class ClassificationResult(BaseModel):
    project: str = "general"
    topic: str = "unknown"
    activity: Activity = Activity.DISCUSSION
    worth_remembering: bool = True
    # NUOVI campi strutturati
    title: str | None = None
    summary: str | None = None
    facts: list[str] = []
    concepts: list[str] = []
    files_touched: list[str] = []
```

### Test: `tests/test_classifier.py` (MODIFICA, +4 test)
16. classify_returns_title
17. classify_returns_facts
18. classify_returns_concepts
19. classify_graceful_fallback_on_missing_fields

---

## Fase 3: Dual-write in cortex.ingest()

### File: `src/neo_cortex/cortex.py` (MODIFICA)

**Modifiche a `__init__`:**
```python
def __init__(self, store, embedder, classifier, index: MemoryIndex | None = None):
    self._index = index  # opzionale per backward compat
```

**Modifiche a `ingest()`:**
- Dopo `self._store.insert(record, embedding)`:
```python
if self._index:
    structured = StructuredFields(
        title=classification.title,
        summary=classification.summary,
        facts=classification.facts,
        concepts=classification.concepts,
        files_touched=classification.files_touched,
    )
    self._index.insert(record, structured)
```

**Modifiche a `server.py` lifespan:**
```python
_index = MemoryIndex(config.MEMORY_INDEX_DB_PATH)
_cortex = CortexService(store, _embedder, _classifier, index=_index)
```

**Nuovo in config.py:**
```python
MEMORY_INDEX_DB_PATH = os.environ.get(
    "MEMORY_INDEX_DB_PATH",
    str(Path.home() / "neo-ram" / "memory_index.db"),
)
```

### Test: `tests/test_cortex.py` (MODIFICA, +3 test)
20. ingest_writes_to_both_stores
21. ingest_works_without_index (backward compat)
22. ingest_structured_fields_propagated

---

## Fase 4: Backfill existing memories

### File: `scripts/backfill_index.py` (NUOVO, ~60 LOC)

Script one-shot che:
1. Legge tutti i metadata da ChromaDB via `store.get_all_metadata()`
2. Per ogni memoria, inserisce in SQLite con i campi disponibili
3. Campi strutturati (title/summary/facts/concepts) = None per memorie vecchie
4. Opzionale: flag `--classify` per ri-classificare con il nuovo prompt
5. Report: N inserite, N skippate, N errori

```bash
cd ~/projects/neo-cortex
python -m scripts.backfill_index          # solo metadata
python -m scripts.backfill_index --classify  # + ri-classifica con Groq
```

### Test: nessun test automatico (script one-shot)

---

## Fase 5: Fallback search

### File: `src/neo_cortex/cortex.py` (MODIFICA)

**Modifiche a `_basic_recall()`:**
```python
async def _basic_recall(self, query: str, n: int) -> RecallResult:
    try:
        embedding = await self._embedder.embed_query(query)
        results = self._store.query(embedding, n=n)
        memories = [RecalledMemory(record=r, similarity=s) for r, s in results]
        return RecallResult(query=query, count=len(memories), memories=memories)
    except Exception as e:
        logger.warning("Embedding failed, falling back to FTS: %s", e)
        return self._fts_recall(query, n)

def _fts_recall(self, query: str, n: int) -> RecallResult:
    if not self._index:
        return RecallResult(query=query, count=0, memories=[])
    compact = self._index.search_fts(query, n=n)
    # Hydrate compact -> full MemoryRecord
    ids = [c.id for c in compact]
    records = self._index.get_by_ids(ids)
    memories = [RecalledMemory(record=r, similarity=0.5) for r in records]
    return RecallResult(query=query, count=len(memories), memories=memories)
```

**Stesso pattern per `_smart_recall()`** — try/except around embed, fallback a FTS.

### Test: `tests/test_cortex.py` (+3 test)
23. recall_fallback_to_fts_when_embed_fails
24. recall_fts_returns_results
25. recall_fts_works_without_index (empty result)

---

## Fase 6: Progressive disclosure endpoints

### File: `src/neo_cortex/server.py` (MODIFICA, +3 endpoint)

```python
@app.get("/api/search")
async def search(q: str = "", project: str | None = None,
                 activity: str | None = None, n: int = 10):
    """Search memories — returns compact index (low token cost)."""
    index = _get_index()
    results = index.search_fts(q, project=project, activity=activity, n=n)
    return {"count": len(results), "results": [r.model_dump() for r in results]}

@app.get("/api/memory/{memory_id}")
async def get_memory(memory_id: str):
    """Get full memory detail by ID."""
    index = _get_index()
    record = index.get_by_id(memory_id)
    if not record:
        raise HTTPException(404, "Memory not found")
    return record.model_dump()

@app.get("/api/memories")
async def get_memories(ids: str):
    """Get multiple memories by comma-separated IDs."""
    index = _get_index()
    id_list = [i.strip() for i in ids.split(",") if i.strip()]
    records = index.get_by_ids(id_list)
    return {"count": len(records), "memories": [r.model_dump() for r in records]}
```

### Modifiche a `stats()` e `timeline()` in cortex.py
- Se `_index` disponibile, usa SQLite (O(log n)) invece di `get_all_metadata()` (O(n))
- Fallback a ChromaDB se SQLite non disponibile

### Test: `tests/test_server.py` (+5 test)
26. search_returns_compact
27. search_with_project_filter
28. get_memory_by_id
29. get_memory_not_found_404
30. get_memories_batch

---

## Fase 7: MCP tools upgrade

### File: `src/neo_cortex/mcp.py` (MODIFICA, +2 tool)

```python
@mcp.tool()
def memory_search(query: str, project: str | None = None,
                  activity: str | None = None, n: int = 10) -> dict:
    """Search memory index. Returns compact results (title/project/date).
    Use memory_get for full details of specific memories.

    Args:
        query: Search text (or empty for recent)
        project: Filter by project name
        activity: Filter by activity type
        n: Max results
    """
    return _get("/api/search", q=query, project=project, activity=activity, n=n)

@mcp.tool()
def memory_get(ids: str) -> dict:
    """Get full memory details by IDs (comma-separated).

    Args:
        ids: Comma-separated memory IDs (e.g. "v3_abc_1_12345,v3_def_2_67890")
    """
    return _get("/api/memories", ids=ids)
```

**Tool esistenti invariati** — memory_query, memory_stats, memory_dream, memory_ingest, memory_timeline restano identici.

---

## Riepilogo modifiche

| File | Azione | LOC stimato |
|------|--------|-------------|
| `memory_index.py` | NUOVO | ~180 |
| `models.py` | MODIFICA (+CompactMemory, +StructuredFields, ClassificationResult esteso) | +30 |
| `classifier.py` | MODIFICA (prompt strutturato) | +20 |
| `cortex.py` | MODIFICA (dual-write, fallback, index param) | +40 |
| `server.py` | MODIFICA (+3 endpoint, index init) | +30 |
| `mcp.py` | MODIFICA (+2 tool) | +20 |
| `config.py` | MODIFICA (+1 path) | +5 |
| `scripts/backfill_index.py` | NUOVO | ~60 |
| `test_memory_index.py` | NUOVO | ~200 |
| `test_classifier.py` | MODIFICA | +40 |
| `test_cortex.py` | MODIFICA | +50 |
| `test_server.py` | MODIFICA | +60 |
| **TOTALE** | | ~735 LOC aggiunti |

## Test previsti
- Esistenti: 65 (devono restare GREEN)
- Nuovi: ~30
- Totale finale: ~95 test

## Regole
- Backward compat: TUTTI gli endpoint/tool esistenti invariati
- MemoryIndex opzionale: se non presente, cortex funziona come v3
- Dual-write: ChromaDB rimane primary, SQLite e index secondario
- FTS5 fallback: solo se Jina/ChromaDB down, non come default
- Nessuna migrazione ChromaDB — le 272 memorie restano dove sono
