# TDD Plan — EpisodeDigestor + Dream Consolidation

## Obiettivo
Due feature nuove:
1. **EpisodeDigestor** — background worker che legge conversation_log, raggruppa turn in episodi, ingesta episodi completi
2. **Dream Consolidation** — dream() estrae pattern ricorrenti dal cortex → aggiorna Memory Bank files

Branch: `feature/cortex-v5` (continua da Phase 5)

## Dati di Test Reali
```
tests/fixtures/
├── conversation_log_snapshot.db    # 901 entries, 29 sessions, 138 user + 232 assistant + 454 tool_use
└── cortex_db_snapshot/             # ChromaDB con ~290 memorie reali
    ├── chroma.sqlite3
    └── 5844946a-.../               # HNSW index files
```

Sessioni interessanti per test:
- `3b1a78ef7307` — 57+ turn, debug restart, ricca (user+assistant+tool_use)
- `93583bb0397b` — 57 turn, 13 user, 20 assistant
- `5dc6437d8fdc` — 12 turn, test VS Code con AskUserQuestion
- `f4b9b306cc9f` — 2 turn, "ciao c6?" (puro noise)

---

## Fase 6 — conversation_log migration + get_undigested
**Effort: basso | Test nuovi: ~6**

### File coinvolti
- `src/neo_cortex/conversation_log.py` — migration + nuovi metodi
- `tests/test_conversation_log.py` — test nuovi (file esistente da verificare/creare)

### Test da scrivere PRIMA

```python
# tests/test_conversation_log.py

class TestDigestedColumn:
    def test_migration_adds_digested_column(log):
        """Dopo init, colonna digested esiste."""
        cols = log._conn.execute("PRAGMA table_info(conversation_log)").fetchall()
        col_names = [c[1] for c in cols]
        assert "digested" in col_names

    def test_new_entries_default_undigested(log):
        """Nuove entries hanno digested=0."""
        log.append(ConversationAppendRequest(
            session_id="s1", role="user", content="test"
        ))
        row = log._conn.execute("SELECT digested FROM conversation_log").fetchone()
        assert row[0] == 0


class TestGetUndigested:
    def test_returns_only_undigested(log_with_data):
        """get_undigested() restituisce solo entries con digested=0."""
        entries = log.get_undigested()
        assert len(entries) > 0
        # All should be undigested

    def test_grouped_by_session(log_with_data):
        """get_undigested_by_session() raggruppa per session_id."""
        grouped = log.get_undigested_by_session()
        assert isinstance(grouped, dict)
        for sid, entries in grouped.items():
            assert all(e.session_id == sid for e in entries)

    def test_mark_digested(log_with_data):
        """mark_digested() segna entries come processate."""
        entries = log.get_undigested()
        ids = [e.id for e in entries[:3]]
        log.mark_digested(ids)
        remaining = log.get_undigested()
        for e in remaining:
            assert e.id not in ids

    def test_snapshot_all_undigested(tmp_path):
        """Il DB snapshot reale non ha colonna digested → migration la aggiunge → tutto è undigested."""
        import shutil
        db = str(tmp_path / "log.db")
        shutil.copy("tests/fixtures/conversation_log_snapshot.db", db)
        log = ConversationLog(db)
        entries = log.get_undigested()
        assert len(entries) == 901  # tutto il log reale
```

### Implementazione

```python
# conversation_log.py — migration
def _migrate(self):
    cols = {c[1] for c in self._conn.execute("PRAGMA table_info(conversation_log)").fetchall()}
    if "digested" not in cols:
        self._conn.execute("ALTER TABLE conversation_log ADD COLUMN digested INTEGER DEFAULT 0")
        self._conn.commit()

def get_undigested(self, limit: int = 500) -> list[ConversationEntry]:
    rows = self._conn.execute(
        "SELECT * FROM conversation_log WHERE digested = 0 ORDER BY timestamp ASC LIMIT ?", (limit,)
    ).fetchall()
    return [self._row_to_entry(r) for r in rows]

def get_undigested_by_session(self, limit: int = 500) -> dict[str, list[ConversationEntry]]:
    entries = self.get_undigested(limit)
    grouped: dict[str, list] = {}
    for e in entries:
        grouped.setdefault(e.session_id, []).append(e)
    return grouped

def mark_digested(self, ids: list[int]):
    if not ids:
        return
    placeholders = ",".join("?" for _ in ids)
    self._conn.execute(f"UPDATE conversation_log SET digested = 1 WHERE id IN ({placeholders})", ids)
    self._conn.commit()
```

---

## Fase 7 — EpisodeDigestor core (buffer + flush logic)
**Effort: medio | Test nuovi: ~12**

### File nuovi
- `src/neo_cortex/digestor.py` — EpisodeDigestor
- `tests/test_digestor.py` — tutti i test

### Test da scrivere PRIMA

```python
# tests/test_digestor.py

class TestEpisodeAssembly:
    def test_combine_user_assistant_turns(digestor):
        """Combina turn user+assistant in testo episodio leggibile."""
        entries = [
            make_entry(role="user", content="Usiamo NetworkX o Kuzu?"),
            make_entry(role="assistant", content="NetworkX è zero-dep, Kuzu serve solo >10k nodi..."),
            make_entry(role="user", content="ok, va bene"),
            make_entry(role="assistant", content="Perfetto, uso NetworkX."),
        ]
        text = digestor._combine_entries(entries)
        assert "NetworkX" in text
        assert "ok, va bene" in text
        assert "Kuzu" in text

    def test_combine_skips_tool_use_details(digestor):
        """tool_use/tool_result vengono sintetizzati, non inclusi verbatim."""
        entries = [
            make_entry(role="user", content="leggi il file"),
            make_entry(role="tool_use", content='{"tool": "Read", "input": {"path": "/foo/bar.py"}}'),
            make_entry(role="tool_result", content="contenuto del file lungo 5000 chars..."),
            make_entry(role="assistant", content="Il file contiene la classe Foo..."),
        ]
        text = digestor._combine_entries(entries)
        # Tool calls menzionati ma non verbatim
        assert len(text) < 1000  # non include il tool_result intero
        assert "Read" in text or "bar.py" in text

    def test_combine_respects_max_length(digestor):
        """Episodio troncato se troppo lungo (max ~8000 char)."""
        entries = [make_entry(role="assistant", content="x" * 5000) for _ in range(5)]
        text = digestor._combine_entries(entries)
        assert len(text) <= 8500

    def test_combine_empty_entries(digestor):
        assert digestor._combine_entries([]) == ""


class TestFlushDecision:
    def test_flush_on_buffer_full(digestor):
        """Buffer > 10 turn → flush."""
        entries = [make_entry() for _ in range(12)]
        assert digestor._should_flush(entries, last_ts=time.time()) is True

    def test_no_flush_small_buffer(digestor):
        """Buffer < 3 turn → non flush (aspetta)."""
        entries = [make_entry() for _ in range(2)]
        assert digestor._should_flush(entries, last_ts=time.time()) is False

    def test_flush_on_timeout(digestor):
        """Ultimo turn > 3 min fa → flush."""
        entries = [make_entry() for _ in range(4)]
        old_ts = time.time() - 200  # 3+ min ago
        assert digestor._should_flush(entries, last_ts=old_ts) is True

    def test_no_flush_recent_activity(digestor):
        """Ultimo turn < 1 min fa, buffer piccolo → non flush."""
        entries = [make_entry() for _ in range(4)]
        assert digestor._should_flush(entries, last_ts=time.time()) is False

    def test_flush_on_topic_break(digestor_with_classifier):
        """Groq detecta cambio topic → flush."""
        entries = [
            make_entry(role="user", content="Fix the encoding bug"),
            make_entry(role="assistant", content="Used UTF8Encoding(false)"),
            make_entry(role="user", content="Come si deploya su systemd?"),  # topic break
        ]
        # classifier.is_topic_break mock returns True
        result = digestor._should_flush(entries, last_ts=time.time())
        assert result is True


class TestTickCycle:
    @pytest.mark.asyncio
    async def test_tick_reads_undigested(digestor_with_log):
        """tick() legge turn dal log e li bufferizza."""
        await digestor.tick()
        # Buffer should have entries now
        assert len(digestor._buffers) > 0

    @pytest.mark.asyncio
    async def test_tick_marks_digested_after_flush(digestor_with_log):
        """Dopo flush, i turn sono segnati come digested."""
        # Force flush by setting old timestamp
        await digestor.tick()
        await digestor.flush_all()
        remaining = digestor._log.get_undigested()
        # Some entries should be marked as digested now

    @pytest.mark.asyncio
    async def test_flush_all_empties_buffers(digestor_with_log):
        """flush_all() forza flush di tutti i buffer."""
        await digestor.tick()
        await digestor.flush_all()
        assert all(len(b) == 0 for b in digestor._buffers.values())


class TestRealData:
    @pytest.mark.asyncio
    async def test_digest_snapshot_session(tmp_path):
        """Digesta una sessione reale dal snapshot."""
        import shutil
        db = str(tmp_path / "log.db")
        shutil.copy("tests/fixtures/conversation_log_snapshot.db", db)
        log = ConversationLog(db)
        # Setup digestor with mocked cortex/classifier
        # Pick session 3b1a78ef7307 (57+ turns, ricca)
        # Tick until that session is fully digested
        # Verify: episodes created, entries marked digested
```

### Implementazione

```python
# digestor.py (~120 LOC)

FLUSH_TIMEOUT_SECS = 180    # 3 minuti
FLUSH_MAX_TURNS = 10        # safety cap
FLUSH_MIN_TURNS = 3         # non flush troppo poco
MAX_EPISODE_CHARS = 8000    # troncamento

class EpisodeDigestor:
    def __init__(self, log: ConversationLog, cortex: CortexService,
                 classifier: GroqClassifier):
        self._log = log
        self._cortex = cortex
        self._classifier = classifier
        self._buffers: dict[str, list[ConversationEntry]] = {}
        self._last_ts: dict[str, float] = {}

    async def tick(self):
        """Chiamato periodicamente. Legge turn nuovi, decide flush."""
        grouped = self._log.get_undigested_by_session()
        for sid, entries in grouped.items():
            self._buffers.setdefault(sid, []).extend(entries)
            if entries:
                self._last_ts[sid] = entries[-1].timestamp

            if self._should_flush(self._buffers[sid], self._last_ts.get(sid, 0)):
                await self._flush_session(sid)

    def _should_flush(self, entries: list, last_ts: float) -> bool:
        if len(entries) >= FLUSH_MAX_TURNS:
            return True
        if len(entries) < FLUSH_MIN_TURNS:
            return False
        if time.time() - last_ts > FLUSH_TIMEOUT_SECS:
            return True
        # TODO: Groq topic break detection
        return False

    async def _flush_session(self, session_id: str):
        entries = self._buffers.get(session_id, [])
        if not entries:
            return
        text = self._combine_entries(entries)
        if text:
            await self._cortex.ingest_episode(text, session_id)
        ids = [e.id for e in entries]
        self._log.mark_digested(ids)
        self._buffers[session_id] = []

    def _combine_entries(self, entries: list[ConversationEntry]) -> str:
        """Assembla entries in testo episodio."""
        parts = []
        for e in entries:
            if e.role == "user":
                parts.append(f"User: {e.content}")
            elif e.role == "assistant":
                # Rimuovi <thinking> tags
                content = e.content
                if "<thinking>" in content:
                    import re
                    content = re.sub(r"<thinking>.*?</thinking>", "", content, flags=re.DOTALL).strip()
                if content:
                    parts.append(f"Assistant: {content}")
            elif e.role == "tool_use":
                # Solo il nome del tool, non l'input intero
                try:
                    import json
                    data = json.loads(e.content)
                    tool = data.get("tool", "unknown")
                    parts.append(f"[Tool: {tool}]")
                except:
                    parts.append("[Tool call]")
            # tool_result: skip (troppo verboso)
        combined = "\n".join(parts)
        if len(combined) > MAX_EPISODE_CHARS:
            combined = combined[:MAX_EPISODE_CHARS]
        return combined

    async def flush_all(self):
        """Force flush di tutti i buffer (chiamato su shutdown)."""
        for sid in list(self._buffers.keys()):
            await self._flush_session(sid)
```

---

## Fase 8 — cortex.ingest_episode() + Groq topic break
**Effort: basso | Test nuovi: ~5**

### Test da scrivere PRIMA

```python
# tests/test_cortex.py

class TestIngestEpisode:
    @pytest.mark.asyncio
    async def test_ingest_episode_stores_memory(cortex_with_index):
        """ingest_episode() crea una memoria dall'episodio."""
        text = "User: Usiamo NetworkX?\nAssistant: Sì, zero dep.\nUser: ok va bene"
        mid = await cortex_with_index.ingest_episode(text, "sess-1")
        assert mid is not None
        assert cortex_with_index._store.count() == 1

    @pytest.mark.asyncio
    async def test_ingest_episode_dedup(cortex_with_dedup):
        """Episodi duplicati bloccati dal dedup."""
        text = "User: Fix encoding\nAssistant: UTF8Encoding(false)"
        mid1 = await cortex_with_dedup.ingest_episode(text, "s1")
        mid2 = await cortex_with_dedup.ingest_episode(text, "s2")
        assert mid1 is not None
        assert mid2 is None

    @pytest.mark.asyncio
    async def test_ingest_episode_populates_graph(cortex_with_graph):
        """Episodio popola il grafo concetti."""
        text = "User: Come funziona encoding?\nAssistant: BOM header..."
        await cortex_with_graph.ingest_episode(text, "s1")
        assert cortex_with_graph._graph.node_count() > 0

    @pytest.mark.asyncio
    async def test_ingest_episode_empty_text(cortex):
        """Testo vuoto → None, no crash."""
        mid = await cortex.ingest_episode("", "s1")
        assert mid is None

    @pytest.mark.asyncio
    async def test_ingest_episode_no_noise_filter(cortex):
        """ingest_episode NON applica noise filter (il digestor ha già deciso)."""
        text = "User: ok\nAssistant: Perfetto, procedo con NetworkX."
        mid = await cortex.ingest_episode(text, "s1")
        assert mid is not None  # "ok" NON filtrato


# tests/test_classifier.py (o test_cortex.py)

class TestTopicBreak:
    @pytest.mark.asyncio
    async def test_is_topic_break_true(classifier):
        """Cambio argomento → True."""
        entries = [
            "User: Fix the encoding bug",
            "Assistant: Used UTF8Encoding(false)",
            "User: Come si deploya su systemd?",
        ]
        result = await classifier.is_topic_break(entries)
        assert result is True

    @pytest.mark.asyncio
    async def test_is_topic_break_false(classifier):
        """Stesso argomento → False."""
        entries = [
            "User: Fix the encoding bug",
            "Assistant: The BOM header is the problem",
            "User: ok, and how about stdin?",
        ]
        result = await classifier.is_topic_break(entries)
        assert result is False
```

### Implementazione

```python
# cortex.py — nuovo metodo
async def ingest_episode(self, text: str, session_id: str) -> str | None:
    """Ingest un episodio completo (multi-turn). NO noise filter."""
    if not text.strip():
        return None

    # Dedup
    if self._dedup and self._dedup.is_duplicate(text):
        return None

    # Embed
    embedding = await self._embedder.embed_passage(text)

    # Classify (vede tutto il contesto!)
    classification = await self._classifier.classify_episode(text)

    # Build record
    ts = time.time()
    memory_id = f"ep_{session_id[:8]}_{int(ts) % 100000}"
    record = MemoryRecord(
        id=memory_id, session_id=session_id, timestamp=ts,
        question="[episode]", answer_preview=text[:500],
        document=text, project=classification.project,
        topic=classification.topic, activity=classification.activity,
        energy=1.0, source="digestor",
    )

    self._store.insert(record, embedding)
    if self._dedup:
        self._dedup.add(text, memory_id)

    # Index + graph (come ingest normale)
    if self._index:
        structured = StructuredFields(...)
        self._index.insert(record, structured)
    if self._graph and classification.concepts:
        self._graph.add_memory(memory_id, classification.concepts)

    return memory_id

# classifier.py — nuovo metodo
async def is_topic_break(self, conversation_lines: list[str]) -> bool:
    """Chiedi a Groq se l'ultimo turn ha cambiato argomento."""
    prompt = "Does the LAST line change the topic from the previous lines? Answer ONLY 'yes' or 'no'."
    text = "\n".join(conversation_lines[-5:])  # ultimi 5 turn max
    raw = await self._call(prompt, text, max_tokens=10, temperature=0.1)
    return raw.strip().lower().startswith("y")
```

---

## Fase 9 — Dream Consolidation → Memory Bank
**Effort: medio | Test nuovi: ~8**

### File nuovi
- `src/neo_cortex/consolidator.py` — MBConsolidator
- `tests/test_consolidator.py`

### Test da scrivere PRIMA

```python
# tests/test_consolidator.py

class TestPatternExtraction:
    def test_find_recurring_concepts(consolidator_with_data):
        """Concetti in 3+ memorie ad alta energy → pattern candidati."""
        patterns = consolidator.find_recurring_patterns()
        assert len(patterns) > 0
        for p in patterns:
            assert p["count"] >= 3
            assert p["avg_energy"] > 0.5

    def test_no_patterns_from_low_energy(consolidator):
        """Memorie a bassa energy non producono pattern."""
        # Tutte le memorie con energy < 0.3
        patterns = consolidator.find_recurring_patterns()
        assert len(patterns) == 0

    def test_pattern_includes_source_memories(consolidator_with_data):
        """Ogni pattern ha lista di memory_id sorgente."""
        patterns = consolidator.find_recurring_patterns()
        for p in patterns:
            assert "memory_ids" in p
            assert len(p["memory_ids"]) >= 3


class TestMBGeneration:
    @pytest.mark.asyncio
    async def test_generate_mb_entry(consolidator_with_data):
        """Genera entry MB da un pattern (via Groq)."""
        patterns = consolidator.find_recurring_patterns()
        entry = await consolidator.generate_mb_entry(patterns[0])
        assert isinstance(entry, str)
        assert len(entry) > 10

    @pytest.mark.asyncio
    async def test_classify_mb_target(consolidator):
        """Classifica un pattern → file MB target."""
        pattern = {"concept": "encoding", "facts": ["UTF8Encoding(false)", "BOM"]}
        target = await consolidator.classify_target(pattern)
        assert target in ["systemPatterns.md", "techContext.md", "history.md", "activeContext.md"]


class TestMBWrite:
    def test_append_to_mb_file(tmp_path, consolidator):
        """Appende entry a un file MB senza sovrascrivere."""
        mb_path = tmp_path / "techContext.md"
        mb_path.write_text("# TechContext\n\nexisting content\n")
        consolidator.append_to_mb(str(mb_path), "## New Pattern\nnew knowledge")
        content = mb_path.read_text()
        assert "existing content" in content
        assert "New Pattern" in content
        assert "new knowledge" in content

    def test_append_idempotent(tmp_path, consolidator):
        """Non appende lo stesso pattern due volte."""
        mb_path = tmp_path / "techContext.md"
        mb_path.write_text("# TechContext\n")
        consolidator.append_to_mb(str(mb_path), "## Encoding\nUTF8Encoding(false)")
        consolidator.append_to_mb(str(mb_path), "## Encoding\nUTF8Encoding(false)")
        content = mb_path.read_text()
        assert content.count("UTF8Encoding(false)") == 1

    def test_create_mb_file_if_missing(tmp_path, consolidator):
        """Se il file MB non esiste, crealo."""
        mb_path = tmp_path / "new_topic.md"
        consolidator.append_to_mb(str(mb_path), "## New Knowledge")
        assert mb_path.exists()


class TestRealData:
    def test_patterns_from_snapshot(consolidator_with_snapshot):
        """Trova pattern reali dal cortex snapshot (~290 memorie)."""
        patterns = consolidator.find_recurring_patterns()
        # Dovremmo trovare almeno encoding, deploy, neo-reloaded patterns
        concepts = [p["concept"] for p in patterns]
        # Non assertiamo concetti specifici, ma che ne trovi alcuni
        assert len(patterns) >= 1
```

### Implementazione

```python
# consolidator.py (~100 LOC)

class MBConsolidator:
    def __init__(self, index: MemoryIndex, graph: ConceptGraph,
                 classifier: GroqClassifier, mb_path: str):
        self._index = index
        self._graph = graph
        self._classifier = classifier
        self._mb_path = mb_path  # ~/neo-ram/memory_bank/
        self._consolidated: set[str] = set()  # track cosa è già stato scritto

    def find_recurring_patterns(self, min_count: int = 3, min_energy: float = 0.5) -> list[dict]:
        """Trova concetti che appaiono in 3+ memorie ad alta energy."""
        all_concepts = self._index.get_all_concepts()
        patterns = []
        for concept, count in all_concepts.items():
            if count < min_count:
                continue
            # Check average energy of memories with this concept
            memories = self._index.search_by_concept(concept, n=count)
            avg_energy = sum(m.energy for m in memories) / len(memories) if memories else 0
            if avg_energy < min_energy:
                continue
            patterns.append({
                "concept": concept,
                "count": count,
                "avg_energy": avg_energy,
                "memory_ids": [m.id for m in memories],
            })
        return patterns

    async def generate_mb_entry(self, pattern: dict) -> str:
        """Usa Groq per sintetizzare un pattern in entry MB."""
        # Raccogli facts/summaries dalle memorie sorgente
        ...

    async def classify_target(self, pattern: dict) -> str:
        """Classifica il pattern → file MB target."""
        ...

    def append_to_mb(self, file_path: str, content: str):
        """Appende contenuto a file MB (idempotente)."""
        ...
```

### Integrazione in dream()

```python
# cortex.py — alla fine di dream()
if self._consolidator:
    patterns = self._consolidator.find_recurring_patterns()
    for p in patterns:
        entry = await self._consolidator.generate_mb_entry(p)
        target = await self._consolidator.classify_target(p)
        target_path = os.path.join(self._mb_path, target)
        self._consolidator.append_to_mb(target_path, entry)
```

---

## Riepilogo Test

| Fase | Feature | Test nuovi | Test totali |
|------|---------|-----------|-------------|
| Baseline | Phases 0-5 | 0 | 182 |
| 6 | conversation_log migration | ~6 | ~188 |
| 7 | EpisodeDigestor core | ~12 | ~200 |
| 8 | ingest_episode + topic break | ~7 | ~207 |
| 9 | Dream consolidation → MB | ~8 | ~215 |
| **Totale** | | **~33** | **~215** |

## File nuovi

```
src/neo_cortex/
├── digestor.py          # ~120 LOC — EpisodeDigestor
└── consolidator.py      # ~100 LOC — MBConsolidator

tests/
├── test_digestor.py     # ~12 test
├── test_consolidator.py # ~8 test
└── fixtures/
    ├── conversation_log_snapshot.db    # 901 entries reali
    └── cortex_db_snapshot/             # ChromaDB ~290 memorie
```

## File modificati

```
src/neo_cortex/
├── conversation_log.py  # +digested column, +get_undigested, +mark_digested
├── cortex.py            # +ingest_episode(), +consolidator in dream()
├── classifier.py        # +is_topic_break(), +classify_episode()
└── config.py            # +MB_PATH

tests/
├── test_conversation_log.py  # +6 test migration/undigested
└── test_cortex.py            # +5 test ingest_episode
```

## Workflow per ogni Fase

```
1. Scrivi test (RED — tutti falliscono)
2. Implementa il minimo per farli passare (GREEN)
3. Refactora se necessario (REFACTOR)
4. Verifica: uv run pytest tests/ → TUTTI verdi (182 vecchi + N nuovi)
5. Commit: "Phase N: <feature> — X/Y tests green"
```

## Ordine di Esecuzione

```
Fase 6 → 7 → 8 → 9
  (ogni fase dipende dalla precedente)
```

## Note Architetturali

### Due strade, non una che sostituisce l'altra
- **Strada 1 (ingest esplicito)**: Claude chiama `memory_ingest` → noise filter → pipeline classica
- **Strada 2 (digest background)**: conversation_log → digestor → episodi → `ingest_episode` (NO noise filter)
- Entrambe scrivono su ChromaDB + SQLite + Graph
- SimHash dedup previene duplicati tra le due strade

### Il digestor NON filtra noise
Il punto è che "ok, va bene" nel contesto di un episodio ha valore.
Il digestor combina i turn, Groq vede il contesto intero e classifica correttamente.

### La MB non viene sovrascritta
Il consolidator solo appende. È idempotente (non scrive lo stesso pattern due volte).
Laco può editare a mano — il consolidator non tocca roba esistente.
