# TDD Plan — neo-cortex v5 "Grafo + Memoria Biologica"

## Obiettivo
Estendere neo-cortex con grafo concetti, dedup, decay biologico e retrieval reinforcement.
Tutto test-driven: RED → GREEN → REFACTOR per ogni step.

## Setup

```bash
git checkout -b feature/cortex-v5
```

Baseline: 116 test verdi su master. Ogni fase aggiunge test, mai rompe quelli esistenti.

---

## Fase 0 — P3: Usare i dati già estratti dal Classifier
**Impatto: alto | Effort: basso | Test nuovi: ~8**

Il classifier GIA' estrae `concepts[]`, `facts[]`, `files_touched[]` e li salva in SQLite.
Nessuno li usa nel recall. Questa fase li collega.

### File coinvolti
- `src/neo_cortex/memory_index.py` — nuovi metodi di query
- `src/neo_cortex/cortex.py` — recall usa structured fields
- `tests/test_memory_index.py` — test nuovi
- `tests/test_cortex.py` — test recall con concepts

### Test da scrivere PRIMA

```python
# tests/test_memory_index.py

class TestConceptSearch:
    def test_search_by_concept_finds_matching(index_with_data):
        """Cerco 'encoding' → trovo memorie che hanno 'encoding' nei concepts"""
        results = index.search_by_concept("encoding")
        assert len(results) >= 1
        assert any("encoding" in r.concepts for r in results)

    def test_search_by_concept_no_match(index_with_data):
        results = index.search_by_concept("nonexistent-xyz")
        assert len(results) == 0

    def test_search_by_concept_partial_match(index_with_data):
        """'process' matcha 'process-lifecycle'"""
        results = index.search_by_concept("process")
        assert len(results) >= 1

    def test_search_by_file_touched(index_with_data):
        """Cerco 'store.py' → memorie che toccano quel file"""
        results = index.search_by_file("store.py")
        assert len(results) >= 1

    def test_get_all_concepts(index_with_data):
        """Lista tutti i concetti unici con frequenza"""
        concepts = index.get_all_concepts()
        assert isinstance(concepts, dict)  # {"encoding": 3, "deploy": 1, ...}
        assert all(isinstance(v, int) for v in concepts.values())

    def test_get_related_memories(index_with_data):
        """Due memorie che condividono concetti sono 'related'"""
        related = index.get_related_by_concepts("memory-id-1")
        assert len(related) >= 1


# tests/test_cortex.py

class TestRecallWithConcepts:
    async def test_recall_uses_concept_boost(cortex):
        """Recall con concept match dovrebbe pesare di più"""
        # Ingest due memorie: una con concept 'encoding', una senza
        # Recall query su 'encoding' → la prima deve venire prima
        pass

    async def test_recall_fallback_without_index(cortex_no_index):
        """Se index è None, recall funziona come prima (solo vector)"""
        result = await cortex.recall("test query")
        assert result.count >= 0  # no crash
```

### Implementazione

**memory_index.py** — nuovi metodi:
```python
def search_by_concept(self, concept: str, n: int = 10) -> list[CompactMemory]:
    """Cerca memorie che contengono il concept (JSON array search)"""
    # SELECT * FROM memories WHERE concepts LIKE '%"concept"%' ORDER BY timestamp DESC

def search_by_file(self, file_path: str, n: int = 10) -> list[CompactMemory]:
    """Cerca memorie che toccano un file"""
    # SELECT * FROM memories WHERE files_touched LIKE '%file_path%'

def get_all_concepts(self) -> dict[str, int]:
    """Tutti i concepts unici con frequenza"""
    # Itera tutte le righe, parse JSON concepts, conta

def get_related_by_concepts(self, memory_id: str, n: int = 5) -> list[CompactMemory]:
    """Memorie che condividono almeno 1 concept con memory_id"""
    # Get concepts of memory_id → search each → deduplicate
```

**cortex.py** — modifica recall():
```python
# Dopo vector search, se index esiste:
# 1. Estrai concepts dalla query (via classifier analysis o keyword match)
# 2. Cerca memorie con quei concepts via index.search_by_concept()
# 3. Merge risultati vector + concept (boost score per chi appare in entrambi)
```

---

## Fase 1 — P2: Deduplicazione SimHash
**Impatto: medio | Effort: basso | Test nuovi: ~10**

Blocca memorie duplicate PRIMA di ingest. Zero API calls.

### File nuovi
- `src/neo_cortex/dedup.py` — SimHash + Hamming distance
- `tests/test_dedup.py` — tutti i test

### Test da scrivere PRIMA

```python
# tests/test_dedup.py

class TestSimHash:
    def test_simhash_deterministic():
        """Stesso testo → stesso hash"""
        h1 = simhash("the quick brown fox")
        h2 = simhash("the quick brown fox")
        assert h1 == h2

    def test_simhash_similar_texts_close():
        """Testi simili → Hamming distance bassa"""
        h1 = simhash("fix encoding bug in process stdin")
        h2 = simhash("fix encoding bug in process stdout")
        assert hamming_distance(h1, h2) <= 5

    def test_simhash_different_texts_far():
        """Testi diversi → Hamming distance alta"""
        h1 = simhash("fix encoding bug in stdin")
        h2 = simhash("deploy neo-cortex to production")
        assert hamming_distance(h1, h2) > 10

    def test_hamming_distance_zero_identical():
        assert hamming_distance(0xABCD, 0xABCD) == 0

    def test_hamming_distance_known():
        assert hamming_distance(0b1010, 0b1000) == 1


class TestDedupStore:
    def test_is_duplicate_false_for_new(dedup_store):
        """Prima memoria → mai duplicata"""
        assert dedup_store.is_duplicate("some new text") is False

    def test_is_duplicate_true_for_same(dedup_store):
        """Testo identico → duplicato"""
        dedup_store.add("fix the encoding bug")
        assert dedup_store.is_duplicate("fix the encoding bug") is True

    def test_is_duplicate_true_for_similar(dedup_store):
        """Testo quasi identico (Hamming ≤ 3) → duplicato"""
        dedup_store.add("fix encoding bug in process stdin")
        assert dedup_store.is_duplicate("fix encoding bug in process stdout") is True

    def test_is_duplicate_false_for_different(dedup_store):
        """Testo diverso → non duplicato"""
        dedup_store.add("fix encoding bug in process stdin")
        assert dedup_store.is_duplicate("deploy neo-cortex to pypi") is False

    def test_dedup_persistence(tmp_path):
        """Hash sopravvive restart"""
        store1 = DedupStore(str(tmp_path / "dedup.db"))
        store1.add("some text")
        del store1
        store2 = DedupStore(str(tmp_path / "dedup.db"))
        assert store2.is_duplicate("some text") is True
```

### Implementazione

**dedup.py** (~60 LOC):
```python
def simhash(text: str, hashbits: int = 64) -> int:
    """Compute SimHash of text"""
    # Tokenize → hash each token → weighted bit sum → threshold to 0/1

def hamming_distance(a: int, b: int) -> int:
    """Count differing bits"""
    return bin(a ^ b).count('1')

class DedupStore:
    """SQLite-backed SimHash store"""
    def __init__(self, db_path: str, threshold: int = 3):
        # CREATE TABLE simhashes (hash INTEGER, memory_id TEXT, timestamp REAL)

    def is_duplicate(self, text: str) -> bool:
        # Compute simhash → scan all hashes → return True if any hamming ≤ threshold

    def add(self, text: str, memory_id: str = "") -> int:
        # Store simhash → return hash value
```

**cortex.py** — agganciare a ingest():
```python
# Dopo noise filter, prima di embedding:
if self._dedup and self._dedup.is_duplicate(combined_text):
    log("DEDUP: skipping duplicate memory")
    return None
# Dopo insert:
if self._dedup:
    self._dedup.add(combined_text, memory_id)
```

---

## Fase 2 — P1: Grafo Concetti con NetworkX
**Impatto: alto | Effort: medio | Test nuovi: ~15**

Il cuore della v5. Grafo di concetti con spreading activation.

### File nuovi
- `src/neo_cortex/graph.py` — ConceptGraph (NetworkX)
- `tests/test_graph.py` — tutti i test

### Dipendenza nuova
```toml
# pyproject.toml
"networkx>=3.0"
```

### Test da scrivere PRIMA

```python
# tests/test_graph.py

class TestConceptGraph:
    def test_add_memory_creates_nodes(graph):
        """Aggiungere memoria crea nodi concetto"""
        graph.add_memory("mem1", ["encoding", "BOM", "stdin"])
        assert graph.has_node("encoding")
        assert graph.has_node("BOM")
        assert graph.has_node("stdin")

    def test_add_memory_creates_cooccurrence_edges(graph):
        """Concetti co-occorrenti nella stessa memoria → archi"""
        graph.add_memory("mem1", ["encoding", "BOM", "stdin"])
        assert graph.has_edge("encoding", "BOM")
        assert graph.has_edge("encoding", "stdin")
        assert graph.has_edge("BOM", "stdin")

    def test_cooccurrence_weight_increments(graph):
        """Stessi concetti in più memorie → peso arco cresce"""
        graph.add_memory("mem1", ["encoding", "BOM"])
        graph.add_memory("mem2", ["encoding", "BOM"])
        w = graph.edge_weight("encoding", "BOM")
        assert w == 2

    def test_mentions_increment(graph):
        """Ogni apparizione incrementa mentions del nodo"""
        graph.add_memory("mem1", ["encoding"])
        graph.add_memory("mem2", ["encoding"])
        assert graph.mentions("encoding") == 2

    def test_memory_links(graph):
        """Nodo concetto traccia quali memorie lo contengono"""
        graph.add_memory("mem1", ["encoding"])
        graph.add_memory("mem2", ["encoding"])
        assert set(graph.memories_for("encoding")) == {"mem1", "mem2"}

    def test_remove_memory(graph):
        """Rimuovere memoria decrementa edge weights"""
        graph.add_memory("mem1", ["encoding", "BOM"])
        graph.add_memory("mem2", ["encoding", "BOM"])
        graph.remove_memory("mem1", ["encoding", "BOM"])
        assert graph.edge_weight("encoding", "BOM") == 1
        assert graph.mentions("encoding") == 1


class TestSpreadingActivation:
    def test_direct_concept_activation(graph_with_data):
        """Query concept → attivazione 1.0"""
        activated = graph.spreading_activation(["encoding"])
        assert activated["encoding"] == 1.0

    def test_neighbor_activation_decays(graph_with_data):
        """Vicini attivati con decay (< 1.0)"""
        activated = graph.spreading_activation(["encoding"])
        # BOM è vicino di encoding
        assert "BOM" in activated
        assert activated["BOM"] < 1.0

    def test_two_hop_activation(graph_with_data):
        """2 hop: encoding → BOM → utf8 (decay²)"""
        activated = graph.spreading_activation(["encoding"], hops=2)
        assert "utf8" in activated
        assert activated["utf8"] < activated["BOM"]

    def test_activation_empty_concepts(graph_with_data):
        activated = graph.spreading_activation([])
        assert activated == {}

    def test_activation_unknown_concept(graph_with_data):
        activated = graph.spreading_activation(["nonexistent"])
        assert activated == {}

    def test_get_related_memories(graph_with_data):
        """Spreading activation → trova memorie collegate"""
        memories = graph.get_related_memories(["encoding"], n=5)
        assert len(memories) >= 1
        assert all(isinstance(m, str) for m in memories)  # memory IDs


class TestGraphPersistence:
    def test_save_and_load(tmp_path):
        """Grafo sopravvive restart (JSON)"""
        g1 = ConceptGraph(str(tmp_path / "graph.json"))
        g1.add_memory("mem1", ["encoding", "BOM"])
        g1.save()
        g2 = ConceptGraph(str(tmp_path / "graph.json"))
        assert g2.has_node("encoding")
        assert g2.has_edge("encoding", "BOM")

    def test_empty_graph_on_missing_file(tmp_path):
        g = ConceptGraph(str(tmp_path / "missing.json"))
        assert g.node_count() == 0

    def test_auto_save_on_add(tmp_path):
        """add_memory() salva automaticamente"""
        g = ConceptGraph(str(tmp_path / "graph.json"))
        g.add_memory("mem1", ["encoding"])
        g2 = ConceptGraph(str(tmp_path / "graph.json"))
        assert g2.has_node("encoding")
```

### Implementazione

**graph.py** (~150 LOC):
```python
import networkx as nx
import json

class ConceptGraph:
    def __init__(self, path: str):
        self._path = path
        self._graph = nx.Graph()
        self._memory_concepts: dict[str, list[str]] = {}  # mem_id → concepts
        self._load()

    def add_memory(self, memory_id: str, concepts: list[str]):
        """Aggiungi nodi + archi co-occorrenza + traccia memoria"""

    def remove_memory(self, memory_id: str, concepts: list[str]):
        """Decrementa weights, rimuovi archi a peso 0"""

    def spreading_activation(self, concepts: list[str], hops: int = 2, decay: float = 0.5) -> dict[str, float]:
        """BFS pesato dal concetto → vicini con decay per hop"""

    def get_related_memories(self, concepts: list[str], n: int = 5) -> list[str]:
        """Spreading activation → raccogli memory IDs dai nodi attivati"""

    def memories_for(self, concept: str) -> list[str]:
    def mentions(self, concept: str) -> int:
    def edge_weight(self, a: str, b: str) -> int:
    def has_node(self, concept: str) -> bool:
    def has_edge(self, a: str, b: str) -> bool:
    def node_count(self) -> int:

    def save(self):
        """Serializza graph + memory_concepts a JSON"""

    def _load(self):
        """Carica da JSON se esiste"""
```

**cortex.py** — agganciare a ingest() e recall():
```python
# Ingest: dopo SQLite insert
if self._graph and structured and structured.concepts:
    self._graph.add_memory(memory_id, structured.concepts)

# Recall: dopo vector search
if self._graph:
    # Estrai concepts dalla query analysis
    activated = self._graph.spreading_activation(query_concepts)
    related_mids = self._graph.get_related_memories(query_concepts)
    # Merge con risultati vector: boost score per memorie in entrambi
```

---

## Fase 3 — P5: Retrieval Reinforcement
**Impatto: medio | Effort: basso | Test nuovi: ~5**

Ogni recall rafforza le memorie trovate.

### Test da scrivere PRIMA

```python
# tests/test_cortex.py

class TestRetrievalReinforcement:
    async def test_recall_boosts_energy(cortex_with_data):
        """Dopo recall, le memorie trovate hanno energy più alta"""
        before = get_energy(cortex, "mem1")
        await cortex.recall("encoding bug")
        after = get_energy(cortex, "mem1")
        assert after > before

    async def test_recall_boost_capped_at_one(cortex_with_data):
        """Energy non supera 1.0"""
        # Set energy a 0.95, recall → max 1.0
        set_energy(cortex, "mem1", 0.95)
        await cortex.recall("encoding bug")
        assert get_energy(cortex, "mem1") <= 1.0

    async def test_recall_propagates_to_neighbors(cortex_with_graph):
        """Recall rafforza anche memorie vicine nel grafo"""
        # mem1 e mem2 condividono concetto "encoding"
        before = get_energy(cortex, "mem2")
        await cortex.recall("encoding bug")  # trova mem1
        after = get_energy(cortex, "mem2")
        assert after > before  # propagazione

    async def test_recall_no_boost_without_results(cortex):
        """Se recall non trova nulla, nessun boost"""
        await cortex.recall("completely unknown topic xyz")
        # No crash, no side effects

    async def test_recall_reinforcement_formula(cortex_with_data):
        """energy += eta * (1 - energy), eta = 0.1"""
        set_energy(cortex, "mem1", 0.5)
        await cortex.recall("encoding bug")
        expected = 0.5 + 0.1 * (1.0 - 0.5)  # 0.55
        assert abs(get_energy(cortex, "mem1") - expected) < 0.01
```

### Implementazione

**cortex.py** — alla fine di recall():
```python
REINFORCEMENT_ETA = 0.1

# Dopo aver trovato risultati:
for record, similarity in results:
    new_energy = min(1.0, record.energy + REINFORCEMENT_ETA * (1.0 - record.energy))
    self._store.update_energy(record.id, new_energy)
    if self._index:
        self._index.update_energy(record.id, new_energy)

# Propagazione via grafo (se esiste):
if self._graph:
    for record, _ in results:
        concepts = self._get_concepts_for(record.id)
        related = self._graph.get_related_memories(concepts, n=3)
        for mid in related:
            # Boost ridotto per vicini (eta/2)
            ...
```

---

## Fase 4 — P6: Ebbinghaus Decay (Dream potenziato)
**Impatto: alto | Effort: medio | Test nuovi: ~8**

Sostituisce il flat ±0.03/+0.1 con decay biologico.

### File nuovi
- `src/neo_cortex/decay.py` — Ebbinghaus model
- `tests/test_decay.py` — test

### Nuovi campi in SQLite
```sql
ALTER TABLE memories ADD COLUMN stability REAL DEFAULT 1.0;
ALTER TABLE memories ADD COLUMN last_accessed REAL;  -- timestamp ultimo recall
ALTER TABLE memories ADD COLUMN access_count INTEGER DEFAULT 0;
```

### Test da scrivere PRIMA

```python
# tests/test_decay.py

class TestEbbinghausDecay:
    def test_fresh_memory_high_weight():
        """Memoria appena creata → weight ~1.0"""
        w = ebbinghaus_weight(oldness_days=0, stability=1.0)
        assert w > 0.99

    def test_old_memory_low_weight():
        """Memoria vecchia + bassa stability → weight basso"""
        w = ebbinghaus_weight(oldness_days=30, stability=1.0)
        assert w < 0.5

    def test_stable_memory_resists_decay():
        """Alta stability → decade più lentamente"""
        w_unstable = ebbinghaus_weight(oldness_days=30, stability=1.0)
        w_stable = ebbinghaus_weight(oldness_days=30, stability=5.0)
        assert w_stable > w_unstable

    def test_reinforcement_increases_stability():
        """Dopo recall, stability += 1"""
        s = reinforce(stability=2.0)
        assert s == 3.0

    def test_weight_never_below_floor():
        """Peso minimo 0.05 (non zero)"""
        w = ebbinghaus_weight(oldness_days=365, stability=0.1)
        assert w >= 0.05

    def test_weight_never_above_one():
        w = ebbinghaus_weight(oldness_days=0, stability=100.0)
        assert w <= 1.0


class TestDreamWithDecay:
    async def test_dream_applies_ebbinghaus(cortex):
        """Dream usa Ebbinghaus invece di flat ±0.03"""
        # Ingest 2 memorie: una vecchia, una recente
        # Dream → la vecchia perde più energy della recente

    async def test_dream_prunes_below_threshold(cortex):
        """Memorie con weight < 0.05 dopo dream → segnalate per pruning"""
        # Non cancellate automaticamente, ma flaggate
```

### Implementazione

**decay.py** (~40 LOC):
```python
import math

FLOOR = 0.05

def ebbinghaus_weight(oldness_days: float, stability: float) -> float:
    """weight = max(FLOOR, exp(-(oldness / stability)))"""
    if stability <= 0:
        return FLOOR
    w = math.exp(-(oldness_days / stability))
    return max(FLOOR, min(1.0, w))

def reinforce(stability: float, increment: float = 1.0) -> float:
    """Ogni recall → stability += increment"""
    return stability + increment
```

**cortex.py** — nuovo dream():
```python
async def dream(self) -> DreamResult:
    all_meta = self._store.get_all_metadata()
    now = time.time()
    for meta in all_meta:
        oldness_days = (now - meta["timestamp"]) / 86400
        stability = meta.get("stability", 1.0)
        new_energy = ebbinghaus_weight(oldness_days, stability)
        self._store.update_energy(meta["id"], new_energy)
    # ... cluster logic resta per il boost
```

---

## Fase 5 — P7: Spontaneous Recall (opzionale)
**Impatto: medio | Effort: basso | Test nuovi: ~3**

Memorie importanti + non accedute recentemente emergono.

### Test

```python
class TestSpontaneousRecall:
    async def test_spontaneous_returns_forgotten_important(cortex):
        """Alta energy + vecchio last_accessed → candidato spontaneo"""
        results = await cortex.spontaneous(n=3)
        assert len(results) >= 0  # può essere vuoto
        for r in results:
            assert r.energy > 0.5  # abbastanza importante
            # last_accessed vecchio (> 7 giorni)

    async def test_spontaneous_excludes_recent(cortex):
        """Memorie accedute di recente NON sono spontanee"""
        pass

    async def test_spontaneous_empty_cortex(empty_cortex):
        results = await empty_cortex.spontaneous()
        assert results == []
```

---

## Riepilogo Test

| Fase | Feature | Test nuovi | Test totali |
|------|---------|-----------|-------------|
| Baseline | — | 0 | 116 |
| 0 | P3: Usare dati classifier | ~8 | ~124 |
| 1 | P2: SimHash dedup | ~10 | ~134 |
| 2 | P1: Grafo NetworkX | ~15 | ~149 |
| 3 | P5: Retrieval reinforcement | ~5 | ~154 |
| 4 | P6: Ebbinghaus decay | ~8 | ~162 |
| 5 | P7: Spontaneous recall | ~3 | ~165 |
| **Totale** | | **~49** | **~165** |

## File nuovi

```
src/neo_cortex/
├── dedup.py          # ~60 LOC — SimHash + DedupStore
├── graph.py          # ~150 LOC — ConceptGraph (NetworkX)
└── decay.py          # ~40 LOC — Ebbinghaus model

tests/
├── test_dedup.py     # ~10 test
├── test_graph.py     # ~15 test
└── test_decay.py     # ~8 test
```

## File modificati

```
src/neo_cortex/
├── cortex.py         # +graph, +dedup, +reinforcement, +decay in dream
├── memory_index.py   # +search_by_concept, +search_by_file, +get_all_concepts
├── models.py         # +stability, +last_accessed, +access_count fields
└── config.py         # +GRAPH_PATH, +DEDUP_DB_PATH

pyproject.toml        # +networkx dependency
```

## Workflow per ogni Fase

```
1. Scrivi test (RED — tutti falliscono)
2. Implementa il minimo per farli passare (GREEN)
3. Refactora se necessario (REFACTOR)
4. Verifica: pytest tests/ → TUTTI verdi (116 vecchi + N nuovi)
5. Commit: "Phase N: <feature> — X/Y tests green"
```

## Merge

Quando tutte le fasi sono complete e 165 test sono verdi:
```bash
git checkout master
git merge feature/cortex-v5
# Bump version in pyproject.toml → 5.0.0
# uv build && publish
```

## Backfill

Dopo merge, script per popolare il grafo con le ~300 memorie esistenti:
```bash
python scripts/backfill_graph.py  # Legge concepts da SQLite → popola NetworkX
python scripts/backfill_dedup.py  # Calcola SimHash per memorie esistenti
```
