# Spike 5 Implementation Plan — Operational Hardening + Search Quality

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Fix all known bugs from code review, then add temporal filtering, heading context in embeddings, and structured JSON output to make search noticeably better for all agents.

**Architecture:** Two sequential phases. Phase 1 fixes bugs in pool.py, server.py, routes.py, events.py, and templates. Phase 2 adds features to store.py, indexer.py, and server.py. TDD throughout — tests first, then implementation.

**Tech Stack:** Python 3.12, FastMCP, ChromaDB, pytest, Starlette/Jinja2

---

## Phase 1: Operational Hardening

### Task 1: Unify StorePool between server and dashboard

The most significant bug. `create_server()` creates its own StorePool (server.py:142), and `main()` creates a second one for the dashboard (server.py:533). They have independent ChromaDB connections and divergent state.

**Files:**
- Modify: `src/annal/server.py:129-157` (create_server signature + main)
- Modify: `src/annal/server.py:492-538` (main function)
- Test: `tests/test_server.py`

**Step 1: Write the failing test**

Add to `tests/test_server.py`:

```python
def test_create_server_returns_pool(server_env):
    """create_server should return both mcp and pool."""
    result = create_server(config_path=server_env["config_path"])
    assert isinstance(result, tuple)
    mcp, pool = result
    assert mcp.name == "annal"
    assert pool is not None


def test_create_server_accepts_external_pool(server_env):
    """create_server should use a provided pool instead of creating its own."""
    from annal.pool import StorePool
    from annal.config import AnnalConfig
    config = AnnalConfig.load(server_env["config_path"])
    external_pool = StorePool(config)
    mcp, returned_pool = create_server(
        config_path=server_env["config_path"],
        pool=external_pool,
    )
    assert returned_pool is external_pool
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/test_server.py::test_create_server_returns_pool tests/test_server.py::test_create_server_accepts_external_pool -v`
Expected: FAIL — create_server returns FastMCP, not a tuple

**Step 3: Update create_server signature and return**

In `src/annal/server.py`, change `create_server`:

```python
def create_server(
    config_path: str = DEFAULT_CONFIG_PATH,
    pool: StorePool | None = None,
) -> tuple[FastMCP, StorePool]:
    """Create and configure the Annal MCP server."""
    config = AnnalConfig.load(config_path)

    mcp = FastMCP(
        "annal",
        instructions=SERVER_INSTRUCTIONS,
        host="127.0.0.1",
        port=config.port,
    )

    if pool is None:
        pool = StorePool(config)
    # ... rest unchanged ...
    return mcp, pool
```

**Step 4: Fix all existing callers of create_server**

In `main()` (server.py:492-538), replace the double config load + double pool:

```python
    config = AnnalConfig.load(config_path)
    pool = StorePool(config)
    mcp, _ = create_server(config_path=config_path, pool=pool)

    if not no_dashboard:
        dashboard_port = config.port if transport == "stdio" else config.port + 1
        _start_dashboard(pool, config, port=dashboard_port)

    mcp.run(transport=transport)
```

Fix all test fixtures that unpack `create_server()` — the `mcp` fixture in test_server.py:

```python
@pytest.fixture
def mcp(server_env):
    mcp, _pool = create_server(config_path=server_env["config_path"])
    return mcp
```

And `test_create_server`:
```python
def test_create_server(server_env):
    mcp, pool = create_server(config_path=server_env["config_path"])
    assert mcp is not None
    assert mcp.name == "annal"
```

**Step 5: Run all tests to verify nothing broke**

Run: `pytest tests/test_server.py -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/annal/server.py tests/test_server.py
git commit -m "fix: unify StorePool — create_server returns (mcp, pool), main shares single pool"
```

---

### Task 2: Add `_startup_reconcile` error handling + `index_failed` event

Two related fixes: the startup loop needs per-project error handling, and `index_failed` needs to be emitted when reconciliation fails.

**Files:**
- Modify: `src/annal/server.py:146-153` (_startup_reconcile)
- Modify: `src/annal/pool.py:68-92` (reconcile_project_async _run)
- Modify: `src/annal/events.py:16` (Event docstring)
- Test: `tests/test_pool.py`

**Step 1: Write the failing tests**

Add to `tests/test_pool.py`:

```python
def test_reconcile_project_async_emits_index_failed(tmp_data_dir, tmp_config_path, tmp_path):
    """If reconciliation fails, an index_failed event should be emitted."""
    from annal.events import event_bus

    # Create a project with a watch path that will cause issues
    config = AnnalConfig(config_path=tmp_config_path, data_dir=tmp_data_dir)
    config.add_project("failtest", watch_paths=[str(tmp_path / "nonexistent")])
    config.save()
    pool = StorePool(config)

    events = []
    q = event_bus.subscribe()

    def on_complete(count):
        pass

    pool.reconcile_project_async(
        "failtest",
        on_complete=on_complete,
    )
    time.sleep(2)

    # Drain the queue
    import queue as queue_mod
    while True:
        try:
            events.append(q.get_nowait())
        except queue_mod.Empty:
            break
    event_bus.unsubscribe(q)

    # Should not have crashed — pool should still be usable
    assert pool.is_indexing("failtest") is False
```

**Step 2: Run tests to verify they fail (or pass — this test may already pass since nonexistent paths are handled)**

Run: `pytest tests/test_pool.py::test_reconcile_project_async_emits_index_failed -v`

**Step 3: Add `index_failed` event emission to pool.py**

In `src/annal/pool.py`, modify `reconcile_project_async`'s `_run()`:

```python
from annal.events import event_bus, Event

def _run() -> None:
    lock = self._get_index_lock(project)  # will be updated in Task 4
    if not lock.acquire(blocking=False):
        logger.info("Indexing already in progress for '%s', waiting", project)
        lock.acquire()
    try:
        self._index_started[project] = datetime.now(timezone.utc)
        if project not in self._config.projects:
            return
        store = self.get_store(project)
        if clear_first:
            store.delete_by_source("file:")
        proj_config = self._config.projects[project]
        watcher = FileWatcher(store=store, project_config=proj_config)
        count = watcher.reconcile(progress_callback=on_progress)
        self._last_reconcile[project] = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "file_count": count,
        }
        logger.info("Reconciled %d files for project '%s'", count, project)
        if on_complete:
            on_complete(count)
    except Exception as e:
        logger.exception("Reconciliation failed for project '%s'", project)
        event_bus.push(Event(
            type="index_failed", project=project, detail=str(e)
        ))
    finally:
        self._index_started.pop(project, None)
        lock.release()
```

Add the import at the top of pool.py:
```python
from annal.events import event_bus, Event
```

Update the Event docstring in `src/annal/events.py`:
```python
@dataclass
class Event:
    """A dashboard event."""
    type: str      # "memory_stored", "memory_deleted", "index_started", "index_complete", "index_failed"
    project: str
    detail: str = ""
```

**Step 4: Add per-project error handling to `_startup_reconcile`**

In `src/annal/server.py`, modify `_startup_reconcile`:

```python
def _startup_reconcile() -> None:
    for project_name in config.projects:
        try:
            logger.info("Reconciling project '%s'...", project_name)
            event_bus.push(Event(type="index_started", project=project_name))
            count = pool.reconcile_project(project_name)
            event_bus.push(Event(type="index_complete", project=project_name, detail=f"{count} files"))
            pool.start_watcher(project_name)
        except Exception:
            logger.exception("Startup reconciliation failed for project '%s'", project_name)
            event_bus.push(Event(type="index_failed", project=project_name, detail="startup reconciliation failed"))
    logger.info("Startup reconciliation complete")
```

**Step 5: Run all tests**

Run: `pytest tests/test_pool.py tests/test_server.py -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/annal/pool.py src/annal/server.py src/annal/events.py tests/test_pool.py
git commit -m "fix: emit index_failed event on reconciliation errors, add per-project error handling to startup"
```

---

### Task 3: Fix activity-indicator element + dashboard `index_failed` handling

**Files:**
- Modify: `src/annal/dashboard/templates/memories.html:7-9` (add element)
- Modify: `src/annal/dashboard/templates/memories.html:153-160` (update JS)

**Step 1: Add the missing element**

In `src/annal/dashboard/templates/memories.html`, change the page header div:

```html
<div class="page-header">
  <h1>{{ project }}</h1>
  <span class="page-count" id="memory-count">{{ total }} memories</span>
  <span id="activity-indicator"></span>
</div>
```

**Step 2: Update JS to handle `index_failed` and add the SSE trigger**

In the same file, update the SSE listener div to include `index_failed`:

```html
<div hx-ext="sse" sse-connect="/events"
     hx-trigger="sse:memory_stored, sse:memory_deleted, sse:index_complete"
     hx-get="/memories/table"
     hx-target="#memory-table"
     hx-include="[name]"
     style="display:none"></div>
```

Update the script block:

```javascript
document.body.addEventListener('sse:index_started', function() {
  var el = document.getElementById('activity-indicator');
  if (el) { el.textContent = 'indexing\u2026'; el.classList.add('active'); }
});
document.body.addEventListener('sse:index_complete', function() {
  var el = document.getElementById('activity-indicator');
  if (el) { el.textContent = ''; el.classList.remove('active'); }
});
document.body.addEventListener('sse:index_failed', function() {
  var el = document.getElementById('activity-indicator');
  if (el) { el.textContent = 'index failed'; el.classList.remove('active'); el.classList.add('error'); }
});
```

**Step 3: Run dashboard tests**

Run: `pytest tests/test_dashboard.py -v`
Expected: ALL PASS

**Step 4: Commit**

```bash
git add src/annal/dashboard/templates/memories.html
git commit -m "fix: add missing activity-indicator element, handle index_failed in dashboard JS"
```

---

### Task 4: Narrow SSE exception catch + `_index_locks` thread safety

**Files:**
- Modify: `src/annal/dashboard/routes.py:232` (SSE catch)
- Modify: `src/annal/pool.py:26` (_index_locks)
- Modify: `src/annal/pool.py:69,99` (callers)
- Test: `tests/test_pool.py`

**Step 1: Write the failing test for `_get_index_lock`**

Add to `tests/test_pool.py`:

```python
def test_get_index_lock_returns_same_lock(tmp_data_dir, tmp_config_path):
    """_get_index_lock should return the same lock for the same project."""
    config = AnnalConfig(config_path=tmp_config_path, data_dir=tmp_data_dir)
    config.save()
    pool = StorePool(config)

    lock1 = pool._get_index_lock("testproject")
    lock2 = pool._get_index_lock("testproject")
    assert lock1 is lock2


def test_get_index_lock_different_projects(tmp_data_dir, tmp_config_path):
    """_get_index_lock should return different locks for different projects."""
    config = AnnalConfig(config_path=tmp_config_path, data_dir=tmp_data_dir)
    config.save()
    pool = StorePool(config)

    lock1 = pool._get_index_lock("project_a")
    lock2 = pool._get_index_lock("project_b")
    assert lock1 is not lock2
```

**Step 2: Run test to verify it fails**

Run: `pytest tests/test_pool.py::test_get_index_lock_returns_same_lock -v`
Expected: FAIL — AttributeError: 'StorePool' has no attribute '_get_index_lock'

**Step 3: Implement `_get_index_lock` and update callers**

In `src/annal/pool.py`, replace the defaultdict and add the method:

```python
class StorePool:
    def __init__(self, config: AnnalConfig) -> None:
        self._config = config
        self._stores: dict[str, MemoryStore] = {}
        self._watchers: dict[str, FileWatcher] = {}
        self._lock = threading.Lock()
        self._index_locks: dict[str, threading.Lock] = {}  # was defaultdict
        self._index_started: dict[str, datetime] = {}
        self._last_reconcile: dict[str, dict] = {}

    def _get_index_lock(self, project: str) -> threading.Lock:
        """Get or create an index lock for a project (thread-safe)."""
        with self._lock:
            if project not in self._index_locks:
                self._index_locks[project] = threading.Lock()
            return self._index_locks[project]
```

Update `reconcile_project_async` (line 69):
```python
lock = self._get_index_lock(project)
```

Update `is_indexing` (line 99):
```python
lock = self._get_index_lock(project)
```

Remove the `defaultdict` import if no longer needed.

**Step 4: Narrow the SSE exception catch**

In `src/annal/dashboard/routes.py`, add the import at the top:
```python
import queue
```

Change line 232 from:
```python
except Exception:
```
to:
```python
except queue.Empty:
```

**Step 5: Run all tests**

Run: `pytest tests/test_pool.py tests/test_dashboard.py -v`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add src/annal/pool.py src/annal/dashboard/routes.py tests/test_pool.py
git commit -m "fix: thread-safe _get_index_lock, narrow SSE catch to queue.Empty"
```

---

### Task 5: Dead code removal + dependency cleanup + README

**Files:**
- Modify: `src/annal/store.py:199-207` (remove get_file_mtime)
- Modify: `pyproject.toml:33` (add httpx)
- Modify: `README.md:190` (test count)

**Step 1: Remove `get_file_mtime`**

Delete the `get_file_mtime` method from `src/annal/store.py` (lines 199-207).

**Step 2: Add httpx to dev dependencies**

In `pyproject.toml`, change:
```toml
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.23.0",
]
```
to:
```toml
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.23.0",
    "httpx>=0.27.0",
]
```

**Step 3: Fix README test count**

In `README.md`, change "95 tests" to just describe the coverage without a hard number.

**Step 4: Run all tests to verify nothing references the removed method**

Run: `pytest -v`
Expected: ALL PASS

**Step 5: Commit**

```bash
git add src/annal/store.py pyproject.toml README.md
git commit -m "chore: remove dead get_file_mtime, add httpx to dev deps, fix README test count"
```

---

## Phase 2: Search & Retrieval

### Task 6: Heading context in embeddings

**Files:**
- Modify: `src/annal/indexer.py:95-104` (prepend heading to content)
- Test: `tests/test_indexer.py`

**Step 1: Write the failing test**

Add to `tests/test_indexer.py`:

```python
def test_index_file_includes_heading_context_in_content(tmp_data_dir, tmp_path):
    """File-indexed chunks should include heading path in stored content for better embeddings."""
    md_file = tmp_path / "test.md"
    md_file.write_text("# Overview\nIntro text\n\n## Architecture\nThree layers.\n")

    store = MemoryStore(data_dir=tmp_data_dir, project="headingtest")
    index_file(store, str(md_file))

    results = store.search("Architecture", limit=5)
    # The chunk content should include the heading context, not just the body
    arch_chunk = [r for r in results if "Architecture" in r["source"]]
    assert len(arch_chunk) > 0
    assert "Architecture" in arch_chunk[0]["content"]
    assert "Three layers" in arch_chunk[0]["content"]
```

**Step 2: Run test to verify it fails**

Run: `pytest tests/test_indexer.py::test_index_file_includes_heading_context_in_content -v`
Expected: May pass already (Architecture appears in heading). Add a more precise test:

```python
def test_index_file_prepends_heading_path_to_content(tmp_data_dir, tmp_path):
    """Stored content should start with the heading path for embedding context."""
    md_file = tmp_path / "doc.md"
    md_file.write_text("# Project\nIntro\n\n## Design\n### Backend\nUses Python.\n")

    store = MemoryStore(data_dir=tmp_data_dir, project="headingtest2")
    index_file(store, str(md_file))

    # Get all stored chunks
    results = store.search("Python", limit=5)
    backend_chunk = [r for r in results if "Backend" in r["source"]]
    assert len(backend_chunk) > 0
    # Content should start with heading path like "doc.md > Design > Backend:"
    assert backend_chunk[0]["content"].startswith("doc.md")
    assert "Backend" in backend_chunk[0]["content"]
    assert "Uses Python" in backend_chunk[0]["content"]
```

**Step 3: Run test to verify it fails**

Run: `pytest tests/test_indexer.py::test_index_file_prepends_heading_path_to_content -v`
Expected: FAIL — content is just "Uses Python." without heading prefix

**Step 4: Implement heading context prepend**

In `src/annal/indexer.py`, change the store loop (lines 95-104):

```python
    # Store each chunk
    for chunk in chunks:
        tags = _derive_tags(path)
        # Prepend heading path to content for better embedding context
        heading_context = chunk["heading"]
        content_with_context = f"{heading_context}: {chunk['content']}"
        store.store(
            content=content_with_context,
            tags=tags,
            source=f"file:{file_path}|{chunk['heading']}",
            chunk_type="file-indexed",
            file_mtime=file_mtime,
        )
```

**Step 5: Run indexer tests**

Run: `pytest tests/test_indexer.py -v`
Expected: Some existing tests may need content assertion updates since stored content now includes heading prefix. Fix any that broke by updating assertions to account for the heading prefix.

**Step 6: Run all tests**

Run: `pytest -v`
Expected: ALL PASS (may need to fix test_index_file_stores_chunks and test_reindex_file_replaces_old_chunks assertions)

**Step 7: Commit**

```bash
git add src/annal/indexer.py tests/test_indexer.py
git commit -m "feat: prepend heading context to file-indexed chunks for better embedding quality"
```

---

### Task 7: Temporal filtering

**Files:**
- Modify: `src/annal/store.py:44-86` (search method — add after/before params)
- Modify: `src/annal/server.py:190-243` (search_memories tool — add after/before params)
- Test: `tests/test_store.py`
- Test: `tests/test_server.py`

**Step 1: Write the failing store-level test**

Add to `tests/test_store.py`:

```python
def test_search_with_after_filter(tmp_data_dir):
    store = MemoryStore(data_dir=tmp_data_dir, project="temporal")

    store.store(content="Old decision about auth", tags=["decision"])
    # Store a second memory — both will have created_at within the same second,
    # so we test with a date range instead
    store.store(content="New decision about billing", tags=["decision"])

    # Search with after set to tomorrow — should find nothing
    from datetime import datetime, timezone, timedelta
    tomorrow = (datetime.now(timezone.utc) + timedelta(days=1)).isoformat()
    results = store.search("decision", after=tomorrow)
    assert len(results) == 0

    # Search with after set to yesterday — should find both
    yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
    results = store.search("decision", after=yesterday)
    assert len(results) == 2


def test_search_with_before_filter(tmp_data_dir):
    store = MemoryStore(data_dir=tmp_data_dir, project="temporal2")

    store.store(content="Some memory", tags=["test"])

    from datetime import datetime, timezone, timedelta
    # Before yesterday — should find nothing (memory was just created)
    yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
    results = store.search("memory", before=yesterday)
    assert len(results) == 0

    # Before tomorrow — should find it
    tomorrow = (datetime.now(timezone.utc) + timedelta(days=1)).isoformat()
    results = store.search("memory", before=tomorrow)
    assert len(results) == 1


def test_search_with_after_and_before(tmp_data_dir):
    store = MemoryStore(data_dir=tmp_data_dir, project="temporal3")

    store.store(content="Memory in range", tags=["test"])

    from datetime import datetime, timezone, timedelta
    yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
    tomorrow = (datetime.now(timezone.utc) + timedelta(days=1)).isoformat()

    results = store.search("Memory", after=yesterday, before=tomorrow)
    assert len(results) == 1
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/test_store.py::test_search_with_after_filter tests/test_store.py::test_search_with_before_filter tests/test_store.py::test_search_with_after_and_before -v`
Expected: FAIL — search() got unexpected keyword argument 'after'

**Step 3: Implement temporal filtering in store.search()**

In `src/annal/store.py`, update the `search` method signature and add the `where` filter:

```python
def search(
    self,
    query: str,
    limit: int = 5,
    tags: list[str] | None = None,
    after: str | None = None,
    before: str | None = None,
) -> list[dict]:
    if self._collection.count() == 0:
        return []

    # Over-fetch when filtering by tags since filtering is post-query
    limit_query = max(limit * 3, 20) if tags else limit

    # Build ChromaDB where filter for temporal constraints
    where = None
    if after and before:
        where = {"$and": [
            {"created_at": {"$gte": after}},
            {"created_at": {"$lte": before}},
        ]}
    elif after:
        where = {"created_at": {"$gte": after}}
    elif before:
        where = {"created_at": {"$lte": before}}

    results = self._collection.query(
        query_texts=[query],
        n_results=min(limit_query, self._collection.count()) or 1,
        where=where,
    )
    # ... rest unchanged ...
```

**Step 4: Run store tests**

Run: `pytest tests/test_store.py -v`
Expected: ALL PASS

**Step 5: Write the MCP tool-level test**

Add to `tests/test_server.py`:

```python
@pytest.mark.asyncio
async def test_search_with_temporal_filter(mcp):
    """search_memories should accept after/before date params."""
    await _call(mcp, "store_memory", {
        "project": "test",
        "content": "Decision made today about API design",
        "tags": ["decision"],
    })

    from datetime import datetime, timezone, timedelta
    tomorrow = (datetime.now(timezone.utc) + timedelta(days=1)).isoformat()
    yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()

    # Should find with yesterday-tomorrow range
    result = await _call(mcp, "search_memories", {
        "project": "test",
        "query": "API design",
        "after": yesterday,
        "before": tomorrow,
    })
    assert "API design" in result

    # Should not find with future-only range
    result = await _call(mcp, "search_memories", {
        "project": "test",
        "query": "API design",
        "after": tomorrow,
    })
    assert "No matching memories" in result
```

**Step 6: Update search_memories tool to pass through temporal params**

In `src/annal/server.py`, update the `search_memories` tool:

```python
@mcp.tool()
def search_memories(
    project: str,
    query: str,
    tags: list[str] | str | None = None,
    limit: int = 5,
    mode: str = "full",
    min_score: float = 0.0,
    after: str | None = None,
    before: str | None = None,
) -> str:
    """Search project memories using natural language.

    Args:
        project: Project name to search in
        query: Natural language search query
        tags: Optional tag filter — only return memories with at least one of these tags
        limit: Maximum number of results (default 5)
        mode: "full" (default) returns complete content; "probe" returns compact
              summaries — use probe to scan relevance, then expand_memories for details
        min_score: Minimum similarity score to include (default 0.0, suppresses negative scores)
        after: Optional ISO 8601 date — only return memories created after this date
        before: Optional ISO 8601 date — only return memories created before this date
    """
    tags = _normalize_tags(tags)
    store = pool.get_store(project)
    results = store.search(query=query, tags=tags, limit=limit, after=after, before=before)
    # ... rest unchanged ...
```

**Step 7: Run all tests**

Run: `pytest tests/test_server.py tests/test_store.py -v`
Expected: ALL PASS

**Step 8: Commit**

```bash
git add src/annal/store.py src/annal/server.py tests/test_store.py tests/test_server.py
git commit -m "feat: temporal filtering — after/before date params on search_memories"
```

---

### Task 8: Structured JSON output

**Files:**
- Modify: `src/annal/server.py:190-269` (search_memories + expand_memories)
- Test: `tests/test_server.py`

**Step 1: Write the failing tests**

Add to `tests/test_server.py`:

```python
@pytest.mark.asyncio
async def test_search_json_output(mcp):
    """output='json' should return valid JSON with results and meta."""
    import json

    await _call(mcp, "store_memory", {
        "project": "test",
        "content": "JSON output test memory",
        "tags": ["test"],
        "source": "test-source",
    })

    result = await _call(mcp, "search_memories", {
        "project": "test",
        "query": "JSON output",
        "output": "json",
    })

    data = json.loads(result)
    assert "results" in data
    assert "meta" in data
    assert len(data["results"]) > 0
    assert data["meta"]["project"] == "test"
    assert data["meta"]["mode"] == "full"
    assert data["meta"]["query"] == "JSON output"

    first = data["results"][0]
    assert "id" in first
    assert "content" in first
    assert "tags" in first
    assert "score" in first
    assert "source" in first
    assert "created_at" in first


@pytest.mark.asyncio
async def test_search_json_probe_mode(mcp):
    """output='json' + mode='probe' should include content_preview but not full content."""
    import json

    long_content = "A" * 300 + " unique marker"
    await _call(mcp, "store_memory", {
        "project": "test",
        "content": long_content,
        "tags": ["test"],
    })

    result = await _call(mcp, "search_memories", {
        "project": "test",
        "query": "AAAA",
        "mode": "probe",
        "output": "json",
    })

    data = json.loads(result)
    first = data["results"][0]
    assert "content_preview" in first
    assert len(first["content_preview"]) <= 200
    assert "content" not in first


@pytest.mark.asyncio
async def test_search_text_output_unchanged(mcp):
    """output='text' (default) should return the same format as before."""
    await _call(mcp, "store_memory", {
        "project": "test",
        "content": "Text output test",
        "tags": ["test"],
    })

    result = await _call(mcp, "search_memories", {
        "project": "test",
        "query": "Text output",
    })

    # Should be the existing text format, not JSON
    assert "[test]" in result
    assert "results" not in result or "1 results" in result


@pytest.mark.asyncio
async def test_expand_json_output(mcp):
    """expand_memories with output='json' should return structured results."""
    import json

    store_result = await _call(mcp, "store_memory", {
        "project": "test",
        "content": "Expand JSON test",
        "tags": ["test"],
    })
    mem_id = store_result.split("Stored memory ")[-1].strip()

    result = await _call(mcp, "expand_memories", {
        "project": "test",
        "memory_ids": [mem_id],
        "output": "json",
    })

    data = json.loads(result)
    assert "results" in data
    assert len(data["results"]) == 1
    assert data["results"][0]["id"] == mem_id
    assert data["results"][0]["content"] == "Expand JSON test"
    assert "score" not in data["results"][0]
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/test_server.py::test_search_json_output tests/test_server.py::test_search_json_probe_mode tests/test_server.py::test_expand_json_output -v`
Expected: FAIL — search_memories got unexpected keyword argument 'output'

**Step 3: Implement JSON output in search_memories**

In `src/annal/server.py`, update `search_memories` to add the `output` parameter and a JSON formatting path:

```python
@mcp.tool()
def search_memories(
    project: str,
    query: str,
    tags: list[str] | str | None = None,
    limit: int = 5,
    mode: str = "full",
    min_score: float = 0.0,
    after: str | None = None,
    before: str | None = None,
    output: str = "text",
) -> str:
    """Search project memories using natural language.

    Args:
        project: Project name to search in
        query: Natural language search query
        tags: Optional tag filter — only return memories with at least one of these tags
        limit: Maximum number of results (default 5)
        mode: "full" (default) returns complete content; "probe" returns compact
              summaries — use probe to scan relevance, then expand_memories for details
        min_score: Minimum similarity score to include (default 0.0, suppresses negative scores)
        after: Optional ISO 8601 date — only return memories created after this date
        before: Optional ISO 8601 date — only return memories created before this date
        output: "text" (default) for formatted text, "json" for structured JSON
    """
    tags = _normalize_tags(tags)
    store = pool.get_store(project)
    results = store.search(query=query, tags=tags, limit=limit, after=after, before=before)
    if not results:
        if output == "json":
            return json.dumps({"results": [], "meta": {"query": query, "mode": mode, "project": project, "total": 0, "returned": 0}})
        return f"[{project}] No matching memories found."

    results = [r for r in results if r["score"] >= min_score]
    if not results:
        if output == "json":
            return json.dumps({"results": [], "meta": {"query": query, "mode": mode, "project": project, "total": 0, "returned": 0}})
        return f"[{project}] No matching memories found."

    if output == "json":
        json_results = []
        for r in results:
            entry = {
                "id": r["id"],
                "tags": r["tags"],
                "score": round(r["score"], 4),
                "source": r["source"],
                "created_at": r["created_at"],
                "updated_at": r.get("updated_at", ""),
            }
            if mode == "probe":
                entry["content_preview"] = r["content"][:200]
            else:
                entry["content"] = r["content"]
                entry["content_preview"] = r["content"][:200]
            json_results.append(entry)
        return json.dumps({
            "results": json_results,
            "meta": {
                "query": query,
                "mode": mode,
                "project": project,
                "total": len(results),
                "returned": len(results),
            },
        })

    # Existing text formatting below — unchanged
    lines = []
    # ... rest of existing text formatting ...
```

Add `import json` at the top of server.py if not already present.

**Step 4: Implement JSON output in expand_memories**

```python
@mcp.tool()
def expand_memories(project: str, memory_ids: list[str], output: str = "text") -> str:
    """Retrieve full content for specific memories by ID.

    Use after a probe-mode search to fetch details for relevant results.

    Args:
        project: Project name the memories belong to
        memory_ids: List of memory IDs to expand
        output: "text" (default) for formatted text, "json" for structured JSON
    """
    store = pool.get_store(project)
    results = store.get_by_ids(memory_ids)
    if not results:
        if output == "json":
            return json.dumps({"results": []})
        return f"[{project}] No memories found for the given IDs."

    if output == "json":
        json_results = []
        for r in results:
            json_results.append({
                "id": r["id"],
                "content": r["content"],
                "tags": r["tags"],
                "source": r["source"],
                "created_at": r["created_at"],
                "updated_at": r.get("updated_at", ""),
            })
        return json.dumps({"results": json_results})

    # Existing text formatting below — unchanged
    lines = []
    # ... rest unchanged ...
```

**Step 5: Run all tests**

Run: `pytest tests/test_server.py -v`
Expected: ALL PASS

**Step 6: Run full suite**

Run: `pytest -v`
Expected: ALL PASS

**Step 7: Commit**

```bash
git add src/annal/server.py tests/test_server.py
git commit -m "feat: structured JSON output — output='json' param on search_memories and expand_memories"
```

---

### Task 9: Update SERVER_INSTRUCTIONS + version bump + final verification

**Files:**
- Modify: `src/annal/server.py` (SERVER_INSTRUCTIONS — document new params)
- Modify: `pyproject.toml` (version bump to 0.3.0)
- Modify: `docs/plans/2026-02-20-feature-backlog.md` (mark shipped items)

**Step 1: Update SERVER_INSTRUCTIONS**

Add to the "Searching" section in SERVER_INSTRUCTIONS:

```
## Temporal filtering
Scope searches by date using `after` and `before` (ISO 8601 dates):
  search_memories(query="auth decision", after="2026-02-01", before="2026-02-28")

## Structured output
For programmatic access, use output="json" to get structured results:
  search_memories(query="...", output="json")
Returns {"results": [...], "meta": {...}} instead of formatted text.
```

**Step 2: Bump version**

In `pyproject.toml`, change `version = "0.2.0"` to `version = "0.3.0"`.

**Step 3: Mark shipped items in backlog**

In `docs/plans/2026-02-20-feature-backlog.md`, mark the shipped items with strikethrough under a new "Shipped (spike 5)" section.

**Step 4: Run the full test suite one final time**

Run: `pytest -v`
Expected: ALL PASS, increased test count (should be ~85-90)

**Step 5: Commit**

```bash
git add src/annal/server.py pyproject.toml docs/plans/2026-02-20-feature-backlog.md
git commit -m "chore: bump to 0.3.0, update SERVER_INSTRUCTIONS and backlog for spike 5 release"
```
