# Spike 4: Async Indexing, Operational Visibility, Thread Safety — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Make `init_project` and `index_files` non-blocking, add operational visibility for indexing progress, fix thread safety issues, and clean up spike 3 code review findings.

**Architecture:** Background threads for reconciliation with per-project locks. Progress events pushed through the existing EventBus. New `index_status` MCP tool for agents to poll. Dashboard badges for indexing state.

**Tech Stack:** Python 3.11+, threading, FastMCP, ChromaDB, HTMX SSE, Starlette, Jinja2, pytest

---

## Context for the implementer

Annal is a semantic memory MCP server. Agents call MCP tools (`init_project`, `index_files`, `store_memory`, `search_memories`, etc.) to store and retrieve knowledge. The server runs as an HTTP daemon. File indexing scans directories, chunks markdown by heading, and embeds each chunk into ChromaDB.

The problem: `init_project` and `index_files` run reconciliation synchronously. On the kubernetes repo (~80k files), this blocked the entire server for 15+ minutes. Multiple agents trying to use the server simultaneously got starved.

Key files you'll work with:
- `src/annal/pool.py` — StorePool manages per-project MemoryStore and FileWatcher instances
- `src/annal/server.py` — MCP tool definitions, `init_project` and `index_files` tools
- `src/annal/events.py` — EventBus for SSE dashboard updates
- `src/annal/store.py` — ChromaDB wrapper
- `src/annal/watcher.py` — File watcher and reconciliation logic
- `src/annal/indexer.py` — Markdown chunking and file indexing
- `src/annal/config.py` — YAML config management
- `src/annal/cli.py` — Install/uninstall CLI
- `src/annal/dashboard/routes.py` — Dashboard HTTP routes
- `src/annal/dashboard/templates/` — Jinja2 templates

Run tests with: `pytest -v` from the project root. The project uses `pip install -e ".[dev]"` for editable install.

---

### Task 1: Thread safety — EventBus lock

The EventBus `_queues` list is mutated from multiple threads (MCP tool handlers push, SSE clients subscribe/unsubscribe). CPython's GIL makes this safe in practice, but we need a lock by contract.

**Files:**
- Modify: `src/annal/events.py:20-49`
- Test: `tests/test_dashboard.py`

**Step 1: Write the failing test**

Add to `tests/test_dashboard.py`:

```python
def test_event_bus_thread_safety():
    """EventBus should handle concurrent subscribe/push/unsubscribe without errors."""
    import threading

    from annal.events import EventBus, Event

    bus = EventBus()
    errors = []

    def subscriber():
        try:
            q = bus.subscribe()
            for _ in range(100):
                bus.push(Event(type="test", project="t"))
            bus.unsubscribe(q)
        except Exception as e:
            errors.append(e)

    threads = [threading.Thread(target=subscriber) for _ in range(10)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert errors == [], f"Thread safety errors: {errors}"
```

**Step 2: Run test to verify it passes (it will — GIL protects us)**

Run: `pytest tests/test_dashboard.py::test_event_bus_thread_safety -v`
Expected: PASS (this is a correctness-by-contract change, not fixing a visible bug)

**Step 3: Add threading.Lock to EventBus**

In `src/annal/events.py`, add `import threading` and a `self._lock = threading.Lock()` in `__init__`. Wrap `self._queues.append(q)` in subscribe, `self._queues.remove(q)` in unsubscribe, and `list(self._queues)` in push with `with self._lock:`.

```python
import threading

class EventBus:
    def __init__(self) -> None:
        self._queues: list[queue.Queue[Event]] = []
        self._lock = threading.Lock()

    def subscribe(self) -> queue.Queue[Event]:
        q: queue.Queue[Event] = queue.Queue(maxsize=256)
        with self._lock:
            self._queues.append(q)
        return q

    def unsubscribe(self, q: queue.Queue[Event]) -> None:
        with self._lock:
            try:
                self._queues.remove(q)
            except ValueError:
                pass

    def push(self, event: Event) -> None:
        with self._lock:
            snapshot = list(self._queues)
        for q in snapshot:
            try:
                q.put_nowait(event)
            except queue.Full:
                logger.warning("SSE client queue full, dropping event")
```

**Step 4: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 5: Commit**

```bash
git add src/annal/events.py tests/test_dashboard.py
git commit -m "fix: add threading lock to EventBus for correct concurrent access"
```

---

### Task 2: Thread safety — StorePool lock

The StorePool `_stores` dict is accessed from multiple MCP tool handler threads. Add a lock around dict mutation in `get_store`.

**Files:**
- Modify: `src/annal/pool.py:1-35`
- Test: `tests/test_pool.py` (new file)

**Step 1: Write the failing test**

Create `tests/test_pool.py`:

```python
import threading

import pytest

from annal.config import AnnalConfig
from annal.pool import StorePool


def test_store_pool_concurrent_get_store(tmp_data_dir, tmp_config_path):
    """Multiple threads calling get_store for a new project should not race."""
    config = AnnalConfig(config_path=tmp_config_path, data_dir=tmp_data_dir)
    config.save()
    pool = StorePool(config)

    stores = []
    errors = []

    def get():
        try:
            s = pool.get_store("racetest")
            stores.append(s)
        except Exception as e:
            errors.append(e)

    threads = [threading.Thread(target=get) for _ in range(10)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert errors == [], f"Race condition errors: {errors}"
    # All threads should get the same store instance
    assert len(set(id(s) for s in stores)) == 1
```

**Step 2: Run test to verify behavior**

Run: `pytest tests/test_pool.py::test_store_pool_concurrent_get_store -v`
Expected: May pass due to GIL, but correctness requires a lock

**Step 3: Add threading.Lock to StorePool**

In `src/annal/pool.py`, add `import threading` and `self._lock = threading.Lock()` in `__init__`. Wrap the `get_store` body with `with self._lock:`.

```python
import threading

class StorePool:
    def __init__(self, config: AnnalConfig) -> None:
        self._config = config
        self._stores: dict[str, MemoryStore] = {}
        self._watchers: dict[str, FileWatcher] = {}
        self._lock = threading.Lock()

    def get_store(self, project: str) -> MemoryStore:
        with self._lock:
            if project not in self._stores:
                logger.info("Creating store for project '%s'", project)
                self._stores[project] = MemoryStore(
                    data_dir=self._config.data_dir, project=project
                )
                if project not in self._config.projects:
                    self._config.add_project(project)
                    self._config.save()
                    logger.info("Auto-registered project '%s' in config", project)
            return self._stores[project]
```

**Step 4: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 5: Commit**

```bash
git add src/annal/pool.py tests/test_pool.py
git commit -m "fix: add threading lock to StorePool for concurrent get_store safety"
```

---

### Task 3: Per-project indexing locks and async reconciliation

This is the core change. Add per-project `threading.Lock` instances for indexing, and make `init_project` and `index_files` kick off reconciliation on background threads instead of blocking.

**Files:**
- Modify: `src/annal/pool.py`
- Modify: `src/annal/server.py:319-373` (init_project and index_files tools)
- Modify: `src/annal/watcher.py:106-137` (reconcile method — add progress callback)
- Test: `tests/test_pool.py`
- Test: `tests/test_server.py`

**Step 1: Write the failing tests**

Add to `tests/test_pool.py`:

```python
import time


def test_reconcile_project_async(tmp_data_dir, tmp_config_path, tmp_path):
    """reconcile_project_async should return immediately and reconcile in the background."""
    watch_dir = tmp_path / "docs"
    watch_dir.mkdir()
    (watch_dir / "test.md").write_text("# Test\nSome content\n")

    config = AnnalConfig(config_path=tmp_config_path, data_dir=tmp_data_dir)
    config.add_project("asynctest", watch_paths=[str(watch_dir)])
    config.save()
    pool = StorePool(config)

    # Should return immediately
    pool.reconcile_project_async("asynctest")

    # Give the background thread time to finish
    time.sleep(2)

    store = pool.get_store("asynctest")
    assert store.count() > 0


def test_is_indexing(tmp_data_dir, tmp_config_path, tmp_path):
    """is_indexing should report True while reconciliation is running."""
    watch_dir = tmp_path / "docs"
    watch_dir.mkdir()
    (watch_dir / "test.md").write_text("# Test\nContent\n")

    config = AnnalConfig(config_path=tmp_config_path, data_dir=tmp_data_dir)
    config.add_project("idxtest", watch_paths=[str(watch_dir)])
    config.save()
    pool = StorePool(config)

    # Before any indexing
    assert pool.is_indexing("idxtest") is False
```

Add to `tests/test_server.py`:

```python
@pytest.mark.asyncio
async def test_init_project_returns_immediately(server_env):
    """init_project should return immediately with 'Indexing in progress' message."""
    mcp = create_server(config_path=server_env["config_path"])
    result = await _call(mcp, "init_project", {
        "project_name": "asyncinit",
        "watch_paths": [server_env["watch_dir"]],
    })
    assert "Indexing in progress" in result or "initialized" in result.lower()


@pytest.mark.asyncio
async def test_index_files_returns_immediately(server_env):
    """index_files should return immediately with progress message."""
    mcp = create_server(config_path=server_env["config_path"])
    # First init the project
    await _call(mcp, "init_project", {
        "project_name": "asyncidx",
        "watch_paths": [server_env["watch_dir"]],
    })
    import time
    time.sleep(1)  # let init reconcile finish

    result = await _call(mcp, "index_files", {"project": "asyncidx"})
    # Should return immediately — message indicates async operation
    assert "asyncidx" in result.lower() or "index" in result.lower()
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/test_pool.py::test_reconcile_project_async tests/test_pool.py::test_is_indexing tests/test_server.py::test_init_project_returns_immediately tests/test_server.py::test_index_files_returns_immediately -v`
Expected: FAIL — `reconcile_project_async` and `is_indexing` don't exist yet

**Step 3: Add progress callback to FileWatcher.reconcile**

In `src/annal/watcher.py`, add an optional `progress_callback` parameter to `reconcile()` that gets called every 50 files with the current count:

```python
def reconcile(self, progress_callback: Callable[[int], None] | None = None) -> int:
    """Scan all watch paths and index new or changed files. Returns file count."""
    total = 0
    skipped = 0
    for watch_path in self._config.watch_paths:
        root = Path(watch_path)
        if not root.exists():
            continue
        for path in root.rglob("*"):
            if path.is_dir():
                continue
            try:
                rel = str(path.relative_to(root))
                if not matches_patterns(rel, self._config.watch_patterns, self._config.watch_exclude):
                    continue

                file_path = str(path)
                current_mtime = path.stat().st_mtime
                stored_mtime = self._store.get_file_mtime(f"file:{file_path}")

                if stored_mtime is not None and stored_mtime == current_mtime:
                    skipped += 1
                    continue

                index_file(self._store, file_path, file_mtime=current_mtime)
                total += 1

                if progress_callback and total % 50 == 0:
                    progress_callback(total)
            except Exception:
                logger.exception("Failed to reconcile file: %s", path)

    if skipped:
        logger.info("Skipped %d unchanged files", skipped)
    return total
```

Add `from collections.abc import Callable` to the imports at the top of watcher.py.

**Step 4: Add async reconciliation and per-project locks to StorePool**

In `src/annal/pool.py`:

```python
import threading
from collections import defaultdict
from datetime import datetime, timezone

class StorePool:
    def __init__(self, config: AnnalConfig) -> None:
        self._config = config
        self._stores: dict[str, MemoryStore] = {}
        self._watchers: dict[str, FileWatcher] = {}
        self._lock = threading.Lock()
        self._index_locks: dict[str, threading.Lock] = defaultdict(threading.Lock)
        self._last_reconcile: dict[str, dict] = {}  # project -> {timestamp, file_count}

    def get_store(self, project: str) -> MemoryStore:
        with self._lock:
            if project not in self._stores:
                logger.info("Creating store for project '%s'", project)
                self._stores[project] = MemoryStore(
                    data_dir=self._config.data_dir, project=project
                )
                if project not in self._config.projects:
                    self._config.add_project(project)
                    self._config.save()
                    logger.info("Auto-registered project '%s' in config", project)
            return self._stores[project]

    def reconcile_project(self, project: str) -> int:
        """Reconcile file indexes for a project (blocking). Returns file count."""
        if project not in self._config.projects:
            return 0
        store = self.get_store(project)
        proj_config = self._config.projects[project]
        watcher = FileWatcher(store=store, project_config=proj_config)
        count = watcher.reconcile()
        self._last_reconcile[project] = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "file_count": count,
        }
        logger.info("Reconciled %d files for project '%s'", count, project)
        return count

    def reconcile_project_async(
        self, project: str, on_progress: Callable[[int], None] | None = None,
        on_complete: Callable[[int], None] | None = None,
        clear_first: bool = False,
    ) -> None:
        """Kick off reconciliation on a background thread. Returns immediately."""
        def _run():
            lock = self._index_locks[project]
            if not lock.acquire(blocking=False):
                logger.info("Indexing already in progress for '%s', queuing", project)
                lock.acquire()  # wait for the other to finish, then run
            try:
                if clear_first:
                    store = self.get_store(project)
                    store.delete_by_source("file:")
                count = self.reconcile_project(project)
                if on_progress:
                    # Wire up progress through the reconcile — for now just fire complete
                    pass
                if on_complete:
                    on_complete(count)
            finally:
                lock.release()

        # Actually wire progress into the reconcile call
        def _run_with_progress():
            lock = self._index_locks[project]
            if not lock.acquire(blocking=False):
                logger.info("Indexing already in progress for '%s', waiting", project)
                lock.acquire()
            try:
                if project not in self._config.projects:
                    return
                store = self.get_store(project)
                if clear_first:
                    store.delete_by_source("file:")
                proj_config = self._config.projects[project]
                watcher = FileWatcher(store=store, project_config=proj_config)
                count = watcher.reconcile(progress_callback=on_progress)
                self._last_reconcile[project] = {
                    "timestamp": datetime.now(timezone.utc).isoformat(),
                    "file_count": count,
                }
                logger.info("Reconciled %d files for project '%s'", count, project)
                if on_complete:
                    on_complete(count)
            finally:
                lock.release()

        thread = threading.Thread(target=_run_with_progress, daemon=True)
        thread.start()

    def is_indexing(self, project: str) -> bool:
        """Check if a project is currently being indexed."""
        lock = self._index_locks[project]
        acquired = lock.acquire(blocking=False)
        if acquired:
            lock.release()
            return False
        return True

    def get_last_reconcile(self, project: str) -> dict | None:
        """Get the last reconcile info for a project."""
        return self._last_reconcile.get(project)

    def start_watcher(self, project: str) -> None:
        # ... unchanged ...

    def shutdown(self) -> None:
        # ... unchanged ...
```

Add `from collections import defaultdict` and `from collections.abc import Callable` and `from datetime import datetime, timezone` to imports.

**Step 5: Update init_project and index_files in server.py to use async reconciliation**

In `src/annal/server.py`, modify the `init_project` tool:

```python
@mcp.tool()
def init_project(
    project_name: str,
    watch_paths: list[str] | None = None,
    watch_patterns: list[str] | None = None,
    watch_exclude: list[str] | None = None,
) -> str:
    config.add_project(
        project_name,
        watch_paths=watch_paths,
        watch_patterns=watch_patterns,
        watch_exclude=watch_exclude,
    )
    config.save()
    proj = config.projects[project_name]
    if watch_paths:
        def on_progress(count: int) -> None:
            event_bus.push(Event(type="index_progress", project=project_name, detail=f"{count} files"))

        def on_complete(count: int) -> None:
            pool.start_watcher(project_name)
            event_bus.push(Event(type="index_complete", project=project_name, detail=f"{count} files"))

        event_bus.push(Event(type="index_started", project=project_name))
        pool.reconcile_project_async(
            project_name,
            on_progress=on_progress,
            on_complete=on_complete,
        )
        return (
            f"Project '{project_name}' initialized. "
            f"Indexing in progress — use index_status to check progress. "
            f"Patterns: {proj.watch_patterns}, excludes: {proj.watch_exclude}."
        )
    return (
        f"Project '{project_name}' initialized with "
        f"watch paths: {proj.watch_paths}, "
        f"patterns: {proj.watch_patterns}, "
        f"excludes: {proj.watch_exclude}."
    )
```

Modify the `index_files` tool:

```python
@mcp.tool()
def index_files(project: str) -> str:
    if project not in config.projects:
        return f"[{project}] No watch paths configured. Use init_project first."

    if pool.is_indexing(project):
        return f"[{project}] Indexing already in progress. Use index_status to check progress."

    def on_progress(count: int) -> None:
        event_bus.push(Event(type="index_progress", project=project, detail=f"{count} files"))

    def on_complete(count: int) -> None:
        event_bus.push(Event(type="index_complete", project=project, detail=f"{count} files"))

    event_bus.push(Event(type="index_started", project=project))
    pool.reconcile_project_async(
        project,
        on_progress=on_progress,
        on_complete=on_complete,
        clear_first=True,
    )
    return f"[{project}] Re-indexing started in background. Use index_status to check progress."
```

**Step 6: Run all tests**

Run: `pytest -v`
Expected: All pass. Some existing tests (like `test_index_files_clears_stale_chunks`) may need a `time.sleep(2)` added since indexing is now async. Adjust as needed.

**Step 7: Commit**

```bash
git add src/annal/pool.py src/annal/server.py src/annal/watcher.py tests/test_pool.py tests/test_server.py
git commit -m "feat: async indexing — init_project and index_files return immediately"
```

---

### Task 4: index_status MCP tool

New tool that returns per-project diagnostics so agents know what's happening.

**Files:**
- Modify: `src/annal/server.py`
- Test: `tests/test_server.py`

**Step 1: Write the failing test**

Add to `tests/test_server.py`:

```python
@pytest.mark.asyncio
async def test_index_status(mcp):
    """index_status should return project diagnostics."""
    # Store something so there's data
    await _call(mcp, "store_memory", {
        "project": "statustest",
        "content": "Some memory",
        "tags": ["test"],
    })

    result = await _call(mcp, "index_status", {"project": "statustest"})
    assert "statustest" in result
    assert "chunks" in result.lower() or "total" in result.lower()
```

**Step 2: Run test to verify it fails**

Run: `pytest tests/test_server.py::test_index_status -v`
Expected: FAIL — tool doesn't exist

**Step 3: Implement index_status tool**

Add to `src/annal/server.py` after the `index_files` tool:

```python
@mcp.tool()
def index_status(project: str) -> str:
    """Check indexing status and collection diagnostics for a project.

    Args:
        project: Project name to check
    """
    store = pool.get_store(project)
    total = store.count()
    stats = store.stats()
    indexing = pool.is_indexing(project)
    last = pool.get_last_reconcile(project)

    lines = [f"[{project}] Status:"]
    if indexing:
        lines.append("  Indexing: IN PROGRESS")
    else:
        lines.append("  Indexing: idle")
    lines.append(f"  Total chunks: {total}")
    lines.append(f"  File-indexed: {stats['by_type'].get('file-indexed', 0)}")
    lines.append(f"  Agent memories: {stats['by_type'].get('agent-memory', 0)}")
    if last:
        lines.append(f"  Last reconcile: {last['timestamp']} ({last['file_count']} files)")
    else:
        lines.append("  Last reconcile: never")
    return "\n".join(lines)
```

**Step 4: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 5: Commit**

```bash
git add src/annal/server.py tests/test_server.py
git commit -m "feat: index_status tool — per-project diagnostics for agents"
```

---

### Task 5: Dashboard indexing visibility

Add an "Indexing..." badge on the project cards and a progress counter driven by SSE events.

**Files:**
- Modify: `src/annal/dashboard/templates/index.html`
- Modify: `src/annal/dashboard/static/style.css`
- Test: `tests/test_dashboard.py`

**Step 1: Write the test**

Add to `tests/test_dashboard.py`:

```python
def test_index_page_has_sse_connection(dashboard_client):
    """The index page should include SSE connection for live updates."""
    response = dashboard_client.get("/")
    assert response.status_code == 200
    html = response.text
    assert "sse-connect" in html or "events" in html
```

**Step 2: Run test to verify it fails**

Run: `pytest tests/test_dashboard.py::test_index_page_has_sse_connection -v`
Expected: FAIL — index.html doesn't have SSE connection yet

**Step 3: Add SSE connection and indexing badge to index.html**

Update `src/annal/dashboard/templates/index.html`. Add an SSE connection div and an indexing badge on each project card. The badge is hidden by default and shown/hidden via JS listeners on SSE events:

```html
{% extends "base.html" %}

{% block title %}Annal - Projects{% endblock %}

{% block content %}
<div class="page-header">
  <h1>Projects</h1>
</div>

<div hx-ext="sse" sse-connect="/events" style="display:none"></div>

{% if project_stats %}
<div class="card-grid">
  {% for p in project_stats %}
  <a href="/memories?project={{ p.name }}" class="project-card" id="card-{{ p.name }}">
    <div class="card-header">
      <h2 class="card-title">{{ p.name }}</h2>
      <span class="indexing-badge" id="indexing-{{ p.name }}" style="display:none">indexing…</span>
      <span class="card-count">{{ p.total }}</span>
    </div>
    <div class="card-stats">
      <div class="stat">
        <span class="stat-value">{{ p.agent_memory }}</span>
        <span class="stat-label">agent</span>
      </div>
      <div class="stat">
        <span class="stat-value">{{ p.file_indexed }}</span>
        <span class="stat-label">indexed</span>
      </div>
    </div>
    {% if p.top_tags %}
    <div class="card-tags">
      {% for tag, count in p.top_tags %}
      <span class="tag-pill">{{ tag }} <span class="tag-count">{{ count }}</span></span>
      {% endfor %}
    </div>
    {% endif %}
  </a>
  {% endfor %}
</div>
{% else %}
<div class="empty-state">
  <div class="empty-icon">&gt;_</div>
  <p>No projects configured yet.</p>
  <p class="empty-hint">Use the <code>init_project</code> MCP tool to add your first project.</p>
</div>
{% endif %}

<script>
document.body.addEventListener('sse:index_started', function(e) {
  var project = e.detail.value.split('|')[0];
  var badge = document.getElementById('indexing-' + project);
  if (badge) { badge.style.display = 'inline'; badge.textContent = 'indexing…'; }
});
document.body.addEventListener('sse:index_progress', function(e) {
  var parts = e.detail.value.split('|');
  var project = parts[0];
  var detail = parts[1] || '';
  var badge = document.getElementById('indexing-' + project);
  if (badge) { badge.style.display = 'inline'; badge.textContent = detail; }
});
document.body.addEventListener('sse:index_complete', function(e) {
  var project = e.detail.value.split('|')[0];
  var badge = document.getElementById('indexing-' + project);
  if (badge) { badge.style.display = 'none'; }
});
</script>
{% endblock %}
```

**Step 4: Add indexing badge CSS**

Add to `src/annal/dashboard/static/style.css`:

```css
.indexing-badge {
  font-size: 0.75rem;
  color: var(--accent);
  background: var(--surface);
  padding: 2px 8px;
  border-radius: 4px;
  animation: pulse 1.5s ease-in-out infinite;
}
```

(The `pulse` animation and `--accent`/`--surface` variables already exist in the stylesheet from spike 3.)

**Step 5: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 6: Commit**

```bash
git add src/annal/dashboard/templates/index.html src/annal/dashboard/static/style.css tests/test_dashboard.py
git commit -m "feat: dashboard indexing badge — live progress via SSE on project cards"
```

---

### Task 6: SSE slow client fix

Replace `q.get()` (blocks indefinitely) with `q.get(timeout=30)` in a loop so the thread can check for cancellation.

**Files:**
- Modify: `src/annal/dashboard/routes.py:219-238`

**Step 1: Update the SSE endpoint**

In `src/annal/dashboard/routes.py`, change the `events` handler:

```python
async def events(request: Request) -> Response:
    """SSE endpoint for live dashboard updates."""
    q = event_bus.subscribe()
    loop = asyncio.get_running_loop()

    async def generate():
        try:
            while True:
                try:
                    event = await loop.run_in_executor(
                        None, lambda: q.get(timeout=30)
                    )
                    yield f"event: {event.type}\ndata: {event.project}|{event.detail}\n\n"
                except Exception:
                    # queue.Empty on timeout — send keepalive comment
                    yield ": keepalive\n\n"
        except asyncio.CancelledError:
            pass
        finally:
            event_bus.unsubscribe(q)

    return StreamingResponse(generate(), media_type="text/event-stream", headers={
        "Cache-Control": "no-cache",
        "Connection": "keep-alive",
    })
```

Add `import queue as queue_module` at the top of routes.py if you need to catch `queue.Empty` specifically, or just use bare `except Exception` since the only exception `q.get(timeout=N)` raises is `queue.Empty`.

**Step 2: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 3: Commit**

```bash
git add src/annal/dashboard/routes.py
git commit -m "fix: SSE endpoint uses get(timeout=30) to avoid blocking threads indefinitely"
```

---

### Task 7: updated_at consistency

Add `updated_at` to `search()` and `browse()` return values — currently only `get_by_ids()` returns it.

**Files:**
- Modify: `src/annal/store.py:44-85` (search method)
- Modify: `src/annal/store.py:189-233` (browse method)
- Test: `tests/test_store.py`

**Step 1: Write the failing test**

Add to `tests/test_store.py`:

```python
def test_search_returns_updated_at(tmp_data_dir):
    store = MemoryStore(data_dir=tmp_data_dir, project="updated_at_search")
    mem_id = store.store(content="Will be updated", tags=["test"])
    store.update(mem_id, content="Updated content")

    results = store.search("Updated content", limit=1)
    assert len(results) == 1
    assert "updated_at" in results[0]
    assert results[0]["updated_at"] != ""


def test_browse_returns_updated_at(tmp_data_dir):
    store = MemoryStore(data_dir=tmp_data_dir, project="updated_at_browse")
    mem_id = store.store(content="Will be updated", tags=["test"])
    store.update(mem_id, content="Updated content")

    results, total = store.browse()
    assert total == 1
    assert "updated_at" in results[0]
    assert results[0]["updated_at"] != ""
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/test_store.py::test_search_returns_updated_at tests/test_store.py::test_browse_returns_updated_at -v`
Expected: FAIL — `updated_at` not in search/browse results

**Step 3: Add updated_at to search() and browse()**

In `src/annal/store.py`, in the `search` method, add `"updated_at": meta.get("updated_at", ""),` to the dict in the `memories.append(...)` call (around line 83).

In the `browse` method, add `"updated_at": meta.get("updated_at", ""),` to the dict in the `all_items.append(...)` call (around line 229).

**Step 4: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 5: Commit**

```bash
git add src/annal/store.py tests/test_store.py
git commit -m "fix: include updated_at in search() and browse() results for consistency"
```

---

### Task 8: HTMX SSE trigger cleanup

The `memories.html` template uses both `sse-swap` and `hx-trigger` for the same event, which is confusing. Simplify.

**Files:**
- Modify: `src/annal/dashboard/templates/memories.html:79-86`

**Step 1: Simplify the SSE listener div**

Replace the current SSE div in `memories.html`:

```html
<div hx-ext="sse" sse-connect="/events"
     hx-trigger="sse:memory_stored, sse:memory_deleted, sse:index_complete"
     hx-get="/memories/table"
     hx-target="#memory-table"
     hx-include="[name]"
     style="display:none"></div>
```

This removes the redundant `sse-swap` attribute and keeps only `hx-trigger` with SSE events.

**Step 2: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 3: Commit**

```bash
git add src/annal/dashboard/templates/memories.html
git commit -m "fix: simplify HTMX SSE triggers — remove redundant sse-swap"
```

---

### Task 9: Heading depth fix

Change the markdown chunking regex from `#{1,3}` to `#{1,6}` so `####` through `######` headings are recognized as chunk boundaries.

**Files:**
- Modify: `src/annal/indexer.py:23`
- Test: `tests/test_indexer.py` (check if file exists, may need to create)

**Step 1: Write the failing test**

Check if `tests/test_indexer.py` exists. If not, create it. Add:

```python
from annal.indexer import chunk_markdown


def test_chunk_markdown_recognizes_h4_through_h6():
    """Headings #### through ###### should create chunk boundaries."""
    content = """# Top Level
Intro text

## Section
Section text

### Subsection
Sub text

#### Detail
Detail text

##### Fine Detail
Fine detail text

###### Finest Detail
Finest detail text
"""
    chunks = chunk_markdown(content, "test.md")
    headings = [c["heading"] for c in chunks]
    # All heading levels should create separate chunks
    assert any("Detail" in h for h in headings)
    assert any("Fine Detail" in h for h in headings)
    assert any("Finest Detail" in h for h in headings)
```

**Step 2: Run test to verify it fails**

Run: `pytest tests/test_indexer.py::test_chunk_markdown_recognizes_h4_through_h6 -v`
Expected: FAIL — h4-h6 headings not matched

**Step 3: Fix the regex**

In `src/annal/indexer.py` line 23, change:

```python
heading_match = re.match(r"^(#{1,3})\s+(.+)$", line)
```

to:

```python
heading_match = re.match(r"^(#{1,6})\s+(.+)$", line)
```

**Step 4: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 5: Commit**

```bash
git add src/annal/indexer.py tests/test_indexer.py
git commit -m "fix: recognize #### through ###### as markdown chunk boundaries"
```

---

### Task 10: Spike 3 test gaps

Fill the test gaps identified in code review: nonexistent ID update, no-op update guard, install idempotency.

**Files:**
- Test: `tests/test_server.py`
- Test: `tests/test_store.py`
- Test: `tests/test_cli.py`

**Step 1: Write the tests**

Add to `tests/test_server.py`:

```python
@pytest.mark.asyncio
async def test_update_memory_nonexistent_id(mcp):
    """update_memory with a nonexistent ID should return a not-found message."""
    result = await _call(mcp, "update_memory", {
        "project": "test",
        "memory_id": "nonexistent-id-12345",
        "content": "This should fail",
    })
    assert "not found" in result.lower()


@pytest.mark.asyncio
async def test_update_memory_no_op(mcp):
    """update_memory with no content, tags, or source should return a no-op message."""
    result = await _call(mcp, "update_memory", {
        "project": "test",
        "memory_id": "doesnt-matter",
    })
    assert "nothing to update" in result.lower()
```

Add to `tests/test_store.py`:

```python
def test_update_nonexistent_memory_raises(tmp_data_dir):
    store = MemoryStore(data_dir=tmp_data_dir, project="update_missing")
    with pytest.raises(ValueError, match="not found"):
        store.update("nonexistent-id", content="Should fail")
```

Add to `tests/test_cli.py`:

```python
def test_install_idempotent(fake_home):
    """Calling install twice should not crash or duplicate config entries."""
    with patch("annal.cli.Path.home", return_value=fake_home):
        result1 = install(start_service=False)
        result2 = install(start_service=False)

    assert "config.yaml" in result1
    assert "config.yaml" in result2

    # MCP json should still have exactly one annal entry
    mcp_json = fake_home / ".mcp.json"
    data = json.loads(mcp_json.read_text())
    assert "annal" in data["mcpServers"]

    # Codex config should not have duplicate sections
    codex = (fake_home / ".codex" / "config.toml").read_text()
    assert codex.count("[mcp_servers.annal]") == 1
```

**Step 2: Run tests to verify**

Run: `pytest tests/test_server.py::test_update_memory_nonexistent_id tests/test_server.py::test_update_memory_no_op tests/test_store.py::test_update_nonexistent_memory_raises tests/test_cli.py::test_install_idempotent -v`
Expected: All PASS (these test existing behavior that works but wasn't covered)

**Step 3: Commit**

```bash
git add tests/test_server.py tests/test_store.py tests/test_cli.py
git commit -m "test: fill spike 3 test gaps — nonexistent update, no-op guard, install idempotency"
```

---

### Task 11: _annal_executable launchd fix

The `_annal_executable()` fallback returns a single string `"python -m annal.server"` which breaks launchd's ProgramArguments (needs separate elements per argument).

**Files:**
- Modify: `src/annal/cli.py:21-29`
- Modify: `src/annal/cli.py:136-161` (launchd plist generation)
- Test: `tests/test_cli.py`

**Step 1: Write the failing test**

Add to `tests/test_cli.py`:

```python
def test_annal_executable_returns_list():
    """_annal_executable should return a list of strings, not a single string."""
    from annal.cli import _annal_executable
    result = _annal_executable()
    assert isinstance(result, list)
    assert all(isinstance(s, str) for s in result)
    assert len(result) >= 1
```

**Step 2: Run test to verify it fails**

Run: `pytest tests/test_cli.py::test_annal_executable_returns_list -v`
Expected: FAIL — currently returns a `str`

**Step 3: Change _annal_executable to return a list**

In `src/annal/cli.py`:

```python
def _annal_executable() -> list[str]:
    """Find the annal executable path. Returns a list for subprocess/plist compatibility."""
    venv_bin = Path(sys.executable).parent / "annal"
    if venv_bin.exists():
        return [str(venv_bin)]
    found = shutil.which("annal")
    if found:
        return [found]
    return [sys.executable, "-m", "annal.server"]
```

Then update all callers of `_annal_executable()`:

In the systemd service section (Linux), change `ExecStart={exe}` to `ExecStart={" ".join(exe)}` (systemd ExecStart takes a single command string).

In the launchd plist section (Darwin), change the single `<string>{exe}</string>` to generate one `<string>` per element:

```python
exe = _annal_executable()
# ...
prog_args = "\n        ".join(f"<string>{arg}</string>" for arg in exe)
plist_file.write_text(f"""\
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" \
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key><string>com.annal.server</string>
    <key>ProgramArguments</key>
    <array>
        {prog_args}
        <string>--transport</string>
        <string>streamable-http</string>
    </array>
    <key>RunAtLoad</key><true/>
    <key>KeepAlive</key><true/>
    <key>StandardOutPath</key><string>/tmp/annal.stdout.log</string>
    <key>StandardErrorPath</key><string>/tmp/annal.stderr.log</string>
    <key>EnvironmentVariables</key>
    <dict><key>PYTHONUNBUFFERED</key><string>1</string></dict>
</dict>
</plist>
""")
```

**Step 4: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 5: Commit**

```bash
git add src/annal/cli.py tests/test_cli.py
git commit -m "fix: _annal_executable returns list for launchd ProgramArguments compatibility"
```

---

### Task 12: Update startup reconciliation to use async

The `_startup_reconcile` function in `server.py` already runs on a background thread, but it calls `pool.reconcile_project` (blocking). Update it to push events and track last-reconcile metadata.

**Files:**
- Modify: `src/annal/server.py:143-152`

**Step 1: Update _startup_reconcile**

```python
def _startup_reconcile() -> None:
    for project_name in config.projects:
        logger.info("Reconciling project '%s'...", project_name)
        event_bus.push(Event(type="index_started", project=project_name))
        count = pool.reconcile_project(project_name)
        event_bus.push(Event(type="index_complete", project=project_name, detail=f"{count} files"))
        pool.start_watcher(project_name)
    logger.info("Startup reconciliation complete")
```

Note: startup reconciliation stays synchronous within its own background thread (projects are reconciled one after another). This is fine — the thread is already off the main event loop.

**Step 2: Run all tests**

Run: `pytest -v`
Expected: All pass

**Step 3: Commit**

```bash
git add src/annal/server.py
git commit -m "feat: startup reconciliation pushes index events for dashboard visibility"
```

---

### Task 13: Final verification

**Step 1: Run the full test suite**

Run: `pytest -v`
Expected: All tests pass

**Step 2: Check test count**

The test count should be higher than the previous 78. Verify new tests are included.

**Step 3: Manual smoke test (optional)**

Restart the annal service and call `init_project` — it should return immediately. Call `index_status` to see progress. Check the dashboard for indexing badges.

```bash
pip install -e ".[dev]"
systemctl --user restart annal
# Then use curl or an agent to test init_project and index_status
```
