# Design: Incremental Sync

**Date**: 2026-02-17
**Status**: Approved, not yet implemented
**Related**: ADR-007 (sync architecture)

---

## Problem

The engine currently only supports full re-ingestion: clone or read the entire repo, parse every file, embed everything, write all graph nodes. There is no way to process only changed files. This means every commit would trigger a full re-index — unacceptably slow for any repo with more than a few hundred files, and impossible to trigger from a git hook without the user noticing.

Incremental sync must handle three operations on a set of changed files: **add**, **modify**, and **remove**.

---

## Prerequisite: Deterministic Vector IDs

The single biggest blocker to incremental sync is that vector IDs are currently generated as lazy UUIDs. There is no way to compute "what is the Qdrant point ID for `src/foo.py` in repo `org/repo`?" without a lookup table. Deletion is therefore impossible without a round-trip search-by-metadata first, which is fragile.

**Decision**: IDs must be deterministic, computed as:

```python
import hashlib

def make_vector_id(repo_id: str, file_path: str) -> str:
    raw = f"{repo_id}:{file_path}"
    return hashlib.sha256(raw.encode()).hexdigest()
```

This ID is stable across re-ingests for the same file, cheap to compute anywhere without a database, and collision-free within a repo. The `upsert` operation in both the vector store and graph store becomes idempotent: running it twice produces the same state.

This change affects the full ingestion pipeline as well — it should be adopted there too so that a full re-ingest overwrites rather than duplicates existing entries.

---

## What Each Operation Requires

### Added files

Identical to full ingest scoped to one file:

1. Parse with tree-sitter → extract definitions, imports, calls
2. Build file summary → embed → upsert to vector store (deterministic ID)
3. Create file node in graph → create edges for IMPORTS and CALLS relationships

No new protocol methods required.

### Modified files

Same as add, but edges must be refreshed — a file's import statements may have changed.

1. Re-parse → re-embed → upsert to vector store (same ID, overwrites)
2. **Delete all edges where this file is the source** from graph
3. Re-create edges based on new parse

The graph node itself can stay (preserving its `qdrant_id` link). Only its outgoing edges need refreshing.

New protocol method required:

```python
# GraphStore protocol addition
async def delete_file_edges(self, file_path: str, repo_id: str) -> None:
    """Delete all outgoing edges (IMPORTS, CALLS) from a file node."""
    ...
```

### Removed files

1. Delete from vector store by deterministic ID
2. Delete graph node and all its edges (incoming and outgoing)

New protocol methods required:

```python
# VectorStore protocol addition
async def delete(self, id: str) -> None:
    """Delete a vector by ID."""
    ...

# GraphStore protocol addition
async def delete_file_node(self, file_path: str, repo_id: str) -> None:
    """Delete a file node and all its edges (incoming and outgoing)."""
    ...
```

**Dangling edge policy**: when a file is removed, delete all edges to *and* from it. If `bar.py` imports from the now-deleted `foo.py`, the IMPORTS edge pointing at the ghost node is removed. This keeps the graph clean. Nodes representing the *callers* of deleted code are not themselves deleted — only their relationship to the removed file.

---

## Protocol Changes Summary

The following additions are needed across the adapter protocols:

| Protocol | Method | Purpose |
|----------|--------|---------|
| `VectorStore` | `delete(id: str)` | Remove a single vector by ID |
| `GraphStore` | `delete_file_edges(file_path, repo_id)` | Refresh edges on modify |
| `GraphStore` | `delete_file_node(file_path, repo_id)` | Full removal on delete |

These must be implemented in:
- `QdrantStore` and `PgVectorStore` (for `VectorStore.delete`)
- `Neo4jStore` (for both `GraphStore` additions)
- `NoOpGraphStore` (no-op stubs)

---

## IncrementalIngestionPipeline

A new pipeline class alongside the existing `IngestionPipeline`. It takes the delta from the trigger (git hook or platform job) and runs scoped stages.

```python
@dataclass
class ChangeSet:
    repo_id: str
    repo_path: str        # local path to the repo at the relevant commit
    commit_sha: str
    added: list[str]      # relative file paths
    modified: list[str]
    removed: list[str]


class IncrementalIngestionPipeline:
    def __init__(self, vector_store, graph_store, parser): ...

    def run(self, changeset: ChangeSet) -> ProcessingResult:
        return asyncio.run(self._run_async(changeset))

    async def _run_async(self, changeset: ChangeSet) -> ProcessingResult:
        # Process in dependency order:
        # 1. Removals first (clean state before any adds/modifies)
        # 2. Additions
        # 3. Modifications (which are remove-edges + re-add)
        ...
```

Processing order matters: removals go first so that if a file is renamed (appears in both `removed` and `added`), the old entry is cleanly gone before the new one is written.

Stages are scoped versions of the existing pipeline stages — no full repo discovery, no cloning. The repo must already be checked out at the target commit.

---

## Phase Plan

### Phase 1 — Make incremental sync possible

All foundational changes. Nothing user-visible yet.

- [ ] Adopt deterministic ID scheme in `ParsingStage` (affects full ingestion too)
- [ ] Add `VectorStore.delete(id)` to protocol + implement in `QdrantStore`, `PgVectorStore`
- [ ] Add `GraphStore.delete_file_edges(file_path, repo_id)` to protocol + implement in `Neo4jStore`, `NoOpGraphStore`
- [ ] Add `GraphStore.delete_file_node(file_path, repo_id)` to protocol + implement in `Neo4jStore`, `NoOpGraphStore`
- [ ] Implement `IncrementalIngestionPipeline` with `ChangeSet` input
- [ ] Unit tests for each delete operation
- [ ] Integration test: ingest fixture repo, modify a file, verify vector and graph reflect new state

### Phase 2 — Local trigger (git hooks)

Builds on Phase 1. Validates incremental sync end-to-end with a real git workflow.

- [ ] `clew sync` CLI subcommand: accepts `--from <sha> --to <sha>` or defaults to `HEAD~1..HEAD`
- [ ] Git diff parser: produces `ChangeSet` from `git diff --name-status`
- [ ] `clew hooks install` command: writes `post-commit` and `post-merge` hook scripts into `.git/hooks/`
- [ ] `clew hooks uninstall` command
- [ ] End-to-end test: `git commit` in a fixture repo triggers re-index of changed files only

### Phase 3 — Platform WebSocket connection

Builds on Phase 1. Requires platform-side work (job queue + WS endpoint) to be coordinated with `clew-platform`.

- [ ] Platform: WebSocket endpoint at `/ws/engine` — authenticated, per-repo subscription
- [ ] Platform: job queue that buffers jobs when no engine connected, drains on reconnect
- [ ] Engine: `PlatformSyncClient` — outbound WS connection, activated by `CLEW_PLATFORM_TOKEN`
- [ ] Engine: reconnect with exponential backoff
- [ ] Engine: job receipt → `IncrementalIngestionPipeline.run(changeset)` → report result
- [ ] Engine: PR node population on `pull_request` job type (restores the orphaned graph endpoints)
- [ ] Integration test: platform pushes job, engine processes it, platform receives completion

---

## Full Re-Ingest Escape Hatch

`clew sync --full` forces a complete re-ingest of the repo regardless of what changed. This is the recovery path when incremental state drifts (e.g., a bug in edge cleanup leaves stale data, or the vector store is wiped and needs rebuilding).

Because the deterministic ID scheme makes `upsert` idempotent, a full re-ingest is safe to run at any time — it overwrites what's there rather than duplicating it.

---

## What This Unlocks

Once Phase 1 is complete:
- The two orphaned PR graph endpoints get their write path back (Phase 3)
- Full re-ingest becomes idempotent (bonus, no extra work)
- The reranker and graph boost features operate on a continuously fresh index rather than a stale snapshot
- Local-only users get a usable developer workflow without a platform account
