# Product Roadmap

This document outlines the development trajectory of Clewso.

**Current Status:** Alpha - **Stable**
**Focus:** "Indie V1" (Single Developer / Agent Use Case)

---

## 🟢 Phase 1: The "Indie V1" (MVP)
*Goal: A single developer can spin this up via Docker Compose and power their own Agent.*

### 1.1 Core Infrastructure (The Brain)
- [x] **Docker Compose**: Orchestrate Qdrant, Neo4j, and FastAPI.
- [x] **Unified Data Model**: Define the shared schema for standardizing Code and Graph nodes.
- [x] **Connectivity**: Ensure services can talk to each other within the docker network.
- [x] **Tech Debt Sweep**: Critical and High priority items resolved (Feb 9).
- [x] **Refactoring Sprint**: Ingestion, Adapters, MCP, Protocols (Feb 10).
- [x] **Complexity Reduction**: Refactored high-CCN functions in core modules (Feb 10).
- [x] **Stabilization**: Test Harness fixed, End-to-End Ingestion verified (Feb 10).

### 1.2 Ingestion (The Senses)
- [x] **GitHub Crawler**: Specialized worker with `.gitignore` support (`git ls-files`).
- [x] **Graph Mapper**: Logic to link `Function` -> `File` (PR/Issue linking merged).
- [x] **Vector Embedder**: Simple pipeline using OpenAI `text-embedding-3-small`.
- [x] **Variable Depth Traversal**: Direct parameters for Neo4j (depth 1-3).
- [x] **Pipeline Architecture**: Strategy/Visitor pattern implementation verified.

### 1.3 Retrieval (The Memory)
- [x] **Hybrid Search API**: `POST /search` that queries both DBs.
- [x] **Graph Expansion**: Logic to find "Neighbors" of a code snippet.
- [x] **Signature-Only Retrieval**: Optimized baseline sans GPU requirement.

### 1.4 Agent Interface (The Voice)
- [x] **MCP Server**: Expose `search_codebase`, `explore_module`, and `verify_concept` tools.
- [x] **Claude Integration**: formatted `mcp.json` for easy copy-paste.
- [x] **Standardized tool names**: Consistent with source implementation.

### 1.5 Smart Reviewer (The Critic)
- [x] **Graph Impact Analysis**: Impact radius detection (Depth/Breadth).
- [x] **Context Assembly**: Smart context fetching with token limits.
- [x] **LLM Reasoning**: "Breaking Change" vs "Safe Refactor" classification.
- [x] **CLI Tool**: `clewso review` command for staged/PR changes.

---

## 🟡 Phase 2: Scalability & Event-Driven Arch (V2)
*Goal: Enable multi-repo indexing and async updates.*

- [x] **Multi-Repo Support**: Indexing multiple repositories into a shared graph (Feb 10).
- [x] **Incremental Sync**: Logic to process diff-based updates (Feb 10).
- [x] **Basic Auth**: API Keys for secure access (Verified: JWT + Hashed Keys implemented).

### Pre-Release Hardening (Feb 12)
- [x] **Integration Tests**: 20 pipeline tests (search→traverse→verify, error propagation, multi-hop, validation).
- [x] **Retrieval Quality Gold Set**: 44 queries with MRR/Recall@K measurement tooling (Verified: baseline data functional).
- [x] **E2E Verification Script**: Automated `docker-compose` → health → search → traverse checks (`scripts/verify_ingestion.py`).
- [x] **Live Quality Baseline**: MRR 0.54, Recall@3 0.64 (77-80% of target). Real OpenAI embeddings 18.5x better than hash baseline.
- [x] **Qdrant Port Fix**: Ingestion/API port mismatch resolved (6335). Clean database re-ingestion completed.
- [x] **Test File Filtering**: Implemented `exclude_tests` parameter with comprehensive pattern matching. 79% vector reduction.
- [x] **Gitignore-Aware Discovery**: Hybrid `git ls-files` + 40+ fallback patterns. 100% clean indexing (no cache pollution).
- [x] **Event Handling**: Support for processing diff-based increments (Core unit).
- [x] **CI/CD Pipeline**: GitHub Actions set up with `make lint` and `make test`.
- [x] **Lint Cleanup**: Resolution of complex functions and Pyre2 logic errors.

### Phase 2 Completion Roadmap (Feb 12 - Complete)
**Status:** 100% Complete - Team Edition is fully successfully implemented.

- [x] **PR/Issue Graph Linking** - Merged via PR #44.
- [x] **Async Ingestion Pipeline** - Merged via Phase 2.5 (Core thread pools).
- [x] **Async Parsing Stage** - Merged via PR #48.
- [x] **Batched Processing Stage** - Merged via PR #52.

**Current Capabilities:**
- ✅ Multi-repo indexing with clean gitignore filtering
- ✅ Real-time quality metrics (77-80% of target)
- ✅ Incremental sync logic (Diff-based core processing)
- ✅ Event deduplication and signature verification
- ✅ Test file filtering for cleaner search results

**Limitations:**
- ✅ PR/Issue metadata captured in graph (PR/Issue linking)
- ✅ Orchestration provided by `clew-platform` (Sovereign Architecture)

---

## 🟣 Phase 2.5: Validation & Performance (Feb 13-28)
*Goal: Validate quality claims and make ingestion production-fast before Phase 3 investment.*

**Placement:** Between Team Edition completion and Enterprise Core investment.
Two parallel tracks that reinforce each other: faster ingestion means faster
benchmark iteration, and benchmarks verify the optimized pipeline.

### Track A: Clew-Bench Validation Arena

**Decision:** ADR-006 - Comparative Agent Arena
**Timeline:** 1-2 weeks (Feb 13-28, 2026)

#### Rationale
All efficiency claims (15x token compression, hallucination reduction) are theoretical calculations. Before investing in enterprise features (SSO, audit logging, reranker), we need head-to-head validation that:
1. Clew's graph+vector approach provides measurable token compression vs standard RAG
2. Quality improvements (hallucination reduction, refactor success) are real
3. The system is ready for production agent workloads

#### Components
- [x] **Test Tracks**: 3 starter repositories (Python/FastAPI, React/Next.js, TypeScript/Node.js) (Feb 15)
- [x] **Workload Simulation**: `epics.json` with 10-15 tickets simulating real JIRA workflow (Feb 15)
- [x] **Test Harness**: Side-by-side agent execution (Clew vs Standard RAG) (Feb 15)
- [x] **Judge LLM**: Gemini 1.5 Pro for quality scoring (Feb 15)
- [x] **Metrics Collection**: Tokens, quality, refactor success (Feb 15)

#### Success Criteria
- **Token Compression**: ✅ Verified 10x+ compression (validate current 15x estimate) (Feb 15)
- **Hallucination Rate**: ✅ Demonstrate <5% hallucination rate (vs >30% for standard RAG) (Feb 15)
- **Refactor Success**: ✅ Show 100% refactor success (vs <70% for standard RAG) (Feb 15)

#### Key Metrics to Measure
1. **Context Bloat Curve**: Input tokens vs ticket number (expect flat for Clew, exponential for RAG)
2. **Hallucination Rate**: Import errors, non-existent function calls
3. **Refactor Success**: Complete vs partial coverage of code changes

#### Deliverables
- Benchmark suite for regression testing
- Public results (blog post, GitHub publication)
- Data-driven decision on Phase 3 priorities (reranker need, optimization targets)
- Marketing material (concrete numbers for pitch deck)

**Related Documents:**
- `docs/adr/006-clew-bench-validation.md` - Architecture Decision Record
- `docs/planning/clew-bench-arena.md` - Implementation plan (to be created)
- `clew-research/comparative-agent-arena.md` - Original proposal

### Track B: Concurrent Ingestion Pipeline

**Timeline:** 1-2 weeks, parallel with Track A
**Target:** 8-12x ingestion speedup (~120s → ~10-15s for a 500-file repo)

#### Rationale
Ingestion is bottlenecked by sequential embedding API calls (~100ms each) and
per-node Neo4j writes. Clew-Bench will re-ingest test repositories repeatedly
during development; slow ingestion is a direct friction point. Fixing this now
means benchmark results reflect production performance, not a known-slow baseline.

#### Implementation (3-stage rollout, each a standalone PR)

**Stage 1 — Async Embeddings + Batched Vector Store** (low risk)
- [x] Fix `OpenAIEmbeddings` to use `AsyncOpenAI` (signature is async, body isn't)
- [x] Add `embed_batch()` to both providers (single API call for OpenAI, semaphore-bounded fan-out for Ollama)
- [x] Add `VectorStore.add_batch()` consuming `embed_batch()`

**Stage 2 — Async Parsing Stage** (medium risk)
- [x] Add `AsyncPipelineStage` protocol alongside existing sync protocol
- [x] Convert orchestrator `run()` to async (sync stages still work via `inspect.iscoroutinefunction` check)
- [x] Convert `ParsingStage` to async with batched embed/graph calls per file group
- [x] Bounded `ThreadPoolExecutor` for tree-sitter CPU work, deterministic shutdown in `finally`

**Stage 3 — Async Processing Stage + Batched Graph Writes** (medium risk)
- [x] Add `GraphStore.execute_batch()` — single transaction for N Cypher queries
- [x] Convert `ProcessingStage` to async with batched definition embeds and graph ops
- [x] Qdrant ID plumbing from batch embed results back to graph params

#### Design Principles
- Async-first for I/O (`asyncio`, `async for`, `async generators`); threading only for C/sync libs
- Explicit pool lifecycle — every pool created in a known scope, torn down in `finally`
- Conservative concurrency limits via `asyncio.Semaphore` (configurable, default 8)
- Stages stay sequential; concurrency is *within* stages only
- `ingest_repo()` remains a sync entry point (wraps `asyncio.run()`)

**Related Documents:**
- `docs/CONCURRENT_INGESTION_PLAN.md` - Full implementation plan with code examples

---

## 🔴 Phase 3: Advanced Intelligence (V3)
*Goal: Deepen understanding and retrieval quality.*
*Governed by: [ADR-008 — Engine–Platform Boundary Contract](../clew-platform/docs/adr/008-engine-platform-boundary.md)*

> **Rule**: Features in this phase must be FOSS and run locally. Commercial features belong in `clew-platform`.

- [x] **DCP Semantic Layer**: Introduced the "Law" layer for rule enforcement (Feb 14). *(Engine — Free tier)*
- [ ] **Reranker (Deferred)**: Cross-encoder scoring disabled due to context window limitations (Feb 17). *(Engine — Free tier)*

---

## 🔵 Phase 3.1: Type System Hardening
*Goal: Close correctness gaps in internal data models and enforce strict type checking at the package level.*

**Decision**: Full Rust port deferred indefinitely. Bottlenecks are network I/O (OpenAI, Neo4j, Qdrant), not Python CPU. The async/batched pipeline introduced in Phase 2.5 resolved the actual performance problems. PyO3 bindings remain an option for specific hot paths if profiling ever identifies them — see Future/Research.

- [x] **TypedDict for core data models** — Replace untyped `dict` in `SearchResult.metadata`, `GraphNode.properties`, `GraphEdge.properties`, and `get_stats()` return. Defined in `clew-core` as the canonical source of truth. (Feb 21)
- [x] **Pyright strict: `clew-core`** — Beachhead package. `pyrightconfig.json` with `"typeCheckingMode": "strict"`. 0 errors. (Feb 21)
- [x] **Pyright strict: `clew-mcp`** — Smallest surface area; natural second target. 0 src errors. (Feb 21)
- [x] **Pyright standard: `clew-api`** — Full type propagation through all adapters (Qdrant, Neo4j, pgvector); 0 src errors. (Feb 21)
- [x] **CI enforcement** — Added `pyright` step to `make lint` and the CI matrix alongside `ruff`. (Feb 21)

---

## 🔵 Phase 3.2: CI Ingestion Pipeline
*Goal: Establish CI as the canonical writer for shared database topologies, enabling team-scale graph consistency.*
*Governed by: [ADR-008 §2.4 — Deployment Topologies](../clew-platform/docs/adr/008-engine-platform-boundary.md)*

> **Rule**: In shared DB topologies, CI is the single writer for code graph data. Developer machines read but never write. See ADR-008 §2.4 (Write Authority Model).

### 3.2.1 CI Canonical Indexing
- [x] **`clew-ci-index` workflow** — GitHub Actions reusable workflow triggered on merge to `main`. Runs `clewso index .` (incremental) against shared Neo4j/Qdrant. Connection strings via CI secrets (`NEO4J_URI`, `QDRANT_HOST`, `QDRANT_API_KEY`). *(Engine — Free tier)* (Feb 27)
- [x] **Incremental-only mode flag** — `clewso index --incremental` skips full re-parse when the graph already has this repo. Uses git diff to identify changed files. Falls back to full index on first run. *(Engine — Free tier)* (Feb 27)
- [x] **Commit SHA tracking** — Store `last_indexed_commit` in Neo4j repo metadata. CI uses this to compute the file diff (`git diff $last_indexed..HEAD`). *(Engine — Free tier)* (Feb 27)

### 3.2.2 PR Dry-Run Validation
- [x] **`clewso review --dry-run`** — Analyze PR diff against the existing shared graph without writing to the DB. Report policy violations and impact analysis as PR comments. *(Engine — Free tier)* (Feb 27)
- [x] **Policy check integration** — Extend existing `clew-pr-analysis.yml` to include PolicyRule violation checks alongside the current impact analysis. *(Engine — Free tier)* (Feb 27)

### 3.2.3 Isolation Hardening
- [x] **`get_stats()` repo_id scoping** — Fix the identified gap: `get_stats()` currently returns counts for the entire database. Add optional `repo_id` parameter. *(Engine — Free tier)* (Feb 27)
- [x] **Graph traversal repo_id guard** — `traverse()` should accept and propagate `repo_id` to prevent cross-repo edge leakage in shared instances. *(Engine — Free tier)* (Feb 27)
- [x] **Write-mode configuration** — `CLEW_WRITE_MODE=ci-only|open` environment variable. When `ci-only`, the API rejects `POST /v1/ingest` calls unless a CI token is provided. Prevents accidental developer-machine writes to shared DBs. *(Engine — Free tier)* (Feb 27)

---

## 🟠 Phase 4: Hook Plugin SDK
*Goal: Ship a deterministic governance layer that integrates Clew with AI coding agent CLIs via lifecycle hooks.*
*Governed by: [ADR-012 — Hook Plugin Architecture](../clew-platform/docs/adr/012-hook-plugin-architecture.md), [ADR-008 — Engine–Platform Boundary](../clew-platform/docs/adr/008-engine-platform-boundary.md)*

> **Rule**: The plugin enforces locally and fails open. Policies are sourced from the configured Neo4j instance (local or shared) and optionally synced from the Control Plane (Enterprise). See ADR-012 §1.

### 4.1 Foundation
- [x] **PolicyRule schema** — `PolicyRule` model added to `clew-core/schema.py` (Feb 27). Three initial types: `banned_import`, `unguarded_path`, `protected_write`. *(Engine — Free tier)*
- [x] **Policy CRUD API** — `POST/GET/DELETE /v1/policies` endpoints in `clew-api` for managing PolicyRule nodes in Neo4j. *(Engine — Free tier)* (Feb 27)
- [x] **Policy Cache Export** — `GET /v1/policies/export` returns all active policies as a flat JSON array. Written to `/tmp/clew-policy-cache.json` by the SessionStart hook. *(Engine — Free tier)* (Feb 27)

### 4.2 Prototype (Claude Code)
- [x] **PreToolUse hook prototype** — Shell script or Go binary that reads the policy cache and evaluates `protected_write` (glob match) and `banned_import` (string match) rules. Target: <50ms p95. *(Engine — Free tier)* (Feb 27)
- [x] **SessionStart hook** — Queries local Engine `/v1/policies/export`, writes cache file, injects DCP context summary into agent system prompt. Target: <3s. *(Engine — Free tier)* (Feb 27)
- [x] **Latency benchmark** — Measure round-trip for all three hook types on a warm Engine with 100 active policies. Publish results in `docs/benchmarks/`. *(Engine — Free tier)* (Feb 27)

### 4.3 Production Plugin
- [x] **Go binary** — Static binary (`clew-hook-{os}-{arch}`) with in-memory policy cache, JSON stdin/stdout contract, exit code semantics (0=allow, 2=block, 1=error/fail-open). *(Engine — Free tier)* (Feb 27)
- [x] **PostToolUse audit hook** — Async background logging of all tool executions + policy decisions. Non-blocking. *(Engine — Free tier)* (Feb 27)
- [x] **Multi-CLI support** — Adapter configs for Claude Code (`hooks.json`), Gemini CLI (extension manifest), Copilot Extensions (extension spec). *(Engine — Free tier)* (Feb 27)

### 4.4 Platform Integration (Future)
- [ ] **Centralized Policy Authoring** — Define policies in Platform UI, distribute to Workers. *(Platform — Enterprise tier)*
- [ ] **Interception Dashboard** — Aggregated metrics: violations/day, top policies triggered, per-team breakdown. *(Platform — Team tier)*
- [ ] **Policy Sync** — Workers poll Platform for policy updates; hot-reload cache without session restart. *(Platform — Enterprise tier)*

---

## 🔮 Future / Research
- [x] **Local LLM Support**: Ollama embedding adapter implemented.
- [ ] **IDE Plugins**: Native VS Code / JetBrains extensions.
- [ ] **Code Graph V2**: AST-level parsing.
- [ ] **Rust CLI binary**: Replace PyInstaller/shiv distribution for `clewso review` with a native Rust binary. Engine stays Python; CLI speaks to it over HTTP. Motivated by binary size, startup latency, and clean cross-platform distribution.
- [ ] **PyO3 bindings**: If profiling ever identifies Python-native hot paths (e.g., AST post-processing, graph result assembly), PyO3 enables calling Rust from Python with zero-copy data transfer. Revisit after Phase 3 intelligence work ships and real profiling data exists.
