# Phase 2 Roadmap: The Path to Internal Alpha

**Goal**: Move from "Technically Feasible" to "Ready for Users".
**Target Date**: End of February 2026.
**Milestone**: "Internal Alpha" - Zero-touch ingestion and persistence.

---

## Stream Status Overview

| Stream | Status | Description |
|--------|--------|-------------|
| Stream 1 | ✅ Complete | Infrastructure Resilience |
| Stream 2 | ✅ Complete | Visibility & Management |
| Stream 3 | ✅ Complete | The "First Wow" |
| Stream 4 | ✅ Complete | GitHub Integration |

---

## Stream 1: Operational Scalability
**STATUS: COMPLETED** (Feb 7, 2026)

### Delivered
- [x] **Redis State**: WebSocket worker state moved to Redis with TTL
- [x] **Arq Job Queue**: Background task processing for heavy operations
- [x] **Job Recovery**: Orphaned job detection and cleanup on startup
- [x] **Health Endpoints**: `/health/workers`, `/health/jobs/stale`, `/health/jobs/summary`

### Key Files
- `clew-platform`: `src/redis_manager.py`, `src/worker.py`, `src/health_endpoints.py`
- ADR: `docs/adr/003-async-job-queue.md`, `docs/adr/004-infrastructure-resilience.md`

---

## Stream 2: Visibility & Management
**STATUS: COMPLETED**

### Delivered
- [x] **Dashboard**: Next.js "Single Pane of Glass" for Workers, Jobs, Logs
- [x] **BYO-DB**: Encrypted credential storage and management
- [x] **Observability**: Live MCP tool call feed

---

## Stream 3: The "First Wow"
**STATUS: COMPLETED**

### Delivered
- [x] **Clew-Bot**: PR reviewer script with Mermaid dependency graphs
- [x] **Mermaid Diagrams**: Visual traversal paths in MCP output
- [x] **Zero-Touch Ingestion**: Git URL → automatic clone/parse/index
- [x] **Path Filtering**: Exact-match search in Qdrant adapter

### Key Files
- `clew-engine`: `examples/clew_bot.py`, `docs/adr/004-stream3-first-wow.md`

---

## Stream 4: GitHub Integration
**STATUS: COMPLETED** (Feb 7, 2026)

### Delivered
- [x] **GitHub Action**: `.github/workflows/clew-pr-analysis.yml`
- [x] **PR Comments**: Automatic Mermaid graph posting on PRs
- [x] **Smart Updates**: Edits existing comment instead of spamming

### Activation Required
Set repository secrets:
- `CLEW_API_URL`: API endpoint (e.g., `https://api.clewengine.dev`)
- `CLEW_API_KEY`: Authentication key

### Key Files
- `clew-engine`: `.github/workflows/clew-pr-analysis.yml`, `examples/clew_bot.py`
- ADR: `docs/adr/005-github-integration.md`

---

## Pre-Release Hardening (Alpha Blockers)
**STATUS: IN PROGRESS** (Updated: Feb 12, 2026)

| Priority | Item | Status | Rationale |
|:---------|:-----|:-------|:----------|
| **1** | Integration Tests | ✅ Done | Can't ship what we can't verify. |
| **2** | Deep Graph Traversal (>1 hop) | ✅ Done | This is our stated differentiator. |
| **3** | Retrieval Quality Baseline | 🔄 In Progress | Measure before optimizing. |
| **4** | Error Handling / Degradation | ✅ Done | Silent failures are not acceptable. |
| **5** | E2E Demo on Real Repo | 🔄 In Progress | Prove it works outside the lab. |

### Tasks
- [x] Add integration tests for `clew-engine` (20 tests in `test_pipeline_integration.py`: search→traverse, verify, error propagation, multi-hop, validation)
- [x] Deep graph traversal: Variable-depth 1-3 hops implemented and tested
- [ ] Create "Gold Set" of 44 queries (`tests/quality/gold_set.json`) with MRR measurement tool (`tests/quality/measure_mrr.py`) - baseline data updated, needs live testing
- [x] Error handling: Standardized across all adapters. Embedding fallback uses deterministic hash vectors.
- [ ] E2E verification script (`scripts/verify_e2e.sh`): health → search → traverse → validation checks - script updated, needs end-to-end validation

### Remaining
- [ ] Run `measure_mrr.py` against a live instance with an indexed real repo to establish baseline metrics
- [ ] Wire up webhook `dispatch_sync_job()` to async task queue for background ingestion (`webhooks.py:247,262`)

---

## Post-Alpha: Bookmarked

### Cognitive Layer (`clew-model-trainer`)
- **Status**: Paused. (Specs are coherent; blocked on alpha user feedback.)
- **Trigger**: Resume if MRR < 0.7 or Recall@3 < 0.8 after alpha launch.


---

## Completed Milestones
- **Phase 1 MVP**: Core Engine, Platform, Worker, Auth, Persistence
- **Benchmarking Harness**: Validated Universal MCP approach
- **System Hardening**: Implemented `verify_concept` backstop

## Success Criteria (Internal Alpha)
1. ✅ **Zero-Touch**: User inputs Git URL → Clew handles everything
2. ✅ **Context**: `explore_module` finds hidden dependencies/callers
3. ✅ **Persistence**: Data survives restarts; history preserved
4. ✅ **Integration**: GitHub Actions post Mermaid graphs on PRs
