# ossctx Task List

## Completed

- Create the unified Python package and root `ossCtx` CLI.
- Build the shared `common` platform layer for config, DB setup, versioning, and FastAPI app creation.
- Implement `codectx` indexing, querying, stats, cleanup, and API routes.
- Add `codectx` parser coverage for Go, Python, JavaScript, TypeScript, and Java.
- Add persisted `codectx` relation extraction, call graph queries, and cross-file relation resolution.
- Implement `docsctx` local indexing, crawling, upload ingestion, search, finalize flow, and API routes.
- Add deterministic `docsctx` structure/entity/community extraction.
- Add provider-backed `docsctx` finalize support with deterministic fallback.
- Implement `otelctx` trace ingestion, stats, trace/service APIs, and compatibility CLI aliases.
- Add OTLP HTTP JSON trace compatibility and `/v1/traces` ingestion route.
- Add OTLP gRPC trace receiver support with integration coverage.
- Add OTLP HTTP JSON logs and metrics ingestion, persistence, stats, listing APIs, and `/v1/logs` and `/v1/metrics` compatibility routes.
- Improve `codectx` relation resolution so local Python imports drive target selection instead of falling back to global name order.
- Deepen `docsctx` provider robustness with retries, persisted per-document provider status metadata, and deterministic fallback tracking.
- Replace the `codectx` MCP and setup command stubs with a working FastMCP server and VS Code `mcp.json` config writer.

## Remaining — Full Migration Plan (Updated)

The Go workspace has four projects (codecontext, docscontext, otelcontext, central-ops).
The Python `ossctx` has migrated the core loops of all three services.

**Key Direction (Phase based on user feedback):**
- DocsCtx: LLM/Embedding ONLY for GraphRAG generation and storage. Agents will query MCP tools for context.
- CodeCtx & OtelCtx: NO AI features. Agents will use exposed MCP tools for analysis/synthesis instead.

Below is the updated gap-by-gap plan grouped into phases.

---

### Phase 1 — High-Leverage Parity (MCP + gRPC + Search) ⬅️ NEXT

These close the biggest functional gaps and unblock downstream work.

#### 1.1 OTLP gRPC Logs & Metrics Receiver (`otelctx`) ✅ PLANNED
- Extend the existing gRPC server to handle `LogsService/Export` and `MetricsService/Export`.
- Re-use `normalize_log_payload()` / `normalize_metric_payload()` for persistence.
- Add integration tests mirroring the trace gRPC tests.
- _Depends on:_ nothing new; HTTP path already works.

#### 1.2 otelctx MCP Server & Setup Command ✅ PLANNED
- Create `otelctx/mcp.py` with a FastMCP server exposing 12 tools matching Go: `get_system_graph`, `get_service_health`, `search_logs`, `tail_logs`, `get_trace`, `search_traces`, `get_metrics`, `get_dashboard_stats`, `get_storage_status`, `find_similar_logs`, `get_alerts`, `search_cold_archive`.
- Add `ossCtx otel mcp` and `ossCtx otel setup` CLI commands (same pattern as codectx/docsctx).
- _Depends on:_ 2.1–2.3 for full tool coverage; stub those initially.

#### 1.3 Expand codectx MCP Tools (5 → 14) ✅ PLANNED
- Add the 9 missing MCP tools: `get_docs`, `list_files`, `get_file_outline`, `find_usages`, `search_entities`, `get_entity_code`, `get_file_imports`, `get_graph_context`, `get_file_context`.
- Wire into the existing FastMCP server in `codectx/cli.py`.
- _Depends on:_ repository methods already exist for most.

#### 1.4 docsctx Vector Embedding & Semantic Search ✅ COMPLETED (Phase 1)
- ✅ Added embedding layer (`docsctx/embedder.py`) supporting Ollama, OpenAI, Azure OpenAI providers.
- ✅ Extended provider.py with `extract_graph()` and LLM-based entity/relationship/claim extraction.
- ✅ Implemented **local search** (vector similarity + graph walk BFS to configurable depth) in `search.py`.
- ✅ Implemented **global search** (community summary aggregation) in `search.py`.
- ✅ Built docsctx MCP server (`mcp.py`) with 12 tools: `search_documents`, `local_search_vector`, `global_search_communities`, `query_entity`, `find_relationships`, `get_graph_neighborhood`, `get_document_structure`, `list_entities`, `list_documents`, `get_community_report`, `get_chunk`, `stats`.
- ✅ Added CLI commands: `ossCtx docs mcp` and `ossCtx docs setup`.

---

### Phase 2 — otelctx Advanced Storage & Analytics

These migrate the in-memory accelerator stack from Go otelcontext.

#### 2.1 Service Dependency Graph (`otelctx`)
- Build an in-memory graph rebuilt periodically from span data.
- Track per-edge call count, error rate, and latency percentiles.
- Compute health scores (0–1) and status enum (healthy / degraded / critical).
- Expose via `/api/graph` endpoint and the `get_system_graph` / `get_service_health` MCP tools.

#### 2.2 In-Memory TSDB Ring Buffer (`otelctx`)
- Port the sliding-window ring buffer with tumbling-window aggregation.
- Pre-compute p50/p95/p99 percentiles per metric series.
- Add cardinality controls and overflow buckets.
- Expose via `/api/metrics/timeseries` (or similar) and the `get_metrics` MCP tool.

#### 2.3 Embedded Vector Index for Semantic Log Search (`otelctx`)
- Implement TF-IDF or lightweight embedding index over log bodies.
- Cosine-similarity retrieval with FIFO eviction.
- Expose via `find_similar_logs` MCP tool and `/api/logs/search` endpoint.

#### 2.4 Hot/Cold Archive Tiering (`otelctx`)
- Implement configurable retention (default 7 days).
- Archive cold data as zstd-compressed JSONL files with daily manifests and SHA-256 checksums.
- Expose `search_cold_archive` MCP tool.
- _Depends on:_ 2.5 (compression utils).

#### 2.5 Compression Utilities (`otelctx`)
- Port zstd compress/decompress helpers (Python `zstandard` package).

#### 2.6 TTL Cache & Dead Letter Queue (`otelctx`)
- Add a generic TTL cache with background eviction thread.
- Add a DLQ table for failed ingestion with retry semantics.

---

### Phase 3 — Extended Parser Coverage (`codectx`)

Port parsers from Go codecontext in order of community usage.
**Note:** No LLM analysis in codectx; agents will use MCP tools on the extracted graph.

#### 3.1 Tier 1 — High Usage (6 languages)
- **Rust** (`.rs`) — regex-based, functions/structs/enums/traits/impl blocks
- **C#** (`.cs`) — regex-based, classes/interfaces/methods/properties/namespaces
- **Ruby** (`.rb`) — regex-based, classes/modules/methods/blocks
- **SQL** (`.sql`) — regex-based, tables/views/procedures/functions
- **Scala** (`.scala`, `.sc`, `.sbt`) — regex-based, classes/objects/traits/defs
- **Kotlin** (`.kt`, `.kts`) — regex-based, classes/interfaces/functions/objects

#### 3.2 Tier 2 — Scripting & Shell (4 languages)
- **Bash/Shell** (`.sh`, `.bash`, `.zsh`) — functions, aliases, exports
- **Lua** (`.lua`) — functions, local functions, tables
- **Perl** (`.pl`, `.pm`) — subs, packages, use/require
- **R** (`.r`, `.R`) — functions, assignments

#### 3.3 Tier 3 — Markup & Config (9 languages + Dockerfile)
- **HTML** (`.html`, `.htm`) — tags, ids, scripts, links
- **CSS/SCSS** (`.css`, `.scss`, `.sass`) — selectors, variables, mixins
- **Markdown** (`.md`) — headings, links, code blocks
- **JSON** (`.json`) — top-level keys
- **XML/XSD/SVG/XSLT** (`.xml`, `.xsd`, `.svg`, `.xsl`, `.xslt`) — elements, attributes
- **YAML** (`.yml`, `.yaml`) — top-level keys
- **TOML** (`.toml`) — sections, keys
- **HCL/Terraform** (`.hcl`, `.tf`, `.tfvars`) — resources, variables, outputs, modules
- **Properties** (`.properties`) — key-value pairs
- **Dockerfile** — stages, instructions

---

### Phase 4 — Real-Time & Observability

#### 4.1 WebSocket Real-Time Hub (`otelctx`)
- Buffered hub for log + metric streaming.
- Slow-client eviction and connection callbacks.
- FastAPI WebSocket endpoint (`/ws`).

#### 4.2 Prometheus Metrics Instrumentation (`otelctx`)
- Port the 19 Prometheus metrics from Go.
- Add HTTP/gRPC middleware instrumentation.
- Expose `/metrics` endpoint (prometheus-client library).

---

### Phase 5 — REST API Parity & Document Loaders

#### 5.1 docsctx Additional API Endpoints
- `/api/graph/neighborhood` — graph walk from entity
- `/api/entities` — list/filter entities
- `/api/communities` — list communities
- `/api/communities/{id}` — community detail + report

#### 5.2 docsctx PDF & DOCX Loaders
- **PDF** loader via `pymupdf` or `pdfplumber`.
- **DOCX** loader via `python-docx`.
- Register in the loader dispatcher.

#### 5.3 codectx Additional API Endpoints
- Port any missing REST endpoints from Go codecontext (file list, entity detail, etc.).

---

### Phase 6 — Web UIs (Low Priority)

#### 6.1 codectx Web UI
- Port or rebuild the React frontend for code graph exploration.
- Embed static assets via FastAPI `StaticFiles`.

#### 6.2 docsctx Web UI
- Port the vis-network graph explorer, semantic search, and document browser.

#### 6.3 otelctx Web UI
- Port the React 19 + Mantine + ECharts + ReactFlow dashboard.
- WebSocket integration for real-time updates.

---

### Phase 7 — Polish & Packaging

#### 7.1 README & Documentation Polish
- Comprehensive CLI reference for all three subsystems.
- API endpoint documentation.
- MCP tool reference.

#### 7.2 Multi-DB Support (`otelctx`)
- Extend SQLAlchemy config to support MySQL, PostgreSQL, MSSQL connection strings.

#### 7.3 YAML Config File Support (`docsctx`)
- Add optional YAML config file loading alongside env vars.

#### 7.4 codectx File Aggregation (Legacy)
- Port the combine-files-into-single-context-block mode.

#### 7.5 JS/TS Extension Variants
- Add `.mjs`, `.cjs`, `.mts`, `.cts` to the JavaScript/TypeScript parser file extension list.

---

### Recommended Execution Order

| Order | Task | Rationale | Agent Role |
|-------|------|-----------|-----------|
| 1 | 1.1 OTLP gRPC Logs & Metrics | Smallest gap; extends existing gRPC server | Direct implementation |
| 2 | 1.3 Expand codectx MCP Tools | Repository methods exist; wiring only | Direct implementation |
| 3 | 1.2 otelctx MPC Server | Enables agent workflows for otel | Direct implementation |
| 4 | 2.1 Service Dependency Graph | Enables health tools in otelctx MCP | Agent queries MCP for context |
| 5 | 2.2 TSDB Ring Buffer | Enables metrics tools in otelctx MCP | Agent queries MCP for context |
| 6 | 3.1 Tier 1 Parsers | Broadens language support | Agent uses codectx MCP for analysis |
| 7 | 2.3–2.6 Remaining otelctx storage | Completes otelctx parity | Agent queries MCP for context |
| 8 | 4.1–4.2 Real-time & metrics | Live streaming support | Websocket-based agent monitoring |
| 9 | 5.1–5.3 API parity & loaders | REST completeness | Agent uses REST + MCP interchangeably |
| 10 | 6.1–6.3 Web UIs | Visual interfaces | UI for agents & users |
| 11 | 7.1–7.5 Polish | Packaging and edge cases | Final polish |

---

**Key Architectural Principles (Phase 1 completion):**
- ✅ DocsCtx LLM/embedding for GraphRAG generation only (production-quality extraction into DB)
- ✅ CodeCtx & OtelCtx: MCP tools expose extracted data; agents do synthesis via their own LLMs
- ✅ All 3 subsystems: MCP server for tool access + VS Code setup command for config
- ✅ Vector search only in docsctx (community + entity context for agents)