# Milestones

Phased delivery plan for Red Hat Oracle. Each milestone builds on its dependencies and includes testable acceptance criteria.

## Design Decisions

| Decision | Choice |
|----------|--------|
| First LLM provider | Ollama |
| Embedding consistency | Defer until basic pipeline works |
| Concurrency model | Async from the start (asyncio + httpx) |
| Re-indexing strategy | Full re-index per repo on each sync |
| Chunk ID format | `{source}/{org}/{repo}/{path}::sha256[:12]` |
| Ollama unavailable | Prompt user interactively (retry / pick another / quit) |
| Async + UI integration | Separate async thread for core; CLI/GUI main threads stay synchronous |
| Python version | 3.11+ |
| Progress UI | Rich progress bars (per-repo, per-batch) |
| Session naming | Auto-named by timestamp, resume via `--resume` (latest) or `--session <name>` |
| Code quality tooling | Ruff + mypy from the start |
| Git clone async | GitPython wrapped in `asyncio.to_thread()`, concurrent via thread pool |
| Clone concurrency | Configurable `max_concurrent_clones` in oracle.yaml, default 5, via asyncio.Semaphore |

---

## Milestone 0: Project Scaffolding

**Depends on:** nothing

**Scope:**

- `pyproject.toml` — metadata, dependencies (core + `[gui]` extra), console_scripts entry point (`oracle`), ruff + mypy config
- `src/redhat_oracle/__init__.py`, `__main__.py`
- `tests/conftest.py`
- `.gitignore`
- Ruff and mypy configuration (in pyproject.toml)
- Python 3.11+ requirement

**Acceptance criteria:**

- [ ] `pip install -e .` succeeds
- [ ] `python -m redhat_oracle` runs without error (prints help or placeholder)
- [ ] `ruff check src/ tests/` passes with no errors
- [ ] `mypy src/` passes with no errors

---

## Milestone 1: Config System

**Depends on:** Milestone 0

**Scope:**

- `core/config.py` — Pydantic models for all config sections, YAML loading, 3-level merge (system → user → project), env var fallback for secrets
- `oracle.yaml` — example config with all fields documented
- `max_concurrent_clones` config field (default: 5)
- `tests/test_config.py`

**Acceptance criteria:**

- [ ] Loads and merges system/user/project YAML configs correctly
- [ ] Env vars (`ORACLE_GITHUB_TOKEN`, `ORACLE_GITLAB_TOKEN`, `ORACLE_SLACK_TOKEN`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) override null YAML values
- [ ] Pydantic validation rejects invalid config (wrong types, missing required fields)
- [ ] `max_concurrent_clones` defaults to 5 when not specified
- [ ] All tests pass

---

## Milestone 2: Source Sync (async)

**Depends on:** Milestone 1

**Scope:**

- `core/sources/github.py` — async org enumeration via httpx, clone/pull via GitPython in `asyncio.to_thread()`, `asyncio.Semaphore(max_concurrent_clones)`
- `core/sources/gitlab.py` — same pattern for GitLab API + clone
- `core/sources/slack.py` — async message fetching, rate-limit retry on 429, thread resolution
- Rich progress bars for clone/fetch progress
- `tests/test_sources.py`

**Acceptance criteria:**

- [ ] Can enumerate repos from a GitHub org via API (mocked in tests)
- [ ] Can clone/pull repos concurrently, respecting semaphore limit
- [ ] Slack fetcher handles 429 rate limits with retry
- [ ] Progress bars display during ingestion
- [ ] All tests pass (using mocked HTTP responses)

---

## Milestone 3: Chunking

**Depends on:** Milestone 0 (no dependency on sources — can chunk test fixtures)

**Scope:**

- `core/vectordb/chunker.py` — AST chunking for Python (functions/classes), fixed-size word chunking with overlap for everything else
- File filtering: extension whitelist, max size 512KB, skip `.git/`, `node_modules/`, `__pycache__/`, vendored dirs
- Chunk ID generation: `{source}/{org}/{repo}/{path}::sha256[:12]`
- `tests/test_chunker.py`

**Acceptance criteria:**

- [ ] Python file produces one chunk per function/class via AST
- [ ] Falls back to fixed chunking if AST parse fails
- [ ] Non-Python files chunked by word count with overlap
- [ ] Files >512KB and vendored directories are skipped
- [ ] Chunk IDs are deterministic (same content produces same ID)
- [ ] All tests pass

---

## Milestone 4: Vector Store + Embeddings

**Depends on:** Milestones 1, 3

**Scope:**

- `core/vectordb/embeddings.py` — sentence-transformers provider (`all-MiniLM-L6-v2`, 384-dim), lazy-loaded
- `core/vectordb/store.py` — ChromaDB embedded store, three collections (`oracle_github`, `oracle_gitlab`, `oracle_slack`), upsert with content-hash IDs, full re-index per repo (delete all chunks for repo, then re-insert)
- Batch embedding (64 chunks at a time) with Rich progress
- `tests/test_vectordb.py`

**Acceptance criteria:**

- [ ] Embedding model loads lazily on first use
- [ ] Chunks are embedded in batches of 64
- [ ] Upsert with content-hash IDs deduplicates identical content
- [ ] Re-index deletes all existing chunks for a repo before inserting new ones
- [ ] Vector search returns ranked results with similarity scores
- [ ] All tests pass

---

## Milestone 5: RAG Pipeline + Ollama

**Depends on:** Milestone 4

**Scope:**

- `core/llm/base.py` — abstract `LLMProvider` (generate, stream)
- `core/llm/ollama.py` — async HTTP to localhost:11434, streaming responses
- Error handling: if Ollama unreachable, prompt user (retry / pick another / quit)
- `core/rag.py` — embed query → search ChromaDB top-k → filter by min score 0.3 → assemble context prompt → stream LLM response
- `core/session.py` — chat history, JSON persistence, auto-named by timestamp
- `tests/test_rag.py`

**Acceptance criteria:**

- [ ] Ollama provider streams responses token-by-token
- [ ] When Ollama is unreachable, user is prompted with options (retry / pick another / quit)
- [ ] RAG pipeline retrieves relevant chunks and includes them in the LLM prompt
- [ ] Sessions are saved as `{timestamp}.json` and can be resumed
- [ ] All tests pass (Ollama responses mocked)

---

## Milestone 6: CLI

**Depends on:** Milestone 5

**Scope:**

- `cli/app.py` — click commands:
  - `oracle` — top-level group
  - `oracle chat [--provider] [--resume] [--session <name>]` — interactive chat
  - `oracle ingest [--source]` — sync and index sources
  - `oracle status` — collection stats (chunk counts, last sync time)
  - `oracle search "query"` — vector search without LLM
  - `oracle config [--edit]` — show or edit config
- Interactive chat with prompt-toolkit + Rich
- Streaming response display with source citations
- Async core runs in a separate thread; CLI main thread stays synchronous

**Acceptance criteria:**

- [ ] `oracle ingest` syncs sources and indexes with progress bars
- [ ] `oracle chat` starts interactive session with streaming responses
- [ ] `oracle chat --resume` loads the latest session
- [ ] `oracle search "query"` returns vector search results without LLM
- [ ] `oracle status` shows collection stats
- [ ] All commands exit cleanly on Ctrl+C

---

## Milestone 7: Additional LLM Providers

**Depends on:** Milestone 5

**Scope:**

- `core/llm/external.py` — OpenAI + Anthropic SDKs, streaming
- `core/llm/openvino.py` — Intel hardware inference via optimum-intel
- Provider selection via config `llm.default_provider` + CLI `--provider` override
- Fallback prompting when selected provider unavailable

**Acceptance criteria:**

- [ ] OpenAI and Anthropic providers stream responses
- [ ] `--provider openai` overrides the default
- [ ] OpenVINO provider loads and runs a quantized model
- [ ] All tests pass (API responses mocked)

---

## Milestone 8: GUI

**Depends on:** Milestone 5

**Scope:**

- `gui/app.py` — PySide6 desktop app (optional `[gui]` dependency)
- Chat panel with markdown rendering
- Source panel (sidebar) showing indexed sources and stats
- Settings and session management
- Async core runs in a separate background thread; Qt main thread stays synchronous

**Acceptance criteria:**

- [ ] GUI launches and displays chat interface
- [ ] Can send messages and receive streaming responses
- [ ] Source panel shows indexed sources and stats
- [ ] Session save/load works from GUI

---

## Dependency Graph

```
Milestone 0 ──┬── Milestone 1 ── Milestone 2
              │        │
              │        └────┐
              │             ▼
              └── Milestone 3 ── Milestone 4 ── Milestone 5 ──┬── Milestone 6
                                                              ├── Milestone 7
                                                              └── Milestone 8
```
