# Red Hat Oracle — Data Flow

## Ingestion Pipeline

### Phase 1: Source Sync

All source syncing is incremental — only new or changed data is fetched.

#### GitHub Repos

```
1. Read oracle.yaml → sources.github.orgs
2. For each org:
   a. GET /orgs/{org}/repos (paginated, PAT auth)
   b. Filter by include/exclude glob patterns
   c. For each matching repo:
      - If not cloned: git clone (SSH preferred) --depth=1
      - If cloned: git pull (fast-forward only)
   d. Store in: {base_dir}/repos/github/{org}/{repo}/
3. Report: N repos synced, M skipped, K failed
```

**Authentication flow:**
```
PAT token lookup order:
  1. oracle.yaml → sources.github.token
  2. Environment variable: ORACLE_GITHUB_TOKEN

Clone URL selection:
  use_ssh: true  → git@github.com:{org}/{repo}.git
  use_ssh: false → https://github.com/{org}/{repo}.git
                   (PAT injected into URL for HTTPS)
```

#### GitLab Repos

```
1. Read oracle.yaml → sources.gitlab.instances
2. For each instance (e.g., gitlab.cee.redhat.com):
   a. For each group:
      - GET /api/v4/groups/{group}/projects?include_subgroups=true
      - Filter by include/exclude
      - Clone/pull each project
   b. Store in: {base_dir}/repos/gitlab/{hostname}/{namespace}/{project}/
3. Report results
```

**Authentication flow:**
```
PAT token lookup order:
  1. Instance-level: instances[].token
  2. Global GitLab: sources.gitlab.token
  3. Environment variable: ORACLE_GITLAB_TOKEN

API auth: PRIVATE-TOKEN header
Clone auth: SSH key or PAT-injected HTTPS URL
```

#### Slack Messages

```
1. Read oracle.yaml → sources.slack
2. List all accessible channels (conversations.list)
3. Filter by configured channels (empty = all)
4. For each channel:
   a. Fetch messages since (now - history_days) via conversations.history
   b. For messages with threads: fetch replies via conversations.replies
   c. Resolve user IDs to display names (cached)
   d. Build SlackMessage objects with full thread context
5. Return list of messages for vectorization
```

**Rate limiting:** Slack API returns 429 with Retry-After header. The client
automatically waits and retries.

---

### Phase 2: Chunking

Raw source content must be split into chunks suitable for embedding.

#### Code Files

```
For each repo:
  Walk all files
  Filter by:
    - Extension in index_extensions (.py, .yaml, .js, etc.)
    - Size < max_file_size (512KB default)
    - Not in .git/, node_modules/, __pycache__/, etc.

  For Python files (when method=ast):
    Parse AST → extract functions, classes
    Each function/class = one chunk
    Metadata: file path, line range, node type, node name
    Fallback to fixed chunking if AST parse fails

  For all other files (or method=fixed):
    Split into chunks of chunk_size words
    Overlap of chunk_overlap words between chunks
    Metadata: file path, line range
```

**Example chunk (AST):**
```python
# From: insights-core/insights/client/upload.py
# Lines: 45-92
# Type: class
# Name: Uploader

class Uploader:
    """Handles uploading archives to the Insights API."""
    def __init__(self, config):
        self.config = config
        ...
```

**Example chunk (fixed):**
```yaml
# From: ansible/ansible/lib/ansible/config/data.yaml
# Lines: 1-50

DEFAULT_BECOME:
  description: Toggle the use of privilege escalation...
  ...
```

#### Slack Messages

```
For each message:
  Build document text:
    "[#channel-name] User Name: message text"
    "  Reply User: reply text"
    "  Reply User: reply text"

  If combined text > chunk_size:
    Split with fixed chunking (preserve thread grouping)

  Metadata: channel, user, timestamp, date, has_thread
```

---

### Phase 3: Embedding

```
Collect all chunks from Phase 2
Batch into groups of batch_size (64)
For each batch:
  Generate embeddings via configured provider:
    - local: sentence-transformers all-MiniLM-L6-v2 (384-dim)
    - ollama: ollama embeddings API
    - openai: text-embedding-3-small (1536-dim)
  Yield (chunks, embeddings)
```

**Memory management:**
- Embedding model loaded lazily (first use)
- Batched processing prevents loading all chunks into memory
- Model unloaded after ingestion if not needed for queries

---

### Phase 4: Storage

```
For each (chunks, embeddings) batch:
  Upsert into ChromaDB collection:
    - oracle_github → GitHub code chunks
    - oracle_gitlab → GitLab code chunks
    - oracle_slack  → Slack message chunks

  ChromaDB stores:
    - Document text (for retrieval)
    - Embedding vector (for similarity search)
    - Metadata dict (for filtering and citations)
    - Unique ID (for deduplication on re-index)
```

**ChromaDB storage format:**
```
{vectordb_dir}/chroma/
├── chroma.sqlite3          # Metadata and document storage
└── {collection_uuid}/
    └── data_level0.bin     # HNSW index (hnswlib C++)
```

---

## Query Pipeline

### Step 1: User Input

```
User types question in CLI or GUI
  → "How does the insights-core upload process work?"
```

### Step 2: Query Embedding

```
Generate embedding for user's question
  → Using same model as ingestion (important for consistency)
  → 384-dim vector (for all-MiniLM-L6-v2)
```

### Step 3: Vector Search

```
Search all ChromaDB collections:
  query_embeddings = [question_embedding]
  n_results = top_k (10)
  include = [documents, metadatas, distances]

For each result:
  Convert cosine distance to similarity score: 1.0 - distance
  Filter by min_score (0.3)

Sort all results by score descending
Take top_k results
```

### Step 4: Context Assembly

```
Build augmented prompt:

  SYSTEM: {rag.system_prompt}
          "You are the Red Hat Oracle..."

  CONTEXT:
    "Relevant code and conversations:
     ---
     Source: github/RedHatInsights/insights-core/insights/client/upload.py:45-92
     class Uploader:
         ...
     ---
     Source: slack/#platform-dev (2025-01-15)
     John Smith: the upload process was refactored to...
     ---
     ..."

  HISTORY:
    Previous messages in this chat session (last N)

  USER: "How does the insights-core upload process work?"
```

### Step 5: LLM Generation

```
Send assembled prompt to configured LLM provider:
  - Ollama: POST http://localhost:11434/api/chat (streaming)
  - OpenVINO: Direct model.generate() call
  - OpenAI: client.chat.completions.create(stream=True)
  - Anthropic: client.messages.create(stream=True)

Stream response tokens back to UI as they arrive
```

### Step 6: Response Display

```
Render response with:
  - Formatted text (markdown)
  - Source citations (file paths, Slack links)
  - Confidence indicators (based on retrieval scores)

Append to session history
Auto-save session if configured
```

---

## Data Volume Estimates

| Source | Estimated Repos/Channels | Estimated Chunks |
|--------|-------------------------|-----------------|
| RedHatInsights (GitHub) | ~350 repos | ~500K chunks |
| ansible (GitHub) | ~100 repos | ~300K chunks |
| GitLab groups (6 groups) | ~200+ repos | ~400K chunks |
| Slack (1 year) | varies | ~100K chunks |
| **Total** | | **~1.3M chunks** |

**Storage estimates:**
- ChromaDB index: ~2-5 GB (for 1.3M chunks with 384-dim embeddings)
- Cloned repos (shallow): ~10-30 GB
- Total disk: ~15-35 GB

**Ingestion time estimates:**
- Repo cloning: 1-4 hours (network dependent)
- Embedding generation: 2-6 hours (CPU), 30-60 min (GPU)
- Total first-time setup: 3-10 hours
- Incremental update: 5-30 minutes
