# Async Webhook Queue - Implementation Plan

**Status:** Phase 2 Completion - Remaining Work
**Priority:** Medium
**Estimated Effort:** 1-2 hours
**Dependencies:** None (can be done independently)

---

## 🎯 Objective

Convert webhook sync jobs from inline execution to background processing using async task queue.

**Benefits:**
- ✅ Webhooks respond immediately (no timeout on large repos)
- ✅ Failed jobs can retry with backoff
- ✅ Job status tracking via API
- ✅ Horizontal scaling (multiple workers)
- ✅ Rate limiting and queue prioritization

## 📊 Current State

**What Works:**
- ✅ Webhook endpoints receive GitHub/GitLab events
- ✅ Event deduplication with Redis
- ✅ `dispatch_sync_job()` extracts changes and logs them
- ✅ Integration tests pass

**What's Missing:**
- ❌ `dispatch_sync_job()` is a placeholder (just logs, doesn't queue)
- ❌ No background workers
- ❌ No job status tracking
- ❌ No retry logic for failed syncs
- ❌ Sync runs inline (can timeout on large repos)

**Current Flow:**
```
Webhook → Verify → Deduplicate → dispatch_sync_job() → Log (placeholder)
                                                      ↓
                                                   TODO: Queue job
```

**Desired Flow:**
```
Webhook → Verify → Deduplicate → dispatch_sync_job() → Enqueue → Return job_id
                                                                 ↓
                                                           Worker picks up
                                                                 ↓
                                                         Run incremental sync
                                                                 ↓
                                                         Update job status
```

---

## 📋 Implementation Tasks

### Task 1: Choose Task Queue Technology

**Recommended:** Celery + Redis (already using Redis for deduplication)

**Alternatives Considered:**
- ARQ (async Python, lighter than Celery)
- RQ (simpler but less feature-rich)
- Dramatiq (good for high throughput)

**Decision:** Celery for Phase 2 (industry standard, good docs, feature-complete)

### Task 2: Set Up Celery

**File:** Create `packages/clew-api/celery_app.py`

```python
"""
Celery application for async webhook processing.
"""
from celery import Celery
import os

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")

app = Celery(
    "clew-webhooks",
    broker=REDIS_URL,
    backend=REDIS_URL,
)

app.conf.update(
    task_serializer="json",
    accept_content=["json"],
    result_serializer="json",
    timezone="UTC",
    enable_utc=True,
    task_track_started=True,
    task_time_limit=3600,  # 1 hour max per task
    task_soft_time_limit=3000,  # 50 min soft limit
    worker_prefetch_multiplier=1,  # One task at a time
    worker_max_tasks_per_child=100,  # Restart worker after 100 tasks
)

# Auto-discover tasks
app.autodiscover_tasks(['src.tasks'])
```

### Task 3: Create Sync Task

**File:** Create `packages/clew-api/src/tasks/sync_task.py`

```python
"""
Background task for incremental repository sync.
"""
import logging
from celery import Task
from celery_app import app

logger = logging.getLogger(__name__)


class SyncTask(Task):
    """Base task for sync operations with retry logic."""

    autoretry_for = (Exception,)
    retry_kwargs = {"max_retries": 3}
    retry_backoff = True  # Exponential backoff
    retry_backoff_max = 600  # 10 minutes max
    retry_jitter = True


@app.task(base=SyncTask, bind=True)
def process_incremental_sync(
    self,
    repo_id: str,
    repo_url: str,
    commit_sha: str,
    changed_files: dict[str, list[str]],
):
    """
    Process incremental sync for changed files.

    Args:
        self: Task instance (for retry/status updates)
        repo_id: Repository identifier
        repo_url: Repository URL
        commit_sha: Commit SHA
        changed_files: Dict with 'added', 'modified', 'removed' lists

    Returns:
        Dict with sync result
    """
    logger.info(f"Starting incremental sync for {repo_id}@{commit_sha}")

    try:
        # Update task state
        self.update_state(state="PROCESSING", meta={"repo_id": repo_id})

        # TODO: Call incremental sync logic from packages/clew-ingestion
        # from packages.clew_ingestion.src.sync import sync_changed_files
        # result = sync_changed_files(repo_url, commit_sha, changed_files)

        # Placeholder: Simulate processing
        import time
        time.sleep(2)

        logger.info(f"Sync complete for {repo_id}@{commit_sha}")

        return {
            "status": "success",
            "repo_id": repo_id,
            "commit_sha": commit_sha,
            "files_processed": sum(len(f) for f in changed_files.values()),
        }

    except Exception as e:
        logger.error(f"Sync failed for {repo_id}@{commit_sha}: {e}")
        # Update state before retry
        self.update_state(
            state="FAILURE",
            meta={"repo_id": repo_id, "error": str(e)},
        )
        raise  # Let Celery handle retry
```

### Task 4: Update Webhook Dispatch

**File:** `packages/clew-api/src/routes/webhooks.py`

Replace the TODO placeholder:

```python
async def dispatch_sync_job(changes: dict[str, Any]) -> str:
    """
    Dispatch incremental sync job to Celery queue.

    Args:
        changes: Change information extracted from webhook

    Returns:
        Job ID (Celery task ID)
    """
    from src.tasks.sync_task import process_incremental_sync

    # Dispatch to Celery
    task = process_incremental_sync.delay(
        repo_id=changes["repo_id"],
        repo_url=changes["repo_url"],
        commit_sha=changes["commit_sha"],
        changed_files=changes["files"],
    )

    logger.info(
        f"Queued sync job {task.id} for {changes['repo_id']}@{changes['commit_sha']}"
    )

    return task.id
```

### Task 5: Add Job Status Endpoint

**File:** `packages/clew-api/src/routes/jobs.py`

```python
"""
Job status endpoints for tracking webhook sync jobs.
"""
from fastapi import APIRouter, HTTPException
from celery.result import AsyncResult
from celery_app import app

router = APIRouter()


@router.get("/jobs/{job_id}")
async def get_job_status(job_id: str):
    """
    Get status of a sync job.

    Returns:
        {
            "job_id": "abc-123",
            "state": "PENDING|PROCESSING|SUCCESS|FAILURE",
            "meta": {...},
            "result": {...}  # If completed
        }
    """
    task = AsyncResult(job_id, app=app)

    if task.state == "PENDING":
        return {
            "job_id": job_id,
            "state": "PENDING",
            "status": "Job is queued, waiting for worker",
        }

    elif task.state == "PROCESSING":
        return {
            "job_id": job_id,
            "state": "PROCESSING",
            "meta": task.info,
        }

    elif task.state == "SUCCESS":
        return {
            "job_id": job_id,
            "state": "SUCCESS",
            "result": task.result,
        }

    elif task.state == "FAILURE":
        return {
            "job_id": job_id,
            "state": "FAILURE",
            "error": str(task.info),
        }

    else:
        return {
            "job_id": job_id,
            "state": task.state,
            "meta": task.info,
        }
```

Register in `src/main.py`:
```python
from src.routes import jobs

app.include_router(jobs.router, prefix="/v1/jobs", tags=["Jobs"])
```

### Task 6: Create Worker Startup Script

**File:** Create `scripts/start-worker.sh`

```bash
#!/bin/bash
set -e

echo "Starting Celery worker for webhook processing..."

cd packages/clew-api

celery -A celery_app worker \
  --loglevel=info \
  --concurrency=2 \
  --max-tasks-per-child=100 \
  --task-events \
  --without-heartbeat \
  --without-gossip \
  --without-mingle
```

### Task 7: Update Docker Compose

**File:** `docker-compose.yml`

Add worker service:

```yaml
services:
  # ... existing services ...

  celery-worker:
    build:
      context: ./packages/clew-api
      dockerfile: Dockerfile
    command: celery -A celery_app worker --loglevel=info --concurrency=2
    environment:
      - REDIS_URL=redis://redis:6379/0
      - NEO4J_URI=bolt://neo4j:7687
      - QDRANT_HOST=qdrant
      - QDRANT_PORT=6333
    depends_on:
      - redis
      - neo4j
      - qdrant
    restart: unless-stopped
    networks:
      - clew-network

  # Optional: Celery monitoring UI (Flower)
  celery-flower:
    build:
      context: ./packages/clew-api
      dockerfile: Dockerfile
    command: celery -A celery_app flower --port=5555
    ports:
      - "5555:5555"
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
      - celery-worker
    networks:
      - clew-network
```

### Task 8: Add Dependencies

**File:** `packages/clew-api/pyproject.toml`

```toml
[tool.poetry.dependencies]
# ... existing dependencies ...
celery = "^5.3.0"
flower = "^2.0.0"  # Optional: monitoring UI
```

Run:
```bash
cd packages/clew-api
poetry add celery flower
```

---

## ✅ Acceptance Criteria

1. Celery app configured with Redis broker
2. `process_incremental_sync` task defined with retry logic
3. `dispatch_sync_job()` enqueues tasks (no longer just logs)
4. Worker can pick up and process tasks
5. Job status endpoint (`GET /v1/jobs/{job_id}`) returns task state
6. Failed jobs retry with exponential backoff
7. Webhook returns immediately with job_id
8. Can monitor queue via Flower UI (optional)
9. Integration tests still pass

---

## 🧪 Testing Plan

**Manual Testing:**

1. Start Redis, API, and worker:
   ```bash
   docker-compose up redis
   docker-compose up celery-worker
   poetry run uvicorn src.main:app --port 8001
   ```

2. Send webhook event:
   ```bash
   curl -X POST http://localhost:8001/v1/webhooks/github \
     -H "Content-Type: application/json" \
     -d @tests/fixtures/github_push_event.json
   ```

3. Check response for job_id:
   ```json
   {
     "status": "accepted",
     "job_id": "abc-123-def-456",
     ...
   }
   ```

4. Check job status:
   ```bash
   curl http://localhost:8001/v1/jobs/abc-123-def-456
   ```

5. Watch worker logs:
   ```bash
   docker-compose logs -f celery-worker
   ```

6. (Optional) Open Flower UI: http://localhost:5555

**Integration Testing:**
- Update webhook tests to verify job_id returned
- Add test for job status endpoint
- Verify existing tests still pass

---

## 📚 Resources

**Celery Documentation:**
- https://docs.celeryq.dev/en/stable/
- Task retry: https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying

**Flower Monitoring:**
- https://flower.readthedocs.io/

**Redis as Broker:**
- https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html

**Current Webhook Implementation:**
- `packages/clew-api/src/routes/webhooks.py` (line 234: `dispatch_sync_job`)
- `packages/clew-ingestion/src/sync.py` (incremental sync logic)

---

## 🚀 Getting Started

1. Install Celery: `cd packages/clew-api && poetry add celery`
2. Create `celery_app.py` with basic config
3. Create `src/tasks/sync_task.py` with placeholder task
4. Update `dispatch_sync_job()` to call `task.delay()`
5. Test locally: Start worker + send webhook
6. Verify job is picked up and processed
7. Add job status endpoint
8. Add Docker Compose worker service

---

## ⚠️ Known Limitations

- No job persistence beyond Redis TTL (consider PostgreSQL backend for production)
- No queue prioritization (all jobs treated equally)
- No rate limiting per repo (could overwhelm system)
- Worker scaling requires manual configuration

These can be addressed in Phase 3 or as needed for production deployment.

---

## 💡 Future Enhancements (Phase 3+)

- PostgreSQL result backend for job persistence
- Priority queues (urgent repos processed first)
- Rate limiting per repository
- Dead letter queue for permanently failed jobs
- Webhook replay from failed jobs
- Auto-scaling workers based on queue depth
