---
title: "Queue System"
description: "Background job queue system for TraceRTM"
keywords: [queue, jobs, background, tasks, celery]
---

# Queue System

Background job queue system for processing long-running tasks.

## Queue Architecture

### Job Types

- **Export jobs**: Generate exports
- **Import jobs**: Process imports
- **Notification jobs**: Send emails/notifications
- **Search indexing**: Update search index
- **Webhook delivery**: Deliver webhooks

## Basic Queue

### Simple Queue

```python
import asyncio
from collections import deque

class SimpleQueue:
    def __init__(self):
        self._queue = deque()
        self._workers = []
    
    async def enqueue(self, job_type: str, data: dict):
        """Add job to queue."""
        self._queue.append({"type": job_type, "data": data})
    
    async def process(self, handler: Callable):
        """Process jobs."""
        while True:
            if self._queue:
                job = self._queue.popleft()
                await handler(job)
            else:
                await asyncio.sleep(0.1)
```

## Job Processing

### Job Handler

```python
async def process_export_job(job: dict):
    """Process export job."""
    project_id = job["data"]["project_id"]
    format = job["data"]["format"]
    
    # Generate export
    export_data = await export_service.export(project_id, format)
    
    # Save to storage
    await storage_service.save(f"exports/{project_id}.{format}", export_data)
    
    # Notify completion
    await notification_service.send_export_ready(project_id)
```

### Worker

```python
async def worker(queue: SimpleQueue):
    """Background worker."""
    while True:
        job = await queue.dequeue()
        if job:
            try:
                await process_job(job)
            except Exception as e:
                logger.error(f"Job failed: {e}", exc_info=True)
```

## Job Status

### Track Job Status

```python
class JobStatus:
    PENDING = "pending"
    PROCESSING = "processing"
    COMPLETED = "completed"
    FAILED = "failed"

async def create_job(job_type: str, data: dict) -> str:
    """Create job and return job ID."""
    job_id = str(uuid.uuid4())
    
    await redis_client.hset(
        f"job:{job_id}",
        mapping={
            "type": job_type,
            "status": JobStatus.PENDING,
            "data": json.dumps(data),
            "created_at": datetime.utcnow().isoformat()
        }
    )
    
    # Enqueue job
    await queue.enqueue(job_id)
    
    return job_id

async def get_job_status(job_id: str) -> dict:
    """Get job status."""
    job = await redis_client.hgetall(f"job:{job_id}")
    return job
```

## Retry Logic

### Retry Failed Jobs

```python
async def process_job_with_retry(job: dict, max_retries: int = 3):
    """Process job with retry logic."""
    for attempt in range(max_retries):
        try:
            await process_job(job)
            await update_job_status(job["id"], JobStatus.COMPLETED)
            return
        except Exception as e:
            if attempt == max_retries - 1:
                await update_job_status(job["id"], JobStatus.FAILED)
                raise
            await asyncio.sleep(2 ** attempt)  # Exponential backoff
```

## Related Documentation

- [Job Processing](./07a-job-processing) - Job processing details
