# PyCharter Worker

Asynchronous validation processing using a **database-backed job queue**. No Redis or other message broker required.

## Overview

The worker is a long-running process that:

- Polls the `validation_jobs` table for pending jobs
- Claims one job at a time (atomic claim so each job is processed by exactly one worker)
- Runs data quality validation (contract + quality check)
- Updates job status (completed/failed) and stores results in the database

Multiple worker processes can run against the same database; jobs are distributed by atomic claim.

## Documentation

**Full documentation:** [Validation Worker](https://optophi.github.io/pycharter/guides/worker/) (in the main PyCharter docs).

Build the docs locally: `pycharter docs serve` then open the "Validation Worker" guide under How-To Guides.

## Quick start

### 1. Database

Set `PYCHARTER_DATABASE_URL` (or `PYCHARTER_WORKER_DB_URL`). The worker uses the same database as the API; ensure migrations are applied (`pycharter db upgrade`).

### 2. Start the worker

```bash
pycharter worker start
```

Optional: `--concurrency N` (default 5), `--poll-interval MS` (default 500).

### 3. Submit jobs

**From Python:**

```python
from pycharter.worker import submit_job_sync

job_id = submit_job_sync(
    contract_name="user_schema",
    contract_version="1.0",
    data_source="/path/to/data.json",
)
```

**From API:** `POST /api/v1/validation/jobs` (see REST API docs).

## Architecture

```
API / Client  →  validation_jobs (DB)  →  Worker(s)  →  complete/fail in DB
```

- Jobs are rows in `validation_jobs` with status `pending` / `claimed` / `completed` / `failed`.
- Workers poll, claim, run validation, then update the row. No Redis or Spark in the current implementation.

## Configuration

| Env / CLI | Description |
|-----------|-------------|
| `PYCHARTER_DATABASE_URL` | Database URL (shared with API). |
| `PYCHARTER_WORKER_DB_URL` | Override DB URL for worker only. |
| `--concurrency` | Number of poller tasks (default 5). |
| `--poll-interval` | Poll interval in ms (default 500). |

## Package layout

- `worker/__init__.py` — `Worker`, `submit_job`, `submit_job_sync`, `get_job`
- `worker/config.py` — `WorkerConfig` (db_url, concurrency, poll_interval_ms, etc.)
- `worker/runner.py` — `Worker` run loop, signal handling, drain
- `worker/processor.py` — Runs validation for a claimed job
- `worker/job_sources/database.py` — `DatabaseJobSource` (submit, claim, complete, fail)
- `worker/cli.py` — `cmd_worker_start` for `pycharter worker start`
