🧠 Nuvom Architecture¶

This document explains the internal architecture of Nuvom, a lightweight, plugin-first task execution engine for Python.

🧩 High-Level Overview¶

Nuvom is designed to decouple task definition, discovery, execution, queuing, and result storage. Each layer is pluggable and follows a clearly defined contract via abstract base classes.

     +-------------------------+
     |      @task decorator    |
     +-------------------------+
                  |
                  v
      +------------------------+
      |     Task Registry      | <--- loaded from manifest
      +------------------------+
                  |
                  v
+-------------+     +-------------------+
| Dispatcher  | --> |  Job Queue        |
+-------------+     +-------------------+
                         |
                         v
            +----------------------+ 
            |   Worker Pool        |
            | (Threads + Runner)   |
            +----------------------+ 
                         |
                         v
            +----------------------+ 
            |  Result Backend      |
            +----------------------+ 
````

---

## 🧱 Core Components

### ✅ `@task` Decorator

**Location:** `nuvom/task.py`

* Wraps a function to register it as a Nuvom task.
* Adds metadata (`retries`, `timeout_secs`, etc.).
* Supports `.delay()` and `.map()` for job dispatch.
* All tasks are auto-registered via AST and manifest system.

---

### ✅ Task Discovery

**Location:** `nuvom/discovery/`

* Uses AST parsing (not imports) to detect decorated `@task` functions.
* Avoids side-effects, safe for large codebases.
* Uses `.nuvomignore` to skip paths.
* Output is cached in `.nuvom/manifest.json` for fast reloading.

Key files:

* `walker.py` – file traversal
* `parser.py` – AST parsing
* `manifest.py` – manifest file I/O
* `auto_register.py` – registry loader

---

### ✅ Task Registry

**Location:** `nuvom/registry/registry.py`

* Thread-safe global registry for tasks.
* Validates task names (prevents duplicates unless `force=True`).
* Used by the dispatcher and job runner to resolve function names.

---

### ✅ Dispatcher

**Location:** `nuvom/dispatcher.py`

* Orchestrates job submission: serializes, enqueues, retries.
* Provides `.delay()`, `.map()`, and job creation utilities.
* Uses `msgpack` for efficient, cross-platform job serialization.

---

### ✅ Job Queues

**Location:** `nuvom/queue_backends/`

Built-in backends:

* `MemoryJobQueue`
* `FileJobQueue`
* `SQLiteJobQueue` (v0.10)

Required interface methods:

```python
enqueue(job)
dequeue(timeout=None)
pop_batch(batch_size)
qsize()
clear()

Custom backends can be added via the plugin system.

✅ Workers & Job Execution¶

Location: nuvom/worker.py, nuvom/execution/job_runner.py

Each worker runs in its own thread.
Jobs are executed with timeouts, retries, and lifecycle hooks:
before_job()
after_job()
on_error()
ThreadPoolExecutor is used internally for concurrency.
Supports graceful shutdown with log flushing and safe teardown.

✅ Result Backends¶

Location: nuvom/result_backends/

Built-in backends:

MemoryResultBackend
FileResultBackend
SQLiteResultBackend

All result backends implement:

set_result(job_id, ...)
get_result(job_id)
set_error(job_id, ...)
get_error(job_id)
get_full(job_id)
list_jobs()

Use .nuvom_plugins.toml to register custom plugins.

✅ Logging¶

Location: nuvom/log.py

Unified logging across all modules using Rich.
Logs are styled, color-coded, and exception-aware.
Categories: debug, info, warning, error.

🔌 Plugin Architecture¶

Location: nuvom/plugins/

Nuvom supports plugins for:

Queues
Result backends
Monitoring/exporters

Plugins follow a strict Plugin protocol with start() and stop() lifecycle methods.

[plugins]
queue_backend = ["custom.module:MyQueue"]
result_backend = ["custom.module:MyResult"]

Each plugin must register itself via a Plugin subclass, and may use register_queue_backend() or register_result_backend().

🔁 Job Lifecycle¶

Developer defines a task with @task.
nuvom discover tasks parses and caches it.
Job is queued with .delay() or .map().
Worker dequeues the job.
JobRunner:
Triggers lifecycle hooks
Executes task with timeout/retry logic
Stores result or error
Job metadata is saved in the selected result backend.
Results are queried via SDK or CLI.

🎯 Design Principles¶

✅ Plugin-first, interface-driven
✅ No global daemons or dependencies like Redis
✅ Developer-first: minimal config, rich logging, CLI tooling
✅ Native on Windows, Linux, macOS
✅ Built to teach: readable source, clean separation

For more, see: