---
id: 002
status: proposed
title: ADR-002: Split Scheduler into Rufus (In-Process) and Systemd Timers (Batch)
---

## Status
Proposed

## Date
2026-02-28

## Context

Onetime Secret runs three long-lived Podman containers as Quadlet units: web (Puma/Rack), worker (RabbitMQ/Kicks consumers), and scheduler (rufus-scheduler). The scheduler process exists solely to keep rufus timers alive. It runs all scheduled work in-process — both high-frequency application-coupled jobs (HeartbeatJob at 1m, DLQMonitorJob at 60s) and low-frequency batch maintenance jobs that perform self-contained Redis operations and exit.

This conflates two execution models in one process:

1. **Event-loop jobs** that need in-process application state, Familia model access, and sub-minute frequency
2. **Batch jobs** that run infrequently (hourly to weekly), do bulk Redis SCAN/ZREM operations, and have no application coupling

The batch jobs gain nothing from a persistent process. They pay for it with shared failure domains (one crash takes all jobs down), no per-job resource limits, hidden failures in application logs, no catch-up for missed runs after downtime, and schedule changes requiring process restarts.

Validated against codebase: the five in-process jobs (HeartbeatJob, DLQMonitorJob, PlanCacheRefreshJob, ExpirationWarningJob, CatalogRetryJob) all directly access Familia/Horreum models or RabbitMQ state. The batch maintenance jobs (phantom cleanup, data consistency audit, participation GC, index rebuild, instances rebuild) perform standalone Redis operations with no queue coupling. The existing `scripts/detect_phantom_receipts.rb` script already demonstrates the batch pattern with 6 inconsistency categories.

## Decision

**Split the scheduler into two execution models:**

1. **Keep rufus-scheduler** for jobs that are high-frequency (sub-5-minute) or application-state-coupled: HeartbeatJob, DLQMonitorJob, PlanCacheRefreshJob, ExpirationWarningJob, CatalogRetryJob. These remain in the existing scheduler Quadlet unit.

2. **Move batch maintenance jobs to systemd timer + oneshot container pairs.** Each job gets its own `.timer` and `.container` unit. The container runs `bin/ots maintenance <job>`, performs the work, and exits. systemd manages scheduling, catch-up (`Persistent=true`), jitter (`RandomizedDelaySec`), and per-job resource limits.

3. **Stripe billing jobs follow a three-tier model**: immediate event processing via Kicks/RabbitMQ workers, interval safety-nets via rufus (plan cache refresh, webhook retry sweeps), and heavyweight batch operations via systemd timers (subscription reconciliation, catalog validation).

The placement boundary: any job that modifies customer-facing state (plan assignments, subscription status, feature gates) runs in-process where it shares Familia connection context with web request handlers. Jobs that are self-contained Redis sweeps go to systemd.

## Trade-offs

**We lose:**
- Single deployment unit for all scheduled work
- Simpler mental model (one process = all timers)
- Ability to adjust all schedules by editing one Ruby file

**We gain:**
- Per-job failure isolation (one crash does not affect others)
- Per-job resource limits (MemoryMax, CPUQuota via container units)
- Native observability (`systemctl list-timers` shows next/last run and exit codes)
- `Persistent=true` catch-up for missed runs after downtime or deployment
- `RandomizedDelaySec` jitter to spread load
- Schedule changes via `systemctl daemon-reload` without container restarts
- Exit code semantics (0=success, 75=EX_TEMPFAIL for transient Redis errors)
- Structured JSON output to journald for monitoring

## Implementation Notes

### Batch Maintenance Jobs (2026-02-28)

Five jobs to implement as oneshot containers:

| Job | Frequency | Operation |
|-----|-----------|-----------|
| PhantomCleanup | Hourly | SCAN + HLEN + DEL for empty-hash keys |
| DataConsistencyAudit | Every 6h | Verify referential integrity across Familia models |
| ParticipationGC | Daily | ZSCAN instances sorted sets, ZREM orphans |
| IndexRebuild | Daily | Rebuild Familia multi-indexes from instances.members |
| InstancesRebuild | Weekly | SCAN object keys, rebuild class sorted sets via atomic RENAME |

### CLI Entrypoint Design (2026-02-28)

`MaintenanceCommand < Command` base class with:
- Boot via `OT.boot! :cli` with explicit failure propagation (no swallowed exceptions)
- Exit codes: 0 (success), 1 (failure), 2 (boot failure), 75 (EX_TEMPFAIL)
- JSON to stdout for journald capture
- SIGTERM trap for clean cancellation mid-SCAN
- `FAMILIA_POOL_SIZE=3` for single-threaded oneshot (vs 25 for scheduler)

### Worker Template Units (2026-02-28)

Related decision: scale RabbitMQ workers via `ots-worker@N` container template units (container-level scaling) rather than Sneakers process forking. This avoids the Familia Redis connection inheritance bug where forked children share parent file descriptors. Set `workers: 1` in Sneakers config so each container uses threads only.

### Billing Job Placement (2026-02-28)

PlanCacheRefreshJob stays in rufus as a safety net, with webhook-triggered refresh (`product.updated`, `price.updated`) as primary. Any `Persistent=true` catch-up for billing-adjacent jobs must be idempotent and state-checking — read current state from Redis/Stripe, do not trust event payload age.
