# systemd Services & Timers

**Parent**: [/scripts/CLAUDE.md](/scripts/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

All services run as `--user` units on bigblack. Managed via `mise run deploy:bigblack`.

---

## Services (5 units)

| Service                             | Type            | Purpose                                     | Memory (High/Max)                           |
| ----------------------------------- | --------------- | ------------------------------------------- | ------------------------------------------- |
| `opendeviationbar-sidecar`          | simple (daemon) | Real-time streaming via WebSocket           | 3G / 4G                                     |
| `opendeviationbar-kintsugi`         | notify (daemon) | Unified gap reconciler + trailing-edge fill | 16G / 20G                                   |
| `opendeviationbar-kintsugi-catchup` | notify (daemon) | Aggressive historical gap repair            | 16G / 20G                                   |
| `opendeviationbar-heartbeat`        | oneshot (timer) | Unified health + Stathera audit to Telegram | — / 256M                                    |
| `opendeviationbar-seeder`           | oneshot (timer) | Proactive Parquet tick cache seeder         | — / 6G (observed peak 2.5 GB with mimalloc) |

## Timers (2)

| Timer                              | Interval    | Service   |
| ---------------------------------- | ----------- | --------- |
| `opendeviationbar-heartbeat.timer` | Every 5 min | heartbeat |
| `opendeviationbar-seeder.timer`    | Every 5 min | seeder    |

---

## Dependency Chain

```
sidecar → kintsugi
```

Kintsugi uses `After=opendeviationbar-sidecar.service`. The heartbeat timer is independent (runs regardless of daemon state — that's what it monitors).

**Conflicts**: `kintsugi-catchup` conflicts with `kintsugi` — only one runs at a time. The `Conflicts=` directive in the unit file ensures systemd stops one when the other starts.

---

## Environment Files

All services load TWO env files:

1. `deploy/opendeviationbar.env` — git-tracked, non-secret config
2. `deploy/opendeviationbar.local.env` — **NOT** git-tracked, secrets (Telegram token, etc.)

The `-` prefix in `EnvironmentFile=-` means "ignore if missing" (used for `local.env` so services start even without secrets).

---

## mimalloc

All services that process Polars DataFrames (sidecar, kintsugi, seeder) use:

```ini
Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmimalloc.so.2
Environment=MIMALLOC_PURGE_DELAY=1000
```

Prevents glibc ptmalloc2 RSS creep by returning freed memory to the OS (MEM-021). Without this, glibc holds `LazyFree` pages indefinitely — the seeder peaked at 4.47 GB RSS with glibc, 2.5 GB cgroup peak with mimalloc (−44% peak, −69% process RSS).

**Note**: MEM-024 (`malloc_trim(0)` in monthly `finally:` block) was removed during the daily partitioning migration (#277). With daily files and mimalloc, the glibc arena plateau no longer occurs.

---

## Timezone (Issue #206)

All services set `Environment=TZ=UTC` to prevent naive-datetime bugs regardless of the host's system timezone. This is defense-in-depth — bigblack's system TZ is also UTC (`timedatectl set-timezone UTC`) and ClickHouse has an explicit `<timezone>UTC</timezone>` in `/etc/clickhouse-server/config.d/timezone.xml`.

---

## OOM Protection

All long-running services use `MemorySwapMax=0` (no swap escape) and `ManagedOOMMemoryPressure=kill` for graceful systemd-oomd degradation via PSI instead of hard kernel OOM kill.

Priority order via `OOMScoreAdjust`:

1. sidecar (-500) — highest priority, latency-critical
2. kintsugi / kintsugi-catchup (-200) — background repair + trailing-edge fill
3. oneshot services (0, default) — heartbeat

---

## Operational Drop-ins (Kintsugi Override)

Systemd drop-ins in `~/.config/systemd/user/opendeviationbar-kintsugi.service.d/` override service settings without touching git-tracked unit files. They survive `git reset --hard`. The heartbeat reports `⚠️ overrides active` when drop-ins exist — informational only (v13.15.4+).

### Known Drop-ins on bigblack

| File                   | Purpose                                                                  |
| ---------------------- | ------------------------------------------------------------------------ |
| `focus-mode.conf`      | Restricts kintsugi to `--symbol X --threshold N` during focused backfill |
| `memory-override.conf` | Raises `MemoryHigh=36G MemoryMax=40G` for heavy backfill workloads       |

**memory-override.conf** also sets `OPENDEVIATIONBAR_MEMORY_MAX_BYTES=42949672960` to align the Python soft watermark with systemd's hard limit.

### After any drop-in change

```bash
systemctl --user daemon-reload
systemctl --user restart opendeviationbar-kintsugi
sudo monit monitor kintsugi  # if monit unmonitored it
```

---

## Deploy

`mise run deploy:bigblack` syncs service files, enables timers, and restarts all services. Drop-ins in `~/.config/systemd/user/opendeviationbar-kintsugi.service.d/` survive deploy (they are NOT in git).

---

## Resource Cohabitation (bigblack)

Bigblack hosts non-ODB services alongside the ODB stack. Resource separation ensures no service starves another.

### Non-ODB Services

| Service                    | MemoryMax | Purpose                                  |
| -------------------------- | --------- | ---------------------------------------- |
| prediction-market-backfill | 12G       | Prediction market data collection        |
| whisper-api                | —         | Speech-to-text (intermittent, on-demand) |

### Resource Separation Model

- **Memory**: Each service has explicit `MemoryMax` in its systemd unit. ODB total budget: sidecar 4G + kintsugi 20G + heartbeat 256M = ~24.3G. Non-ODB: ~12G. Host total: 62G.
- **CPU**: No hard CPU limits. OOM priority via `OOMScoreAdjust` ensures sidecar survives pressure.
- **Pueue**: ODB and non-ODB use separate pueue groups to prevent queue contention.
- **ClickHouse**: Shared instance, separate databases. No query resource limits configured (#201).
- **Disk**: Single NVMe (1.8TB), ClickHouse ~104GB. Monit alerts at 80%/90%.

---

## Related

- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) - Operations scripts, gap detection, kintsugi
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache, circuit breaker
- [/CLAUDE.md](/CLAUDE.md) - Project hub
