---
name: odb-telemetry
description: >-
  Retrieve and interpret opendeviationbar production telemetry from bigblack.
  Covers heartbeat messages, kintsugi repair logs, Stathera integrity, sidecar
  health, ClickHouse stats, cross-midnight oracle, and system metrics.
  Use whenever the user asks about production health, recent alerts, repair
  status, gap counts, what the Telegram bot is reporting, what's happening
  on bigblack, monit status, or ODB status. Also use when the user pastes
  a Telegram heartbeat message and wants to understand it or check the source
  data. Do NOT use for deploying, releasing, or modifying production services.
allowed-tools: Read, Bash, Grep, Glob, Agent
---

# ODB Telemetry

Retrieve and interpret opendeviationbar production telemetry from bigblack (el02). All data is reachable via `ssh bigblack`.

---

## Quick Lookups

For the most common questions, run the matching command directly.

### "What did the last heartbeat say?"

```bash
ssh bigblack 'journalctl --user -u opendeviationbar-heartbeat --since "6 min ago" --no-pager'
```

Heartbeat runs every 5 minutes. If no output, widen the window (`--since "15 min ago"`).

### "Is everything healthy?"

```bash
ssh bigblack 'journalctl --user -u opendeviationbar-heartbeat --since "6 min ago" --no-pager | grep -E "HEALTHY|UNHEALTHY|cross-midnight|WARNING|ERROR"'
```

### "How many cross-midnight bars?"

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary @- <<EOSQL
SELECT count() AS cross_midnight_bars
FROM opendeviationbar_cache.open_deviation_bars FINAL
WHERE toDate(open_time_ms / 1000, '"'"'UTC'"'"') <> toDate(close_time_ms / 1000, '"'"'UTC'"'"')
EOSQL'
```

### "What's kintsugi doing?"

```bash
ssh bigblack 'echo "=== Recent NDJSON events ==="; tail -20 ~/opendeviationbar-py/logs/kintsugi.ndjson 2>/dev/null; echo ""; echo "=== Journal (last 30 lines) ==="; journalctl --user -u opendeviationbar-kintsugi -n 30 --no-pager'
```

### "Kintsugi repair history?"

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary @- <<EOSQL
SELECT toDate(timestamp) AS day, ouroboros_mode, count() AS repairs, sum(bars_written) AS bars
FROM opendeviationbar_cache.kintsugi_log
GROUP BY day, ouroboros_mode ORDER BY day DESC LIMIT 10
FORMAT PrettyCompact
EOSQL'
```

### "Stathera gap status?"

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary @- <<EOSQL
WITH inner AS (
    SELECT symbol, threshold_decimal_bps, first_agg_trade_id,
           first_agg_trade_id - lagInFrame(last_agg_trade_id, 1)
               OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) - 1 AS tid_delta,
           toDate(open_time_ms / 1000, '"'"'UTC'"'"') - toDate(
               lagInFrame(close_time_ms, 1) OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) / 1000,
               '"'"'UTC'"'"') AS day_diff,
           row_number() OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) AS rn
    FROM opendeviationbar_cache.open_deviation_bars FINAL
    SETTINGS do_not_merge_across_partitions_select_final = 1
)
SELECT
    countIf(tid_delta < 0) AS overlaps,
    countIf(tid_delta > 0 AND day_diff = 1) AS structural_day_boundary,
    countIf(tid_delta > 0 AND day_diff <> 1) AS real_gaps
FROM inner WHERE rn > 1
FORMAT PrettyCompact
EOSQL'
```

### "Sidecar health?"

```bash
ssh bigblack 'curl -s http://localhost:8081/health | python3 -m json.tool'
```

### "Monit status?"

```bash
ssh bigblack 'sudo monit summary 2>/dev/null'
```

### "System resources?"

```bash
ssh bigblack 'echo "Disk:"; df -h / | tail -1; echo "Memory:"; free -h | head -2; echo "Load:"; uptime'
```

---

## Telemetry Source Map

Every piece of production data lives in one of these locations. Use this table to know **where to look**.

### Primary Sources

| Data                        | Where             | How to Access                                                                                   |
| --------------------------- | ----------------- | ----------------------------------------------------------------------------------------------- |
| Heartbeat message (full)    | systemd journal   | `ssh bigblack 'journalctl --user -u opendeviationbar-heartbeat --since "6 min ago" --no-pager'` |
| Heartbeat healthy/unhealthy | systemd journal   | grep for `HEALTHY\|UNHEALTHY\|WARNING` in above                                                 |
| Cross-midnight bar count    | ClickHouse (live) | See query above                                                                                 |
| Kintsugi daemon log         | systemd journal   | `ssh bigblack 'journalctl --user -u opendeviationbar-kintsugi -n 50 --no-pager'`                |
| Kintsugi NDJSON events      | File on bigblack  | `ssh bigblack 'tail -30 ~/opendeviationbar-py/logs/kintsugi.ndjson'`                            |
| Kintsugi repair history     | ClickHouse table  | `opendeviationbar_cache.kintsugi_log` (see query above)                                         |
| Sidecar log                 | systemd journal   | `ssh bigblack 'journalctl --user -u opendeviationbar-sidecar -n 50 --no-pager'`                 |
| Sidecar health              | HTTP endpoint     | `curl -s http://localhost:8081/health`                                                          |
| Stathera integrity          | ClickHouse (live) | See Stathera query above                                                                        |
| Monit watchdog              | monit CLI         | `sudo monit summary`                                                                            |
| ClickHouse stats            | ClickHouse (live) | See ClickHouse queries below                                                                    |
| Circuit breaker state       | Sidecar /health   | `.checks.circuit_breaker` field                                                                 |
| Disk / memory / load        | OS commands       | `df -h /`, `free -h`, `uptime`                                                                  |

### State Files (on bigblack)

| File                                                        | Purpose                                           |
| ----------------------------------------------------------- | ------------------------------------------------- |
| `~/.cache/opendeviationbar/heartbeat/pending_messages.json` | Telegram message IDs for auto-delete (15 min TTL) |
| `~/.cache/opendeviationbar/heartbeat/stathera_counts.json`  | Previous Stathera counts for delta tracking       |
| `~/opendeviationbar-py/logs/kintsugi.ndjson`                | NDJSON event log (10 MB rolling, 1 backup)        |

### Configuration

| File                                | Content                                         | Git-Tracked |
| ----------------------------------- | ----------------------------------------------- | ----------- |
| `deploy/opendeviationbar.env`       | Non-secret env vars (chat ID, mode, thresholds) | Yes         |
| `deploy/opendeviationbar.local.env` | Secrets (Telegram token, Redis)                 | No          |
| `.mise.toml`                        | Mise tasks, Python version, hosts               | Yes         |

---

## Systemd Units (all `--user` scope)

| Unit                                        | Type    | Cycle       | Purpose                                        |
| ------------------------------------------- | ------- | ----------- | ---------------------------------------------- |
| `opendeviationbar-heartbeat.timer`          | Timer   | Every 5 min | Triggers heartbeat service                     |
| `opendeviationbar-heartbeat.service`        | Oneshot | On timer    | Runs `health_heartbeat.py`                     |
| `opendeviationbar-sidecar.service`          | Simple  | Always-on   | Real-time streaming                            |
| `opendeviationbar-kintsugi.service`         | Notify  | Always-on   | Gap reconciliation daemon                      |
| `opendeviationbar-kintsugi-catchup.service` | Notify  | On-demand   | Historical backfill (Conflicts= with kintsugi) |

**Journal access pattern**: `ssh bigblack 'journalctl --user -u <UNIT> <FLAGS>'`

Common flags:

- `-n 50` — last 50 lines
- `--since "N min ago"` — time window
- `--no-pager` — no less/more
- `--grep="pattern"` — filter
- `-f` — follow (live tail)

---

## ClickHouse Queries

### Bar Statistics

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary @- <<EOSQL
SELECT
    count() AS total_bars,
    uniq(symbol, threshold_decimal_bps) AS pairs,
    uniq(symbol) AS symbols,
    formatReadableSize(sum(bytes_on_disk)) AS disk_size,
    sum(parts) AS active_parts
FROM (
    SELECT database, table, sum(bytes_on_disk) AS bytes_on_disk, count() AS parts
    FROM system.parts
    WHERE database = '"'"'opendeviationbar_cache'"'"' AND table = '"'"'open_deviation_bars'"'"' AND active
    GROUP BY database, table
) AS t
CROSS JOIN (SELECT count() AS total_bars, uniq(symbol, threshold_decimal_bps) AS pairs, uniq(symbol) AS symbols FROM opendeviationbar_cache.open_deviation_bars) AS s
FORMAT PrettyCompact
EOSQL'
```

### 24h Growth

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary @- <<EOSQL
SELECT count() AS bars_last_24h
FROM opendeviationbar_cache.open_deviation_bars
WHERE close_time_ms > (toUnixTimestamp(now()) - 86400) * 1000
EOSQL'
```

### Per-Symbol Freshness (Top 8)

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary @- <<EOSQL
SELECT symbol,
       max(close_time_ms) AS latest_ms,
       round(dateDiff('"'"'second'"'"', fromUnixTimestamp64Milli(toInt64(max(close_time_ms))), now64(3, '"'"'UTC'"'"')) / 60, 1) AS minutes_ago,
       count() AS total_bars
FROM opendeviationbar_cache.open_deviation_bars FINAL
GROUP BY symbol ORDER BY total_bars DESC LIMIT 8
FORMAT PrettyCompact
EOSQL'
```

### Stalest Pairs

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary @- <<EOSQL
SELECT symbol, threshold_decimal_bps,
       round(dateDiff('"'"'second'"'"', fromUnixTimestamp64Milli(toInt64(max(close_time_ms))), now64(3, '"'"'UTC'"'"')) / 3600, 1) AS hours_stale
FROM opendeviationbar_cache.open_deviation_bars FINAL
GROUP BY symbol, threshold_decimal_bps
ORDER BY hours_stale DESC LIMIT 5
FORMAT PrettyCompact
EOSQL'
```

---

## Kintsugi NDJSON Events

The file `~/opendeviationbar-py/logs/kintsugi.ndjson` contains structured events. Key event types:

| Event                | Key Fields                              | Meaning                      |
| -------------------- | --------------------------------------- | ---------------------------- |
| `discovery_complete` | `total_shards`                          | Queue depth after shard scan |
| `repair_start`       | `symbol`, `threshold`, `gap_start_ms`   | Repair initiated             |
| `repair_complete`    | `result`, `bars_written`                | Repair outcome               |
| `pass_summary`       | `healed`, `clean`, `deferred`, `failed` | End-of-pass totals           |

**Filter to current daemon PID**:

```bash
ssh bigblack 'PID=$(systemctl --user show opendeviationbar-kintsugi --property=MainPID --value); grep "\"pid\":$PID" ~/opendeviationbar-py/logs/kintsugi.ndjson | tail -10'
```

---

## Interpreting the Heartbeat Message

The Telegram message from @obdpy_bot follows a fixed structure. Here's what each section means:

| Section        | Healthy                                            | Unhealthy                                        |
| -------------- | -------------------------------------------------- | ------------------------------------------------ |
| **Header**     | `❤️ Heartbeat OK` (silent, auto-deletes in 15 min) | `🔴 UNHEALTHY` (loud, persistent) + likely cause |
| **Services**   | All ✅ with uptime + memory                        | ❌ for down services                             |
| **Version**    | ✅ matches GitHub latest                           | ⚠️ drift detected                                |
| **Kintsugi**   | telemetry fresh, no stuck, low repair rate         | ❌ stale telemetry or stuck repair               |
| **Freshness**  | All symbols <5 min                                 | ⚠️ symbols >30 min stale                         |
| **ClickHouse** | Stats + growth + freshest/stalest                  | ❌ if unreachable                                |
| **Stathera**   | ✅ clean or 🔧 being healed                        | 🔴 anomalies growing and kintsugi not healing    |
| **System**     | ✅ disk/breaker                                    | ⚠️ disk >80% or breaker OPEN                     |

**Stathera health logic**: `anomalies_growing AND NOT kintsugi_healing` = UNHEALTHY. Growing counts are OK if kintsugi is actively repairing.

---

## Troubleshooting Common Alerts

| Alert                                                           | Where to Look             | Resolution                                                                                                                                  |
| --------------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `stathera_gaps: Real gaps detected and kintsugi is not healing` | Kintsugi journal + NDJSON | Check if kintsugi is running (`monit summary`). If running, check for stuck repair in NDJSON. May be false positive during service restart. |
| `cross_midnight_bars=N`                                         | heartbeat journal         | Historical violations being cleaned by kintsugi. Monitor count trending toward 0.                                                           |
| `version drift`                                                 | heartbeat journal         | Package on bigblack doesn't match GitHub. Run `mise run deploy:bigblack`.                                                                   |
| `sidecar down`                                                  | sidecar journal           | Check `journalctl --user -u opendeviationbar-sidecar -n 50`. Monit should auto-restart.                                                     |
| `kintsugi telemetry stale`                                      | NDJSON file mtime         | Kintsugi may be stuck or crashed. Check `monit summary` and journal.                                                                        |
| `disk >80%`                                                     | `df -h /`                 | Check ClickHouse parts, dead letters, log rotation.                                                                                         |
| `breaker OPEN`                                                  | sidecar /health           | ClickHouse likely down. Fix CH first, breaker auto-recovers in 60s.                                                                         |
