---
name: remote-backfill
description: >-
  Orchestrate parallel open deviation bar cache backfill on bigblack via pueue.
  Guides AskUserQuestion flows, formulates correct commands, avoids known anti-patterns.
  TRIGGERS - backfill, populate cache, remote populate, pueue jobs, bigblack,
  cache population, 1000 dbps, 250 dbps, per-year, per-month, parallel jobs.
allowed-tools: Read, Bash, Grep, Glob, AskUserQuestion
---

# Remote Backfill Orchestration

Born from the Feb 2026 BTCUSDT@1000 backfill session where multiple command format failures, pueue mismanagement, and ouroboros conflicts were resolved. This skill encodes the **canonical recipe** so future backfills succeed on the first attempt.

## Prerequisite Skills

- `Skill(opendeviationbar-job-safety)` — memory profiles, per-symbol thresholds, invariants
- `Skill(devops-tools:pueue-job-orchestration)` — pueue commands, group management

## When to Use

- Populating ClickHouse cache on bigblack for any symbol/threshold
- Backfilling historical data for new symbols or thresholds
- Re-populating data after force_refresh or gap detection
- Any multi-month or multi-year cache population task

---

## Phase 1: AskUserQuestion Clarification Flow

**Always run this flow before queuing any jobs.** Each question informs the command construction.

### Question 1: Symbol Selection

```
Q: Which symbols do you want to backfill?
Options:
  - "All 15 batch symbols" — BTCUSDT, ETHUSDT, BNBUSDT, SOLUSDT, XRPUSDT,
    DOGEUSDT, ADAUSDT, AVAXUSDT, DOTUSDT, LINKUSDT, LTCUSDT, ATOMUSDT,
    NEARUSDT, SHIBUSDT, UNIUSDT (pueue-populate.sh set)
  - "All 27+ registered symbols" — everything in symbols.toml
  - "Specific subset" — user specifies which
```

### Question 2: Threshold

```
Q: What threshold in dbps?
Options:
  - 1000 (Phase 1, 3-6 GB/job with microstructure)
  - 250 (Phase 2, 1-2 GB/job, most bars)
  - 500 (Phase 3, 2-4 GB/job)
  - 750 (Phase 3, 2-4 GB/job)
  - 100 (Phase 4, resource-intensive, ~8 GB/job)
Note: BTC/ETH use 2-4x more memory than altcoins at the same threshold.
```

### Question 3: Date Range

```
Q: What date range?
Options:
  - "Full history" — from effective_start in symbols.toml to today
  - "Recent (2024-2025)" — last ~2 years
  - "Specific year(s)" — user specifies
```

### Question 4: Parallelism Strategy

```
Q: How to slice date ranges for parallel pueue jobs?
Options:
  - "per-month" — 12 jobs/year, max parallelism (recommended for light thresholds: 1000, 750, 500)
  - "per-year" — 1 job/year, simpler (recommended for heavy thresholds: 250, 100)
Note: ouroboros mode is always "day" — date slicing is purely a pueue parallelism strategy.
```

### Question 5: Feature Tiers

```
Q: Which feature tiers to include?
Options:
  - "Microstructure + Tier 2" (recommended) — intra-bar + inter-bar lookback
  - "Microstructure only" — intra-bar features, skip Tier 2/3
  - "Full (Tier 2 + 3 + Hurst + PE)" — everything, significantly slower
  - "Bars only" — OHLCV only, fastest
```

### Question 6: Check Cache First

Before queuing, **always** check existing cache to avoid redundant work:

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 bigblack \
  "cd /home/tca/opendeviationbar-py && uv run --python 3.13 python -c \"
from opendeviationbar.clickhouse import OpenDeviationBarCache
with OpenDeviationBarCache() as cache:
    count = cache.count_bars('SYMBOL', THRESHOLD)
    oldest = cache.get_oldest_bar_timestamp('SYMBOL', THRESHOLD)
    newest = cache.get_newest_bar_timestamp('SYMBOL', THRESHOLD)
    print(f'SYMBOL@THRESHOLD: {count:,} bars')
    if oldest and newest:
        from datetime import datetime
        print(f'Range: {datetime.fromtimestamp(oldest/1000).date()} to {datetime.fromtimestamp(newest/1000).date()}')
    else:
        print('Range: N/A (no cached bars)')
\""
```

### Question 7: Resource Check + Parallelism

Check bigblack resources before deciding parallelism:

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 bigblack 'free -h | head -2 && echo "---" && uptime'
```

Then use this decision table (memory per job with `--include-microstructure`):

| Threshold | Memory/Job (BTC/ETH) | Memory/Job (Altcoins) | Available > 30 GB | Available > 20 GB | Available > 10 GB |
| --------- | -------------------- | --------------------- | ----------------- | ----------------- | ----------------- |
| 1000 dbps | ~6 GB                | ~3 GB                 | parallel=6        | parallel=4        | parallel=3        |
| 500 dbps  | ~4 GB                | ~2 GB                 | parallel=6        | parallel=4        | parallel=3        |
| 750 dbps  | ~4 GB                | ~2 GB                 | parallel=6        | parallel=4        | parallel=3        |
| 250 dbps  | ~2 GB                | ~1 GB                 | parallel=8        | parallel=5        | parallel=3        |
| 100 dbps  | ~8 GB                | ~4 GB                 | parallel=3        | parallel=2        | parallel=1        |

> **Key lesson (March 2026 #159 incident)**: BTC/ETH generate 2-4x more bars/day than altcoins → proportionally more memory. A flat 2G cap for @1000 OOM-killed 7 jobs. The SSoT is `pueue-populate.sh` which now uses symbol-aware `get_memory_limit(threshold, symbol)`.

---

## Phase 2: Command Formulation

### The Canonical Command (ALWAYS use this)

**Script**: `scripts/populate_full_cache.py` via pueue on bigblack.

**NEVER use `opendeviationbar populate` CLI for pueue jobs** — it lacks checkpointing, ouroboros mode, and feature flags.

### Per-Month Slice Jobs (day ouroboros, monthly date ranges)

One pueue job per calendar month — 12 jobs/year. Best for light thresholds (1000, 750, 500 dbps) where memory/job is low and parallelism pays off.

```bash
/usr/bin/env bash -c '
SYMBOL="BTCUSDT"
THRESHOLD=1000
GROUP="btc1000"
FEATURES="--include-microstructure"  # or --no-microstructure

for year in 2024 2025; do
    if [ "$year" = "2025" ]; then
        end_month=2  # adjust to current month
    else
        end_month=12
    fi

    for m in $(seq 1 $end_month); do
        month=$(printf "%02d" $m)
        start="${year}-${month}-01"
        last_day=$(python3 -c "import calendar; print(calendar.monthrange($year, $m)[1])")
        end="${year}-${month}-${last_day}"
        label="${SYMBOL}@${THRESHOLD}:${year}-${month}"

        cmd="uv run --python 3.13 python scripts/populate_full_cache.py"
        cmd="$cmd --symbol $SYMBOL --threshold $THRESHOLD"
        cmd="$cmd --start-date $start --end-date $end"
        cmd="$cmd --ouroboros day $FEATURES"

        echo "Queuing: $label ($start to $end)"
        ssh -o BatchMode=yes -o ConnectTimeout=10 bigblack \
            "pueue add --working-directory /home/tca/opendeviationbar-py --group $GROUP --label '"'"'$label'"'"' -- $cmd"
    done
done
'
```

### Per-Year Slice Jobs (day ouroboros, annual date ranges)

```bash
/usr/bin/env bash -c '
SYMBOL="BTCUSDT"
THRESHOLD=250
GROUP="btc250"
EFFECTIVE_START="2018-01-16"  # from symbols.toml

for year in $(seq 2018 2025); do
    if [ "$year" = "2018" ]; then
        start="$EFFECTIVE_START"
    else
        start="${year}-01-01"
    fi

    if [ "$year" = "2025" ]; then
        end="2025-02-28"  # adjust to current date
    else
        end="${year}-12-31"
    fi

    label="${SYMBOL}@${THRESHOLD}:${year}"

    cmd="uv run --python 3.13 python scripts/populate_full_cache.py"
    cmd="$cmd --symbol $SYMBOL --threshold $THRESHOLD"
    cmd="$cmd --start-date $start --end-date $end"
    cmd="$cmd --ouroboros day --include-microstructure"

    echo "Queuing: $label ($start to $end)"
    ssh -o BatchMode=yes -o ConnectTimeout=10 bigblack \
        "pueue add --working-directory /home/tca/opendeviationbar-py --group $GROUP --label '"'"'$label'"'"' -- $cmd"
done
'
```

---

## Phase 3: Execution + Dynamic Scaling

### 1. Create pueue group

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 bigblack \
  'pueue group add GROUPNAME --parallel 4'
```

### 2. Queue jobs (Phase 2 commands above)

### 3. Monitor progress

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 bigblack \
  'pueue status --group GROUPNAME'
```

### 4. Ramp up parallelism as resources allow

```bash
# Check resources
ssh bigblack 'free -h | head -2 && uptime'

# Increase if headroom exists
ssh bigblack 'pueue parallel 8 --group GROUPNAME'
```

### 5. Verify completion

```bash
ssh bigblack "cd /home/tca/opendeviationbar-py && uv run --python 3.13 python -c \"
from opendeviationbar.clickhouse import OpenDeviationBarCache
with OpenDeviationBarCache() as cache:
    count = cache.count_bars('SYMBOL', THRESHOLD)
    print(f'SYMBOL@THRESHOLD: {count:,} bars')
\""
```

---

## Anti-Patterns (Hard Lessons)

### AP-1: NEVER use `opendeviationbar populate` CLI for pueue jobs

**Wrong:**

```
pueue add -- uv run opendeviationbar populate BTCUSDT --start 2024-01-01 --end 2024-01-31
```

**Why:** The Click `populate` command takes SYMBOL as positional (not `--symbol`), lacks `--ouroboros`, `--microstructure`, and uses `get_open_deviation_bars()` (no checkpointing). The CliApp-based `populate range` subcommand was never fully wired.

**Error seen:** `Error: No such option: --symbol`

### AP-2: NEVER use `cd dir &&` inside `pueue add`

**Wrong:**

```
pueue add -- cd /home/tca/opendeviationbar-py && uv run python ...
```

**Why:** SSH double-nests the shell. The `&&` gets interpreted by SSH's shell, not pueue's.

**Right:** Use `--working-directory`:

```
pueue add --working-directory /home/tca/opendeviationbar-py -- uv run python ...
```

### AP-3: NEVER use `pueue restart --all-failed` without explicit task IDs

**Wrong:**

```
pueue restart --all-failed
```

**Why:** `--all-failed` has NO group filter. It restarts **every failed task across all groups** — potentially hundreds of historical failures from months of prior work.

**Right:** Always restart by explicit IDs:

```
pueue restart 1292 1293 1294 1295
```

### AP-4: NEVER use zsh for loop variable with space-separated list

**Wrong (zsh):**

```zsh
months="01 02 03 04 05 06 07 08 09 10 11 12"
for month in $months; do  # zsh splits differently
```

**Right:** Use `bash -c` wrapper with `seq`:

```bash
/usr/bin/env bash -c '
for m in $(seq 1 12); do
    month=$(printf "%02d" $m)
```

### AP-5: NEVER ignore ouroboros mode warnings

**Warning:** `Cannot write month bars for X@Y: existing data uses year`

**Reality:** When `OPENDEVIATIONBAR_OUROBOROS_GUARD=warn` (default), this warning is **informational only**. The guard returns normally, and the INSERT proceeds. Bar count DOES increase. Verify by checking `cache.count_bars()`.

### AP-6: NEVER queue monolithic multi-year jobs

**Wrong:** Single job for 2018-2025 (22+ days for heavy thresholds)

**Right:** Per-year or per-month splits through pueue (3-4 days with per-year, minutes with per-month at light thresholds).

---

## Decision Matrix: Month vs Year Ouroboros

| Factor                    | Month Ouroboros                      | Year Ouroboros                 |
| ------------------------- | ------------------------------------ | ------------------------------ |
| Jobs per year             | 12                                   | 1                              |
| Max parallelism (2 years) | 24 concurrent                        | 2 concurrent                   |
| Memory per job            | Lower (1 month of data)              | Higher (full year of data)     |
| Best for threshold        | 1000, 750, 500 dbps (light)          | 250, 100 dbps (heavy)          |
| Completion time (1000)    | ~1 min/month = ~3 min total @ par=8  | ~12 min/year = ~12 min @ par=2 |
| Completion time (250)     | ~5-10 min/month, high memory per job | ~2-4 hours/year, manageable    |
| Checkpoint granularity    | Day-within-month                     | Day-within-year                |
| Ouroboros alignment       | Resets at month boundary             | Resets at year boundary        |

**Rule of thumb:** Use month ouroboros when `memory_per_job * max_parallel < available_memory` holds comfortably.

---

## Speed Reference (BTCUSDT on bigblack)

Empirically measured during Feb 2026 backfill:

| Threshold | Ouroboros | Time per Month | Time per Year | Parallelism Used |
| --------- | --------- | -------------- | ------------- | ---------------- |
| 1000 dbps | month     | ~1-3 min       | N/A           | 8                |
| 250 dbps  | year      | N/A            | ~2-4 hours    | 4                |

---

## Provenance

| Session | Date       | Lesson                                                                                                                                                                                                            |
| ------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 91149.. | 2026-02-28 | CLI quoting failures, pueue --all-failed disaster, ouroboros conflict                                                                                                                                             |
|         | 2026-03-01 | Canonical recipe established, 14 BTCUSDT@1000 jobs completed                                                                                                                                                      |
|         | 2026-03-04 | #159: 7 OOM kills from flat 2G cap on @1000. Root cause: memory scales with symbol volume, not just threshold. Fixed: symbol-aware `get_memory_limit(threshold, symbol)` in pueue-populate.sh, p1 parallelism 4→6 |

## Future: Temporal Migration Path

For structured backfill workloads, [Temporal](https://temporal.io/) can replace pueue + monitoring agents with built-in workflow ID dedup, retry policies (with `non_retryable_error_types`), and `max_concurrent_activities`. See `Skill(autonomous-agent-ops)` RP-7 for the comparison table. POC: `scripts/temporal_backfill_poc.py`.

**Current approach**: Pueue (simple, zero infra, SSH-native) + `Skill(autonomous-agent-ops)` guardrails.
**Future approach**: Temporal workflows with `populate_cache_resumable()` as activities — eliminates need for monitoring agents entirely.

---

## References

- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) — Pueue ops, per-year parallelization
- `Skill(opendeviationbar-job-safety)` — Memory profiles, per-symbol thresholds
- `Skill(autonomous-agent-ops)` — Multi-agent monitoring guardrails (from 2026-03-04 incident)
- `Skill(devops-tools:pueue-job-orchestration)` — Pueue commands
- `Skill(devops-tools:distributed-job-safety)` — Universal invariants
