# Domain Pitfalls: Seeder REST Throughput Maximization

**Domain:** Binance REST API sole-owner optimization
**Researched:** 2026-03-23

## Critical Pitfalls

Mistakes that cause 429 bans, data loss, or cascading failures.

### Pitfall 1: IP Ban Escalation from 429 Storms

**What goes wrong:** Seeder consumes near-budget weight. Sidecar Puck fires a burst. Combined weight exceeds 6000. Binance returns 429. If the retry logic immediately retries without respecting Retry-After, repeated 429s escalate to HTTP 418 (IP ban) lasting 2 minutes to 3 days.
**Why it happens:** Two independent rate limiter instances (sidecar Rust, seeder Rust) each think they have budget. Server header feedback has ~500ms latency.
**Consequences:** IP ban blocks ALL Binance REST access -- sidecar, seeder, and any manual queries. 3-day ban at maximum escalation.
**Prevention:** (1) 800-weight reserve in budget allocation absorbs burst overlap. (2) AdaptiveRateLimiter already uses max(server, local) -- conservative by design. (3) `send_with_rate_limit_backoff()` in historical.rs retries 429 with exponential backoff (1s-2s-4s-8s-16s cap), not immediate retry.
**Detection:** Rate limiter logs "budget exhausted, waiting for window reset". Heartbeat should alert on 429 count > 0 in production logs.

### Pitfall 2: Removing Kintsugi REST Before Seeder Covers All Days

**What goes wrong:** Kintsugi is made Parquet-only. Seeder hasn't yet seeded days T-2 through T-7. Kintsugi finds a gap at T-3, has no Parquet cache, defers the shard. The gap persists until the next seeder cycle covers T-3.
**Why it happens:** Temporal ordering: kintsugi's repair pass runs faster than seeder's batch rotation (18 symbols / 6 per batch = 3 batches = 15 min for full rotation).
**Consequences:** Delayed gap repair. Not data loss (kintsugi defers, doesn't drop), but increased time-to-repair.
**Prevention:** Ensure seeder's REST fallback window (`rest_fallback_max_age_days=7`) covers the same range kintsugi previously covered via REST. Run seeder at higher frequency during transition (every 2 min instead of 5 min) until the Parquet cache is warm for all recent days.
**Detection:** Kintsugi deferred-shard count in Telegram pass summary. If deferred shards are growing, seeder isn't keeping up.

### Pitfall 3: fromId + endTime = HTTP 400

**What goes wrong:** A code change adds `endTime` parameter to a REST call that also uses `fromId`. Binance returns HTTP 400 (bad request).
**Why it happens:** Binance explicitly forbids combining `fromId` with `endTime` in the same aggTrades request. This constraint is documented in `historical.rs:737-738`.
**Consequences:** REST fetch fails for that page. If not caught, the day's data is incomplete.
**Prevention:** Never add `endTime` to fromId-paginated calls. Filter by timestamp post-fetch instead (already the pattern).
**Detection:** HTTP 400 in seeder logs. The existing `send_with_rate_limit_backoff()` does NOT retry 400s (non-transient 4xx except 429 returned immediately).

## Moderate Pitfalls

### Pitfall 4: Cross-Process Window Desynchronization

**What goes wrong:** Seeder's local rate limiter window resets at T=0s. Sidecar's resets at T=30s. At T=45s, seeder thinks it has 15s of budget remaining, fires a burst. Sidecar simultaneously fires a Puck burst. Combined server-side weight exceeds budget.
**Prevention:** X-MBX-USED-WEIGHT-1m header is the ground truth. Both processes read it from every response. The `fetch_max` atomic operation in `update_from_headers()` ensures the highest server-reported value is always used. Window desync causes brief under-utilization (not over-utilization) because each process's local window reset clears both local and server counters.

### Pitfall 5: Stale RateLimitCoordinator State After Seeder Crash

**What goes wrong:** Seeder crashes mid-fetch. Its weight report (e.g., "seeder: 2000 weight used") stays in the JSON state file. Sidecar reads it, thinks budget is 2000 weight tighter than reality, throttles Puck gap fill unnecessarily.
**Prevention:** Already handled. Each entry has an `updated` timestamp. Entries older than 65 seconds (WINDOW_SECONDS + STALE_GRACE_SECONDS) are ignored when computing aggregate usage (`file_coordinator.py:97-98`).

### Pitfall 6: REST Fallback Producing Partial-Day Parquet Files

**What goes wrong:** REST fetch for yesterday is interrupted by rate limit exhaustion. Only 500K of 750K BTCUSDT trades are fetched. The partial file is written to Parquet and marked `complete=false`. Kintsugi reads the partial file, processes it, and the resulting bars have a Stathera gap where the missing 250K trades should be.
**Prevention:** (1) Trade-ID continuity validation after fetch (D-07) catches gaps. (2) If continuity check fails, the REST fallback returns None and the day is not cached. (3) Vision overwrites REST-sourced files on subsequent runs (D-08).

## Minor Pitfalls

### Pitfall 7: Seeder REST Config Not in SeederConfig

**What goes wrong:** REST parallelism and budget configuration is hardcoded or in a different config location than SeederConfig. Operators cannot tune seeder REST behavior via environment variables.
**Prevention:** Add `rest_concurrency_per_symbol`, `rest_concurrent_symbols`, and `rest_budget_weight_per_min` to SeederConfig with pydantic-settings env var support.

### Pitfall 8: Weight Undercount from fetch_aggtrades_rest First Page

**What goes wrong:** The boundary-finding step uses `fetch_aggtrades_rest()` which costs 4 weight per call but the seeder's local budget tracking might not account for it if it bypasses the rate limiter.
**Prevention:** Verify that `fetch_aggtrades_rest()` uses the same rate limiter singleton as `fetch_aggtrades_parallel()`. Both go through `send_with_rate_limit_backoff()` which acquires weight from the limiter.

## Phase-Specific Warnings

| Phase Topic                 | Likely Pitfall                           | Mitigation                                                                   |
| --------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------- |
| Budget reallocation         | Kintsugi env var not updated on bigblack | Deploy checklist: verify OPENDEVIATIONBAR_REST_CEILING=0 in kintsugi service |
| Seeder parallelism increase | Burst weight exceeds budget              | Empirical tuning on bigblack, like kintsugi 24-worker profiling              |
| Kintsugi REST stripping     | Deferred shards pile up                  | Monitor deferred count; seeder must cover T-1 to T-7 before stripping        |
| Cross-symbol parallelism    | Shared rate limiter contention           | Use Arc<AdaptiveRateLimiter> across tasks (already the pattern)              |

## Sources

- `.planning/debug/binance-rest-rate-limiting.md` -- fill_from_rest budget exhaustion root cause analysis
- `crates/opendeviationbar-providers/src/binance/rate_limiter.rs` -- Rate limiter implementation details
- `crates/opendeviationbar-providers/src/binance/historical.rs:737-738` -- fromId + endTime constraint
- `python/opendeviationbar/ratelimit/file_coordinator.py` -- Stale entry expiry logic
- [Binance API limits docs](https://developers.binance.com/docs/binance-spot-api-docs/rest-api/limits) -- 429/418 escalation
- Production telemetry from `.planning/debug/binance-rest-rate-limiting.md` -- Budget exhaustion timing
