# Architecture Patterns: Seeder REST Throughput Maximization

**Domain:** Binance REST API sole-owner optimization
**Researched:** 2026-03-23

## Recommended Architecture

### Component Boundaries (v2.0)

| Component | Responsibility                                            | REST Usage                                |
| --------- | --------------------------------------------------------- | ----------------------------------------- |
| Sidecar   | WS streaming, Puck real-time gap fill, midnight preflight | Puck: bursty 200+ weight, Shadow: ~50/min |
| Seeder    | Vision download + REST fallback for T-1 to T-7            | Sole heavy consumer: up to 3,600/min      |
| Kintsugi  | Parquet-only gap repair                                   | **Zero REST** (defers cache-miss shards)  |
| Heartbeat | Monitoring + telemetry                                    | Zero REST (reads ClickHouse only)         |

### Data Flow: REST Budget

```
Binance REST API (6000 weight/min per IP)
         |
         v
  X-MBX-USED-WEIGHT-1m header (authoritative, IP-wide)
         |
    +-----------+-----------+
    |           |           |
    v           v           v
 Sidecar    Seeder      Kintsugi
 (Puck)    (primary)   (zero REST)
  2400      3600          0
    |           |           |
    v           v           v
 AdaptiveRateLimiter (per-process, Rust)
    |           |
    v           v
 RateLimitCoordinator (cross-process, Python, file-based)
         |
         v
 /tmp/opendeviationbar-ratelimit.json (shared state)
```

### Rate Limiter Layering

The system has two independent rate limiter layers:

**Layer 1: In-Process (Rust AdaptiveRateLimiter)**

- One per process (OnceLock singleton in binance_bindings.rs)
- CAS-based atomic weight tracking
- Server header feedback via X-MBX-USED-WEIGHT-1m
- Dynamic ceiling via set_ceiling()
- 60-second window with automatic reset

**Layer 2: Cross-Process (Python RateLimitCoordinator)**

- Shared JSON state file at /tmp/opendeviationbar-ratelimit.json
- Atomic rename for write consistency
- 65-second stale expiry for crashed consumer cleanup
- Per-consumer weight reporting

**Key insight:** Layer 1 is the enforcement layer (actually blocks requests). Layer 2 is the coordination layer (informs budget decisions). With seeder as sole heavy REST consumer, Layer 2 becomes less critical -- the server header feedback in Layer 1 already reflects IP-wide usage including sidecar Puck bursts.

## Patterns to Follow

### Pattern 1: Budget-Aware REST Seeding

**What:** Seeder checks remaining budget before each symbol's REST fetch, yielding to sidecar Puck when server header shows unexpected weight consumption.

**When:** Every REST seeding cycle (5-min timer).

```python
def _seed_symbol_rest(symbol: str, date_str: str, rate_limiter) -> bool:
    """Seed one day via REST with budget awareness."""
    # Check if enough budget for this symbol's estimated needs
    estimated_weight = estimate_day_weight(symbol)  # ~3000 for BTCUSDT
    if rate_limiter.remaining() < estimated_weight * 0.5:
        logger.info("Deferring %s REST seed: only %d weight remaining",
                     symbol, rate_limiter.remaining())
        return False  # Defer to next timer cycle

    # Proceed with fetch -- rate limiter handles pacing
    trades = fetch_aggtrades_parallel(symbol, first_id, last_id)
    return True
```

### Pattern 2: Graceful Degradation on Budget Exhaustion

**What:** When rate limiter reports budget exhaustion mid-fetch, save progress and resume next cycle rather than blocking.

**When:** Any REST fetch that spans multiple rate limit windows.

**Why:** A 5-minute timer cycle should not block for 60+ seconds waiting for window reset. Better to write what we have and resume next cycle.

## Anti-Patterns to Avoid

### Anti-Pattern 1: Removing Safety Margin to Get More Budget

**What:** Setting `safety_margin=0.0` to use full 6000 weight/min.
**Why bad:** One Puck burst + seeder load = 429 response = IP ban escalation (2 min to 3 days).
**Instead:** Keep 20% margin. The 4,800 usable budget is more than enough for all use cases.

### Anti-Pattern 2: Separate Rate Limiter Instance Per Fetch

**What:** Creating a new `AdaptiveRateLimiter` for each REST seeding call.
**Why bad:** New instance has no history of current window's weight consumption. Will immediately over-consume.
**Instead:** Use the OnceLock singleton (already the pattern in binance_bindings.rs).

### Anti-Pattern 3: Polling RateLimitCoordinator in a Tight Loop

**What:** Calling `can_acquire()` in a spin loop waiting for budget to become available.
**Why bad:** File I/O on every iteration, no benefit over letting the Rust rate limiter handle pacing.
**Instead:** Let AdaptiveRateLimiter.acquire() sleep for window reset. Check coordinator only at batch boundaries.

## Scalability Considerations

| Concern                      | At current (18 symbols)       | At 50 symbols                            | At 200 symbols   |
| ---------------------------- | ----------------------------- | ---------------------------------------- | ---------------- |
| REST budget per symbol/day   | ~800 weight avg               | ~280 weight avg                          | ~70 weight avg   |
| Full-day REST seed time      | ~4 min                        | ~11 min                                  | ~44 min          |
| Vision vs REST ratio         | Vision primary, REST T-1 only | Vision critical, REST infeasible for all | Vision mandatory |
| Rate limit window efficiency | ~75%                          | ~60% (more wait time)                    | Impractical      |

**Verdict:** At current scale (18 symbols), REST seeding for T-1 is comfortable. At 50+ symbols, Vision becomes the only viable bulk source, with REST reserved for a small critical subset. The v2.0 architecture is correct: Vision-primary, REST-fallback.
