# Stack Research: Seeder REST API Throughput Maximization

**Researched:** 2026-03-23
**Domain:** Binance REST API rate limits, optimal request patterns, throughput ceiling for seeder as sole REST consumer
**Confidence:** HIGH (verified against Binance official docs, codebase implementation, and production telemetry)

---

## 1. Binance REST API Rate Limit Architecture

### Hard Numbers (Verified)

| Parameter                 | Value                      | Source                                                                                                                                    | Confidence |
| ------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| REQUEST_WEIGHT per IP     | **6,000 / minute**         | [Binance announcement](https://www.binance.com/en/support/announcement/detail/9820396bf54644c39e666b4780622846), `exchangeInfo` endpoint  | HIGH       |
| aggTrades endpoint weight | **4 per call**             | `historical.rs:756`, [Binance market data docs](https://developers.binance.com/docs/binance-spot-api-docs/rest-api/market-data-endpoints) | HIGH       |
| aggTrades max `limit`     | **1,000 trades**           | `historical.rs:742`, Binance docs                                                                                                         | HIGH       |
| Rate limit scope          | **IP-based** (not API key) | [Binance limits docs](https://developers.binance.com/docs/binance-spot-api-docs/rest-api/limits)                                          | HIGH       |
| Feedback header           | `X-MBX-USED-WEIGHT-1m`     | `rate_limiter.rs:216`, Binance docs                                                                                                       | HIGH       |
| Window type               | **1 minute rolling**       | Binance docs, `rate_limiter.rs:128` (60-second window)                                                                                    | HIGH       |
| 429 response              | Rate limit exceeded        | Binance docs                                                                                                                              | HIGH       |
| 418 response              | IP auto-banned             | Escalating: 2 min to 3 days for repeat offenders                                                                                          | HIGH       |
| Retry-After header        | Seconds to wait            | Present on both 429 and 418 responses                                                                                                     | HIGH       |

### What "6000 Weight Per Minute" Actually Means

The weight budget is IP-wide, not per-connection or per-API-key. All processes on bigblack (sidecar, kintsugi, seeder) share one 6000/min pool. There is no way to get more budget without a second IP.

The window is rolling: Binance tracks a 1-minute sliding window per IP. Once you hit 6000 used weight in any 60-second period, all subsequent requests return 429 until enough time passes for older weight to expire.

### No Documented Connection Limits for REST

Binance does not publish a hard concurrent TCP connection limit for REST. The practical limit is the weight budget: you can open 100 connections simultaneously, but each request still costs weight. The codebase uses `DEFAULT_CONCURRENCY = 3` (parallel_fetch.rs:37) per symbol and `Semaphore(4)` for concurrent symbols in fill_from_rest.

---

## 2. Current Budget Allocation (v1.4)

From `deploy/opendeviationbar.env` and `ratelimit/file_coordinator.py`:

| Consumer  | Budget (weight/min) | Ceiling Env Var                               | Purpose                                              |
| --------- | ------------------- | --------------------------------------------- | ---------------------------------------------------- |
| Sidecar   | 2,400               | `OPENDEVIATIONBAR_REST_CEILING=2400`          | Puck gap fill, shadow validation, midnight preflight |
| Kintsugi  | 2,000               | `OPENDEVIATIONBAR_REST_CEILING=2400` (in env) | REST fallback for repairs                            |
| Seeder    | 800                 | Not yet set as ceiling                        | REST fallback for T-1/recent days                    |
| Reserve   | 800                 | --                                            | Safety buffer                                        |
| **Total** | **6,000**           |                                               |                                                      |

### v2.0 Opportunity: Seeder as Sole REST Owner

When kintsugi becomes Parquet-only (v2.0 goal), the budget redistribution is:

| Consumer | Old Budget | New Budget | Change                            |
| -------- | ---------- | ---------- | --------------------------------- |
| Sidecar  | 2,400      | 2,400      | No change (Puck still needs REST) |
| Kintsugi | 2,000      | **0**      | Parquet-only, no REST             |
| Seeder   | 800        | **2,800**  | Absorbs kintsugi's budget         |
| Reserve  | 800        | 800        | Safety buffer unchanged           |

With 20% safety margin: usable seeder budget = `min(2800, 6000 * 0.80) = 2800` weight/min.

**But the real ceiling depends on sidecar co-existence.** Sidecar's Puck gap fill is bursty (consumes 200+ weight in seconds when a gap is detected). The safe seeder ceiling with sidecar running:

- **Steady state** (sidecar idle, no gaps): seeder can use up to 4,800 - sidecar_baseline (~50 for shadow validation, preflight) = **~4,700 weight/min**
- **During sidecar Puck burst**: seeder should back off. X-MBX-USED-WEIGHT-1m header provides real-time feedback.
- **Recommended seeder ceiling**: **3,600 weight/min** (leaves 1,200 for sidecar bursts + 800 reserve + margin)

---

## 3. Throughput Ceiling Math

### Per-Request Economics

| Metric                           | Value       | Derivation         |
| -------------------------------- | ----------- | ------------------ |
| Weight per aggTrades call        | 4           | Binance fixed cost |
| Trades per call (max)            | 1,000       | `limit=1000`       |
| Trades per weight unit           | 250         | 1000 / 4           |
| Calls per minute @ 3,600 budget  | 900         | 3,600 / 4          |
| Trades per minute @ 3,600 budget | **900,000** | 900 \* 1000        |

### Per-Symbol Daily Volume (Observed)

| Symbol             | Avg trades/day   | REST calls for 1 day | Weight for 1 day | Time @ 3,600 budget |
| ------------------ | ---------------- | -------------------- | ---------------- | ------------------- |
| BTCUSDT            | ~750,000         | 750                  | 3,000            | ~50s                |
| ETHUSDT            | ~450,000         | 450                  | 1,800            | ~30s                |
| SOLUSDT            | ~300,000         | 300                  | 1,200            | ~20s                |
| Low-vol (FILUSDT)  | ~30,000          | 30                   | 120              | ~2s                 |
| **All 18 symbols** | ~3,500,000 total | ~3,500               | ~14,000          | **~3.9 min**        |

### Throughput Ceiling: Seeder Seeding 1 Full Day

**At 3,600 weight/min budget, seeding all 18 symbols for 1 day takes ~4 minutes.**

This means the 5-minute seeder timer can comfortably seed yesterday's data for all 18 symbols in a single run, with budget to spare.

### Historical Backfill Rate

For catching up on historical days (Vision 404 fallback for recent days):

| Scenario                  | Days seedable / hour | At budget |
| ------------------------- | -------------------- | --------- |
| BTCUSDT only              | ~72 days/hour        | 3,600/min |
| All 18 symbols            | ~15 days/hour        | 3,600/min |
| BTCUSDT only (full 4,800) | ~96 days/hour        | 4,800/min |

These rates are for REST-only fallback. Vision download is vastly faster (no weight cost, 48 files/min at 12 concurrent downloads). REST fallback should only be used for the ~1-7 day window where Vision hasn't published yet.

---

## 4. Optimal Request Patterns

### Pattern 1: fromId Pagination (Recommended, Already Implemented)

```
GET /api/v3/aggTrades?symbol=BTCUSDT&fromId=123456789&limit=1000
```

**Why fromId over startTime/endTime:**

- **Zero-gap guarantee**: `fromId` returns trades starting from that exact ID. No trades dropped at millisecond boundaries.
- **Deterministic pagination**: `next_from_id = last_trade.agg_trade_id + 1`. No overlap, no gap.
- **Already proven**: The codebase uses this exclusively for pagination (`historical.rs:736-741`).

**Anti-pattern**: `startTime + endTime` drops trades when multiple trades share the same millisecond timestamp. The codebase already avoids this (first page only uses startTime to find the entry point, then switches to fromId).

### Pattern 2: Parallel Fetch with Early Termination (Already Implemented)

Current implementation in `parallel_fetch.rs`:

```
Concurrency: 3 per symbol (DEFAULT_CONCURRENCY)
Symbol concurrency: 4 (Semaphore(4) in fill_from_rest)
Page size: 1,000 trades
Early termination: AtomicBool flag on short page
```

**For seeder as sole REST owner, recommended adjustments:**

| Parameter              | Current                    | Recommended | Rationale                                       |
| ---------------------- | -------------------------- | ----------- | ----------------------------------------------- |
| Per-symbol concurrency | 3                          | **5**       | Seeder is sole consumer, can use more bandwidth |
| Symbol concurrency     | N/A (seeder is sequential) | **3**       | Seed 3 symbols in parallel within budget        |
| Chunk size             | 50,000 trades              | **50,000**  | Keep same, proven chunk boundary                |

With 3 symbols x 5 pages concurrent = 15 simultaneous REST calls = 60 weight burst. At 500ms RTT per call, that's 60 weight every ~500ms = 7,200 weight/min theoretical burst rate. The rate limiter will naturally pace this down to the 3,600/min ceiling.

### Pattern 3: X-MBX-USED-WEIGHT-1m Feedback Loop (Already Implemented)

The `AdaptiveRateLimiter` in `rate_limiter.rs` already implements the optimal pattern:

1. **Local tracking**: CAS atomic counter tracks locally-consumed weight
2. **Server sync**: `update_from_headers()` reads `X-MBX-USED-WEIGHT-1m` from every response
3. **Conservative merge**: `max(server_reported, local_tracked)` ensures we never underestimate
4. **Window reset**: Every 60 seconds, both counters reset to 0
5. **Dynamic ceiling**: `set_ceiling()` allows runtime budget adjustment

**Key insight for seeder**: When seeder is sole REST consumer, the server header will always reflect seeder's own usage (plus any sidecar Puck bursts). If `X-MBX-USED-WEIGHT-1m` jumps unexpectedly, it means sidecar had a Puck burst. The seeder should automatically back off because the rate limiter's `max(server, local)` logic will see the higher server value.

### Pattern 4: Batch Size Optimization for REST Seeding

For seeding a full day's data via REST:

**Step 1: Find day boundaries** (2 REST calls = 8 weight)

```python
# First trade of the day
first_trades = fetch_aggtrades_rest(symbol, day_start_ms, day_start_ms + 60_000)
# Last trade of the day (30-min window for low-volume symbols)
last_trades = fetch_aggtrades_rest(symbol, day_end_ms - 1_800_000, day_end_ms)
```

**Step 2: Bulk fetch via parallel fromId** (N calls = 4N weight)

```python
trades = fetch_aggtrades_parallel(symbol, first_trade_id, last_trade_id + 1)
```

**Total weight per day per symbol**: `8 + 4 * ceil(trades_per_day / 1000)`

For BTCUSDT: `8 + 4 * 750 = 3,008 weight` (fits in one 60s window at 3,600 budget).

---

## 5. Recommended Stack Changes for v2.0

### Core Technology (Already in Project)

| Technology                        | Version | Purpose                     | Change for v2.0               |
| --------------------------------- | ------- | --------------------------- | ----------------------------- |
| `AdaptiveRateLimiter` (Rust)      | In-tree | Per-process rate limiting   | Raise seeder ceiling to 3,600 |
| `fetch_aggtrades_parallel` (Rust) | In-tree | Parallel REST fetch         | Increase concurrency to 5     |
| `RateLimitCoordinator` (Python)   | In-tree | Cross-process state sharing | Update budget constants       |
| `TickStorage.write_ticks()`       | In-tree | Parquet write with dedup    | No change                     |

### Configuration Changes

```ini
# deploy/opendeviationbar.env — v2.0 changes

# Kintsugi: no REST budget (Parquet-only)
OPENDEVIATIONBAR_REST_CEILING=0  # Was 2400

# Seeder: absorb kintsugi's budget
# Set via seeder service Environment= directive
# OPENDEVIATIONBAR_REST_CEILING=3600  # Was effectively 800
```

### New Seeder Config Fields

```python
# python/opendeviationbar/config/seeder.py additions
class SeederConfig(BaseSettings):
    # ... existing fields ...
    rest_concurrency_per_symbol: int = 5      # Was 3 (DEFAULT_CONCURRENCY)
    rest_concurrent_symbols: int = 3          # Seed N symbols in parallel
    rest_budget_weight_per_min: int = 3600    # Seeder's share of 6000 budget
```

### Budget Allocation Constants Update

```python
# python/opendeviationbar/ratelimit/file_coordinator.py
BUDGET_SIDECAR: int = 2400    # Unchanged
BUDGET_KINTSUGI: int = 0      # Was 2000 — Parquet-only
BUDGET_SEEDER: int = 2800     # Was 800 — absorbs kintsugi budget
BUDGET_RESERVE: int = 800     # Unchanged
BUDGET_TOTAL: int = 6000
```

### Alternatives Considered

| Category                   | Recommended               | Alternative             | Why Not                                                              |
| -------------------------- | ------------------------- | ----------------------- | -------------------------------------------------------------------- |
| Rate limit budget          | Seeder 2800 + reserve 800 | Seeder 3600 + reserve 0 | Reserve absorbs sidecar Puck bursts                                  |
| Parallel fetch concurrency | 5 per symbol              | 10 per symbol           | Diminishing returns past 5; RTT is the bottleneck                    |
| Symbol concurrency         | 3 parallel                | 6 parallel              | More than 3 causes weight bursts that waste time in rate limit waits |
| Coordination mechanism     | File-based (current)      | Redis-based             | File-based is proven, zero-dependency, already deployed              |

---

## 6. Pitfalls Specific to Sole-Owner Transition

### Pitfall 1: Sidecar Puck Burst During Seeder Heavy Load

**What goes wrong:** Seeder is consuming 3,500 weight/min when sidecar detects a WS gap and fires Puck. Puck needs 200+ weight immediately for parallel REST fill. Combined usage exceeds 4,800 usable budget.
**Prevention:** X-MBX-USED-WEIGHT-1m header feedback loop. When server reports higher usage than seeder's local tracking, the rate limiter automatically throttles. The 800 reserve budget is specifically for this.
**Detection:** Rate limiter logs "budget exhausted, waiting for window reset" during Puck bursts.

### Pitfall 2: fromId + endTime = HTTP 400

**What goes wrong:** Combining `fromId` with `endTime` in the same REST call causes Binance to return HTTP 400.
**Prevention:** Already handled in `historical.rs:737-738`: "Binance does not allow fromId + endTime together (HTTP 400). We filter by end_ms post-fetch instead."
**Relevance:** Seeder must continue using the two-step pattern: time-based first call to find entry point, then fromId-only pagination.

### Pitfall 3: Over-Counting Weight When Server Header Lags

**What goes wrong:** Multiple concurrent REST calls in-flight. Server header on first response shows X=100. By the time second response arrives, server shows X=108 (our 2 calls = 8 weight). But local tracking already counted 8. `max(108, 8) = 108`. Next call thinks 108 weight used when only 8 is ours.
**Why this is actually safe:** Being conservative (overestimating usage) is correct behavior. The rate limiter slows down rather than risking 429. The window resets every 60s, so any over-counting is transient.

### Pitfall 4: Seeder REST Fallback Creating Stathera Gaps

**What goes wrong:** REST fallback for yesterday returns partial data due to rate limiting mid-fetch. Kintsugi later finds a Stathera gap at the partial-complete boundary.
**Prevention:** Trade-ID continuity validation after every REST fetch (D-07). The `.parquet.meta` sidecar file marks REST-sourced files as `complete=false`. Vision overwrites when available (D-08).
**Detection:** Kintsugi Stathera audit discovers gaps; heartbeat alerts.

---

## 7. Throughput Improvement Opportunities (Beyond Budget Reallocation)

### 7a. HTTP Keep-Alive (Already Used)

reqwest's connection pool reuses TCP connections by default. For sequential fromId pagination to the same endpoint, this eliminates TLS handshake overhead after the first request. Verified: `HistoricalDataLoader` creates one `Client` per loader instance, which maintains a connection pool.

### 7b. Reduce Boundary-Finding Overhead

Current REST seeding makes 2 boundary-finding calls (day start + day end) before the bulk fetch. For the seeder, an optimization: maintain a "last known trade ID" per symbol per day from ClickHouse, and use that to skip the boundary-finding step for days where we have partial data.

Weight savings: 8 weight per day per symbol (minor, but 8 _18 symbols_ 7 days = 1,008 weight for a full week catch-up).

### 7c. Pipeline REST Across Symbols

Instead of finishing one symbol before starting the next, interleave requests across symbols. While symbol A waits for a response (500ms RTT), fire requests for symbols B and C. This maximizes REST call throughput within the weight budget.

Already partially implemented via `fetch_aggtrades_parallel` for within-symbol parallelism. For cross-symbol parallelism, the seeder would need to manage multiple `fetch_aggtrades_parallel` calls concurrently with a shared rate limiter.

---

## Sources

### Primary (HIGH confidence)

- `crates/opendeviationbar-providers/src/binance/rate_limiter.rs` -- Full AdaptiveRateLimiter implementation
- `crates/opendeviationbar-providers/src/binance/parallel_fetch.rs` -- Parallel fetch with early termination
- `crates/opendeviationbar-providers/src/binance/historical.rs` -- REST fetch, weight=4 (line 756)
- `python/opendeviationbar/ratelimit/file_coordinator.py` -- Cross-process coordination
- `deploy/opendeviationbar.env` -- Production budget allocation
- `.planning/debug/binance-rest-rate-limiting.md` -- fill_from_rest budget exhaustion analysis
- `.planning/milestones/v1.4-phases/05-seeder-rest-ratelimit/05-RESEARCH.md` -- Phase 5 research

### Official Binance Documentation (HIGH confidence)

- [Binance Spot API Limits](https://developers.binance.com/docs/binance-spot-api-docs/rest-api/limits) -- Rate limit framework
- [Binance Market Data Endpoints](https://developers.binance.com/docs/binance-spot-api-docs/rest-api/market-data-endpoints) -- aggTrades weight=4
- [Binance Rate Limit Increase Announcement](https://www.binance.com/en/support/announcement/detail/9820396bf54644c39e666b4780622846) -- 6000/min since Aug 2023
- [Binance GitHub rest-api.md](https://github.com/binance/binance-spot-api-docs/blob/master/rest-api.md) -- 418 ban, Retry-After

### Codebase Evidence (HIGH confidence)

- `scripts/tick_cache_seeder.py` -- Current REST fallback implementation (\_rest_fallback_day)
- `python/opendeviationbar/config/seeder.py` -- SeederConfig pydantic-settings
