# Research Summary: v2.0 Seeder as Sole REST Owner

**Domain:** Service integration architecture for REST API consolidation
**Researched:** 2026-03-23
**Overall confidence:** HIGH (all findings from direct code analysis + production config)

## Executive Summary

The v2.0 milestone consolidates all Binance REST API usage into the seeder service, making kintsugi a Parquet-only consumer. This is a service boundary change, not a data model change -- the Parquet tick cache becomes the sole interface between the REST API world (seeder) and the bar construction world (kintsugi).

The architecture change is clean because the seeder already populates the Parquet cache proactively (5-min timer, newest-first, REST fallback for T-1). Kintsugi already prefers Parquet as its highest-priority source (`_try_parquet_day_replace()` is Step 1 in the cascade). The change removes Steps 2 (Vision) and 3 (REST) from kintsugi's repair path and replaces them with a cache-miss deferral.

The critical discovery is **P2: Forex symbol-to-cache-key naming mismatch.** Kintsugi uses `shard.symbol` (e.g., "EURUSD") for Parquet lookups, but the Exness seeder writes under `tick_cache_key` (e.g., "EXNESS_RAW_SPREAD_EURUSD"). This WILL break all 10 forex symbols on day 1 unless resolved. Additionally, **P3: the 30-day dismiss window** will permanently lose cache-miss shards unless a new `deferred_cache_miss` state is introduced that bypasses `kintsugi_log`.

The sidecar is unaffected -- its REST usage (startup `fill_from_rest`, midnight preflight, Puck gap-fill) is real-time critical, low-volume, and self-contained for today-only data that the seeder does not cover.

## Key Findings

**Stack:** Rate limit coordinator keeps its current file-based design; update budget constants (kintsugi 2000 -> 0, seeder 800 -> 2800). Seeder REST concurrency can increase from 3 to 5 per symbol. All 18 symbols seedable via REST in ~4 minutes at 3600 weight/min budget.

**Architecture:** Data flow becomes REST -> Seeder -> Parquet -> Kintsugi -> ClickHouse. Sidecar's WS -> ClickHouse path is independent. Three-tier concurrency classification in orchestrator.py is replaced by always-64 Parquet workers.

**Critical pitfall:** Forex naming mismatch (P2) is a day-1 blocker. The `tick_cache_key` from `symbols.toml` must be resolved before Parquet lookups. Additionally, cache-miss shards must never be logged to `kintsugi_log` to avoid 30-day dismissal (P3).

## Implications for Roadmap

Based on research, suggested phase structure:

1. **Kintsugi Parquet-only core** - Strip Vision/REST from `_repair_under_lock()`, add `deferred_cache_miss` state
   - Addresses: FEATURES (Parquet-only repair, cache-miss deferral, 64 workers)
   - Avoids: P3 (dismissal window), P1 (unbounded gap accumulation)

2. **Forex naming fix + pre-migration audit** - Resolve `tick_cache_key` in Parquet lookups, verify coverage
   - Addresses: FEATURES (forex support)
   - Avoids: P2 (naming mismatch), P4 (historical coverage gaps)

3. **Rate limit simplification** - Update budget constants, remove dead config
   - Addresses: FEATURES (budget update, dead config removal)
   - Avoids: Stale config confusion

4. **OOM hardening** - Arrow zero-copy path, allocator purge, us normalization
   - Addresses: FEATURES (OOM prevention, repair_direct_parquet patterns)
   - Avoids: P6 (deferred shard pile-up at 64 workers)

**Phase ordering rationale:**

- Phase 1 before Phase 2: The core Parquet-only behavior must work before forex-specific fixes matter. Forex symbols already fail on Vision/REST fallback today (404/400 errors), so they are no worse off temporarily.
- Phase 2 before Phase 3: Naming fix is a correctness requirement; rate limit simplification is cleanup.
- Phase 3 can run in parallel with Phase 2 (independent changes).
- Phase 4 after Phase 1: OOM hardening is only critical when running at 64 workers (enabled by Phase 1).

**Research flags for phases:**

- Phase 1: Standard patterns, well-understood code paths. Low research risk.
- Phase 2: Needs production audit of Parquet cache coverage per symbol. Run `storage.list_days(symbol)` for all 18 symbols before committing to deployment.
- Phase 4: Arrow zero-copy path needs validation that `process_trades_arrow()` is available in the current Rust build. Confirmed in `repair_direct_parquet.py` usage but kintsugi may need different import paths.

## Confidence Assessment

| Area         | Confidence | Notes                                                                                                                              |
| ------------ | ---------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Stack        | HIGH       | All rate limit mechanics verified from source code + Binance docs                                                                  |
| Features     | HIGH       | Every feature traced to specific lines in repair.py, orchestrator.py                                                               |
| Architecture | HIGH       | Data flow verified from all 3 service entry points                                                                                 |
| Pitfalls     | HIGH       | P2 (forex naming) confirmed by cross-referencing seeder + kintsugi code. P3 (dismiss window) confirmed from kintsugi_log mechanism |

## Gaps to Address

- **Parquet cache coverage audit:** Before deployment, must verify all 18 symbols have complete Parquet coverage for their historical range. Not yet measured.
- **64-worker empirical validation:** The 32-worker cliff was observed for Vision-download shards. Pure Parquet at 64 workers should not hit the same ClickHouse contention (INSERT-only vs DELETE+INSERT), but needs production confirmation.
- **Exness seeder reliability:** `ticks.ex2archive.com` is a single point of failure for all 10 forex symbols. No redundant data source exists. Monitoring needed.
- **`deferred_cache_miss` pass-level behavior:** When 90%+ of shards are deferred, kintsugi's pass takes <1s and immediately polls again at 60s interval. This is harmless (no wasted work) but may look anomalous in telemetry. Needs documentation.

## Sources

All findings from direct code reading:

- `.planning/research/ARCHITECTURE.md` -- Service integration, data flow, rate limit simplification
- `.planning/research/FEATURES.md` -- Code paths to remove, OOM patterns to adopt
- `.planning/research/STACK.md` -- REST API rate limits, throughput math, budget allocation
- `.planning/research/PITFALLS.md` -- 10 ranked pitfalls with prevention strategies
