---
name: source-validation
description: Validate three-tier data source joining (Vision, REST, WebSocket) for Ouroboros-based open deviation bar construction. Use when investigating data gaps, verifying source continuity, debugging agg_trade_id junctions, onboarding new data sources, or planning reconstruction pipelines.
---

# Source Validation

Empirically validate that Binance's three data tiers produce a seamless, gap-free sequence of `agg_trade_id` values across Ouroboros boundaries. This skill captures the first-principles knowledge of how raw trade data flows from historical archives to live streams.

## When to Use

- Verifying Vision → REST → WebSocket junction continuity
- Debugging data gaps in the ClickHouse cache pipeline
- Onboarding a new symbol (prove all 3 sources join cleanly)
- After changing pagination strategy in `opendeviationbar-providers`
- Validating day-mode Ouroboros boundary behavior (UTC midnight)
- Planning or modifying the reconstruction pipeline

## Three-Tier Data Sources

| Tier          | Source                                                      | Latency   | Coverage     | Pagination Key                       |
| ------------- | ----------------------------------------------------------- | --------- | ------------ | ------------------------------------ |
| **Vision**    | `data.binance.vision` daily ZIPs                            | T-2 days  | Historical   | Date-based (daily files)             |
| **REST**      | `/api/v3/aggTrades`                                         | Real-time | Any ID range | `fromId` (MUST use, NOT `startTime`) |
| **WebSocket** | Combined streams (recommended, #279) or per-symbol (legacy) | Live      | Present only | Push stream (monotonic IDs)          |

## Critical Findings

### fromId vs startTime (Empirically Proven)

**`startTime` pagination DROPS TRADES.** Multiple aggTrades share the same millisecond timestamp. When paginating with `startTime = last_ts + 1`, trades at the boundary millisecond are lost. Evidence - Python probe found 3 gaps in 5 pages.

**`fromId` pagination has ZERO gaps.** Always use `fromId = last_agg_trade_id + 1`. Evidence - [telemetry run](./references/telemetry_20260225_235803.ndjson), all junctions id_delta=1.

### Field Identity Across Sources

REST and WebSocket return **bit-identical** data for the same `agg_trade_id` - same price, volume, timestamp (after normalization), and `is_buyer_maker`. Proven via 42/42 field cross-validation with 0 mismatches.

### Timestamp Normalization

All three sources normalize to **microseconds** (16-digit) via `normalize_timestamp()`. Vision CSV provides ms, REST JSON provides ms, WebSocket JSON provides ms. All converted to us uniformly.

## Reconstruction Pipeline (The Real Architecture)

Data retrieval is fast (~10s). The bottleneck is open deviation bar construction (microstructure math). The pipeline design:

```
               fromId = last_agg_trade_id + 1
                         │
    ┌────────────────────┼────────────────────┐
    │                    │                    │
 VISION              REST API            WEBSOCKET
 (T-2 bulk)       (gap fill)            (live push)
    │                    │                    │
    └────────────────────┼────────────────────┘
                         │
                    AggTrade stream
                    (contiguous IDs)
                         │
                ┌────────▼────────┐
                │ OpenDeviationBarProcessor│
                │  - OHLCV        │
                │  - 10 intra-bar │
                │  - 16 inter-bar │
                │  - Ouroboros    │
                └────────┬────────┘
                         │
              ┌──────────▼──────────┐
              │ Ouroboros Boundary   │
              │ - Orphan bar: save  │
              │   last_agg_trade_id │
              │ - Reset processor   │
              │ - Next: fromId =    │
              │   orphan.last_id+1  │
              └─────────────────────┘
```

**Key invariant**: At any Ouroboros boundary, the orphan bar carries `last_agg_trade_id`. Next reconstruction starts at `fromId = last_agg_trade_id + 1`. This is the **single resumption point** across all three data sources.

**Day-mode Ouroboros**: Processor resets at UTC midnight (00:00:00.000 UTC) for every symbol/threshold. One reset per day — the orphan bar (bar in progress at midnight) carries `last_agg_trade_id` for the resumption point.

## Validation Strategy (3 REST calls, ~10s)

1. **Phase 1 - Vision**: Load last 3 available days, verify day-to-day contiguity
2. **Phase 2a - REST head**: `fromId=vision_last+1, limit=1000` - 1 call proves Vision→REST junction
3. **Phase 2b - REST tail**: `limit=1` (no fromId) - 1 call gets latest agg_trade_id
4. **Phase 3 - WebSocket**: 5-second live capture
5. **Phase 4 - Bridge**: `fromId=tail+1, limit=1000` - 1 call bridges REST→WS with field-validated overlap
6. **Phase 5 - Semantics**: Cross-validate field formats (all timestamps in microseconds)
7. **Phase 6 - Ouroboros**: Find UTC-midnight day boundaries in loaded window

## Empirical Results (2026-02-25, field-validated run)

```
Vision:    3,522,003 trades in 4.7s  (3 days, 0 gaps, 0 ts_reversals)
REST head: 1,000 trades in 387ms     Vision→REST id_delta=1 SEAMLESS
REST tail: 1 trade in 287ms          Latest anchor acquired
WebSocket: 42 trades in 5.0s         0 internal gaps
Bridge:    43 trades in 95ms         tail→bridge id_delta=1 SEAMLESS
                                     bridge→WS 42/42 FIELD-IDENTICAL
TOTAL:     10.5s
```

## REST to WebSocket Reliability (3 Failure Modes)

Live junction between REST backfill and WebSocket streaming has three distinct failure modes, each requiring a different mitigation:

| #   | Failure Mode         | Symptom                                                    | Detection                                   | Mitigation                                                                                           |
| --- | -------------------- | ---------------------------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| 1   | **Millisecond gap**  | `startTime` pagination drops trades sharing a millisecond  | `id_delta > 1` between pages                | Use `fromId` pagination (WIRED in `historical.rs`)                                                   |
| 2   | **WS reconnect gap** | WebSocket drops during reconnection window                 | `id_delta > 1` in WS stream after reconnect | REST backfill: `fromId = last_ws_id + 1` to fill gap                                                 |
| 3   | **REST→WS race**     | REST tail fetched before WS starts, small gap between them | `bridge_last_id + 1 < ws_first_id`          | Overlap strategy: fetch REST bridge AFTER WS is live, use `fromId` overlap + dedup by `agg_trade_id` |

### Overlap Strategy (REST→WS Junction)

```
REST backfill              REST bridge         WebSocket live
─────────────── ... ──────┬──────────────┬─────────────────
                          │   OVERLAP    │
                          │  (dedup by   │
                          │  agg_trade_id)│
                          └──────────────┘
```

The bridge page (`fromId = rest_tail + 1`) is fetched AFTER the WebSocket connection is established. This guarantees overlap — bridge trades include some trades that the WebSocket also receives. Dedup by `agg_trade_id` removes duplicates, producing a seamless sequence.

## Production Resilience Summary

| Component                    | Status          | Location                                       | What It Does                                                |
| ---------------------------- | --------------- | ---------------------------------------------- | ----------------------------------------------------------- |
| `fromId` pagination          | **WIRED**       | `historical.rs:fetch_aggtrades_by_id()`        | Zero-gap single-page REST fetch                             |
| `fetch_latest_aggtrade()`    | **WIRED**       | `historical.rs`                                | Anchor point for cursor pagination                          |
| 429 backoff                  | **WIRED**       | `historical.rs:send_with_rate_limit_backoff()` | Exponential backoff 1s→16s, 5 retries                       |
| `From<RestAggTrade>`         | **WIRED**       | `historical.rs`                                | Consistent timestamp normalization (ms→us)                  |
| PyO3 bindings                | **WIRED**       | `binance_bindings.rs`                          | `fetch_aggtrades_by_id()`, `fetch_latest_aggtrade()`        |
| Python wrappers              | **WIRED**       | `backfill.py`                                  | `_fetch_aggtrades_by_id()`, `_fetch_latest_aggtrade()`      |
| WS reconnect backfill        | **DESIGN-ONLY** | This skill doc                                 | REST `fromId` fill after WS gap detected                    |
| Overlap dedup                | **DESIGN-ONLY** | This skill doc                                 | Bridge page + WS overlap, dedup by ID                       |
| `startTime` caller migration | **PENDING**     | `fetch_aggtrades_rest()` callers               | Existing callers still use `startTime`; migrate to `fromId` |

## TodoWrite Task Templates

### Template A - Run Full Validation

```
1. [Preflight] Verify network access to Binance APIs
2. [Execute] cd .claude/skills/source-validation/scripts && cargo build --release
3. [Execute] ./target/release/opendeviationbar-source-validation
4. [Verify] All junctions report SEAMLESS or OVERLAP (never GAP)
5. [Verify] Field cross-validation shows 0 mismatches
6. [Verify] Save telemetry NDJSON to references/ for evidence
```

### Template B - Validate New Symbol

```
1. [Preflight] Confirm symbol is registered in symbols.toml
2. [Execute] Modify scripts/src/main.rs line 219: let symbol = "NEWSYMBOL"
3. [Execute] Build and run validation
4. [Verify] All 3 source junctions are seamless
5. [Verify] Ouroboros boundaries present in loaded window
6. [Verify] Update SKILL.md empirical results if significantly different
```

### Template C - Investigate Data Gap

```
1. [Preflight] Get the gap's agg_trade_id range from ClickHouse
2. [Execute] curl "https://api.binance.com/api/v3/aggTrades?symbol=BTCUSDT&fromId={GAP_START}&limit=1000"
3. [Verify] Trades exist on Binance (gap is in our pipeline, not source)
4. [Execute] Check if gap spans an Ouroboros boundary
5. [Verify] Determine root cause - pagination bug, connection drop, or boundary handling
```

### Template D - Wire fromId into opendeviationbar-providers (DONE 2026-02-25)

```
 1. [Preflight] Read crates/opendeviationbar-providers/src/binance/historical.rs
 2. [Rust] Add HistoricalError::RateLimited variant
 3. [Rust] Add From<RestAggTrade> for AggTrade (uses normalize_timestamp, not raw *1000)
 4. [Rust] Add send_with_rate_limit_backoff() helper (1s→2s→4s→8s→16s, 5 retries)
 5. [Rust] Retrofit 429 backoff into existing fetch_aggtrades_rest()
 6. [Rust] Refactor fetch_aggtrades_rest() to use From trait instead of inline conversion
 7. [Rust] Add fetch_aggtrades_by_id(from_id, limit) — single-page fromId fetch
 8. [Rust] Add fetch_latest_aggtrade() — anchor point (limit=1, no fromId)
 9. [Rust] Add 3 unit tests (From impl, CSV consistency, backoff constants)
10. [PyO3] Add fetch_aggtrades_by_id() binding in binance_bindings.rs
11. [PyO3] Add fetch_latest_aggtrade() binding in binance_bindings.rs
12. [PyO3] Register both in lib.rs (data-providers feature block)
13. [Python] Add _fetch_aggtrades_by_id() wrapper in backfill.py
14. [Python] Add _fetch_latest_aggtrade() wrapper in backfill.py
15. [Verify] cargo nextest run -p opendeviationbar-providers (41/41 pass)
```

### Template E - Reconstruct Open Deviation Bars for Ouroboros Window

```
1. [Preflight] Determine Ouroboros boundary (day: UTC midnight 00:00:00.000)
2. [Preflight] Get last_agg_trade_id from checkpoint/ClickHouse for this boundary
3. [Execute] Fetch trades: Vision (bulk) → REST fromId (gap fill) → WS (live)
4. [Execute] Feed contiguous AggTrade stream into OpenDeviationBarProcessor
5. [Execute] At boundary: save orphan bar with last_agg_trade_id
6. [Verify] No agg_trade_id gaps in produced bars (first_agg_trade_id..last_agg_trade_id)
7. [Verify] Orphan bar's last_agg_trade_id matches next boundary's fromId - 1
```

### Template F - Verify REST→WS Junction Reliability

```
 1. [Preflight] Ensure opendeviationbar-providers built with latest fromId changes
 2. [Execute] Fetch latest trade: loader.fetch_latest_aggtrade()
 3. [Execute] Fetch 1000 trades by ID: loader.fetch_aggtrades_by_id(latest_id - 2000, 1000)
 4. [Execute] Fetch next page: loader.fetch_aggtrades_by_id(page1_last_id + 1, 1000)
 5. [Verify] page2[0].agg_trade_id == page1_last.agg_trade_id + 1 (contiguity)
 6. [Verify] Internal contiguity: all windows(2) have delta=1
 7. [Execute] Connect WebSocket for 5s, capture trades
 8. [Execute] Fetch bridge: loader.fetch_aggtrades_by_id(ws_first_id - 100, 200)
 9. [Verify] Bridge overlaps with WS — overlapping trades are field-identical
10. [Verify] No gaps in the full sequence: REST pages + bridge + WS
```

## Post-Execution Reflection

After this skill completes, reflect before closing the task:

0. **Locate yourself.** — Find this SKILL.md's canonical path (Glob for this skill's name) before editing. All corrections target THIS file and its sibling references/ — never other documentation.
1. **What failed?** — Fix the instruction that caused it. If it could recur, add it as an anti-pattern.
2. **What worked better than expected?** — Promote it to recommended practice. Document why.
3. **What drifted?** — Any script, reference, or external dependency that no longer matches reality gets fixed now.
4. **Log it.** — Every change gets an evolution-log entry with trigger, fix, and evidence.

Do NOT defer. The next invocation inherits whatever you leave behind.
---

## Post-Change Checklist

- [ ] Validation script still compiles with workspace path deps
- [ ] All 6 phases complete without error
- [ ] Telemetry NDJSON captures all junction reports
- [ ] Field cross-validation shows 0 mismatches in overlap
- [ ] References updated with latest telemetry evidence
- [ ] Evolution log updated with change description
- [ ] Verify against Skill Quality Checklist (skill-architecture)

## Resources

- [Validation script (Rust)](./scripts/) - Standalone binary with path deps to opendeviationbar-core/providers
- [Telemetry - field-validated run](./references/telemetry_20260225_235803.ndjson) - 42/42 field-identical
- [Telemetry - initial run](./references/telemetry_20260225_235020.ndjson) - First successful 6-phase run
- [Python probes](./references/python-probes/) - Quick CLI probes used during initial investigation
- [Evolution log](./references/evolution-log.md) - Change history
