# Storage Layer (Tier 1: Parquet Tick Cache)

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Tier-1 local Parquet cache for raw aggTrades. Stores downloaded ticks on disk to avoid re-downloading from Binance Vision/REST on subsequent runs.

---

## Files

| File                   | Purpose                                                      |
| ---------------------- | ------------------------------------------------------------ |
| `parquet.py`           | `TickStorage` — primary read/write API, monthly partitioning |
| `parquet_io.py`        | Low-level Parquet I/O helpers                                |
| `parquet_query.py`     | Query-time utilities (filter, schema, timestamp conversion)  |
| `checksum_registry.py` | SHA-256 checksum verification for downloaded Vision ZIPs     |

---

## TickStorage (parquet.py)

**Path**: `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM.parquet`

Monthly partitioned files per symbol. Each file is an atomic unit — writes are to `.parquet.tmp` then renamed (atomic on same filesystem).

### Key Methods

| Method                                                | Description                                                             |
| ----------------------------------------------------- | ----------------------------------------------------------------------- |
| `write_ticks(symbol, ticks)`                          | Atomic write with dedup on `agg_trade_id`. Merges with existing month.  |
| `read_ticks(symbol, start_ts_ms, end_ts_ms)`          | Lazy Polars scan with timestamp filter. Returns materialized DataFrame. |
| `read_ticks_streaming(symbol, start, end, size)`      | Chunked iterator — bounded memory for large date ranges.                |
| `has_ticks(symbol, start_ts_ms, end_ts_ms, cov=0.95)` | **Metadata-only scan** — checks row count without loading data.         |
| `list_cached_months(symbol)`                          | Returns list of `YYYY-MM` strings with cached data.                     |

### `has_ticks()` Is Metadata-Only

```python
# Kintsugi uses this in _shard_is_parquet_cached() — no data loaded, just schema scan
if storage.has_ticks("BTCUSDT", shard_start_ms, shard_end_ms, min_coverage=0.95):
    # fast-path: uses MAX_CONCURRENT_PARQUET=64
```

The `min_coverage=0.95` default tolerates up to 5% missing rows (e.g., final day of month not yet complete). This prevents false negatives from day-boundary edge cases.

### Compression

| Parameter            | Value  | Notes                                      |
| -------------------- | ------ | ------------------------------------------ |
| Codec                | ZSTD-3 | Level 3 balances speed and ratio           |
| Observed ratio       | 5.37×  | 6.50 MB for 761K trades (BTCUSDT daily)    |
| `_COMPRESSION_RATIO` | 4.0    | Conservative empirical in-memory expansion |

`_COMPRESSION_RATIO` is used by `estimate_tick_memory()` (MEM-004) to predict how much RAM a dataset will consume before materializing it.

---

## Memory Guards (MEM-001, MEM-004)

| Guard   | Mechanism                                                 | Location     |
| ------- | --------------------------------------------------------- | ------------ |
| MEM-001 | Native Polars `dt` operations instead of `map_elements()` | `parquet.py` |
| MEM-004 | Pre-flight memory estimate via `_COMPRESSION_RATIO`       | `parquet.py` |

MEM-001: Partitioning by month uses `pl.col("timestamp_ms").dt.truncate("1mo")` — no Python map over rows. This is the key fix that dropped peak memory from 13.4 GB to ~100 MB for the monthly partitioning step.

MEM-004: `storage.estimate_memory_mb(symbol, start_ts, end_ts)` returns an estimate before loading. If estimate exceeds budget, raises `MemoryError` with suggestion to use streaming path.

---

## Storage Layout

```
~/.cache/opendeviationbar/
├── ticks/
│   ├── BINANCE_SPOT_BTCUSDT/
│   │   ├── 2018-01.parquet    # monthly files
│   │   ├── 2018-02.parquet
│   │   └── ...
│   └── BINANCE_SPOT_ETHUSDT/
│       └── ...
├── checkpoints/               # populate_cache_resumable() state
└── heartbeat/                 # pending_messages.json, stathera_counts.json
```

Cache key format: `{SOURCE}_{MARKET}_{SYMBOL}` (uppercased). Examples: `BINANCE_SPOT_BTCUSDT`, `BINANCE_UM_BTCUSDT`.

---

## When the Parquet Cache Is Used

| Consumer                              | When                                 | Notes                                 |
| ------------------------------------- | ------------------------------------ | ------------------------------------- |
| `get_open_deviation_bars()`           | `has_ticks()` returns True           | Skips network download                |
| `populate_cache_resumable()`          | Before each day's Vision download    | Skips Vision if day already cached    |
| `kintsugi._shard_is_parquet_cached()` | Before each repair pass              | Switches to MAX_CONCURRENT_PARQUET=64 |
| `fetch_ticks_with_backoff()`          | All count-bounded backoff iterations | Reads cached months, fetches new ones |

**Multi-threshold efficiency**: Once BTCUSDT@250 populates the cache for a date range, BTCUSDT@500/750/1000 on the same dates reads from local Parquet — zero additional network downloads. Same raw aggTrades, different processor threshold.

---

## Dead-Letter Store (Adjacent)

`/tmp/opendeviationbar-dead-letter/*.parquet` — bars that failed ClickHouse writes. NOT managed by `TickStorage`. Managed by Charon (periodic replay) and Niffler (immediate replay). Each file is named by `{symbol}-{threshold}-{timestamp}.parquet`.

---

## Related

- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) — parent: caching architecture, MEM guards
- [/python/opendeviationbar/orchestration/CLAUDE.md](/python/opendeviationbar/orchestration/CLAUDE.md) — orchestration layer that reads/writes ticks
- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) — pueue Parquet seeder (seeder service), kintsugi parallelism
- [/CLAUDE.md](/CLAUDE.md) — project hub
