# Python Layer

**Parent**: [/CLAUDE.md](/CLAUDE.md) | **API Reference**: [/docs/api/INDEX.md](/docs/api/INDEX.md)

This directory contains the Python API layer for opendeviationbar-py.

---

## AI Agent Quick Reference

### Common Tasks & Entry Points

| When Claude is asked to...                       | Primary File          | Function/Class                         |
| ------------------------------------------------ | --------------------- | -------------------------------------- |
| Generate open deviation bars (date-bounded)      | `__init__.py`         | `get_open_deviation_bars()`            |
| Generate open deviation bars (count-bounded, ML) | `__init__.py`         | `get_n_open_deviation_bars()`          |
| Generate open deviation bars (existing data)     | `__init__.py`         | `process_trades_to_dataframe()`        |
| Generate open deviation bars (Polars)            | `__init__.py`         | `process_trades_polars()`              |
| Process large datasets                           | `__init__.py`         | `process_trades_chunked()`             |
| Run streaming sidecar                            | `sidecar.py`          | `run_sidecar()`, `SidecarConfig`       |
| Populate cache for long ranges                   | `checkpoint.py`       | `populate_cache_resumable()`           |
| Read/write tick data                             | `storage/parquet.py`  | `TickStorage` class                    |
| Bar-count cache operations                       | `clickhouse/cache.py` | `count_bars()`, `get_n_bars()`         |
| Validate microstructure features                 | `validation/tier1.py` | `validate_tier1()`                     |
| Self-healing gap reconciliation                  | `kintsugi.py`         | `kintsugi_pass()`, `kintsugi_daemon()` |

### API Selection Guide

```
Starting Point?
├── Date range > 30 days (≤ 366)? → populate_cache_resumable() first, then get_open_deviation_bars()
├── Date range > 366 days? → mise run kintsugi:catchup (historical backfill)
├── Need data fetching (date range ≤ 30 days)? → get_open_deviation_bars() [DATE-BOUNDED]
├── Need exactly N bars (ML/walk-forward)? → get_n_open_deviation_bars() [COUNT-BOUNDED]
├── Have pandas DataFrame → process_trades_to_dataframe()
├── Have Polars DataFrame → process_trades_polars() [2-3x faster]
├── Have Polars LazyFrame → process_trades_polars_lazy() [predicate pushdown]
└── Have Iterator (large data) → process_trades_chunked()
```

### File-to-Responsibility Mapping

| File                    | Responsibility                                                        |
| ----------------------- | --------------------------------------------------------------------- |
| `__init__.py`           | Public Python API                                                     |
| `__init__.pyi`          | Re-export index (stubs)                                               |
| `sidecar.py`            | Streaming sidecar orchestrator (Puck/Niffler/Charon/Asclepius)        |
| `sidecar_api.py`        | Consumer API: ariadne, gap-fill, catchup (#213, #224)                 |
| `sidecar.pyi`           | Sidecar type stubs (SidecarConfig, run_sidecar)                       |
| `storage/parquet.py`    | Tier 1 cache (local Parquet)                                          |
| `clickhouse/cache.py`   | Tier 2 cache (ClickHouse)                                             |
| `clickhouse/schema.sql` | ClickHouse table schema                                               |
| `validation/tier1.py`   | Fast validation (<30 sec)                                             |
| `validation/tier2.py`   | Statistical validation (~10 min)                                      |
| `exness.py`             | Exness data source utilities                                          |
| `kintsugi.py`           | Self-healing gap reconciliation (Issue #115, sealed today-guard #280) |

### Performance Optimization Checklist

When optimizing data processing:

1. Use `pl.scan_parquet()` instead of `pl.read_parquet()` (lazy loading)
2. Apply filters on LazyFrame before `.collect()` (predicate pushdown)
3. Select only required columns before `.to_dicts()` (minimal conversion)
4. Use `process_trades_chunked()` for datasets >10M trades

---

## Structure

```
python/opendeviationbar/
├── __init__.py          # Public API (get_open_deviation_bars, process_trades_*)
├── __init__.pyi         # Type stubs for IDE/AI
├── _core.abi3.so        # PyO3 binary extension (built by maturin)
├── sidecar.py           # Streaming sidecar orchestrator (v12.20+)
├── sidecar_api.py       # Consumer API business logic (ariadne, gap-fill)
├── sidecar.pyi          # Sidecar type stubs
├── clickhouse/          # Tier 2 cache (ClickHouse, bigblack)
│   ├── cache.py         # Open deviation bar cache operations
│   └── schema.sql       # Table schema (v7.0: 10 microstructure columns)
├── data/                # Package data files (Issue #79)
│   ├── __init__.py      # importlib.resources marker
│   └── symbols.toml     # Unified Symbol Registry (SSoT)
├── symbol_registry.py   # Registry loader + mandatory gate (Issue #79)
├── storage/             # Tier 1 cache (local Parquet)
│   └── parquet.py       # TickStorage class
├── validation/          # Microstructure feature validation (v7.0+)
│   ├── tier1.py         # Fast validation (<30 sec)
│   └── tier2.py         # Statistical validation (~10 min)
└── exness.py            # Exness data source utilities
```

---

## Key Files

### `__init__.py` - Public API

**Entry points**:

| Function                           | Purpose                           |
| ---------------------------------- | --------------------------------- |
| `get_open_deviation_bars()`        | Date-bounded, auto-fetch, caching |
| `get_n_open_deviation_bars()`      | Count-bounded, ML training        |
| `process_trades_to_dataframe()`    | From existing pandas DataFrame    |
| `process_trades_polars()`          | From Polars, 2-3x faster          |
| `process_trades_polars_lazy()`     | Lazy Polars, predicate pushdown   |
| `process_trades_chunked()`         | Streaming, memory-safe            |
| `get_open_deviation_bars_pandas()` | Pandas variant for backtesting.py |
| `precompute_open_deviation_bars()` | Batch precompute to ClickHouse    |

**Constants**:

- `TIER1_SYMBOLS` - 18 high-liquidity symbols
- `THRESHOLD_PRESETS` - Named thresholds (micro, tight, standard, etc.)

### `.pyi` Type Stubs (PEP 561 Per-Module Layout)

Type stubs follow PEP 561 per-module layout. Each `.pyi` file lives alongside its `.py` source.

- `__init__.pyi` — **Re-export index only** (no definitions). Uses `from .x import Y as Y` pattern.
- Per-module stubs (e.g., `constants.pyi`, `ouroboros.pyi`, `orchestration/open_deviation_bars.pyi`) — Contain actual type definitions.

**Enforced by**: `pretooluse-pyi-stub-guard.ts` hook (blocks definitions in `__init__.pyi`).

### `_core.abi3.so` - PyO3 Extension

Binary built by `maturin develop`. Contains:

- `PyOpenDeviationBarProcessor` - Wraps Rust processor
- `PyAggTrade` - Trade data type
- Data fetching (when providers feature enabled)

**Rebuild after Rust changes**: `maturin develop`

---

## Symbol Registry (Issue #79)

**Every symbol must be registered in `symbols.toml` before processing.** Unregistered symbols raise `SymbolNotRegisteredError`.

### Files

| File                       | Purpose                                      |
| -------------------------- | -------------------------------------------- |
| `data/symbols.toml`        | TOML registry (SSoT for all symbol metadata) |
| `symbol_registry.py`       | Loader module + gate + query functions       |
| `exceptions.py`            | `SymbolNotRegisteredError` exception         |
| `symbols.toml` (repo root) | Symlink for developer convenience            |

### Adding New Symbols

1. Edit `python/opendeviationbar/data/symbols.toml`
2. Run `maturin develop`
3. Process the symbol

### Gate Integration Points

Gates are inserted at the top of each entry point function, **before** threshold validation:

```python
from opendeviationbar.symbol_registry import validate_symbol_registered, validate_and_clamp_start_date

validate_symbol_registered(symbol, operation="function_name")
start_date = validate_and_clamp_start_date(symbol, start_date)  # only if start_date param exists
```

### Environment Variable

`OPENDEVIATIONBAR_SYMBOL_GATE` controls gate behavior:

- `"strict"` (default): `SymbolNotRegisteredError`
- `"warn"`: `UserWarning` + continue
- `"off"`: No gating (dev/testing)

### Telemetry

Gate violations emit:

1. **NDJSON log**: `logs/events.jsonl` with `component="symbol_registry"`
2. **Pushover alert**: Emergency priority (repeats every 30s until acknowledged)

---

## Caching Architecture

### Tier 1: Local Parquet Ticks Cache

**Location**: `storage/parquet.py` | **Path**: `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM-DD.parquet`

Stores raw aggTrades locally as daily Parquet files (Issue #277). Key behaviors:

- **Pre-download check**: `populate_cache_resumable()` (`checkpoint.py:636`) calls `ctx.storage.has_ticks()` before each day's Vision/REST download. If the day is already cached, reads from local Parquet — zero network cost.
- **Multi-threshold efficiency**: Running BTCUSDT@250 first populates the ticks cache for all dates. Subsequent runs for BTCUSDT@500/750/1000 on the same dates read from local Parquet instead of re-downloading from Vision. Same raw trades, different processor threshold.
- **Dead-letter store**: When ClickHouse writes fail, bars are saved to `/tmp/opendeviationbar-dead-letter/*.parquet` and replayed by Charon.

```python
from opendeviationbar.storage.parquet import TickStorage
storage = TickStorage()
# Check before download (as checkpoint.py does internally):
if storage.has_ticks("BTCUSDT", day_start_ms, day_end_ms):
    ticks = storage.read_ticks("BTCUSDT", day_start_ms, day_end_ms)  # local, no network
```

### Tier 2: ClickHouse

**Location**: `clickhouse/cache.py`

- Stores precomputed open deviation bars
- Used by `get_open_deviation_bars()` and `get_n_open_deviation_bars()`
- Requires ClickHouse connection (bigblack via SSH tunnel)

```python
from opendeviationbar.clickhouse.cache import OpenDeviationBarCache

cache = OpenDeviationBarCache()
bars_df = cache.get_open_deviation_bars("BTCUSDT", threshold=250, start_date=start, end_date=end)
n = cache.count_bars("BTCUSDT", threshold=250)
```

**Schema**: `clickhouse/schema.sql` (v7.0 adds 10 microstructure columns)

---

## Validation Framework (v7.0+)

**Location**: `validation/`

Validates microstructure features for ML readiness.

### Tier 1: Smoke Test (`tier1.py`)

- Runtime: <30 seconds
- Runs automatically on every `precompute_open_deviation_bars()`
- Checks: NaN, Inf, bounds, basic correlations

```python
from opendeviationbar.validation.tier1 import validate_tier1
result = validate_tier1(df)
assert result["tier1_passed"]
```

### Tier 2: Statistical (`tier2.py`)

- Runtime: ~10 minutes
- **Mandatory before production ML training**
- Checks: Stationarity, predictive power, mutual information

```python
from opendeviationbar.validation.tier2 import validate_tier2
df["forward_return"] = df["Close"].shift(-1) / df["Close"] - 1
result = validate_tier2(df)
assert result["tier2_passed"]
```

---

## Notifications (`notify/`)

Telegram and Pushover notification modules for cache operations and monitoring.

| Module        | Bot        | Use Case                            |
| ------------- | ---------- | ----------------------------------- |
| `telegram.py` | @obdpy_bot | Hook events, gap alerts, heartbeats |
| `pushover.py` | —          | Critical alerts (checksum failures) |

### Telegram Integration

```python
from opendeviationbar.notify.telegram import send_telegram, enable_telegram_notifications

# Enable for all hook events (populate, cache write, validation, etc.)
enable_telegram_notifications()

# Or failures only
from opendeviationbar.notify.telegram import enable_failure_notifications
enable_failure_notifications()

# Direct message
send_telegram("<b>Alert:</b> Custom message", disable_notification=False)
```

**Env vars**: `OPENDEVIATIONBAR_TELEGRAM_TOKEN` in `.mise.local.toml` (non-git-tracked) or `deploy/opendeviationbar.local.env` (bigblack) + `OPENDEVIATIONBAR_TELEGRAM_CHAT_ID` in `.mise.toml`

### Safe Populate Pattern (Post-Incident)

`populate_cache_resumable()` uses `_fatal_cache_write()` (not `try_cache_write()`) to ensure ClickHouse failures raise exceptions instead of being silently swallowed. The checkpoint only advances for days where ClickHouse confirmed the write.

| Guard                          | What It Prevents                          |
| ------------------------------ | ----------------------------------------- |
| Fatal cache write              | Silent loss of bars when SSH tunnel drops |
| Circuit breaker (3 failures)   | Cascade writes during ClickHouse outages  |
| T-1 date clamping              | Crash on unavailable Binance Vision data  |
| `bars_written` from ClickHouse | Checkpoint/ClickHouse count divergence    |

---

## Common Patterns

### Date-Bounded Bars (Backtesting)

```python
from opendeviationbar import get_open_deviation_bars

df = get_open_deviation_bars(
    "BTCUSDT",
    "2024-01-01",
    "2024-06-30",
    threshold_decimal_bps=250,  # 0.25%
)
```

### Count-Bounded Bars (ML)

```python
from opendeviationbar import get_n_open_deviation_bars

df = get_n_open_deviation_bars(
    "BTCUSDT",
    n_bars=10000,
    threshold_decimal_bps=250,
)
assert len(df) == 10000  # Exact count guaranteed
```

### With Microstructure Features (v7.0+)

```python
df = get_open_deviation_bars(
    "BTCUSDT",
    "2024-01-01",
    "2024-06-30",
    include_microstructure=True,
)
# Includes: ofi, vwap_close_deviation, kyle_lambda_proxy, etc.
```

### Large Dataset Processing

```python
from opendeviationbar import process_trades_chunked

for bars_chunk in process_trades_chunked(trades_iterator, chunk_size=100_000):
    # Process incrementally, bounded memory
    save_to_database(bars_chunk)
```

### backtesting.py Integration

```python
from backtesting import Backtest, Strategy
from opendeviationbar import get_open_deviation_bars

# Fetch data and generate open deviation bars in one call
data = get_open_deviation_bars("BTCUSDT", "2024-01-01", "2024-06-30")

# Use directly with backtesting.py
bt = Backtest(data, MyStrategy, cash=10000, commission=0.0002)
stats = bt.run()
bt.plot()
```

**Output Format** (backtesting.py compatible):

```python
# DataFrame with DatetimeIndex and OHLCV columns
                          Open      High       Low     Close  Volume
timestamp
2024-01-01 00:00:15  42000.00  42105.00  41980.00  42100.00   15.43
2024-01-01 00:03:42  42100.00  42220.00  42100.00  42215.00    8.72
```

### backtesting.py Compatibility Checklist

- [x] **OHLCV column names**: Capitalized (Open, High, Low, Close, Volume)
- [x] **DatetimeIndex**: Pandas DatetimeIndex with timezone-naive timestamps
- [x] **No NaN values**: All bars complete (backtesting.py raises on NaN)
- [x] **Sorted chronologically**: Timestamps in ascending order
- [x] **OHLC invariants**: High ≥ max(Open, Close), Low ≤ min(Open, Close)

**Validation Script**:

```python
def validate_for_backtesting_py(df: pd.DataFrame) -> bool:
    """Validate DataFrame is compatible with backtesting.py."""
    assert list(df.columns) == ["Open", "High", "Low", "Close", "Volume"]
    assert isinstance(df.index, pd.DatetimeIndex)
    assert df.index.is_monotonic_increasing
    assert not df.isnull().any().any()
    assert (df["High"] >= df["Open"]).all()
    assert (df["High"] >= df["Close"]).all()
    assert (df["Low"] <= df["Open"]).all()
    assert (df["Low"] <= df["Close"]).all()
    return True
```

---

## Long Range Processing (MEM-013)

For date ranges > 30 days, direct `get_open_deviation_bars()` is blocked to prevent OOM.

### Workflow

```python
# For ranges > 366 days, use kintsugi catch-up mode:
#   mise run kintsugi:catchup
#
# For bounded ranges (≤ 366 days):
from opendeviationbar import populate_cache_resumable, get_open_deviation_bars

populate_cache_resumable(
    "BTCUSDT",
    "2025-01-01",
    "2025-12-31",
    threshold_decimal_bps=250,
)

df = get_open_deviation_bars("BTCUSDT", "2025-01-01", "2025-12-31")
```

### Parameters

| Parameter                | Default | Description                                           |
| ------------------------ | ------- | ----------------------------------------------------- |
| `force_refresh`          | `False` | Wipe existing cache and checkpoint                    |
| `threshold_decimal_bps`  | `250`   | Threshold (same as get_open_deviation_bars)           |
| `include_microstructure` | `False` | Include microstructure features                       |
| `ouroboros`              | `"day"` | Reset mode for reproducibility (only "day" supported) |
| `max_days`               | `366`   | Max date range span; `None` to bypass (kintsugi only) |
| `notify`                 | `True`  | Send progress notifications                           |
| `verbose`                | `True`  | Show tqdm progress bar and logging                    |

### Resumability

- **Local checkpoint**: `~/.cache/opendeviationbar/checkpoints/` (fast)
- **ClickHouse checkpoint**: `population_checkpoints` table (cross-machine)
- **Bar-level**: Incomplete bars preserved across interrupts

---

## Error Handling

| Exception                                                      | When                         | Fix                       |
| -------------------------------------------------------------- | ---------------------------- | ------------------------- |
| `ValueError`                                                   | Invalid threshold, bad dates | Check parameters          |
| `RuntimeError`                                                 | Processing failure           | Check data sorting        |
| `ConnectionError`                                              | ClickHouse unavailable       | Check network/credentials |
| `FileNotFoundError`                                            | Tick data not cached         | Set `use_cache=False`     |
| `AttributeError: OpenDeviationBarProcessor has no attribute X` | Rust binding outdated        | Run `maturin develop`     |
| `AssertionError: High < Low`                                   | OHLC invariant violation     | Check input data sorting  |

---

## Development

### Adding New Features

1. Update `__init__.py` with new function
2. Add type stub to the per-module `.pyi` file (e.g., `orchestration/open_deviation_bars.pyi`)
3. Add re-export to `__init__.pyi` (e.g., `from .orchestration.open_deviation_bars import new_func as new_func`)
4. Add tests in `/tests/`
5. Update `/docs/api/INDEX.md`

### Testing

```bash
# Run Python tests
mise run test-py

# Test specific file
pytest tests/test_microstructure_features.py -v

# E2E tests
pytest tests/test_e2e_optimized.py -v
pytest tests/test_get_n_open_deviation_bars.py -v
```

### Portable Validation Scripts

For GPU workstations without full dev environment:

- `scripts/validate_n_open_deviation_bars.py` - Count-bounded API validation
- `scripts/validate_microstructure_features.py` - v7.0 feature validation

---

## Breaking Changes

### gap_fill_trades() Response (v12.62+, #220)

`GET /trades/gap-fill` and `gap_fill_trades()` now return a dict instead of a bare list:

```json
{"trades": [...], "partial": false, "count": 1234}
```

- `trades`: list of `{a, p, q, T, m}` dicts (same format as before)
- `partial`: `true` when REST bridge failed after Parquet returned partial data
- `count`: number of trades returned

Consumers must update: `trades = response["trades"]` (was: `trades = response`).

### ariadne_lookup() includes sources_tried (v12.62+, #221)

`GET /ariadne/{symbol}/{threshold}` responses now include `sources_tried` — a diagnostic list showing which sources were attempted and their outcomes. No consumer code change required (additive field).

### Server-Side Gap-Fill (v12.62+, #224-#228)

`GET /catchup/{symbol}/{threshold}` — single-call gap-fill from last committed CH bar. Composes `ariadne_lookup(committed_only=True)` + paginated Parquet→REST gap-fill. Replaces 3-step client flow.

**New ariadne query param**: `?committed_only=true` skips Sources 1-2 (live processor, checkpoint file), returns only committed ClickHouse data.

**New functions** in `sidecar_api.py`:

- `gap_fill_trades_paginated(symbol, from_agg_id, max_trades)` — server-side pagination, 100K hard cap
- `catchup_lookup(symbol, threshold, max_trades, engine)` — composite endpoint

```bash
# Usage
curl 'http://localhost:8081/catchup/BTCUSDT/250?max_trades=50000' | jq '{count, partial, source_breakdown}'
curl 'http://localhost:8081/ariadne/BTCUSDT/250?committed_only=true' | jq '.source'
```

---

## ClickHouse Infrastructure

All open deviation bar data served from **bigblack** (remote GPU host). No local ClickHouse.

| Host        | Thresholds (dbps)   | ClickHouse | Total Bars     |
| ----------- | ------------------- | ---------- | -------------- |
| bigblack    | 250, 500, 750, 1000 | Native     | ~42M (growing) |
| littleblack | 100                 | Docker     | —              |

**Connection mode**: `OPENDEVIATIONBAR_MODE=remote` (set in `.mise.toml`). Preflight: `mise run db:ensure`.

**Cache operations**: [clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md)

**Plugin columns**: Not in `schema.sql` — managed via `ALTER TABLE ADD COLUMN IF NOT EXISTS` by each `FeatureProvider` plugin. See [plugins/CLAUDE.md](/python/opendeviationbar/plugins/CLAUDE.md).

**Tier-1 Parquet ticks cache**: Raw aggTrades stored locally in `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM-DD.parquet`. Multi-threshold efficiency: BTCUSDT@500/750/1000 reuse files already downloaded by @250, zero re-downloads. See [storage/CLAUDE.md](/python/opendeviationbar/storage/CLAUDE.md).

---

## Related

- [/CLAUDE.md](/CLAUDE.md) - Project hub (architecture, principles, terminology)
- [/docs/OPEN-DEVIATION-BAR.md](/docs/OPEN-DEVIATION-BAR.md) - Full algorithm explanation: layman + bit-exact ClickHouse oracle examples
- [/docs/api/INDEX.md](/docs/api/INDEX.md) - Full API reference
- [/crates/CLAUDE.md](/crates/CLAUDE.md) - Rust crates, microstructure feature formulas
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache, dedup hardening
- [/python/opendeviationbar/orchestration/CLAUDE.md](/python/opendeviationbar/orchestration/CLAUDE.md) - Orchestration layer: request flow, circuit breaker, count-bounded API
- [/python/opendeviationbar/plugins/CLAUDE.md](/python/opendeviationbar/plugins/CLAUDE.md) - FeatureProvider plugin system, entry-point discovery
- [/python/opendeviationbar/storage/CLAUDE.md](/python/opendeviationbar/storage/CLAUDE.md) - Tier-1 Parquet ticks cache: TickStorage, MEM guards, daily partitions
- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) - Pueue ops, per-year parallelization (default strategy), anti-patterns
- [/src/lib.rs](/src/lib.rs) - PyO3 bindings source
