# PyO3 FFI Boundary Memory Audit

**Scope**: `src/helpers.rs`, `src/core_bindings.rs`, `src/streaming_bindings.rs`, `src/stream_manager_bindings.rs`, `src/arrow_bindings.rs`

**Date**: 2026-03-22

---

## Executive Summary

The dict-based path (`dict_to_agg_trade` / `opendeviationbar_to_dict`) dominates FFI overhead.
The Arrow path (`process_trades_arrow`) is already near-optimal.
The largest win is **avoiding the dict path entirely** in hot loops by routing
callers through Arrow. GIL release during pure-Rust computation is the second
largest opportunity.

---

## Finding 1: `dict_to_agg_trade()` — 7-8 Python API calls per trade (hot path)

**File**: `src/helpers.rs:26-90`
**Priority**: P0

### Current cost per trade

| Operation                                  | Python API calls | Notes                                   |
| ------------------------------------------ | ---------------- | --------------------------------------- |
| `get_item("timestamp")` + `extract()`      | 2                | Required                                |
| `get_item("price")` + `extract()`          | 2                | Required                                |
| `get_item("quantity")` + `extract()`       | 2                | Required (fallback to "volume" adds +2) |
| `get_item("agg_trade_id")` + `extract()`   | 2                | Optional                                |
| `get_item("first_trade_id")` + `extract()` | 2                | Optional                                |
| `get_item("last_trade_id")` + `extract()`  | 2                | Optional                                |
| `get_item("is_buyer_maker")` + `extract()` | 2                | Optional                                |
| **Total**                                  | **14**           | 7 `get_item` + 7 `extract`              |

Each `get_item` is a Python `__getitem__` call (hash lookup + refcount ops).
Each `extract` is a `PyLong_AsLongLong` / `PyFloat_AsDouble` C API call.

For a typical batch of 100K trades: **1.4M Python C-API calls** just for input conversion.

### Proposed fix

**Option A (recommended): Route all batch callers through Arrow.**
`process_trades()` and `process_trades_streaming()` accept `Vec<Bound<PyDict>>`.
If callers already have DataFrames or Arrow tables (the common case for backfill),
they should call `process_trades_arrow()` which does zero Python API calls for
trade extraction — the `record_batch_to_aggtrades()` function operates entirely
on Arrow column arrays in Rust.

**Option B: Columnar batch extraction from Python lists.**
Instead of iterating dicts, accept parallel Python lists (`timestamps: list[int]`,
`prices: list[float]`, etc.) and use `PyList::get_item(i).extract()` which avoids
hash-based dict lookups. ~40% fewer C-API calls (no key hashing).

**Option C: `FromPyObject` derive for struct-level extraction.**
Use `#[derive(FromPyObject)]` on a mirror struct to let PyO3 generate optimized
extraction code. Marginal gain; still one C-API call per field.

### Quantification

- **Arrow path**: 0 Python API calls per trade (already implemented)
- **Dict path**: 14 Python API calls per trade
- **Savings if callers migrate to Arrow**: 100% elimination of input conversion overhead

---

## Finding 2: `opendeviationbar_to_dict()` — 48+ `set_item` calls per bar

**File**: `src/helpers.rs:95-259`
**Priority**: P1

### Current cost per bar

The function calls `dict.set_item(ikey(py, "..."), value)` for every field:

- 7 OHLCV core fields
- 3 volume accumulators
- 4 trade tracking fields
- 10 microstructure features
- 7 inter-bar core features (always set)
- 9 inter-bar optional features (conditional)
- 8 intra-bar ITH features (conditional)
- 12 intra-bar statistical features (conditional)
- 2 intra-bar complexity features (conditional)

**Minimum**: 31 `set_item` calls (no optional features)
**Maximum**: ~55 `set_item` calls (all features enabled)

Each `set_item` call:

1. `ikey()` — `PyString::intern()` (fast after first call, cached by Python)
2. Value conversion — `i64 → PyLong`, `f64 → PyFloat` (one `PyObject` allocation each)
3. `PyDict_SetItem` — hash + insert + incref

**Per bar**: 31-55 `PyObject` allocations (one per numeric value).

### Existing optimization

`PyString::intern()` via `ikey()` already eliminates key string allocations after
the first bar. This is good.

### Proposed fix

**Option A (recommended): Return Arrow RecordBatch instead of list-of-dicts.**
Already implemented as `process_trades_streaming_arrow()`. The Arrow path builds
columns in Rust with zero per-value `PyObject` allocations. Callers get a
RecordBatch they can convert to Polars/pandas with zero-copy.

The `opendeviationbar_vec_to_record_batch()` function in `arrow_export.rs` builds
columnar arrays entirely in Rust — no Python API calls at all.

**Option B: Pre-sized dict with `PyDict::new_with_capacity()` (if PyO3 exposes it).**
Currently `PyDict::new(py)` creates a dict that resizes as items are added.
Pre-sizing to 55 entries would avoid 2-3 internal dict resizes.
However, CPython `PyDict_New()` doesn't accept a capacity hint, so this isn't
available through Python's C API.

**Option C: Return a `#[pyclass]` struct instead of a dict.**
Define `#[pyclass] struct PyOpenDeviationBar` with `#[pyo3(get)]` on each field.
Eliminates all `set_item` calls. Attributes are lazily converted on Python access.
**Tradeoff**: breaks API compatibility (bars are currently dicts, used in
`df = pd.DataFrame(bars)`).

### Quantification

For a batch producing 1,000 bars with all features:

- **Dict path**: ~55,000 `set_item` + ~55,000 `PyObject` allocations
- **Arrow path**: 0 Python API calls, 0 `PyObject` allocations
- **Savings**: ~110K C-API calls eliminated per 1,000 bars

---

## Finding 3: `Vec<Bound<PyDict>>` materialization in `process_trades()`

**File**: `src/core_bindings.rs:267-291`
**Priority**: P1

### Current behavior

```rust
fn process_trades(
    &mut self,
    py: Python,
    trades: Vec<Bound<PyDict>>,  // <-- entire list materialized
    return_format: &str,
) -> PyResult<PyObject> {
```

PyO3 extracts ALL elements from the Python list into a `Vec<Bound<PyDict>>` before
any processing begins. For 100K trades, this means:

1. 100K `PyList_GetItem()` calls
2. 100K `Bound<PyDict>` references held simultaneously
3. The original Python list AND the Rust Vec coexist in memory

### Proposed fix

**Option A: Accept `&Bound<PyList>` and iterate lazily.**

```rust
fn process_trades(
    &mut self,
    py: Python,
    trades: &Bound<PyList>,
) -> PyResult<PyObject> {
    let mut agg_trades = Vec::with_capacity(trades.len());
    for (i, item) in trades.iter().enumerate() {
        let dict: &Bound<PyDict> = item.cast()?;
        agg_trades.push(dict_to_agg_trade(py, dict, i)?);
    }
    // ...
}
```

This avoids materializing the full `Vec<Bound<PyDict>>` — each dict reference is
short-lived. Memory savings: eliminates the ~100K-element `Vec<Bound<PyDict>>`.

**Note**: The `agg_trades: Vec<AggTrade>` still needs to be fully materialized
because `process_agg_trade_records()` requires a slice. The savings are in avoiding
the intermediate `Vec<Bound<PyDict>>`, not the `Vec<AggTrade>`.

### Quantification

- `Vec<Bound<PyDict>>` overhead: ~8 bytes per element (pointer) = ~800KB for 100K trades
- Not a large memory savings, but reduces peak memory by not holding both representations
- More importantly, enables **streaming conversion** where each dict is converted and dropped

---

## Finding 4: GIL held during pure Rust computation

**File**: `src/core_bindings.rs:286-297`
**Priority**: P0

### Current behavior

```rust
// GIL is held for ALL of this:
let agg_trades: Vec<AggTrade> = trades
    .iter()
    .enumerate()
    .map(|(i, trade_dict)| dict_to_agg_trade(py, trade_dict, i))
    .collect::<PyResult<Vec<_>>>()?;

// GIL STILL held here — pure Rust computation:
let bars = self
    .processor
    .process_agg_trade_records(&agg_trades)
    .map_err(|e| PyRuntimeError::new_err(format!("Processing failed: {e}")))?;
```

`process_agg_trade_records()` is pure Rust — no Python interaction. Yet the GIL
is held throughout, blocking all other Python threads.

Similarly in `process_trades_streaming()` (line 342-370): the inner loop of
`process_single_trade()` is pure Rust but GIL is held because each iteration
also calls `opendeviationbar_to_dict()` on completion.

### Already correct

- `LiveBarEngine.next_bar()` — releases GIL via `py.detach()` while waiting
- `StreamManager.next_bar()` — releases GIL via `py.detach()` while waiting
- `StreamManager.fill_from_rest()` — releases GIL via `py.detach()`
- All async operations properly use `py.detach()` for blocking waits

### Methods that DO NOT release GIL during Rust computation

| Method                             | File:Line                  | Pure Rust work                                                | GIL held? |
| ---------------------------------- | -------------------------- | ------------------------------------------------------------- | --------- |
| `process_trades()`                 | `core_bindings.rs:286-297` | `process_agg_trade_records()`                                 | YES       |
| `process_trades_streaming()`       | `core_bindings.rs:352-370` | `process_single_trade()` loop                                 | YES       |
| `process_trades_streaming_arrow()` | `core_bindings.rs:412-432` | `process_single_trade()` loop                                 | YES       |
| `process_trades_arrow()`           | `core_bindings.rs:455-480` | `record_batch_to_aggtrades()` + `process_single_trade()` loop | YES       |
| `process_trades_arrow_native()`    | `core_bindings.rs:498-522` | same as above                                                 | YES       |

### Proposed fix

For `process_trades()` batch mode, the pattern would be:

```rust
// Phase 1: Extract trades (needs GIL)
let agg_trades = /* ... dict extraction ... */;

// Phase 2: Process (release GIL)
let bars = py.detach(|| {
    self.processor.process_agg_trade_records(&agg_trades)
});
let bars = bars.map_err(|e| PyRuntimeError::new_err(format!("Processing failed: {e}")))?;

// Phase 3: Convert output (needs GIL)
let dicts = bars.iter().map(|bar| opendeviationbar_to_dict(py, bar)).collect()?;
```

**Complication**: `&mut self.processor` is borrowed across the `py.detach()` boundary.
PyO3's `py.detach()` requires the closure to be `Send`, but `&mut self` (which borrows
the `#[pyclass]`) is not `Send` because PyO3 manages its lifecycle.

**Workaround**: Use `unsafe` to transmute the reference, or restructure to take
ownership of the processor temporarily. Alternatively, extract the processor's
inner state into a standalone Rust struct that IS `Send`.

For `process_trades_arrow()` and `process_trades_arrow_native()`, the entire
function body (after extracting the RecordBatch reference) is pure Rust and could
theoretically release the GIL. However, `&mut self.processor` has the same `Send` issue.

### Quantification

- Processing 100K trades takes ~50-200ms of pure Rust computation
- During this time, ALL Python threads are blocked
- In the sidecar (multi-symbol streaming), this blocks the heartbeat thread, Telegram
  alerting, and HTTP health endpoint responses
- For single-threaded batch processing, GIL release has zero benefit

---

## Finding 5: `checkpoint_to_dict()` — double conversion of incomplete bar

**File**: `src/helpers.rs:262-351`
**Priority**: P2

### Current behavior

When a checkpoint has an `incomplete_bar`, it is converted TWICE:

1. Full conversion via `opendeviationbar_to_dict()` (all 55 fields) → stored as `"incomplete_bar"`
2. Redundant partial conversion (15 fields) → stored as `"incomplete_bar_raw"`

```rust
if let Some(ref bar) = checkpoint.incomplete_bar {
    dict.set_item("incomplete_bar", opendeviationbar_to_dict(py, bar)?)?;  // 55 fields

    // ALSO store raw OHLCV for easy access — DUPLICATE work
    let bar_dict = PyDict::new(py);
    bar_dict.set_item("open", bar.open.to_f64())?;
    // ... 14 more fields ...
    dict.set_item("incomplete_bar_raw", bar_dict.into_any().unbind())?;
}
```

This creates ~70 `set_item` calls and ~70 `PyObject` allocations for a single
checkpoint with an incomplete bar.

### Proposed fix

Remove one of the two representations. If `incomplete_bar_raw` is needed for
checkpoint round-trip (it IS — `dict_to_checkpoint()` reads from `incomplete_bar_raw`),
then skip the full `opendeviationbar_to_dict()` call and only emit `incomplete_bar_raw`.

Or, have `dict_to_checkpoint()` read from `incomplete_bar` directly, eliminating
the need for `incomplete_bar_raw` entirely.

### Quantification

- Saves ~55 `set_item` calls per checkpoint
- Checkpoints are infrequent (once per file boundary or shutdown), so impact is LOW

---

## Finding 6: String conversions at boundary

**File**: `src/core_bindings.rs:611-612`, `src/streaming_bindings.rs:109`
**Priority**: P2

### Current behavior

- `PyOpenDeviationBarProcessor.symbol()` returns `Option<String>` — clones the stored
  `String` and PyO3 creates a new `PyString` on each call
- `PyBinanceLiveStream::new()` calls `symbol.to_uppercase()` — allocates a new `String`

### Assessment

These are infrequent operations (constructor, getter). No meaningful optimization
possible or needed. The `ikey()` interning pattern already handles the hot-path
string keys in `opendeviationbar_to_dict()`.

---

## Finding 7: Numeric type conversions — no unnecessary round-trips found

**Priority**: P2 (no action needed)

### Assessment

Numeric conversions are direct:

- **Input**: `Python int → Rust i64` via `extract()` — single C-API call
- **Output**: `Rust i64 → Python int` via `set_item()` — single C-API call
- No intermediate representations or unnecessary conversions detected

The `FixedPoint` ↔ `f64` conversions in `opendeviationbar_to_dict()` are necessary
(Python doesn't have a `FixedPoint` type, so `f64` is the correct output type).

The `i128` volume fields use `as f64` division by `VOLUME_SCALE` which is the
correct approach — no extra allocations.

---

## Finding 8: `process_trades_streaming()` converts bars one-at-a-time during processing loop

**File**: `src/core_bindings.rs:358-370`
**Priority**: P1

### Current behavior

```rust
for trade in agg_trades {
    match self.processor.process_single_trade(&trade) {
        Ok(Some(bar)) => bars.push(opendeviationbar_to_dict(py, &bar)?),  // <-- GIL-heavy
        Ok(None) => {}
        Err(e) => return Err(...),
    }
}
```

Each completed bar triggers `opendeviationbar_to_dict()` (31-55 `set_item` calls)
inline within the trade processing loop. This interleaves Rust computation with
Python API calls, destroying CPU cache locality.

### Proposed fix

Collect `Vec<OpenDeviationBar>` first, then batch-convert:

```rust
let mut rust_bars = Vec::with_capacity(agg_trades.len() / 10);
for trade in agg_trades {
    match self.processor.process_single_trade(&trade) {
        Ok(Some(bar)) => rust_bars.push(bar),
        Ok(None) => {}
        Err(e) => return Err(...),
    }
}
// Batch convert after all processing
let dicts: Vec<PyObject> = rust_bars
    .iter()
    .map(|bar| opendeviationbar_to_dict(py, bar))
    .collect::<PyResult<_>>()?;
```

This separates pure-Rust work from Python API work, improving cache locality.
Also enables future GIL release during the processing loop (Finding 4).

Compare with `process_trades_streaming_arrow()` (line 401-432) which already
does this correctly — it collects `Vec<OpenDeviationBar>` first, then converts.

### Quantification

- No memory savings (same number of bars)
- ~5-15% throughput improvement from better cache locality
- Enables GIL release during the processing loop

---

## Summary: Priority Matrix

| #   | Finding                                        | Priority | Impact                           | Effort                   |
| --- | ---------------------------------------------- | -------- | -------------------------------- | ------------------------ |
| 1   | `dict_to_agg_trade` — 14 C-API calls/trade     | P0       | HIGH (100K+ trades/batch)        | LOW (route to Arrow)     |
| 4   | GIL held during pure Rust computation          | P0       | HIGH (blocks all Python threads) | MED (`Send` bound issue) |
| 2   | `opendeviationbar_to_dict` — 55 `set_item`/bar | P1       | MED (1K bars/batch typical)      | LOW (route to Arrow)     |
| 3   | `Vec<Bound<PyDict>>` full materialization      | P1       | LOW (memory, not speed)          | LOW                      |
| 8   | Interleaved Rust/Python in streaming loop      | P1       | MED (cache locality)             | LOW                      |
| 5   | Double checkpoint bar conversion               | P2       | LOW (infrequent)                 | LOW                      |
| 6   | String conversions                             | P2       | NEGLIGIBLE                       | N/A                      |
| 7   | Numeric conversions                            | P2       | NONE (already optimal)           | N/A                      |

---

## Recommended Action Plan

### Phase 1: Eliminate dict path from hot loops (P0)

The Arrow path already exists and is well-implemented. The key action is
**migrating callers** from dict-based methods to Arrow-based methods:

- `process_trades(trades, return_format="arrow")` — already supported
- `process_trades_arrow(batch)` — Arrow in, Arrow out (zero Python API calls)
- `process_trades_streaming_arrow(trades)` — streaming with Arrow output

Audit Python callers in `python/opendeviationbar/` to find which still use the
dict path and can be migrated.

### Phase 2: GIL release for batch processing (P0)

The `process_trades_arrow()` and `process_trades_arrow_native()` methods are
entirely pure Rust after extracting the RecordBatch. Investigate whether the
`&mut self.processor` can be temporarily moved out to satisfy `Send` bounds
for `py.detach()`.

### Phase 3: Batch output conversion (P1)

Fix `process_trades_streaming()` to collect `Vec<OpenDeviationBar>` first,
then batch-convert — matching the pattern already used by
`process_trades_streaming_arrow()`.

### Phase 4: Cleanup (P2)

Remove duplicate `incomplete_bar` / `incomplete_bar_raw` in checkpoint conversion.
