# Arrow Zero-Copy Audit: Memory Efficiency at the Arrow Boundary

**Date**: 2026-03-22
**Scope**: Arrow import/export, AggTrade materialization, pipeline double-conversions

---

## Finding 1: `record_batch_to_aggtrades()` Materializes Full Vec<AggTrade> (P0)

**File**: `crates/opendeviationbar-core/src/arrow_export.rs:79-186`
**File**: `src/core_bindings.rs:455-479` (`process_trades_arrow`)

**Current behavior**: Every call to `process_trades_arrow()` or `process_trades_arrow_native()` converts the entire Arrow RecordBatch into a `Vec<AggTrade>` before processing trade-by-trade. For a 6-hour chunk (~250K trades), this allocates:

- `AggTrade` struct size: 56 bytes (5x i64 = 40B, 2x FixedPoint(i64) = 16B, 1x bool = 1B, 1x Option<bool> = 2B, padding to 56B)
- 250K trades x 56B = **14 MB per chunk** of intermediate allocation

The Arrow RecordBatch already holds this data in columnar buffers. The materialization into row-oriented `Vec<AggTrade>` is a full copy that exists only to satisfy `process_single_trade(&AggTrade)`.

**Proposed fix**: Add a `process_trades_from_columns()` method to `OpenDeviationBarProcessor` that accepts column slices directly:

```rust
pub fn process_from_arrow_columns(
    &mut self,
    timestamps: &[i64],
    prices: &[i64],      // FixedPoint raw values
    volumes: &[i64],     // FixedPoint raw values
    agg_trade_ids: &[i64],
    first_trade_ids: &[i64],
    last_trade_ids: &[i64],
    is_buyer_maker: &BooleanArray,
) -> Result<Vec<OpenDeviationBar>, ProcessingError>
```

This reads directly from Arrow's contiguous memory buffers without creating `AggTrade` structs. The processor would construct a transient `AggTrade` on the stack per iteration (not heap-allocated Vec), or better, inline the field access.

**Estimated savings**: 14 MB per 250K-trade chunk. For a full-day BTCUSDT (~1M trades), saves ~56 MB of transient allocation. For kintsugi processing 24 symbols, this is ~1.3 GB of avoided allocation throughput per pass.

**Complexity**: Medium. The processor's `process_single_trade()` takes `&AggTrade`, so the minimal change is stack-constructing AggTrade per iteration without Vec preallocation. The maximal change is refactoring the processor to accept individual fields.

---

## Finding 2: `f64_to_fixed_point()` in Tight Loop -- Already Zero-Cost (P2, no action)

**File**: `crates/opendeviationbar-core/src/arrow_export.rs:193-195`

```rust
#[inline]
fn f64_to_fixed_point(value: f64) -> crate::fixed_point::FixedPoint {
    crate::fixed_point::FixedPoint((value * SCALE as f64).round() as i64)
}
```

This is `#[inline]`, operates on stack values, and does no heap allocation. The `round()` call is an FPU instruction. **Zero-cost confirmed.** No action needed.

However, the conversion IS unnecessary when data originates from Arrow columns that were themselves produced from FixedPoint values. See Finding 5.

---

## Finding 3: `opendeviationbar_vec_to_record_batch()` -- Well-Optimized, Minor Improvement Possible (P2)

**File**: `crates/opendeviationbar-core/src/arrow_export.rs:315-431`

**Current behavior**: Uses Arrow builders with `with_capacity(n)` pre-allocation, single-pass iteration over bars. This is already well-optimized:

- All builders pre-allocated to exact size (no realloc)
- Single pass over bars (good L1 cache use per the comment)
- `Arc::new()` wraps each finished array (reference-counted, no copy)

**Minor improvement**: The `data_source` StringBuilder allocates `n * 16` bytes for string data. Since DataSource is an enum with 3 variants of known lengths (11, 17, 17 chars), this could use a `DictionaryArray<UInt8Type>` instead, saving ~15 bytes per bar on the string copy.

For 10K bars: current = 160 KB string allocation; dictionary = ~200 bytes (3 unique values + 10K u8 indices = ~10 KB). Savings: ~150 KB per batch. Negligible.

**Estimated savings**: Negligible. Not worth the complexity.

---

## Finding 4: Double Arrow Conversion in `_stream_bars_with_processor()` Pipeline (P0)

**File**: `python/opendeviationbar/orchestration/helpers.py:316-326`

**Current flow** (production populate path):

```
Polars DataFrame (trades)
  -> .to_arrow()                     # Polars -> Arrow RecordBatch (zero-copy if lucky, copy if rechunk needed)
  -> processor.process_trades_arrow()  # Arrow -> Vec<AggTrade> -> process -> Vec<ODB> -> Arrow RecordBatch
  -> pl.from_arrow()                 # Arrow -> Polars DataFrame (zero-copy)
  -> ts conversion exprs            # Polars column ops (us -> ms rename)
  -> validation                     # pandera schema check
  -> pl.concat()                    # accumulate
```

The key inefficiency: `process_trades_arrow()` in `core_bindings.rs:455-479`:

1. Converts Arrow RecordBatch -> `Vec<AggTrade>` (14 MB copy for 250K trades)
2. Iterates `Vec<AggTrade>` calling `process_single_trade()` per trade
3. Accumulates `Vec<OpenDeviationBar>`
4. Converts `Vec<OpenDeviationBar>` -> Arrow RecordBatch

Step 1 is the avoidable copy (Finding 1). Steps 3-4 are unavoidable since bars are produced one at a time.

**Proposed fix**: Same as Finding 1 -- eliminate the `Vec<AggTrade>` materialization.

---

## Finding 5: FixedPoint Round-Trip Through f64 in Arrow-to-Arrow Path (P1)

**File**: `crates/opendeviationbar-core/src/arrow_export.rs:174-176` (import)
**File**: `crates/opendeviationbar-core/src/arrow_export.rs:459-461` (export, `aggtrades_to_record_batch`)

**Current behavior in `stream_binance_trades_arrow` -> `process_trades_arrow_native` pipeline**:

1. Binance CSV parsed into `AggTrade` with `FixedPoint` prices (exact i64)
2. `aggtrades_to_record_batch()` converts `FixedPoint` -> `f64` via `.to_f64()` (line 474)
3. `record_batch_to_aggtrades()` converts `f64` -> `FixedPoint` via `f64_to_fixed_point()` which does `(value * SCALE).round() as i64` (line 176)

This f64 -> FixedPoint -> f64 -> FixedPoint round-trip loses no precision (f64 has 53-bit mantissa, FixedPoint values fit in 53 bits for typical prices), but it wastes CPU cycles on unnecessary float multiply + round operations for every trade.

**Proposed fix**: In the Rust-internal path (`stream_binance_trades_arrow` -> `process_trades_arrow_native`), store prices as `Int64` in the Arrow schema (raw FixedPoint values) instead of `Float64`. This eliminates the round-trip entirely. Would require a separate "internal" schema vs the "Python API" schema.

**Estimated savings**: ~2-5% CPU time reduction on the Arrow import path (eliminates 2x f64 mul + round per trade). No memory savings, but improved throughput.

**Complexity**: Medium. Requires a second Arrow schema for internal use, and conditional logic in `record_batch_to_aggtrades` to detect Int64 price columns.

---

## Finding 6: `bars_to_arrow()` in `arrow_bindings.rs` -- Full Python Dict Round-Trip (P1)

**File**: `src/arrow_bindings.rs:31-48`

**Current behavior**: `bars_to_arrow(bars: Vec<Bound<PyDict>>)` accepts Python dicts, converts each to a full `OpenDeviationBar` Rust struct (250+ bytes each, with 22 `Option<f64>` inter-bar fields all set to None), then converts to Arrow.

This function exists for users who have bars as Python dicts and want Arrow output. The conversion path is:

```
Python Dict -> dict_to_opendeviationbar_full() -> OpenDeviationBar -> Arrow builders -> RecordBatch
```

The `OpenDeviationBar` struct is large (~480 bytes including all Option fields for inter/intra-bar features). For 10K bars, the intermediate `Vec<OpenDeviationBar>` is ~4.8 MB, used briefly and then discarded.

**Proposed fix**: Build Arrow arrays directly from Python dict values without the intermediate Rust struct. Extract each field column-by-column:

```rust
for bar_dict in &bars {
    open_builder.append_value(get_f64_required(bar_dict, "open", i)?);
    high_builder.append_value(get_f64_required(bar_dict, "high", i)?);
    // ...
}
```

This skips the `OpenDeviationBar` allocation entirely and writes directly to Arrow builders.

**Estimated savings**: ~4.8 MB for 10K bars. However, `bars_to_arrow` is a utility function, not on the hot path (the hot path uses `opendeviationbar_vec_to_record_batch` directly from Rust structs). **Low priority.**

---

## Finding 7: Arrow Buffer Lifecycle in Streaming Pipeline (P2)

**File**: `src/core_bindings.rs:455-479` (`process_trades_arrow`)

**Current behavior**: The input `PyRecordBatch` (wrapping the Arrow RecordBatch) is borrowed for the duration of `record_batch_to_aggtrades()`, which extracts all data into `Vec<AggTrade>`. After extraction, the original RecordBatch buffers are still held alive by the `PyRecordBatch` parameter until the function returns.

This means during processing, both the input Arrow buffers AND the `Vec<AggTrade>` copy coexist in memory:

- Input RecordBatch: ~20 MB (250K trades x 8 columns x ~10 bytes avg)
- Vec<AggTrade>: ~14 MB
- Total overlap: ~34 MB

**Proposed fix**: Drop the input batch reference after extraction:

```rust
fn process_trades_arrow(&mut self, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    let agg_trades = {
        let record_batch = batch.as_ref();
        record_batch_to_aggtrades(record_batch, false)?
    };
    drop(batch);  // Release Arrow buffers before processing
    // ... process agg_trades ...
}
```

However, `PyRecordBatch` is owned by Python's refcount, so `drop()` only decrements the refcount -- the actual buffer is freed only when Python's GC runs. This fix has **negligible real-world impact** since Python will release it when the calling frame completes anyway.

**Estimated savings**: Theoretical 20 MB earlier release per chunk, but Python GC timing makes this unpredictable. Not worth the code change.

---

## Finding 8: `fetch_binance_aggtrades` Creates Python Dicts Instead of Arrow (P1)

**File**: `src/binance_bindings.rs:147-224`

**Current behavior**: `fetch_binance_aggtrades()` loads `Vec<AggTrade>` from Binance, then converts each trade to a Python dict (7 `set_item` calls per trade through the GIL). For 1M trades/day, this is 7M GIL-bound calls.

The Arrow-native path (`stream_binance_trades_arrow`) already exists as a separate function and avoids this entirely. But the dict-based `fetch_binance_aggtrades` is still called by the non-streaming path in `get_open_deviation_bars()` for short date ranges.

The same pattern appears in `fetch_aggtrades_rest`, `fetch_aggtrades_by_id`, and `fetch_aggtrades_parallel` (lines 244-438) -- all convert `Vec<AggTrade>` to Python dicts.

**Proposed fix**: Add `_arrow` variants of these functions (or a `return_format="arrow"` parameter) that return `PyRecordBatch` directly via `aggtrades_to_record_batch()`. This is already done for `stream_binance_trades_arrow`. The pattern to replicate exists.

**Estimated savings**: For kintsugi repair of 1 day (1M trades): eliminates 7M Python dict allocations + 7M GIL-bound `set_item` calls. Saves ~200 MB of Python dict overhead (each dict = ~200 bytes for 7 keys) and ~2 seconds of GIL time.

**Complexity**: Low. `aggtrades_to_record_batch()` already exists. Just wire it as an alternative return path.

---

## Architecture-Level Proposal: Columnar Processing Without AggTrade Materialization

### Current Architecture

```
Arrow RecordBatch (columnar, contiguous)
    |
    v  [record_batch_to_aggtrades: FULL COPY]
Vec<AggTrade> (row-oriented, heap-allocated)
    |
    v  [process_single_trade: per-row iteration]
Vec<OpenDeviationBar> (row-oriented, heap-allocated)
    |
    v  [opendeviationbar_vec_to_record_batch: builder pattern]
Arrow RecordBatch (columnar, contiguous)
```

### Proposed Architecture (Zero-Copy Input)

```
Arrow RecordBatch (columnar, contiguous)
    |
    v  [column slice references: ZERO COPY]
(&[i64], &[f64], &[f64], ...) column slices
    |
    v  [process_from_columns: stack-local AggTrade per iteration]
Vec<OpenDeviationBar> (row-oriented, heap-allocated)
    |
    v  [opendeviationbar_vec_to_record_batch: builder pattern]
Arrow RecordBatch (columnar, contiguous)
```

**Key insight**: Arrow `PrimitiveArray<T>` provides `.values()` returning `&[T]` -- a zero-copy slice into the Arrow buffer. We can iterate these slices in lockstep, constructing a stack-local `AggTrade` per iteration (never heap-allocated), and feeding it to `process_single_trade()`.

```rust
fn process_trades_arrow_zerocopy(&mut self, batch: &RecordBatch) -> Result<Vec<OpenDeviationBar>, ...> {
    let timestamps = batch.column_by_name("timestamp").unwrap()
        .as_any().downcast_ref::<Int64Array>().unwrap().values();
    let prices = batch.column_by_name("price").unwrap()
        .as_any().downcast_ref::<Float64Array>().unwrap().values();
    let volumes = batch.column_by_name("volume").unwrap()
        .as_any().downcast_ref::<Float64Array>().unwrap().values();
    // ... other columns ...

    let mut bars = Vec::with_capacity(timestamps.len() / 100);
    for i in 0..timestamps.len() {
        let trade = AggTrade {
            timestamp: timestamps[i] * 1000,
            price: f64_to_fixed_point(prices[i]),
            volume: f64_to_fixed_point(volumes[i]),
            agg_trade_id: agg_ids[i],
            // ...
        };
        if let Some(bar) = self.processor.process_single_trade(&trade)? {
            bars.push(bar);
        }
    }
    Ok(bars)
}
```

**This eliminates**: The `Vec<AggTrade>` heap allocation entirely. Each `AggTrade` lives on the stack for one loop iteration, then is dropped. The Arrow buffer is never copied.

**Cache efficiency**: Arrow columns are contiguous `&[i64]` / `&[f64]` buffers. Sequential iteration over `.values()` is maximally cache-friendly (each column is a single contiguous allocation, prefetcher works perfectly). The current `record_batch_to_aggtrades` already uses iterators which is good, but it writes to a heap-allocated Vec that then gets re-read -- doubling memory traffic.

**Cannot eliminate the output Vec<OpenDeviationBar>**: Bars are produced asynchronously (one bar per N trades), so we cannot stream them to Arrow builders until we know the total count. The builder pattern with `with_capacity()` is already optimal here.

### Feasibility Assessment

The `process_single_trade(&AggTrade)` signature takes a reference, so the stack-local construction works perfectly. No changes needed to the processor core -- only the PyO3 binding layer and `arrow_export.rs`.

**Risk**: Low. The AggTrade construction is identical; only the allocation strategy changes (heap Vec -> stack temporaries).

---

## Priority Summary

| #   | Finding                                                 | Priority | Memory Impact             | Effort     |
| --- | ------------------------------------------------------- | -------- | ------------------------- | ---------- |
| 1   | Vec<AggTrade> materialization from Arrow                | P0       | 14-56 MB/chunk            | Medium     |
| 4   | Double Arrow conversion in populate pipeline            | P0       | Same as #1 (root cause)   | Same as #1 |
| 8   | Dict-based fetch functions lack Arrow variants          | P1       | 200 MB/day (Python dicts) | Low        |
| 5   | FixedPoint f64 round-trip in internal path              | P1       | CPU only, ~2-5%           | Medium     |
| 6   | bars_to_arrow intermediate struct                       | P1       | 4.8 MB/10K bars           | Low        |
| 2   | f64_to_fixed_point already zero-cost                    | P2       | None                      | None       |
| 3   | opendeviationbar_vec_to_record_batch data_source string | P2       | ~150 KB/batch             | Low        |
| 7   | Arrow buffer lifecycle overlap                          | P2       | Theoretical, GC-dependent | Low        |

---

## Implementation Roadmap

**Phase 1 (P0)**: Implement `process_from_arrow_columns()` or equivalent zero-copy input path in `arrow_export.rs` + wire into `process_trades_arrow()` in `core_bindings.rs`. This is the single highest-impact change -- eliminates the `Vec<AggTrade>` allocation for every Arrow-path call.

**Phase 2 (P1)**: Add `_arrow` return variants to `fetch_aggtrades_rest`, `fetch_aggtrades_by_id`, `fetch_aggtrades_parallel` in `binance_bindings.rs`. Low effort, high impact for kintsugi.

**Phase 3 (P1)**: Internal FixedPoint-native Arrow schema for `stream_binance_trades_arrow` -> `process_trades_arrow_native` path, eliminating f64 round-trip.
