# Rust Memory Audit: Unnecessary Copies

Audited: `processor.rs`, `types.rs`, `interbar.rs`, `interbar_types.rs`, `arrow_export.rs`, `intrabar/features.rs`

Struct sizes (estimated):

- `OpenDeviationBar`: ~896 bytes (massive — 48 Option fields for inter/intra-bar features)
- `AggTrade`: ~58 bytes (compact, Copy-eligible but derives Clone)
- `TradeSnapshot`: ~48 bytes (already a lightweight projection of AggTrade)
- `InterBarConfig`: ~40 bytes (3 bools + 2 Option<bool> + enum with usize/i64)
- `OpenDeviationBarState`: ~896 + 16 + 3\*SmallVec overhead = ~1KB+

---

## P0: Hot Path, Measurable Impact

### F1: `AggTrade.clone()` in intra-bar trade accumulation

**File**: `processor.rs:1233,1250`
**Current code**:

```rust
// new_with_trade_accumulation:
sv.push(trade.clone());  // line 1233

// accumulate_trade:
self.accumulated_trades.push(trade.clone());  // line 1250
```

**Context**: Called on EVERY trade when `include_intra_bar_features = true`. AggTrade is 58 bytes. With typical 50-500 trades per bar across 4+ thresholds, this is 200-2000 clones per bar.
**Proposed fix**: AggTrade is all scalar fields (i64, FixedPoint(i64), bool, Option<bool>) — derive `Copy` on AggTrade. Then `push(*trade)` is a memcpy with no Drop overhead. Even without Copy, `push(*trade)` would work since AggTrade has no heap allocations — clone is already a bitwise copy, but the compiler can't elide the Clone trait call overhead.
**Estimated savings**: Eliminates clone trait dispatch overhead per trade. With Copy, the compiler may optimize more aggressively (e.g., register-passing). Marginal per-trade (~1-3ns) but multiplied by millions of trades.
**API change**: Yes — adding `#[derive(Copy)]` to AggTrade in `trade.rs`. Non-breaking (Copy is a subtrait of Clone). Downstream code using `.clone()` continues to work.

### F2: `OpenDeviationBar.clone()` in `get_incomplete_bar()`

**File**: `processor.rs:566`
**Current code**:

```rust
pub fn get_incomplete_bar(&self) -> Option<OpenDeviationBar> {
    self.current_bar_state.as_ref().map(|state| {
        let mut bar = state.bar.clone();  // ~896 byte clone
        bar.compute_microstructure_features();
        bar
    })
}
```

**Context**: Called by LiveBarEngine on every forming bar push. The clone is necessary here since we need to return an owned bar while keeping the state alive. However, the bar is ~896 bytes.
**Proposed fix**: No fix possible without API redesign. The clone is semantically required. Consider whether callers actually need the full bar or just a subset (e.g., OHLCV + a few features). A `FormingBarSnapshot` struct with only the ~100 bytes callers need would reduce copy cost by ~8x.
**Estimated savings**: ~800 bytes per forming bar push (if snapshot approach adopted).
**API change**: Yes — new return type if snapshot approach.

### F3: `OpenDeviationBar.clone()` in checkpoint creation

**File**: `processor.rs:886`
**Current code**:

```rust
pub fn create_checkpoint(&self, symbol: &str) -> Checkpoint {
    let (incomplete_bar, thresholds) = match &self.current_bar_state {
        Some(state) => (
            Some(state.bar.clone()),  // ~896 byte clone
            ...
```

**Context**: Called once per checkpoint (relatively infrequent). But checkpoints happen on every bar completion in the streaming path.
**Proposed fix**: Acceptable cost for checkpointing. Low priority.
**Estimated savings**: N/A — semantically required, infrequent.
**API change**: N/A.

### F4: `OpenDeviationBar.clone()` in `include_incomplete` path

**File**: `processor.rs:818`
**Current code**:

```rust
self.current_bar_state = Some(OpenDeviationBarState {
    bar: state.bar.clone(),  // ~896 byte clone
    upper_threshold: state.upper_threshold,
    lower_threshold: state.lower_threshold,
    accumulated_trades: SmallVec::new(),
    scratch_prices: SmallVec::new(),
    scratch_volumes: SmallVec::new(),
});
```

**Context**: Only in `process_agg_trade_records_with_options` when `include_incomplete=true`. This clone is needed to save checkpoint state while consuming the bar for output.
**Proposed fix**: Already optimized (avoids cloning accumulated_trades SmallVec). The bar clone is semantically required here.
**Estimated savings**: None — semantically required.
**API change**: N/A.

---

## P1: Moderate Impact

### F5: `InterBarConfig.clone()` on processor setup

**File**: `processor.rs:312,391`
**Current code**:

```rust
self.trade_history = Some(TradeHistory::new_with_cache(config.clone(), external_cache));
self.inter_bar_config = Some(config);
```

**Context**: Called once during processor initialization (not hot path). InterBarConfig is ~40 bytes. But the clone is unnecessary — the config could be passed by reference to `TradeHistory::new_with_cache()` since it only needs to read the config to set up internal state.
**Proposed fix**: Change `TradeHistory::new_with_cache()` to accept `&InterBarConfig` and clone internally only where needed (it stores the config). Or restructure to move config into TradeHistory and keep a reference/copy in processor.
**Estimated savings**: ~40 bytes per processor creation. Negligible.
**API change**: Internal only.

### F6: `symbol.to_string()` in checkpoint creation

**File**: `processor.rs:893`
**Current code**:

```rust
Checkpoint::new(
    symbol.to_string(),  // heap allocation
    ...
```

**Context**: Called on every checkpoint. Symbol is always a short string (e.g., "BTCUSDT", 7 bytes). The allocation is ~24 bytes (String header) + 7 bytes.
**Proposed fix**: Use `SmallString` or `CompactString` to avoid heap allocation for short symbols. Or accept `String` in the caller and move it in.
**Estimated savings**: One heap allocation per checkpoint (~31 bytes).
**API change**: Internal Checkpoint API would change.

### F7: `String` allocation in `fallback_permutation_entropy()`

**File**: `intrabar/features.rs:839`
**Current code**:

```rust
let pattern_key: String = indices.iter().map(|&i| i.to_string()).collect();
*pattern_counts.entry(pattern_key).or_insert(0usize) += 1;
```

**Context**: Called inside a loop over `n - m + 1` patterns. For m=3 and 60 trades, that's 58 iterations, each allocating a String. With m=3, pattern is 3 digits like "012", "021", etc.
**Proposed fix**: For m=3 (the common case), encode the pattern as a single `u8` or `u16` (6 permutations of 3 elements = fits in a byte). Use a `[usize; 6]` array instead of HashMap. For general m, use `(u8, u8, u8)` tuple or integer key instead of String.
**Estimated savings**: Eliminates 58+ heap allocations per call. For m=3, replaces HashMap + String keys with a fixed-size array lookup.
**API change**: Internal only.

### F8: `get_lookback_trades()` returns `SmallVec<[&TradeSnapshot; 256]>`

**File**: `interbar.rs:467`
**Current code**:

```rust
pub fn get_lookback_trades(&self, bar_open_time: i64) -> SmallVec<[&TradeSnapshot; 256]> {
    // ... builds SmallVec of references by iterating trades
    let mut result = SmallVec::new();
    for i in 0..cutoff_idx {
        result.push(&self.trades[i]);
    }
    return result;
```

**Context**: Called once per bar for inter-bar features. 256 pointers inline = 2048 bytes on stack. If lookback > 256 trades, spills to heap.
**Proposed fix**: Return a slice or iterator instead of materializing a SmallVec. The `trades` VecDeque already holds the data contiguously (in two slices). Could return `(slice1, slice2)` from VecDeque's `as_slices()` and pass cutoff_idx to callers, avoiding the copy of 256+ pointers.
**Estimated savings**: Eliminates 2048+ bytes of pointer copying per bar (256 \* 8 bytes per pointer).
**API change**: Internal — `compute_features` and tier functions would accept `(&[TradeSnapshot], &[TradeSnapshot])` or a custom iterator.

---

## P2: Minor / Test-Only

### F9: `features.clone()` in interbar_cache test

**File**: `interbar_cache.rs:199`
**Context**: Test-only. No production impact.

### F10: Various `path_str.clone()` in test_data_loader.rs

**File**: `test_data_loader.rs:235,251,259,295,316,324`
**Context**: Test-only data loading. No production impact.

### F11: `global_cache.clone()` (Arc clone)

**File**: `interbar.rs:1988`, `entropy_cache_global.rs:98`
**Context**: Arc clone is just a reference count increment (~1 atomic op). Not a deep copy.

### F12: `runtime.handle().clone()` in bindings

**File**: `src/streaming_bindings.rs:147`, `src/stream_manager_bindings.rs:366`
**Context**: Tokio Handle clone is lightweight (Arc-based). Correct usage.

---

## Summary Table

| ID  | Priority | File:Line              | What                                     | Savings Est.      | API Change |
| --- | -------- | ---------------------- | ---------------------------------------- | ----------------- | ---------- |
| F1  | P0       | processor.rs:1233,1250 | AggTrade.clone() per trade (intra-bar)   | ~1-3ns/trade      | Yes (Copy) |
| F2  | P0       | processor.rs:566       | ODB.clone() in get_incomplete_bar()      | ~800B/push        | Yes        |
| F3  | P1       | processor.rs:886       | ODB.clone() in create_checkpoint()       | ~896B/checkpoint  | N/A        |
| F4  | P1       | processor.rs:818       | ODB.clone() in include_incomplete        | Semantically req. | N/A        |
| F5  | P2       | processor.rs:312,391   | InterBarConfig.clone() on setup          | ~40B (once)       | Internal   |
| F6  | P2       | processor.rs:893       | symbol.to_string() in checkpoint         | ~31B heap alloc   | Internal   |
| F7  | P1       | intrabar/features:839  | String alloc in permutation_entropy loop | 58+ heap allocs   | Internal   |
| F8  | P1       | interbar.rs:467        | SmallVec<[&T; 256]> materialization      | ~2KB/bar          | Internal   |

## Recommendations (Ranked)

1. **F1 (P0)**: Derive `Copy` on `AggTrade`. Zero-risk, eliminates clone overhead on hot path. AggTrade is 58 bytes of pure scalars with no heap data. This is the highest-value, lowest-risk change.

2. **F7 (P1)**: Replace String-keyed HashMap with integer-keyed array in permutation entropy. For m=3 (the hardcoded common case), there are only 6 possible permutations. Use a `[usize; 6]` array with a permutation-to-index function. Eliminates all heap allocation in the inner loop.

3. **F8 (P1)**: Refactor `get_lookback_trades()` to avoid materializing a SmallVec of pointers. Return VecDeque slices directly. This saves ~2KB of pointer copying per bar.

4. **F2 (P0)**: Consider a `FormingBarSnapshot` lightweight struct for `get_incomplete_bar()` if profiling shows this is a bottleneck. The 896-byte clone on every forming bar push is significant in the streaming path.
