# Memory Profiling Guide

This guide explains how to diagnose and fix memory issues in Rust applications, with a focus on unbounded memory growth (memory leaks). We'll cover multiple tools and demonstrate with a real case study from this codebase.

## Table of Contents

1. [Quick Start](#quick-start)
2. [Understanding Memory Issues](#understanding-memory-issues)
3. [Tool Comparison](#tool-comparison)
4. [DHAT: Precision Heap Analysis](#dhat-precision-heap-analysis)
5. [Heaptrack: Visual Timeline](#heaptrack-visual-timeline)
6. [Valgrind Massif: The Fallback](#valgrind-massif-the-fallback)
7. [Case Study: The ElementParser Leak](#case-study-the-elementparser-leak)
8. [Common Rust Memory Patterns](#common-rust-memory-patterns)
9. [Prevention Strategies](#prevention-strategies)

---

## Quick Start

For most memory leak investigations, start with DHAT:

```bash
# 1. Build with profiling (already configured in this project)
cargo build --release --features dhat-heap

# 2. Run your command
./target/release/wasot your_file.wasot -t jsonl -o /tmp/out.jsonl

# 3. Analyze the output
# Upload dhat-heap.json to: https://nnethercote.github.io/dh_view/dh_view.html
# Focus on the "At t-gmax (bytes)" column
```

---

## Understanding Memory Issues

### Types of Memory Problems

| Problem | Symptom | Cause |
|---------|---------|-------|
| **Memory Leak** | Memory grows continuously | Allocations without deallocation |
| **Unbounded Growth** | Memory grows with input size | Collections that accumulate without bounds |
| **Peak Spike** | Sudden high memory, then drops | Large temporary allocations |
| **Fragmentation** | High RSS but low heap usage | Many small allocations/deallocations |

### Key Metrics

- **t-gmax**: The moment of peak heap usage (most important for leaks)
- **t-end**: Memory still allocated when program exits
- **Total bytes**: Sum of all allocations (not current usage)
- **RSS**: Resident Set Size - actual physical memory used

---

## Tool Comparison

| Tool | Best For | Overhead | Line Numbers | Timeline |
|------|----------|----------|--------------|----------|
| **DHAT** | Finding peak allocation sites | Low | Yes | No |
| **Heaptrack** | Visualizing growth patterns | Low | Yes | Yes |
| **Valgrind Massif** | Detailed snapshots | High | Yes | Yes |
| **perf + flamegraph** | CPU + allocation correlation | Very Low | Partial | No |

**Recommendation:** Start with DHAT for diagnosis, use Heaptrack for visualization.

---

## DHAT: Precision Heap Analysis

DHAT (Dynamic Heap Analysis Tool) is the best tool for finding exactly which code is responsible for peak memory usage.

### Setup

**Step 1: Add to Cargo.toml**

```toml
[dependencies]
dhat = { version = "0.3", optional = true }

[features]
dhat-heap = ["dhat"]

[profile.release]
debug = true  # Required for line numbers
strip = false # Don't strip symbols when profiling
```

**Step 2: Instrument main.rs**

```rust
// Use dhat allocator for heap profiling
#[cfg(feature = "dhat-heap")]
#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;

fn main() {
    // Initialize profiler at the very start
    #[cfg(feature = "dhat-heap")]
    let _profiler = dhat::Profiler::new_heap();

    // ... your code ...
}
```

**Step 3: Handle Custom Allocators**

If you use a custom allocator (e.g., mimalloc), conditionally disable it:

```rust
// Use mimalloc normally, but dhat when profiling
#[cfg(all(feature = "optimized-alloc", not(feature = "dhat-heap")))]
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;

#[cfg(feature = "dhat-heap")]
#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;
```

### Running DHAT

```bash
# Build with profiling
cargo build --release --features dhat-heap

# Run your program (creates dhat-heap.json)
./target/release/your_program <args>

# Output will show summary:
# dhat: Total:     42,765,115,456 bytes in 145,935,985 blocks
# dhat: At t-gmax: 124,442,836 bytes in 1,021,356 blocks  <-- PEAK
# dhat: At t-end:  68 bytes in 2 blocks
```

### Analyzing Results

1. Go to [DHAT Viewer](https://nnethercote.github.io/dh_view/dh_view.html)
2. Upload `dhat-heap.json`
3. **Focus on "At t-gmax (bytes)" column** - this shows allocations present at peak memory
4. Click on entries to see full stack traces

### Interpreting DHAT Output

```
PP 1/1000 {
  Total:     37,158,600 bytes in 1,020,214 blocks
  At t-gmax: 37,158,600 bytes in 1,020,214 blocks  <-- Responsible for 30% of peak!
  At t-end:  0 bytes in 0 blocks

  Allocated at {
    #1: alloc::vec::Vec<T>::push
    #2: wasot::parsers::wasot::block::BlockContainer::add_block (block.rs:337)
    #3: wasot::parsers::wasot::element_parser::ElementParser::finish_current_text_block (element_parser.rs:249)
    ...
  }
}
```

Key insights:
- High "At t-gmax" with low "At t-end" = temporary accumulation (potential optimization)
- High "At t-gmax" AND high "At t-end" = true leak (not freed)
- Stack trace shows exact file:line causing allocation

---

## Heaptrack: Visual Timeline

Heaptrack provides a graphical view of memory usage over time - perfect for confirming linear growth.

### Installation

```bash
# Ubuntu/Debian
sudo apt install heaptrack heaptrack-gui

# Fedora
sudo dnf install heaptrack

# macOS (via Homebrew)
brew install heaptrack
```

### Usage

```bash
# Record
heaptrack ./target/release/your_program <args>

# Analyze (GUI)
heaptrack_gui heaptrack.your_program.<pid>.zst

# Analyze (CLI)
heaptrack_print heaptrack.your_program.<pid>.zst
```

### Reading the GUI

1. **Consumed Tab**: Shows heap memory over time
   - Steady diagonal line = linear growth (leak!)
   - Flat line = constant memory (good)
   - Sawtooth pattern = normal allocation/deallocation

2. **Bottom-Up Tab**: Shows which functions are holding memory
   - Sort by "peak" to find worst offenders

3. **Flame Graph Tab**: Visual call stack of allocations

---

## Valgrind Massif: The Fallback

Massif is slower but works when other tools fail.

### Usage

```bash
# Run with Massif
valgrind --tool=massif ./target/release/your_program <args>

# Analyze
ms_print massif.out.<pid>

# Or use massif-visualizer for GUI
massif-visualizer massif.out.<pid>
```

### Output

```
    MB
100.0^                                                       #
     |                                                      @#
     |                                                    @@:#
     |                                                  @@@::#
     |                                                @@@@:::#
     |                                              @@@@@::::#
     |                                            @@@@@@:::::#
     |                                          @@@@@@@::::::#
     |                                        @@@@@@@@:::::::#
     |                                      @@@@@@@@@::::::::#
     |                                    @@@@@@@@@@:::::::::#
     |                                  @@@@@@@@@@@::::::::::#
     |                                @@@@@@@@@@@@:::::::::::#
     |                              @@@@@@@@@@@@@::::::::::::#
     |                            @@@@@@@@@@@@@@:::::::::::::#
   0 +--------------------------------------------------------------->
```

The diagonal climb confirms unbounded growth.

---

## Case Study: The ElementParser Leak

This is a real memory leak fixed in this codebase on 2026-02-10.

### Symptoms

Processing a 50MB wasot file caused 124MB peak memory usage, when it should have been ~1MB constant.

### Investigation

**Step 1: Profile with DHAT**

```bash
cargo build --release --features dhat-heap
./target/release/wasot data/50mb.wasot -t jsonl -o /tmp/out.jsonl
```

Output:
```
dhat: At t-gmax: 124,442,836 bytes in 1,021,356 blocks
```

**Step 2: Analyze dhat-heap.json**

Top allocations at peak:

| Bytes | Location | Function |
|-------|----------|----------|
| 37.1 MB | element_parser.rs:249 | `finish_current_text_block` |
| 19.6 MB | element_parser.rs:249 | Cloning Text |
| 14.0 MB | block.rs:337 | `add_block` |

**Step 3: Find the Code**

```rust
// element_parser.rs:249 - THE CULPRIT
fn finish_current_text_block(&mut self) {
    if let Some(text_block) = self.current_text_block.take() {
        if !text_block.is_empty() {
            // This clones EVERY block into the container!
            self.container.add_block(text_block.clone());
        }
    }
}
```

### Root Cause

The `ElementParser` was designed for two use cases:
1. **Batch parsing**: Returns all blocks in a container
2. **Event streaming**: Only needs to track block open/close

But it ALWAYS accumulated blocks, even when streaming (where the container is never used).

### The Fix

Added a `streaming_mode` that skips container accumulation:

```rust
pub struct ElementParser {
    // ...
    streaming_mode: bool,  // NEW
}

impl ElementParser {
    pub fn new_streaming() -> Self {
        Self {
            streaming_mode: true,
            // ...
        }
    }

    fn finish_current_text_block(&mut self) {
        if let Some(text_block) = self.current_text_block.take() {
            // Only accumulate if NOT in streaming mode
            if !text_block.is_empty() && !self.streaming_mode {
                self.container.add_block(text_block);
            }
        }
    }
}
```

### Results

| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Peak memory | 124 MB | 0.9 MB | **99.3%** |

---

## Common Rust Memory Patterns

### Pattern 1: Unbounded Channels

```rust
// BAD: Unbounded channel grows forever if consumer is slow
let (tx, rx) = std::sync::mpsc::channel();

// GOOD: Bounded channel provides backpressure
let (tx, rx) = std::sync::mpsc::sync_channel(1000);
```

### Pattern 2: Accidental Accumulation

```rust
// BAD: Vec grows inside loop
let mut results = Vec::new();
for item in huge_iterator {
    results.push(process(item));  // Accumulates everything!
}

// GOOD: Process and discard
for item in huge_iterator {
    let result = process(item);
    emit(result);  // Don't accumulate
}
```

### Pattern 3: Cloning in Hot Paths

```rust
// BAD: Clone on every iteration
for block in blocks {
    container.add(block.clone());  // Expensive!
}

// GOOD: Move ownership or use references
for block in blocks {
    container.add(block);  // Move, don't clone
}
```

### Pattern 4: String Interning Gone Wrong

```rust
// BAD: Intern cache grows with unique strings
static CACHE: Mutex<HashSet<String>> = ...;
fn intern(s: &str) -> &'static str {
    CACHE.lock().insert(s.to_string());  // Never shrinks!
}

// GOOD: Use LRU cache with size limit
static CACHE: Mutex<LruCache<String, ()>> = LruCache::new(10000);
```

### Pattern 5: Hidden Allocations in Display/Debug

```rust
// BAD: format! allocates on every log
for item in millions_of_items {
    log::debug!("Processing: {}", item);  // String allocation!
}

// GOOD: Use lazy formatting or check log level
for item in millions_of_items {
    if log::log_enabled!(log::Level::Debug) {
        log::debug!("Processing: {}", item);
    }
}
```

---

## Prevention Strategies

### 1. Design for Streaming

When processing large data, prefer iterators over collecting:

```rust
// Instead of
fn process_all(items: Vec<Item>) -> Vec<Result> {
    items.into_iter().map(process).collect()
}

// Use
fn process_stream(items: impl Iterator<Item=Item>) -> impl Iterator<Item=Result> {
    items.map(process)
}
```

### 2. Bounded Data Structures

Always consider maximum sizes:

```rust
use std::collections::VecDeque;

struct BoundedBuffer<T> {
    buffer: VecDeque<T>,
    max_size: usize,
}

impl<T> BoundedBuffer<T> {
    fn push(&mut self, item: T) {
        if self.buffer.len() >= self.max_size {
            self.buffer.pop_front();
        }
        self.buffer.push_back(item);
    }
}
```

### 3. Periodic Memory Checks in Tests

```rust
#[test]
fn test_memory_bounded() {
    let before = get_memory_usage();

    // Process large input
    process_large_file("test_data/large.wasot");

    let after = get_memory_usage();

    // Memory should not grow significantly
    assert!(after - before < 10 * 1024 * 1024, "Memory grew by {}MB", (after - before) / 1024 / 1024);
}
```

### 4. Profile in CI

Add memory profiling to your CI pipeline:

```yaml
# .github/workflows/memory.yml
memory-check:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - name: Build with profiling
      run: cargo build --release --features dhat-heap
    - name: Run memory test
      run: |
        ./target/release/wasot test_data/large.wasot -t jsonl -o /dev/null 2>&1 | tee memory.log
        grep "At t-gmax" memory.log | awk '{print $3}' | xargs -I {} test {} -lt 10000000
```

---

## Summary

1. **Start with DHAT** for precise allocation site identification
2. **Use Heaptrack** for visual timeline analysis
3. **Focus on t-gmax** - that's where your peak memory lives
4. **Common culprits**: unbounded collections, cloning in loops, accumulating instead of streaming
5. **Prevention**: design for streaming, use bounded structures, test memory usage

For this project, run:
```bash
cargo build --release --features dhat-heap
./target/release/wasot <your-file> -t jsonl -o /tmp/out.jsonl
# Then upload dhat-heap.json to the DHAT viewer
```
