# Memory Benchmark Analysis Report

## Executive Summary

The memory benchmark reveals unexpected memory growth in the Wasot random generator that contradicts the expected constant memory usage. While memory stays relatively constant for up to ~400K events (~60MB), it shows dramatic linear growth beyond that threshold, reaching over 3GB for 100M events.

## Data Analysis

### Memory Usage Pattern

| Events | Memory (MB) | Growth Factor |
|--------|-------------|---------------|
| 1,000 - 400,000 | ~60MB | Constant |
| 4,000,000 | 139MB | 2.3x |
| 10,000,000 | 343MB | 5.7x |
| 100,000,000 | 3,433MB | 57x |

### Key Observations

1. **Two-phase behavior**: Memory usage is constant up to ~400K events, then grows linearly
2. **Linear growth**: Beyond the threshold, memory grows approximately 34MB per million events
3. **Base overhead**: ~60MB baseline memory usage regardless of event count

## Root Cause Analysis

### Architecture Overview

The random command uses this flow:
```
RandomEmitter → Channel → WasotReconstructor → Output
```

### Memory Accumulation Points

After analyzing the code, several potential memory accumulation points were identified:

#### 1. Channel Buffering (Primary Suspect)
```rust
let (sender, receiver) = mpsc::channel();
```

Rust's `mpsc::channel()` creates an **unbounded** channel. When the emitter produces events faster than the consumer processes them, events accumulate in the channel buffer.

**Evidence**: The memory growth correlates with event count, suggesting buffering of event data.

#### 2. Block Stack in WasotReconstructor (Minor)
```rust
pub struct WasotReconstructor<W: Write> {
    block_stack: Vec<usize>, // Track block levels
    // ...
}
```

The reconstructor maintains a stack of open blocks. While this grows with nesting depth, it's unlikely to cause significant memory usage.

#### 3. String Generation (Minimal)
The random emitter generates strings for:
- Type names (from fixed array)
- Random words (temporary)
- Text content (temporary)

These are created per-event but should be dropped immediately after sending.

### Performance Bottleneck Analysis

#### Threading Model
```rust
// Spawn emitter thread
let emitter_handle = thread::spawn(move || {
    let events = emitter.subscribe();
    for event in events {
        if sender.send(event).is_err() {
            break;
        }
    }
});

// Main thread processes events
for event in receiver {
    reconstructor.on_element_event(&event);
}
```

#### The Problem
1. **Producer**: RandomEmitter generates events in memory (no I/O)
2. **Consumer**: WasotReconstructor writes to stdout/file (I/O bound)
3. **Channel**: Unbounded buffer grows when producer outpaces consumer

At high event counts, the emitter generates events faster than the reconstructor can write them, causing the channel to buffer millions of events in memory.

## Event Size Estimation

Each `Event` contains:
- Variant discriminant: ~8 bytes
- MetaLine with strings: ~100-500 bytes estimated
- Text content: Variable length

Estimated average event size: ~200-400 bytes
For 100M events: 20-40GB of data passing through the system
Actual measured: ~3.4GB (consistent with buffering only a fraction)

## Performance Characteristics

### CPU vs I/O Bound Analysis
- **Random generation**: CPU-bound, very fast
- **String formatting**: CPU-bound, fast  
- **File I/O**: I/O-bound, slower
- **stdout**: I/O-bound, potentially slower with terminal rendering

### Scaling Behavior
- Small files: Producer and consumer stay synchronized
- Large files: Producer generates faster than consumer writes, causing backpressure

## Recommended Solutions

### 1. Bounded Channel (Recommended)
```rust
use std::sync::mpsc;
let (sender, receiver) = mpsc::sync_channel(1000); // Bounded buffer
```

**Benefits**:
- Prevents unbounded memory growth
- Creates natural backpressure
- Maintains streaming behavior

### 2. Direct Processing (Alternative)
```rust
// Remove threading, process events directly
let emitter = RandomEmitter::new(event_count);
for event in emitter.subscribe() {
    reconstructor.on_element_event(&event);
}
```

**Benefits**:
- Eliminates channel overhead
- Simpler architecture
- Constant memory usage

### 3. Batch Processing (For Large Files)
Process events in batches to balance memory usage and performance.

## Testing Recommendations

### Verify Fix
1. Implement bounded channel solution
2. Re-run memory benchmark
3. Confirm constant memory usage across all scales

### Additional Tests
1. **Streaming test**: Verify 1GB+ file generation with constant memory
2. **Performance test**: Measure throughput impact of bounded channels
3. **Backpressure test**: Confirm system handles slow consumers gracefully

## Conclusion

The memory growth is caused by unbounded channel buffering when the event producer outpaces the consumer. The solution is to implement a bounded channel to create backpressure and maintain constant memory usage regardless of file size.

This is a classic producer-consumer synchronization issue, not a fundamental algorithmic problem with the Wasot compiler.