# Memory Analysis Report: Wasot Random Command

## Executive Summary

The `wasot random` command (see [random generator](./random-generator.md)) exhibits **exponential memory growth** that scales from ~60MB for small inputs to ~270MB for 400,000 events. This unexpected behavior contradicts the design expectation of constant memory usage for a streaming parser (see [architecture](./architecture.md)).

**Root Cause**: The `RandomEmitter` pre-generates ALL events in memory before streaming, causing memory usage to scale linearly with event count rather than remaining constant.

## Memory Usage Data Analysis

### Observed Memory Pattern

| Lines | Memory (MB) | Growth Factor |
|-------|-------------|---------------|
| 1,000 | 60.29 | Baseline |
| 82,662 | 60.42 | 1.00x (flat) |
| 113,307 | 79.58 | 1.32x (growth starts) |
| 155,313 | 103.95 | 1.31x |
| 212,891 | 145.89 | 1.40x |
| 291,816 | 198.05 | 1.36x |
| 400,000 | 270.82 | 1.37x |

### Critical Observations

1. **Memory remains constant (~60MB) up to ~80K events** - This is the base runtime memory footprint
2. **Sharp increase begins around 113K events** - Memory jumps to 79.58MB
3. **Linear scaling above 113K events** - Each additional ~100K events adds ~100MB of memory
4. **Total growth: 4.5x from baseline** at 400K events

## Root Cause Analysis

### Problem: Eager Event Generation

The `RandomEmitter::generate_events()` method creates a complete `Vec<Event>` containing ALL events before returning:

```rust
// src/emitters/random_emitter.rs:29
fn generate_events(&self) -> Vec<Event> {
    let mut events = Vec::new();
    
    for _ in 0..self.event_count {  // Creates ALL events upfront
        // ... generate event
        events.push(event);         // Accumulates in memory
    }
    
    events  // Returns complete vector
}
```

### Memory Breakdown Analysis

Each `Event` contains:
- **BlockStarted**: `MetaLine` with type name, arguments, keyword arguments, inline text, ID
- **TextFinished**: `Vec<String>` with 1-5 text lines per event
- **BlockFinished**: Minimal data

**Estimated memory per event:**
- Average meta-line: ~50-100 bytes
- Text lines: ~50-200 bytes 
- Rust overhead: ~50 bytes
- **Total: ~150-350 bytes per event**

At 400,000 events: 400K × 250 bytes ≈ **100MB** for event data alone.

### Why the ~60MB Base Memory?

The constant ~60MB for smaller inputs represents:
- Rust runtime initialization
- Cargo/executable overhead  
- Thread stack allocation
- Channel creation for inter-thread communication
- Initial Vec capacity allocation

### Why the Threshold at ~113K Events?

The sharp increase around 113K events likely indicates:
- **Vec reallocation threshold**: Rust Vec capacity grows exponentially (2x each time)
- **Memory fragmentation**: Large allocations trigger different memory management
- **System memory pressure**: OS memory management changes behavior

## Impact Assessment

### Performance Problems

1. **Memory Scaling**: O(n) memory usage instead of O(1)
2. **Startup Latency**: Must generate all events before processing begins
3. **Memory Pressure**: Large inputs can cause system memory pressure
4. **Poor Streaming**: Defeats the purpose of event-based streaming architecture

### Expected vs Actual Behavior

| Aspect | Expected (Streaming) | Actual (Eager) |
|--------|---------------------|----------------|
| Memory Usage | O(1) - constant | O(n) - linear |
| Startup Time | O(1) - immediate | O(n) - proportional |
| Peak Memory | ~60MB always | 60MB → 270MB+ |
| Processing | Lazy/streaming | Eager/batch |

## Solutions

### Immediate Fix: True Lazy Generation

Convert `RandomEmitter` to generate events lazily:

```rust
pub struct RandomEmitter {
    event_count: usize,
    generated: AtomicUsize,
    rng: Mutex<StdRng>,
}

impl Iterator for RandomEmitterIterator {
    fn next(&mut self) -> Option<Event> {
        if self.generated >= self.event_count {
            return None;
        }
        
        // Generate single event on-demand
        let event = self.generate_single_event();
        self.generated += 1;
        Some(event)
    }
}
```

### Alternative: Streaming Generator

Use a generator pattern to yield events one at a time without pre-allocation.

## Verification Strategy

After implementing the fix, memory usage should:
1. **Remain constant at ~60MB** for all input sizes
2. **Show no correlation** between event count and memory usage
3. **Start processing immediately** without upfront generation delay

## Recommendations

1. **Immediate**: Fix `RandomEmitter` to use lazy generation
2. **Validation**: Re-run memory benchmarks to verify O(1) memory usage
3. **Testing**: Add memory usage tests to prevent regression
4. **Documentation**: Update architecture docs to emphasize streaming design

## Additional Notes

### Why This Wasn't Caught Earlier

1. **Testing focused on correctness**, not memory efficiency
2. **Small test inputs** (< 100K events) hide the problem
3. **Development environment** may have different memory behavior than production

### Broader Implications

This issue highlights the importance of:
- **Memory profiling** during development
- **Large-scale testing** to catch scaling issues  
- **Architectural validation** that implementation matches design intent

The Wasot compiler's [event-driven architecture](./architecture.md) is sound, but the [random generator](./random-generator.md) implementation violated the streaming principle by eager evaluation. For related testing strategies, see [testing](./testing.md).