# Testing Infrastructure Guide

A comprehensive guide to Wasot's testing landscape, designed to help new developers understand and navigate the various types of tests, tools, and workflows used in the project. For syntax-specific testing, see [syntax testing](./syntax_testing.md). For code coverage analysis, see [coverage](./coverage.md).

## Notes

Types of testing:

- unit tests
- ./scripts/test_syntax.sh
- roundtrip tests
- fuzzing
- real world use cases
- interesting stuff like converting a massive html and md body of text
- large files testing
- Exploratary testing via `❯ echo '."custom block" "Inline text" .. arg1 "arg 2" key="value" :my-id' | cargo run --bin wasot -- -t jsonl`
- benchmarking

## 📋 Table of Contents

- [Quick Start](#-quick-start)
- [Test Categories](#-test-categories)
- [Fast vs Slow Tests](#%EF%B8%8F-fast-vs-slow-tests)
- [Test Types Overview](#-test-types-overview)
- [Auto-Generated Tests](#-auto-generated-tests)
- [Running Tests](#-running-tests)
- [Test Scripts](#-test-scripts)
- [Performance Analysis](#-performance-analysis)
- [Test Organization](#-test-organization)
- [Adding New Tests](#-adding-new-tests)
- [Troubleshooting](#-troubleshooting)

---

## 🚀 Quick Start

```bash
# Fast development cycle (recommended for regular work)
cargo test                    # ~7 seconds, 796 tests
./scripts/test_fast.sh        # Same with nice output

# Full validation before commits
./scripts/test_all.sh         # Rust + Python + WASM + slow tests

# Run only slow tests (when needed)
cargo test -- --ignored      # 19 slow tests

# Run specific test types
cargo test generated_syntax_tests    # Auto-generated syntax tests
cargo test honey_badger              # Edge case tests
cargo test fuzz                      # Fuzzing tests
```

---

## 🏷️ Test Categories

### By Performance (Fast vs Slow)

| Category | Count | Duration | Usage |
|----------|-------|----------|-------|
| **Fast Tests** | 796 | ~7 seconds | Regular development |
| **Slow Tests** | 19 | ~60+ seconds | Full validation |
| **Total** | 815+ | ~67 seconds | Complete test suite |

### By Type

| Type | Location | Count | Purpose |
|------|----------|-------|---------|
| **Unit Tests** | `src/**/*.rs` | ~600+ | Component testing |
| **Integration Tests** | `tests/*.rs` | ~50+ | End-to-end workflows |
| **Auto-Generated** | `tests/generated_syntax_tests.rs` | 263 | Syntax validation |
| **Fuzz Tests** | `src/parsers/wasot/fuzz_tests.rs` | 7 | Edge case discovery |
| **Honey Badger** | `src/parsers/wasot/honey_badger_tests.rs` | 18 | Extreme edge cases |
| **Performance** | Various locations | ~5 | Benchmarking |

---

## ⚡️ Fast vs Slow Tests

### Fast Tests (~7 seconds)
**Purpose**: Regular development cycle, quick feedback
**Threshold**: <200ms per test (accounting for compilation overhead)
**Usage**: `cargo test` (default)

**Includes**:
- Most unit tests
- Regular integration tests  
- Simple parsing tests
- Basic functionality validation

### Slow Tests (~60+ seconds)
**Purpose**: Comprehensive validation, CI/CD pipelines
**Threshold**: >200ms per test
**Usage**: `cargo test -- --ignored`
**Marked with**: `#[ignore = "slow"]`

**Includes**:
- Fuzz tests (10,000+ iterations)
- Honey badger tests (extreme edge cases)
- Performance benchmarks
- Memory stress tests
- Large input processing

**Slow Test Examples**:
```rust
#[test]
#[ignore = "slow"] // Very slow test: takes 1181ms. Run with --ignored to include.
fn test_extremely_long_lines() {
    // Tests with 10MB type names
}

#[test]
#[ignore = "slow"] // Slow test: takes 786ms. Run with --ignored to include.
fn test_performance_comparison() {
    // Performance benchmarking
}
```

---

## 🧪 Test Types Overview

### 1. Unit Tests
**Location**: Throughout `src/` directories
**Purpose**: Test individual components in isolation
**Examples**:
```rust
// src/base36.rs
#[test]
fn test_base36_to_u64() {
    assert_eq!(base36_to_u64("0").unwrap(), 0);
    assert_eq!(base36_to_u64("Z").unwrap(), 35);
}

// src/parsers/wasot/parser.rs  
#[test]
fn test_extract_id_string() {
    let result = extract_id_string("test :id123 more");
    assert_eq!(result, Some(("test ", "id123", " more")));
}
```

### 2. Integration Tests
**Location**: `tests/*.rs`
**Purpose**: Test complete workflows and component interactions
**Examples**:
- `tests/roundtrip_tests.rs` - Parse → Convert → Parse validation
- `tests/integration/` - Complex scenario testing

### 3. Auto-Generated Syntax Tests
**Location**: `tests/generated_syntax_tests.rs` (generated)
**Source**: `tests/wasot_syntax_tests.md` (human-readable spec)
**Generator**: `scripts/generate_syntax_tests.py`
**Count**: 263 comprehensive syntax tests

**Workflow**:
```bash
# Edit test specifications
vim tests/wasot_syntax_tests.md

# Generate Rust test code
./scripts/test_syntax.sh

# Run generated tests
cargo test generated_syntax_tests
```

### 4. Fuzz Tests
**Location**: `src/parsers/wasot/fuzz_tests.rs`
**Purpose**: Discover edge cases through randomized input
**Characteristics**:
- 10,000+ iterations per test
- Random string generation
- Stress test memory safety
- Property-based testing

**Examples**:
```rust
#[test]
#[ignore = "slow"] // Slow test: takes 60+ seconds
fn fuzz_meta_line_parser() {
    for _ in 0..10000 {
        let input = generate_random_meta_line(&mut rng);
        // Parser should never panic
        if is_meta_line(&input) {
            let _ = MetaLine::parse(&input, 0);
        }
    }
}
```

### 5. Honey Badger Tests
**Location**: `src/parsers/wasot/honey_badger_tests.rs`
**Purpose**: Extreme edge cases, "break everything" testing
**Named after**: Honey badger's fearless, aggressive nature

**Test Categories**:
- **Extreme nesting**: 1000+ dot levels
- **Unicode edge cases**: Zero-width spaces, RTL text, emoji
- **Control characters**: Null bytes, escape sequences
- **Memory stress**: 10MB strings, massive inputs
- **Malformed input**: Unclosed quotes, invalid syntax

### 6. Performance Tests
**Location**: Various (`emitters/`, `selectors/`, etc.)
**Purpose**: Benchmark performance and catch regressions
**Examples**:
- [Random event generation](./random-generator.md) throughput
- [Selector](./selectors.md) matching performance
- Large document processing
- [Memory usage](./memory_analysis.md) validation

---

## 📄 Auto-Generated Tests

One of Wasot's unique features is comprehensive auto-generated syntax testing.

### Architecture

```
tests/wasot_syntax_tests.md    ──────┐
    │ (Human-readable spec)          │
    │                                ▼
    │                    scripts/generate_syntax_tests.py
    │                                │
    │                                ▼
    └─────────────────────▶ tests/generated_syntax_tests.rs
                                     │ (Rust test code)
                                     ▼
                               cargo test generated_syntax_tests
```

### Syntax Test Specification Format

**File**: `tests/wasot_syntax_tests.md`

Each test follows this pattern:
````markdown
### Test Name: descriptive_test_name

```wasot
.document
Content here
```

```yaml
- block-started:
    type: "document"
    level: 1
- text-finished:
    lines: ["Content here"]
- block-finished
```
````

### Generation Process

1. **Parse Markdown**: Extract test cases from markdown
2. **Validate YAML**: Ensure event expectations are valid
3. **Generate Rust**: Create `#[test]` functions
4. **Compile & Run**: Standard Rust testing workflow

### Benefits

- **Human-readable**: Tests documented in natural language
- **Comprehensive**: Covers all syntax combinations
- **Maintainable**: Easy to add new test cases
- **Reliable**: Catches syntax regressions automatically

### Adding New Syntax Tests

1. Edit `tests/wasot_syntax_tests.md`:
````markdown
### Test Name: my_new_feature

```wasot
.my-block .. param=value
Test content
```

```yaml
- block-started:
    type: "my-block"
    level: 1
    kwargs:
      param: "value"
- text-finished:
    lines: ["Test content"]
- block-finished
```
````

2. Generate and run:
```bash
./scripts/test_syntax.sh
```

---

## 🏃 Running Tests

### Development Workflow

```bash
# Quick development cycle
cargo test                    # Fast tests only (~7s)

# Before committing
./scripts/dev/pre-commit.sh   # Includes all fast tests + linting

# Full validation
./scripts/test_all.sh         # Everything including slow tests
```

### Specific Test Categories

```bash
# Auto-generated syntax tests
cargo test generated_syntax_tests

# Fuzzing and stress tests  
cargo test fuzz

# Extreme edge cases
cargo test honey_badger

# Performance benchmarks
cargo test performance

# Integration tests only
cargo test --test roundtrip_tests
```

### Test Filtering

```bash
# Run tests by pattern
cargo test meta_line          # All tests with "meta_line" in name
cargo test -- --ignored      # Only slow tests
cargo test -- --include-ignored # All tests (fast + slow)

# Specific modules
cargo test parsers::wasot::parser::tests
cargo test consumers::sqlite_consumer::tests
```

---

## 📜 Test Scripts

### Core Scripts

| Script | Purpose | Duration | Usage |
|--------|---------|----------|-------|
| `./scripts/test_fast.sh` | Fast tests only | ~7s | Development |
| `./scripts/test_all.sh` | Comprehensive suite | ~5-10 min | Full validation |
| `./scripts/test_syntax.sh` | Generate + run syntax tests | ~1-2s | Syntax development |
| `./scripts/dev/pre-commit.sh` | Pre-commit validation | ~10s | Git hooks |

### Test Generation & Analysis

| Script | Purpose | Usage |
|--------|---------|-------|
| `scripts/generate_syntax_tests.py` | Generate Rust from markdown | Called by test_syntax.sh |
| `scripts/measure_test_performance_fast.py` | Analyze test performance | Debug slow tests |
| `scripts/ci/test_coverage.sh` | Code coverage analysis | CI/CD pipelines |
| `scripts/ci/mutation_test.sh` | Mutation testing | Quality assurance |

### Benchmark Scripts

| Script | Purpose |
|--------|---------|
| `scripts/benchmarks/test_throughput.sh` | Event processing throughput |
| `scripts/benchmarks/test_file_write_performance.sh` | I/O performance |
| `scripts/benchmarks/test_flush_impact.sh` | Buffering impact |

---

## 📊 Performance Analysis

### Test Performance Tool

**Script**: `scripts/measure_test_performance_fast.py`
**Purpose**: Identify slow tests for optimization

```bash
# Analyze test performance
python3 scripts/measure_test_performance_fast.py

# Output shows:
# - Individual test execution times
# - Tests over 200ms threshold
# - Files that need slow test marking
```

### Performance Tiers

| Tier | Duration | Action | Examples |
|------|----------|--------|----------|
| **Fast** | <200ms | Keep as-is | Most unit tests |
| **Slow** | 200ms-1s | Mark `#[ignore = "slow"]` | Honey badger tests |
| **Very Slow** | >1s | Mark + optimize | Extreme fuzz tests |

### Optimization Strategies

1. **Mark appropriately**: Use `#[ignore = "slow"]` for >200ms tests
2. **Reduce iterations**: Lower loop counts in fuzz tests
3. **Mock heavy operations**: Use test doubles for I/O
4. **Parallelize**: Independent tests can run concurrently

---

## 📁 Test Organization

### Source-Embedded Tests

```
src/
├── lib.rs                          # Integration point tests
├── base36.rs                       # base36::{tests}
├── emitters/
│   ├── fast_random_emitter.rs     # Performance tests
│   └── selector_emitter.rs        # Unit tests
├── consumers/
│   ├── sqlite_consumer.rs         # Database integration tests
│   └── xml_consumer.rs            # XML output tests
├── parsers/
│   ├── wasot/
│   │   ├── parser.rs              # Core parsing tests
│   │   ├── fuzz_tests.rs          # Fuzzing (marked slow)
│   │   └── honey_badger_tests.rs  # Edge cases (marked slow)
│   └── selectors/
│       ├── parser.rs              # Selector parsing tests
│       └── integration_tests.rs   # Selector matching tests
└── transformers/
    └── pipeline.rs                # Pipeline tests
```

### Standalone Test Files

```
tests/
├── generated_syntax_tests.rs      # Auto-generated (263 tests)
├── roundtrip_tests.rs             # Parse → Convert → Parse
├── wasot_syntax_tests.md          # Human-readable test specs
└── integration/
    ├── memory_constraint_tests.sh # Memory limit testing
    └── run_all.sh                 # Integration test runner
```

### Test Support Scripts

```
scripts/
├── test_fast.sh                   # Fast test runner
├── test_all.sh                    # Comprehensive test runner  
├── test_syntax.sh                 # Syntax test generator
├── generate_syntax_tests.py       # Markdown → Rust generator
├── measure_test_performance_fast.py # Performance analyzer
├── dev/pre-commit.sh              # Pre-commit validation
├── ci/
│   ├── test_all.sh               # CI test runner
│   ├── test_coverage.sh          # Coverage analysis
│   └── mutation_test.sh          # Mutation testing
├── benchmarks/                    # Performance benchmarks
└── testing/                       # Test utilities
```

---

## ➕ Adding New Tests

### 1. Unit Tests
Add directly to source files:

```rust
// src/my_module.rs
pub fn my_function(input: &str) -> String {
    // implementation
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_my_function() {
        assert_eq!(my_function("test"), "expected");
    }
    
    #[test]
    #[ignore = "slow"] // If test takes >200ms
    fn test_my_function_large_input() {
        // Slow test implementation
    }
}
```

### 2. Integration Tests
Create new file in `tests/`:

```rust
// tests/my_integration_test.rs
use wasot_compiler::*;

#[test]
fn test_end_to_end_workflow() {
    // Test complete workflows
}
```

### 3. Syntax Tests
Edit the markdown specification:

```bash
# Edit specification
vim tests/wasot_syntax_tests.md

# Add new test case following the format
# Generate and run
./scripts/test_syntax.sh
```

### 4. Performance Tests
For performance-sensitive code:

```rust
#[test]
#[ignore = "slow"] // Mark performance tests as slow
fn test_performance_my_feature() {
    use std::time::Instant;
    
    let start = Instant::now();
    // Performance test
    let duration = start.elapsed();
    
    assert!(duration.as_millis() < 100, "Too slow: {:?}", duration);
}
```

---

## 🐛 Troubleshooting

### Common Issues

#### Test Generation Fails
```bash
# Check Python dependencies
python3 -c "import yaml"

# Install if missing
pip3 install pyyaml

# Check file permissions
chmod +x scripts/generate_syntax_tests.py
```

#### Slow Tests Run in Fast Mode
```bash
# Check for unmarked slow tests
python3 scripts/measure_test_performance_fast.py

# Mark slow tests appropriately
#[ignore = "slow"] // Slow test: takes XXXms
```

#### Tests Pass Individually but Fail in Suite
```bash
# Resource conflicts - check for:
# - Shared global state
# - File system conflicts
# - Network ports
# - Environment variables
```

### Performance Issues

#### Test Suite Too Slow
1. **Identify slow tests**:
   ```bash
   python3 scripts/measure_test_performance_fast.py
   ```

2. **Mark slow tests**:
   ```rust
   #[ignore = "slow"] // Slow test: takes XXXms
   ```

3. **Use fast tests for development**:
   ```bash
   cargo test  # Excludes slow tests
   ```

#### Individual Test Too Slow
1. **Reduce iterations**: Lower loop counts
2. **Mock dependencies**: Avoid real I/O
3. **Optimize algorithms**: Profile bottlenecks
4. **Mark as slow**: If optimization isn't feasible

### Debug Test Failures

#### Syntax Test Failures
```bash
# See actual vs expected events
./scripts/test_syntax.sh --verbose

# Debug specific input
cargo run -- listen-demo test.wasot

# Update expectations if parser is correct
vim tests/wasot_syntax_tests.md
```

#### Fuzz Test Failures
```bash
# Reduce iterations for debugging
# Look for panic messages
# Check for memory issues
# Validate input constraints
```

---

## 📈 Best Practices

### Test Design

1. **Fast by Default**: Aim for <200ms per test
2. **Clear Names**: Descriptive test function names
3. **Single Responsibility**: One concept per test
4. **Independent**: Tests shouldn't depend on each other
5. **Deterministic**: Same input → same output

### Performance

1. **Mark Slow Tests**: Use `#[ignore = "slow"]` for >200ms
2. **Optimize Hot Paths**: Profile and optimize bottlenecks
3. **Mock Heavy Operations**: Avoid real I/O in unit tests
4. **Batch Operations**: Group related tests

### Maintenance

1. **Update Documentation**: Keep this guide current
2. **Review Regularly**: Check test performance monthly
3. **Clean Up**: Remove obsolete tests
4. **Automate**: Use scripts for repetitive tasks

---

## 🎯 Testing Philosophy

Wasot's testing philosophy emphasizes:

- **Comprehensive Coverage**: Every syntax feature tested
- **Fast Feedback**: Quick development cycles
- **Real-World Edge Cases**: Honey badger testing
- **Performance Awareness**: Monitor test execution times
- **Documentation as Tests**: Human-readable specifications
- **Automation**: Generate tests from specifications

This infrastructure ensures Wasot remains reliable, performant, and maintainable as it evolves.

---

**Happy Testing!** 🧪✨

*For questions or improvements to this guide, please contribute via GitHub issues or pull requests.*