# Wasot Compiler Scripts

This directory contains various utility scripts for development, testing, and CI/CD, organized into logical subdirectories:

## Directory Structure

- **benchmarks/** - Performance testing and benchmarking scripts
  - Memory usage analysis, throughput testing, base36 benchmarks
  - File write performance tests, flush impact measurements
  
- **build/** - Build and release scripts  
  - Release builds, VSCode extension building, diagram compilation
  
- **ci/** - Continuous integration and quality assurance tools
  - Comprehensive linting, fuzzing, mutation testing
  - Test coverage, automated fixes
  
- **demo/** - Demonstration scripts showcasing features
  - SQLite roundtrip, selector demos, auto-ID generation
  - Python integration examples, CSS selector demonstrations
  
- **dev/** - Development utilities and tools
  - Pre-commit hooks, validation tools, playground scripts
  - Git hook setup, XML validation
  
- **generators/** - Data generation scripts for testing
  - Large file generation, extreme scale testing (trillion events)
  
- **testing/** - Testing scripts and coverage tools
  - Unit tests, documentation tests, selector tests
  - Code coverage analysis

## Pre-Commit Workflow 🚀

### Quick Start

```bash
# Set up git hooks (one-time setup)
./scripts/dev/setup-hooks.sh

# Manual pre-commit check
./scripts/dev/pre-commit.sh

# Fix all issues automatically
./scripts/ci/fix_all.sh
```

### Scripts Overview

| Script | Purpose | Speed | Use Case |
|--------|---------|-------|----------|
| `dev/pre-commit.sh` | ⚡ Fast quality checks | ~8s | Before every commit |
| `dev/setup-hooks.sh` | 🔧 Install git hooks | ~2s | One-time setup |
| `testing/test.sh` | 🧪 Basic functionality test | ~3s | Quick validation |
| `testing/doc-test.sh` | 📚 Run documentation tests | ~4s | Verify examples in docs |
| `demo/sqlite-roundtrip.sh` | 🔄 SQLite roundtrip demo | ~3s | SQLite conversion testing |
| `benchmarks/test_throughput.sh` | 📊 Performance benchmark | ~10s | Throughput testing |
| `benchmarks/measure_memory.sh` | 💾 Memory usage test | ~5s | Memory profiling |
| `ci/fix_all.sh` | 🔧 Auto-fix all issues | ~30s | Weekly cleanup |
| `ci/linter.sh` | 🔍 Comprehensive analysis | ~60s | Deep code review |

## Pre-Commit Script Features

### ⚡ Fast Checks (< 10 seconds)
- **Auto-formatting**: `cargo fmt` 
- **Safe clippy fixes**: Auto-applies non-breaking fixes
- **Compilation check**: Ensures code compiles
- **Unit tests**: Fast test execution
- **Basic quality checks**: TODO/debug print detection

### 🛡️ Quality Gates
- Blocks commits with compilation errors
- Prevents commits with failing tests
- Warns about code quality issues
- Suggests fixes for common problems

### 🔧 Auto-Fixes Applied
- Code formatting (`rustfmt`)
- Safe clippy suggestions
- Compiler suggestions (unused imports, etc.)
- Basic code improvements

## Usage Patterns

### Daily Development
```bash
# Work on code...
git add .
git commit -m "Your message"  # Pre-commit hook runs automatically
```

### Manual Quality Check
```bash
./scripts/dev/pre-commit.sh
```

### Weekly Maintenance
```bash
./scripts/ci/fix_all.sh        # Fix all issues
./scripts/ci/linter.sh         # Full analysis
```

### CI/CD Pipeline
```bash
./scripts/ci/test_all.sh       # Comprehensive testing
./scripts/ci/linter.sh         # Full quality check
```

## Git Hook Setup

The `dev/setup-hooks.sh` script installs a pre-commit hook that:
1. Runs quality checks before each commit
2. Auto-fixes formatting and safe issues
3. Blocks commits if critical issues are found
4. Shows clear feedback on what needs fixing

### Bypass Hook (Not Recommended)
```bash
git commit --no-verify
```

## Script Dependencies

### Required (Built-in)
- `cargo` (Rust toolchain)
- `rustfmt` (code formatting)
- `clippy` (linting)

### Optional (Auto-installed)
- `cargo-audit` (security checking)
- `cargo-machete` (unused dependencies)
- `cargo-outdated` (dependency updates)

## Performance Tips

### For Large Projects
```bash
# Use streaming mode for AST export
cargo run -- export-ast large_file.wasot -o output.jsonl --streaming

# Skip expensive checks in development
SKIP_AUDIT=1 ./scripts/dev/pre-commit.sh
```

### For Faster Feedback
```bash
# Check only staged files
git diff --cached --name-only | xargs cargo clippy --

# Quick format check
cargo fmt -- --check
```

## Customization

### Environment Variables
- `SKIP_AUDIT=1` - Skip security audit
- `SKIP_TESTS=1` - Skip test execution
- `CLIPPY_ARGS` - Additional clippy arguments

### Script Modification
All scripts are designed to be:
- ✅ Fast (< 30s for pre-commit)
- ✅ Reliable (fail-safe defaults)
- ✅ Informative (clear error messages)
- ✅ Extensible (easy to modify)

## Troubleshooting

### Common Issues

**"Script must be run from project root"**
```bash
cd /path/to/wasot/rust
./scripts/dev/pre-commit.sh
```

**"Clippy errors won't auto-fix"**
```bash
# Some issues require manual intervention
./scripts/ci/linter.sh  # See detailed errors
```

**"Tests are slow"**
```bash
# Use quiet mode
cargo test --quiet
```

**"Hook is too strict"**
```bash
# Temporarily disable
git commit --no-verify

# Or adjust script settings
vim scripts/dev/pre-commit.sh
```

## Integration Examples

### VSCode Settings
```json
{
  "rust-analyzer.check.command": "clippy",
  "editor.formatOnSave": true,
  "[rust]": {
    "editor.defaultFormatter": "rust-lang.rust-analyzer"
  }
}
```

### GitHub Actions
```yaml
name: Quality Check
on: [push, pull_request]
jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
      - run: ./scripts/ci/linter.sh
      - run: ./scripts/ci/test_all.sh
```

## SQLite Roundtrip Demo

The `demo/sqlite-roundtrip.sh` script demonstrates the SQLite export/import functionality by:
1. Converting a Wasot file to SQLite database format
2. Reading the SQLite database and converting back to Wasot
3. Outputting the reconstructed Wasot document to stdout

### Usage Examples
```bash
# Basic usage
./scripts/demo/sqlite-roundtrip.sh examples/demo.wasot

# Test with different files
./scripts/demo/sqlite-roundtrip.sh examples/simple.wasot
./scripts/demo/sqlite-roundtrip.sh examples/test_args.wasot

# Pipe output to file
./scripts/demo/sqlite-roundtrip.sh examples/demo.wasot > reconstructed.wasot

# Compare with original
./scripts/demo/sqlite-roundtrip.sh examples/demo.wasot | diff examples/demo.wasot -
```

### Features
- ✅ **Full roundtrip**: Wasot → SQLite → Wasot conversion
- ✅ **Visual feedback**: Colored output showing each step
- ✅ **Database info**: Shows temporary database size
- ✅ **Error handling**: Proper validation and cleanup
- ✅ **Temporary files**: Automatic cleanup of temp databases

## Auto-ID Generation Demo

The `demo/demo_auto_id.sh` script demonstrates how the Wasot parser automatically assigns unique IDs to all blocks:
1. Shows the original Wasot file content
2. Parses the file and reconstructs it with auto-generated IDs
3. Shows the side-by-side comparison in clean Wasot format
4. Explains the ID assignment strategy

### Usage Examples
```bash
# Run the complete demo
./scripts/demo/demo_auto_id.sh

# See the example file used in demo
cat examples/auto_id_demo.wasot
```

### Features
- ✅ **Two-pass scanning**: First pass finds existing IDs, second pass assigns new ones
- ✅ **Base36 encoding**: Compact, human-readable ID format
- ✅ **Manual ID preservation**: Existing IDs like `:manual-id` are kept
- ✅ **Sequential assignment**: Auto-generated IDs continue from max existing ID + 1
- ✅ **Visual demonstration**: Clean Wasot output shows IDs added to each block

## Performance Testing

### Throughput Testing

The `benchmarks/test_throughput.sh` script measures the write performance of Wasot file generation:

```bash
# Run throughput test (uses baseline for comparison)
./scripts/benchmarks/test_throughput.sh

# Reset baseline after performance improvements
./scripts/benchmarks/test_throughput.sh --reset-baseline
```

**Features:**
- 📊 Establishes performance baseline on first run
- ⚖️ Compares current performance against baseline
- 🚨 Fails CI if performance drops below 80% of baseline
- 📈 Provides time estimates for large file generation

### Memory Usage Testing

The `benchmarks/measure_memory.sh` script profiles memory consumption:

```bash
# Test with default 400K events
./scripts/benchmarks/measure_memory.sh

# Test with custom event count
./scripts/benchmarks/measure_memory.sh 1000000

# Test with memory limit (prevents runaway memory)
./scripts/benchmarks/measure_memory.sh 1000000 2  # 2GB limit
```

## Documentation Tests

The `testing/doc-test.sh` script runs all documentation tests to ensure code examples in docs are correct and up-to-date.

### What Doc-Tests Verify
- **Code examples compile**: All ````rust` blocks in documentation compile successfully
- **Examples execute correctly**: Runnable examples produce expected results  
- **API usage is current**: Documentation stays in sync with actual API changes
- **Real-world patterns**: Examples demonstrate realistic usage scenarios

### Usage Examples
```bash
# Run all documentation tests
./scripts/testing/doc-test.sh

# Quick check (same as above but explicit)
cargo test --doc

# Include doc-tests in CI pipeline
./scripts/testing/doc-test.sh && echo "Documentation is reliable!"
```

### Features
- ✅ **27 doc-tests**: Comprehensive coverage of public APIs
- ✅ **Fast execution**: Completes in ~4 seconds
- ✅ **Clear feedback**: Shows which examples pass/fail
- ✅ **CI-friendly**: Exit codes for automated testing
- ✅ **Helpful guidance**: Tips for adding new doc-tests

### Adding New Doc-Tests
```rust
/// Your function description
///
/// # Examples
/// ```
/// use wasot_compiler::your_function;
/// 
/// let result = your_function("input");
/// assert_eq!(result, "expected");
/// ```
///
/// For examples that shouldn't run:
/// ```no_run
/// use wasot_compiler::file_function;
/// file_function("/some/file.wasot").unwrap();
/// ```
pub fn your_function(input: &str) -> String { ... }
```

## Protobuf Roundtrip Demo

The `wasot-via-protobuf.sh` script (if available) demonstrates the protobuf export/import functionality by:
1. Converting a Wasot file to binary protobuf format
2. Reading the binary protobuf and converting to XML
3. Outputting clean XML to stdout (perfect for piping)

### Usage Examples
```bash
# Basic usage
./scripts/wasot-via-protobuf.sh examples/demo.wasot

# Pretty-printed XML
./scripts/wasot-via-protobuf.sh --pretty examples/demo.wasot

# Verbose output (shows all steps)
./scripts/wasot-via-protobuf.sh --verbose --pretty examples/demo.wasot

# Pipe to file
./scripts/wasot-via-protobuf.sh --pretty examples/demo.wasot > output.xml

# Pipe to other commands
./scripts/wasot-via-protobuf.sh examples/demo.wasot | xmllint --format -
```

### Features
- ✅ **Clean output**: Status messages to stderr, XML to stdout
- ✅ **Efficient**: Uses binary protobuf format internally
- ✅ **Robust**: Proper error handling and cleanup
- ✅ **Flexible**: Pretty-print and verbose modes
- ✅ **Pipe-friendly**: Designed for Unix-style command chaining

## Extreme Scale Testing

### Trillion Event Generation

The `generators/generate_trillion_events.sh` script stress-tests the bounded channel memory fix with ultimate scale:

```bash
# Generate 1 trillion events (WARNING: ~100GB file, hours to complete)
./scripts/generators/generate_trillion_events.sh [output_file.wasot]

# Example with custom output
./scripts/generators/generate_trillion_events.sh massive_test.wasot
```

#### Features
- ✅ **Memory monitoring**: Verifies constant ~60MB usage throughout generation
- ✅ **Safety checks**: Validates disk space and sets memory limits  
- ✅ **Progress tracking**: Real-time memory usage display
- ✅ **Graceful interruption**: Clean stop with Ctrl+C
- ✅ **System protection**: ulimit prevents memory runaway

#### Purpose
Created to validate the bounded channel fix (commit 0cf597f) can handle production-scale workloads:
- **Before fix**: Memory would grow to terabytes, crashing system
- **After fix**: Maintains constant ~60MB regardless of scale
- **Validation**: Proves backpressure works even for trillion-event files

⚠️ **WARNING**: This generates ~100GB files and takes hours to complete. Only use for extreme stress testing.

### Memory Benchmarking

```bash
# Quick memory test
./scripts/benchmarks/measure_memory.sh 1000000  # 1M events

# Stress test with memory limit
./scripts/benchmarks/measure_memory.sh 100000000 1  # 100M events, 1GB limit

# Integration tests
./tests/integration/run_all.sh  # Memory constraint validation
```

## Best Practices

1. **Run pre-commit manually** when working on large changes
2. **Use fix_all.sh weekly** to maintain code quality
3. **Review auto-fixes** before committing
4. **Keep hooks fast** (< 30 seconds)
5. **Document script changes** in this README
6. **Use trillion script sparingly** - only for ultimate stress testing

---

💡 **Pro Tip**: The pre-commit script is optimized for speed while maintaining quality. For deeper analysis, use the comprehensive linter script weekly.