# Wasot Contributor Guide

A practical guide for developing wasot. This covers the development workflow, testing infrastructure, and scripts you'll actually use.

## Quick Reference

```bash
# Daily development workflow
./scripts/dev/pre-commit.sh     # Run BEFORE finishing work (~8 seconds)
./scripts/verify.sh --fast      # Full verification minus slow tests

# Before pushing / merging
./scripts/verify.sh             # Complete verification (~65 seconds)

# When things break
./scripts/ci/fix_all.sh         # Auto-fix formatting, clippy, etc.
./scripts/ci/linter.sh          # Deep analysis (14 checks)
```

## Test Architecture Overview

The testing system has multiple layers, each serving a specific purpose:

```
┌─────────────────────────────────────────────────────────────────┐
│                     verify.sh (master script)                    │
├─────────────────────────────────────────────────────────────────┤
│ Step 1: cargo build --release                                   │
│ Step 2: cargo test (fast unit tests, ~7 seconds)                │
│ Step 3: cargo test -- --ignored (slow tests, ~60 seconds)       │
│ Step 4: CLI integration tests (tests/cli_tests.rs)              │
│ Step 5: Generated syntax tests (from markdown spec)             │
│ Step 6: Roundtrip tests on examples (binary/jsonl/xml)          │
│ Step 7: Roundtrip tests on syntax test cases                    │
└─────────────────────────────────────────────────────────────────┘
```

## Test Categories

### 1. Unit Tests (In-Source)

Location: `src/**/*.rs` files with `#[cfg(test)]` modules

```bash
cargo test                      # Fast tests only (~7 seconds)
cargo test -- --ignored         # Slow tests only (~60 seconds)
cargo test -- --include-ignored # Everything
```

**Slow test marker**: Tests taking >200ms should be marked:
```rust
#[test]
#[ignore = "slow"] // Slow test: takes Xms. Run with --ignored to include.
fn expensive_test() {
    // ...
}
```

Slow tests live in:
- `src/parsers/wasot/fuzz_tests.rs` - Fuzz testing (10k+ iterations)
- `src/parsers/wasot/honey_badger_tests.rs` - Edge case resilience
- `src/parsers/selectors/integration_tests.rs` - Selector performance
- `src/emitters/fast_random_emitter.rs` - Performance comparison

### 2. CLI Integration Tests

Location: `tests/cli_tests.rs`

These test the compiled binary as a user would invoke it:

```rust
use assert_cmd::Command;
use predicates::prelude::*;

#[test]
fn normalize_file_to_stdout() {
    let dir = setup_test_dir();
    wasot()
        .arg(dir.path().join("valid.wasot"))
        .assert()
        .success()
        .stdout(predicate::str::contains(".document"));
}
```

Run with:
```bash
cargo test --test cli_tests
# Or against a specific binary:
WASOT_TEST_BINARY=/path/to/wasot cargo test --test cli_tests
```

### 3. Generated Syntax Tests

The parser specification lives in markdown, and tests are auto-generated.

**Workflow**:
```
tests/wasot_syntax_tests.md    # Human-readable spec
        ↓
scripts/generate_syntax_tests.py
        ↓
tests/generated_syntax_tests.rs # Auto-generated (DO NOT EDIT)
        ↓
cargo test --test generated_syntax_tests
```

**Adding a new syntax test**:

1. Edit `tests/wasot_syntax_tests.md`:
```markdown
---

**My new test**

Description of what this tests.

```wasot
.myblock
Some content
```

```yaml
- block-started:
    type: myblock
    level: 1
- text-finished:
    lines: ["Some content"]
- block-finished
```

---
```

2. Run the tests (regenerates automatically):
```bash
./scripts/test_syntax.sh
```

### 4. Roundtrip Tests

Test format conversion fidelity: wasot → format → wasot

**Example file roundtrips** (`scripts/roundtrip_test.py`):
```bash
python3 scripts/roundtrip_test.py -e ./target/release/wasot
```
Tests all files in `examples/` through binary, jsonl, and xml formats.

**Syntax test roundtrips** (`tests/roundtrip_syntax_tests.py`):
```bash
python3 tests/roundtrip_syntax_tests.py -e ./target/release/wasot
```
Tests every syntax test case through format roundtrips.

## Script Reference

### Scripts You'll Use Daily

| Script | Time | Purpose |
|--------|------|---------|
| `./scripts/dev/pre-commit.sh` | ~8s | Quality gate before commits |
| `./scripts/verify.sh --fast` | ~20s | Full verification sans slow tests |
| `./scripts/verify.sh` | ~65s | Complete verification |

### Scripts for Fixing Problems

| Script | Purpose |
|--------|---------|
| `./scripts/ci/fix_all.sh` | Auto-fix formatting, clippy, compiler suggestions |
| `./scripts/ci/linter.sh` | Deep analysis with 14 quality checks |
| `cargo fmt --all` | Format code |
| `cargo clippy --fix --allow-dirty` | Auto-fix clippy warnings |

### Specialized Scripts

| Script | Purpose |
|--------|---------|
| `./scripts/test_syntax.sh` | Regenerate + run syntax tests |
| `./scripts/test_cli.sh` | CLI tests only |
| `./scripts/ci/fuzzer.sh 30` | Fuzz for 30 seconds |
| `./scripts/ci/test_coverage.sh` | Generate coverage report |

### Scripts Not Called by verify.sh

These exist but aren't part of the main workflow:

- `scripts/testing/test.sh` - Legacy simple test runner (called by pre-commit.sh internally)
- `scripts/test_rust.sh` - Minimal wrapper around `cargo test`
- `scripts/test_slow.sh` - Just `cargo test -- --ignored`
- `scripts/build_all.sh` - Build all targets
- `scripts/coverage.sh` - Coverage wrapper

## Development Workflow

### Starting Work

```bash
git pull
cargo build --release
```

### Making Changes

1. **Write tests first** (if applicable)
2. **Make changes**
3. **Run fast tests frequently**:
   ```bash
   cargo test          # Unit tests
   cargo test my_test  # Specific test
   ```

### Before Finishing

Always run the pre-commit checks:

```bash
./scripts/dev/pre-commit.sh
```

This does 10 things:
1. Runs basic tests
2. Auto-formats code
3. Auto-fixes safe clippy issues
4. Applies compiler suggestions
5. Checks compilation
6. Runs clippy linting
7. Runs unit tests (if compilation passed)
8. Checks for TODO/FIXME in staged files
9. Checks for debug prints
10. Final format verification

### Before Merging

Run full verification:

```bash
./scripts/verify.sh
```

## Adding New Features

### New Parser Feature

1. **Add syntax tests** to `tests/wasot_syntax_tests.md`
2. **Implement** in `src/parsers/wasot/`
3. **Run syntax tests**: `./scripts/test_syntax.sh`
4. **Add CLI tests** if user-facing: `tests/cli_tests.rs`

### New Output Format

1. **Create emitter** in `src/emitters/`
2. **Create consumer** in `src/consumers/`
3. **Add CLI subcommand** in `src/main.rs`
4. **Add CLI tests** in `tests/cli_tests.rs`
5. **Add roundtrip test** in `scripts/roundtrip_test.py`

### New CLI Command

1. **Implement** in `src/main.rs` using clap
2. **Add tests** in `tests/cli_tests.rs`:
   ```rust
   mod my_feature {
       use super::*;

       #[test]
       fn basic_usage() {
           wasot()
               .arg("my-command")
               .arg("input.wasot")
               .assert()
               .success();
       }
   }
   ```

## Feature Flags

Cargo features for conditional compilation:

```toml
[features]
default = ["sqlite", "binary"]
python = ["pyo3"]           # Python bindings
binary = ["prost"]          # Protobuf binary format
sqlite = ["sqlx"]           # SQLite support
html = ["scraper"]          # HTML parsing
optimized-alloc = ["mimalloc"]  # Faster allocator
```

Build with specific features:
```bash
cargo build --features "sqlite,binary"
cargo test --features "sqlite"
```

## Python Integration

Build Python bindings:
```bash
maturin develop --features python
```

Test Python:
```bash
python3 -c "import pywasot; print(pywasot.__doc__)"
cd python && python3 -m pytest
```

## Benchmarking

Criterion benchmarks in `benches/`:

```bash
cargo bench                     # Run all benchmarks
cargo bench base36              # Specific benchmark
```

## Common Issues

### Tests Fail After Changes

```bash
# See what's actually broken
cargo test 2>&1 | head -50

# Auto-fix what can be fixed
./scripts/ci/fix_all.sh

# Deep analysis
./scripts/ci/linter.sh
```

### Clippy Complaints

```bash
# Auto-fix safe issues
cargo clippy --fix --allow-dirty

# See what's left
cargo clippy -- -D warnings
```

### Format Issues

```bash
cargo fmt --all
```

### Slow Test Timing

If you add a slow test (>200ms), mark it:

```rust
#[test]
#[ignore = "slow"] // Slow test: takes Xms. Run with --ignored to include.
fn my_slow_test() { ... }
```

## Architecture Quick Reference

```
src/
├── main.rs              # CLI entry (clap)
├── lib.rs               # Library exports
├── parsers/
│   ├── wasot/           # Core wasot parser
│   │   ├── parser.rs    # Main parser logic
│   │   ├── tokenizer.rs # Line tokenization
│   │   └── *.rs         # Other parser components
│   └── selectors/       # CSS-like selector parsing
├── emitters/            # Event producers
│   ├── wasot_file_emitter.rs  # Parse wasot files
│   ├── sqlite_emitter.rs      # Read from SQLite
│   └── jsonl_emitter.rs       # Read from JSONL
├── consumers/           # Event processors
│   ├── xml_listener.rs        # Output XML
│   ├── sqlite_consumer.rs     # Write to SQLite
│   └── to_jsonl_consumer.rs   # Output JSONL
└── events/              # Event types + Emitter trait
```

**Event flow**: `Emitter → Events → Consumer`

Events are streamed through channels, enabling constant memory usage on arbitrarily large files.

## Code Style Rules

1. **No `unwrap()` in production code** - Use `Result<T, E>` or `Option<T>`
2. **Follow clippy pedantic** - Run `cargo clippy -- -W clippy::pedantic`
3. **Test first** - Write tests before implementation
4. **Run pre-commit** - Always before finishing work

## Debug Tips

```bash
# See parser events for a file
cargo run -- listen-demo file.wasot

# Test a single component
cargo test parser_name

# Verbose test output
cargo test -- --nocapture

# Run specific test file
cargo test --test cli_tests
```

### Quick Experimentation via stdin

Test wasot syntax instantly without creating files:

```bash
# Basic: pipe wasot to jsonl output
printf '.block\nsome text\n' | cargo run --bin wasot -- -t jsonl

# Pretty-print JSON with jq
printf '.paragraph\nHello world\n' | cargo run --bin wasot -- -t jsonl | jq '.'

# Extract specific fields
printf '.code language=rust\nfn main() {}\n' | cargo run --bin wasot -- -t jsonl | jq '.block_type, .kwargs'

# Test nested blocks
printf '.section\n..paragraph\nNested content\n' | cargo run --bin wasot -- -t jsonl | jq -s '.'

# Output as XML
printf '.heading\nMy Title\n' | cargo run --bin wasot -- -t xml

# Normalize/pretty-print wasot
printf '.block\ntext\n' | cargo run --bin wasot --
```

Use `printf` (not `echo`) for reliable newline handling across shells.
