# Wasot Developer Guide

**Wasot** is an event-driven markup language that bridges human readability and machine processability. Think Markdown meets XML with streaming capabilities.

## Essential Commands

```bash
# IMPORTANT: Always run before finishing work
./scripts/dev/pre-commit.sh

# Build and test
cargo build --release
cargo test                          # Fast tests only (~7 seconds)
cargo test -- --ignored            # Slow tests only (fuzz/performance tests)
cargo test -- --include-ignored    # All tests including slow ones (~65+ seconds)
./scripts/test_fast.sh              # Quick test runner (fast tests only)
./scripts/verify.sh                 # Full verification (build + all tests)

# Convert formats
wasot xml-listener file.wasot --pretty          # Wasot → XML
wasot to-sqlite file.wasot -d db.sqlite        # Wasot → SQLite
wasot to-jsonl file.wasot -o out.jsonl         # Wasot → JSON Lines
wasot select file.wasot -s "paragraph"          # CSS-like selectors

# Fix issues automatically
./scripts/ci/fix_all.sh
```

## Core Files & Structure

```
src/
├── main.rs              # CLI entry point
├── emitters/            # Event producers (parse Wasot, SQLite, etc.)
│   └── parser/          # Core Wasot parser
├── consumers/           # Event consumers (XML, SQLite, JSONL output)
├── selectors/           # CSS-like selector implementation
├── events/              # Event types and processing
└── facade.rs            # Simple API for common operations

scripts/
├── dev/pre-commit.sh    # MUST run before commits
├── verify.sh            # Full verification suite
└── ci/                  # CI/CD tools

examples/                # Sample Wasot files
docs/                    # Architecture & guides
```

## Wasot Syntax Quick Reference

```wasot
.paragraph                      # Block (one dot)
..nested                        # Nested block (more dots = deeper)
.# Heading                      # Self-closing heading
./                              # Explicit block close

.block arg1 "arg 2" key=value   # Arguments
.block :id123                   # Block with ID
.block .. Inline text here      # Inline text after ..
```

## Architecture: Event-Driven Streaming

```
Emitter → Events → Consumer
│                    │
├─ WasotParser       ├─ XmlListener
├─ SqliteEmitter     ├─ SqliteConsumer
└─ JsonlEmitter      └─ SelectorConsumer
```

**Key Events:**
- `BlockStarted { meta_line }` - New block begins
- `TextFinished { lines }`     - Text content ready
- `BlockFinished`              - Block ends

**Memory-efficient:** Processes line-by-line with bounded channels, handles files larger than RAM.

## Code Style

- **ALWAYS run `./scripts/dev/pre-commit.sh` before finishing work**
- No `unwrap()` in production code - use `Result<T, E>`
- Follow clippy pedantic warnings
- Write tests first (TDD)

## Common Workflows

```bash
# Add new feature
1. Write tests
2. Implement
3. cargo test
4. ./scripts/dev/pre-commit.sh

# Debug parser
cargo run -- listen-demo file.wasot  # See all events

# Test conversions
cargo run -- xml-listener examples/demo.wasot --pretty
./scripts/demo/sqlite-roundtrip.sh

# Performance test
./scripts/generators/generate_large_file.py
cargo run -- to-jsonl large.wasot | wc -l
```

## Key Architecture Points

- **Streaming:** Line-by-line processing, constant memory usage
- **Event-driven:** Emitters produce, consumers process
- **Fault-tolerant:** Parser handles errors gracefully
- **Async SQLite:** Uses SQLx with transactions
- **Bounded channels:** Prevents memory growth
- **Mixed-language tools:** Tools can be implemented in Python or Rust seamlessly

## Tool System Architecture

The tool system allows processing blocks with custom logic implemented in either Python or Rust:

### Tool Registry
- **Global Registry:** Single `PyToolRegistry` instance stored in Rust (`src/python.rs`)
- **Thread-safe:** Uses `OnceLock<Arc<Mutex<PyToolRegistry>>>` for concurrent access
- **Dual access:** Accessible from both Rust and Python code directly
- **No marshalling:** Direct Rust-to-Rust calls eliminate Python roundtrips

### Tool Implementation Patterns
```python
# Python tools - register with Rust registry directly
register_tool("page_tool", ["page"], page_tool_callback)

# Rust tools - register via centralized function  
register_example_tools()  # Loads all Rust tools automatically
```

### Key Functions (src/python.rs)
- `get_global_tool_registry()` - Internal Rust access to registry
- `get_tool_registry()` - Expose registry to Python  
- `register_tool()` - Rust implementation of tool registration
- `register_example_tools()` - Register all Rust example tools
- `rust_document_name_tool()` - Example Rust tool implementation

## Adding New Features

### New Tool (Python)
1. Define tool function that accepts `event` parameter
2. Register with `register_tool("name", ["selector"], callback)`
3. Tool will be called automatically when selector matches

### New Tool (Rust)
1. Add tool implementation to `src/tools/example_tools.rs`
2. Create Python-callable wrapper in `src/python.rs`
3. Register in `register_example_tools()` function
4. Export wrapper function in `pywasot()` module function

### New Emitter
1. Implement `Emitter` trait in `src/emitters/`
2. Provide `get_receiver()` method
3. Emit `ElementEvent` types

### New Consumer
1. Implement `Consumer` trait in `src/consumers/`
2. Handle `BlockStarted`, `TextFinished`, `BlockFinished`
3. Add CLI subcommand in `main.rs`

### New Selector
1. Update grammar in `src/selectors/selectors.pest`
2. Add AST node in `src/selectors/ast.rs`
3. Update parser in `src/selectors/parser.rs`
4. Add matching logic

## Python Integration

```bash
maturin develop --features python
python python/demos/demo1_simple_parsing.py
```

## Performance Notes

- Handles 100M+ events with ~60MB RAM
- Optimized base36 encoding for IDs
- Bounded channels prevent memory growth
- See `docs/memory_analysis.md` for details

## IMPORTANT Reminders

- **YOU MUST run `./scripts/dev/pre-commit.sh` before finishing any work**
- **NEVER use `unwrap()` in production code**
- **ALWAYS write tests for new features**
- **Update README.md for user-facing changes**