# Wasot Parser Implementation Report

This document provides a comprehensive analysis of the parsing system in the Wasot codebase. The parser is an event-driven, streaming parser designed for memory efficiency and fault tolerance.

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Core Parser Components](#core-parser-components)
3. [Data Structures](#data-structures)
4. [Tokenization](#tokenization)
5. [Meta-Line Parsing](#meta-line-parsing)
6. [Block Management](#block-management)
7. [Event System](#event-system)
8. [Selector Parser](#selector-parser)
9. [Error Handling Philosophy](#error-handling-philosophy)
10. [Performance Considerations](#performance-considerations)

---

## Architecture Overview

The Wasot parser follows an **event-driven streaming architecture** that separates concerns into three main layers:

```
Input Source → Emitter → Events → Consumer → Output
```

**Key design principles:**
- **Line-by-line processing** for constant memory usage
- **Event-driven** model allowing flexible consumption patterns
- **Fault-tolerant** ("honey badger") parsing that recovers from malformed input
- **Streaming mode** for processing files larger than RAM

### Entry Points

| Entry Point | File | Purpose |
|-------------|------|---------|
| `WasotParser` | `src/parsers/wasot/facade.rs` | High-level API facade |
| `DocumentParser` | `src/parsers/wasot/document_parser.rs:15` | Event emitter & direct parsing |
| `ElementParser` | `src/parsers/wasot/element_parser.rs:6` | State machine for block nesting |
| `MetaLine::parse()` | `src/parsers/wasot/parser.rs:270` | Low-level meta-line parsing |

---

## Core Parser Components

### 1. DocumentParser (`src/parsers/wasot/document_parser.rs`)

The `DocumentParser` serves dual purposes:

**Direct Parsing Mode (lines 91-140):**
```rust
pub fn parse_file<P: AsRef<Path>>(file_path: P) -> io::Result<BlockContainer>
pub fn parse_reader<R: Read>(reader: R) -> io::Result<BlockContainer>
pub fn parse_string(content: &str) -> Result<BlockContainer, String>
```

**Emitter Mode (lines 325-401):**
- Implements the `Emitter` trait for streaming event emission
- Uses a `Bus<Event>` with capacity 1000 for bounded channel communication
- Creates `ElementParser` in streaming mode to avoid accumulating blocks

**Key fields (lines 15-28):**
```rust
pub struct DocumentParser {
    file_path: Option<String>,
    reader: Option<Box<dyn Read + Send + Sync>>,
    bus: Arc<Mutex<Bus<Event>>>,
    accumulated_text_lines: Vec<String>,
    first_text_line_index: Option<usize>,
    auto_convert_links_to_footnotes: bool,
}
```

### 2. ElementParser (`src/parsers/wasot/element_parser.rs`)

The `ElementParser` maintains parsing state via a **stack-based state machine**:

**State fields (lines 6-19):**
```rust
pub struct ElementParser {
    block_stack: Vec<Block>,           // Open blocks (nesting hierarchy)
    container: BlockContainer,          // Completed blocks (non-streaming only)
    current_text_block: Option<Block>,  // Implicit text accumulator
    current_line_index: usize,
    next_auto_id: Option<Arc<Mutex<u64>>>,
    streaming_mode: bool,
}
```

**Processing flow (lines 72-85):**
1. `process_line()` dispatches to either:
   - `process_meta_line()` for lines starting with `.`
   - `process_content_line()` for regular text

**Block level handling (lines 143-195):**
- Blocks at same or deeper level close previous blocks
- Headings close ALL open blocks
- Void elements don't enter the stack

### 3. MetaLine Parser (`src/parsers/wasot/parser.rs`)

The `MetaLine::parse()` function (lines 270-438) is the core syntax parser:

**Meta-line format:**
```
[dots][type_name][inline_text][.. parameters][:id]
```

**Parsing steps:**
1. **ID extraction** (line 282): Regex `\s:[A-Za-z0-9\-_:\p{L}]+\s*$`
2. **Level counting** (line 289): Count leading dots
3. **Type extraction** (lines 293-294): Via `extract_line_type()`
4. **Part splitting** (line 320): Split on `..` for parameters
5. **Argument parsing** (line 379): Via `parse_arguments()`

---

## Data Structures

### MetaLine (`src/parsers/wasot/parser.rs:102-113`)

```rust
pub struct MetaLine {
    pub type_name: String,              // Block type (e.g., "p", "div")
    pub inline_text: QuotedString,      // Text after type, before ..
    pub level: usize,                   // Nesting depth (dot count)
    pub id: Id,                         // Optional :identifier
    pub line_index: Option<usize>,      // Source line number
    pub arguments: ParsedArguments,     // Key-value arguments
    pub is_void: bool,                  // Self-closing (.hr:)
}
```

### Block (`src/parsers/wasot/block.rs:139-150`)

```rust
pub struct Block {
    pub meta_line: MetaLine,
    pub children: Vec<Element>,
    pub parent_index: Option<usize>,
}
```

**Block types (lines 152-192):**
- `Block::new(meta_line)` - Regular block
- `Block::new_implicit_text(line_index)` - Level-0 text block
- `Block::new_heading(meta_line)` - Self-closing heading (type becomes `##`)

### Element (`src/parsers/wasot/block.rs:4-19`)

```rust
pub enum Element {
    Block(Block),
    Text(Text),
    ContextLine(MetaLine),   // ./tag syntax
    VoidElement(MetaLine),   // .hr: syntax
}
```

### Text (`src/parsers/wasot/block.rs:21-33`)

```rust
pub struct Text {
    pub from: usize,              // Start line (inclusive)
    pub to: usize,                // End line (exclusive)
    pub lines: Vec<String>,       // Content
    pub links: Vec<Vec<Link>>,    // Links per line
}
```

### ParsedArguments (`src/parsers/wasot/arguments.rs:4-8`)

```rust
pub struct ParsedArguments {
    pub args: Vec<QuotedString>,                    // Positional args
    pub kwargs: Vec<(QuotedString, QuotedString)>,  // key=value pairs
}
```

### QuotedString (`src/parsers/wasot/parser.rs:1-6`)

```rust
pub struct QuotedString {
    pub content: String,
    pub quoted: bool,
    pub links: Vec<Link>,
}
```

### Id (`src/parsers/wasot/parser.rs:37-81`)

```rust
pub struct Id {
    pub value: String,
}
```

**Notable methods:**
- `Id::from_u64(value)` - Base36 encoding (line 62)
- `Id::as_u64()` - Reverse conversion (line 70)
- `Id::is_numeric()` - Check if valid base36 (line 78)

---

## Tokenization

### Main Tokenizer (`src/parsers/wasot/tokenizer.rs`)

**`tokenize_with_equals(input)` (lines 108-188):**
- Splits on whitespace and `=` characters
- Handles quoted strings with escape sequences (`\"`, `\\`)
- Returns `Vec<String>`

**`tokenize_with_equals_and_quotes(input)` (lines 1-105):**
- Enhanced version returning `Vec<(String, bool, bool)>`
- Tracks: (token, was_quoted_start, was_quoted_value)
- Used for preserving quote information

### Quote-Aware Tokenizer (`src/parsers/wasot/quote_aware_tokenizer.rs`)

**`tokenize_with_quotes(input)` (lines 14-148):**
- Returns `Vec<QuotedToken>` with `content` and `was_quoted` fields
- Handles escape sequences inside quotes (lines 39-52)
- Unclosed quotes treated as unquoted tokens (honey-badger recovery)

---

## Meta-Line Parsing

### `is_meta_line()` (`src/parsers/wasot/parser.rs:457-459`)

```rust
pub fn is_meta_line(line: &str) -> bool {
    !line.is_empty() && line.starts_with('.')
}
```

### `extract_id_string()` (`src/parsers/wasot/parser.rs:188-229`)

**Regex pattern (line 204):**
```rust
Regex::new(r"\s:[A-Za-z0-9\-_:\p{L}]+\s*$")
```

**Features:**
- Cached regex via `OnceLock<Regex>` for performance
- Supports Unicode letters (`\p{L}`)
- Dots NOT allowed in IDs (reserved for document references)

### `extract_line_type()` (`src/parsers/wasot/parser.rs:590-707`)

Extracts type name from a line starting with `.`:

1. Handles quoted types: `."block type"` (lines 603-658)
2. Detects void elements: `.hr:` (lines 636-643)
3. Processes escape sequences in quoted types (lines 621-629)
4. Falls back to space-delimited parsing (lines 684-702)

### `split_into_parts()` (`src/parsers/wasot/parser.rs:500-574`)

Splits text on `..` while respecting quotes:
- Tracks `in_quotes` state
- Handles escape sequences `\"` and `\\`
- Returns `Vec<String>` including empty parts

---

## Block Management

### Block Stack (`src/parsers/wasot/element_parser.rs`)

**Pushing blocks (lines 198-211):**
```rust
fn start_block(&mut self, block: Block) -> Result<(), String> {
    self.block_stack.push(block);
    Ok(())
}
```

**Popping blocks (lines 214-235):**
```rust
fn finish_block(&mut self) -> Result<(), String> {
    if let Some(block) = self.block_stack.pop() {
        if !self.streaming_mode {
            self.container.add_block(block);
        }
    }
    Ok(())
}
```

**Level-based closing (lines 156-163):**
```rust
while let Some(current_block) = self.block_stack.last() {
    if current_block.level() >= block_level {
        self.finish_block()?;
        closed_count += 1;
    } else {
        break;
    }
}
```

### BlockContainer (`src/parsers/wasot/block.rs:314-396`)

Flat storage with parent-child relationships:

```rust
pub struct BlockContainer {
    blocks: Vec<Block>,
    root_indices: Vec<usize>,
}
```

**Key methods:**
- `add_block()` - Adds block, tracks roots (lines 335-345)
- `get_parent()` - Navigate via `parent_index` (lines 369-373)
- `get_children()` - Returns `&[Element]` (lines 377-383)
- `root_blocks()` - Returns blocks without parents (lines 360-365)

---

## Event System

### Events (`src/events/events.rs:54-86`)

```rust
pub enum Event {
    BlockStarted { meta_line: MetaLine },
    BlockFinished,
    TextFinished {
        lines: Vec<String>,
        links: Vec<Vec<Link>>,
        footnotes: Vec<Footnote>,
    },
    ContextLine { meta_line: MetaLine },
    VoidElement { meta_line: MetaLine },
}
```

### Emitter Trait (`src/events/events.rs:3-22`)

```rust
pub trait Emitter: Send + Sync {
    type Reader: Iterator<Item = Event> + Send;
    fn subscribe(&self) -> Self::Reader;
    fn run(self);
    fn run_with_context(self, context: &mut Context);
}
```

### Consumer Trait (`src/events/events.rs:24-52`)

```rust
pub trait Consumer: Send + Sync {
    fn on_start(&mut self) {}
    fn on_finish(&mut self) {}
    fn handle(&mut self, event: Event) {}
    fn handle_with_context(&mut self, event: Event, context: &mut Context);
    fn run<I>(&mut self, events: I) where I: Iterator<Item = Event>;
}
```

### Event Emission Flow (`src/parsers/wasot/document_parser.rs:254-316`)

```rust
fn process_line_with_events(&mut self, ...) {
    if is_meta_line(line) {
        self.flush_accumulated_text();
        // Emit BlockStarted or ContextLine
        // For headings: immediate BlockFinished
    } else {
        // Accumulate text
        self.accumulate_text_line(line, line_index);
    }

    // Process with ElementParser
    let closed_count = element_parser.process_line(line, line_index)?;

    // Emit BlockFinished for each closed block
    for _ in 0..closed_count {
        bus.broadcast(Event::BlockFinished);
    }
}
```

---

## Selector Parser

### Grammar (`src/parsers/selectors/selectors.pest`)

**Entry point (line 8):**
```pest
selector = { SOI ~ ws* ~ selector_group ~ ws* ~ EOI }
```

**Selector components:**
- `selector_group` - Comma-separated selectors (lines 34-36)
- `compound_selector` - Connected by combinators (lines 40-42)
- `simple_selector` - Chain of basic selectors (lines 46-48)

**Basic selectors:**
- `universal_selector` - `*` (line 55)
- `type_selector` - `paragraph` or `"type with space"` (line 58)
- `id_selector` - `#main` or `#"id with space"` (line 61)
- `class_selector` - `.highlight` (line 64)
- `attribute_selector` - `[key="value"]` (lines 71-81)
- `pseudo_class_selector` - `:first-child`, `:nth-child(n)` (lines 125-153)

**Combinators (lines 162-179):**
- Descendant: `div span` (whitespace)
- Child: `ul > li`
- Adjacent sibling: `h1 + p`
- General sibling: `h1 ~ p`

**Attribute matchers (lines 83-117):**
- `[attr]` - Presence
- `[!attr]` - Negated presence
- `[attr="value"]` - Exact match
- `[attr~="value"]` - Negated match
- `[attr^="value"]` - Starts with
- `[attr$="value"]` - Ends with
- `[attr*="value"]` - Contains

### Selector Parser (`src/parsers/selectors/parser.rs`)

Uses the Pest parser generator to convert grammar matches to AST:

```rust
#[derive(Parser)]
#[grammar = "parsers/selectors/selectors.pest"]
pub struct SelectorParser;

pub fn parse_selector(input: &str) -> Result<SimpleSelectorGroup, String>
```

---

## Error Handling Philosophy

The Wasot parser follows a **"Honey Badger Don't Care"** philosophy:

### Graceful Recovery Patterns

1. **Unclosed quotes** (`src/parsers/wasot/quote_aware_tokenizer.rs:114-123`)
   - Treated as unquoted tokens
   - No parsing failure

2. **Malformed arguments** (`src/parsers/wasot/arguments.rs:90-94`)
   - Best-effort parsing
   - Empty values allowed: `key=`
   - Empty keys allowed: `=value`

3. **Invalid meta-lines** (`src/parsers/wasot/document_parser.rs:100`)
   - Treated as content, not fatal errors
   - Logged to stderr, not thrown

4. **Missing blocks** (`src/parsers/wasot/element_parser.rs:240-243`)
   - Explicit close `./` with empty stack: silent no-op

### Error Reporting

Errors are reported via `eprintln!` rather than panicking:
- `src/parsers/wasot/document_parser.rs:342` - File open errors
- `src/parsers/wasot/document_parser.rs:370` - Line read errors
- `src/parsers/wasot/document_parser.rs:377` - Processing errors

---

## Performance Considerations

### Memory Efficiency

1. **Streaming mode** (`src/parsers/wasot/element_parser.rs:40-49`)
   - Blocks not accumulated in container
   - Suitable for files larger than RAM

2. **Bounded channels** (`src/parsers/wasot/document_parser.rs:34`)
   - Bus capacity: 1000 events
   - Prevents unbounded memory growth

3. **Line-by-line processing** (`src/parsers/wasot/document_parser.rs:366`)
   - BufReader for streaming reads
   - No full-file loading

### CPU Efficiency

1. **Regex caching** (`src/parsers/wasot/parser.rs:192-194`)
   ```rust
   static ID_REGEX: OnceLock<Regex> = OnceLock::new();
   ```
   - Avoids ~300KB allocations per call

2. **Base36 encoding** (`src/parsers/wasot/parser.rs:61-66`)
   - Compact ID representation
   - Fast conversion via `crate::base36`

3. **Lazy link parsing** (`src/parsers/wasot/parser.rs:26-34`)
   - `QuotedString::parse_links()` called on demand
   - Not during initial parsing

---

## File Reference Summary

| Component | Primary File | Key Lines |
|-----------|--------------|-----------|
| Document Parser | `src/parsers/wasot/document_parser.rs` | 15-401 |
| Element Parser | `src/parsers/wasot/element_parser.rs` | 6-321 |
| Meta-Line Parser | `src/parsers/wasot/parser.rs` | 102-707 |
| Block Structures | `src/parsers/wasot/block.rs` | 4-396 |
| Arguments Parser | `src/parsers/wasot/arguments.rs` | 4-138 |
| Tokenizer | `src/parsers/wasot/tokenizer.rs` | 1-188 |
| Quote Tokenizer | `src/parsers/wasot/quote_aware_tokenizer.rs` | 14-148 |
| Events | `src/events/events.rs` | 4-132 |
| Selector Grammar | `src/parsers/selectors/selectors.pest` | 1-179 |
| Selector Parser | `src/parsers/selectors/parser.rs` | 16-49 |
