# Tree-sitter Grammar Development

This document explains the S-expression output format and its role in developing and validating a tree-sitter grammar for wasot.

## Why Build a Tree-sitter Grammar?

Tree-sitter provides:

- **Editor Integration**: Syntax highlighting, code folding, and structural navigation in VS Code, Neovim, Helix, Zed, and other editors
- **Incremental Parsing**: Fast re-parsing on edits for real-time feedback
- **Language-Agnostic Tooling**: Query-based code analysis, refactoring tools, and linters
- **WebAssembly Support**: Browser-based parsing via tree-sitter's WASM builds

## Why the Rust Parser Cannot Be Reused

The existing Rust parser in `src/parsers/wasot/` cannot be directly reused for tree-sitter because the two systems have fundamentally different architectures:

| Aspect | Rust Parser | Tree-sitter |
|--------|-------------|-------------|
| **Grammar Definition** | Hand-written Rust code with tokenizers and state machines | JavaScript DSL (`grammar.js`) |
| **Output Format** | Event stream (`BlockStarted`, `TextFinished`, etc.) | Concrete Syntax Tree (CST) in C |
| **Generated Code** | None (pure Rust) | C parser (`parser.c`) with bindings |
| **Execution Model** | Streaming, line-by-line, bounded memory | Full-document parse into tree |
| **Location** | `src/parsers/wasot/document_parser.rs` | `tree-sitter-wasot/grammar.js` |

Tree-sitter requires its own grammar DSL and generates a C parser. There is no mechanism to translate Rust parsing logic into tree-sitter's grammar format automatically.

### Rust Parser Architecture

The Rust parser (see [`docs/architecture.md`](architecture.md)) uses an event-driven model:

```
WasotFileEmitter → Event Bus → Consumer
         ↓
    Line-by-line
    processing
         ↓
    BlockStarted { meta_line }
    TextFinished { lines, links, footnotes }
    BlockFinished
```

Key implementation files:
- `src/parsers/wasot/document_parser.rs` - Main streaming parser
- `src/parsers/wasot/element_parser.rs` - Block/element construction
- `src/parsers/wasot/tokenizer.rs` - Line tokenization
- `src/parsers/wasot/link_parser.rs` - Embedded link detection

### Tree-sitter Grammar Requirements

Tree-sitter grammars are defined in JavaScript:

```javascript
// Example from .old-implementation/tree-sitter-wasot/grammar.js
module.exports = grammar({
  name: "wasot",
  rules: {
    source_file: $ => repeat(choice($.block, $.context_line)),
    block: $ => seq($.metaline, optional($.text_content)),
    metaline: $ => seq(".", $.block_type, optional($.arguments)),
    // ...
  }
});
```

This generates a C parser that builds a complete syntax tree, fundamentally different from our streaming event model.

## The S-expression Bridge

The `sexp` feature provides a standardized output format that bridges the two parser implementations.

### Purpose

The S-expression format serves as **ground truth** for validating any tree-sitter grammar:

```
┌─────────────────────────────────────────────────────────────────┐
│                    VALIDATION WORKFLOW                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Wasot Source ─────┬────► Rust Parser ────► S-expression        │
│       │            │              (ground truth)                │
│       │            │                    │                       │
│       │            │                    ▼                       │
│       │            │              ┌──────────┐                  │
│       └────────────┴────► Tree-sitter ──► Compare               │
│                                 │                               │
│                                 ▼                               │
│                          S-expression                           │
│                        (must match)                             │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

The Rust parser is the **single source of truth**. Any tree-sitter grammar must produce equivalent S-expression output for all test cases.

### S-expression Format

The format follows tree-sitter conventions (`node-type key: value ...children...`):

```lisp
(block type: "section" level: 1 id: "intro"
  (inline-text content: "Introduction" quoted: false)
  (text
    (line index: 0 content: "Welcome to the documentation.")
    (link target_type: uri start: 15 length: 13 target: "https://example.com"
      (name content: "documentation" quoted: false)
    )
  )
)
```

Key node types:
- `block` - Regular blocks with type, level, optional id
- `void` - Self-closing elements (e.g., `hr:`, `br:`)
- `context-line` - Metadata lines (e.g., `./id`, `./author`)
- `text` - Text content container
- `line` - Individual text line with index
- `link` - Embedded link with target_type (uri, id, footnote)
- `footnote` - Footnote reference
- `arg` - Positional argument
- `kwarg` - Keyword argument with key/value children
- `inline-text` - Inline text on meta line

### Validation

S-expression output is validated using `tree-sitter-scheme` to ensure syntactic correctness:

```rust
// From src/consumers/to_sexp_consumer.rs
pub fn validate_sexp(sexp: &str) -> Result<(), String> {
    let mut parser = tree_sitter::Parser::new();
    parser.set_language(&tree_sitter_scheme::LANGUAGE.into())?;
    let tree = parser.parse(sexp, None)?;
    if tree.root_node().has_error() {
        Err("S-expression contains syntax errors")
    } else {
        Ok(())
    }
}
```

## Using Test Cases

### Source: `tests/wasot_syntax_tests.md`

The syntax test file contains test cases in markdown format (see [`docs/syntax_testing.md`](syntax_testing.md)):

```markdown
**test name**

Description of what this test covers.

```wasot
.section :intro
..paragraph
Hello world
```

```yaml
- block-started:
    type: section
    id: intro
- block-started:
    type: paragraph
- text-finished:
    lines: ["Hello world"]
- block-finished
- block-finished
```
```

### Generating S-expression Ground Truth

Build with the `sexp` feature and convert test snippets:

```bash
# Build with sexp support
cargo build --release --features sexp

# Convert a wasot file to S-expression
wasot input.wasot -t sexp

# Or with pretty-printing
wasot input.wasot -t sexp --pretty

# From stdin
echo '.block\nHello' | wasot -t sexp

# With selector filtering
wasot input.wasot --select=paragraph -t sexp
```

### Workflow for Tree-sitter Development

1. **Extract test snippets** from `tests/wasot_syntax_tests.md`

2. **Generate ground truth S-expressions** using the Rust parser:
   ```bash
   wasot snippet.wasot -t sexp -o expected.sexp
   ```

3. **Parse with tree-sitter** and convert CST to S-expression format

4. **Compare outputs** - they must match structurally

5. **Iterate** on the tree-sitter grammar until all tests pass

### Automated Test Extraction (Planned)

A future tool will automate this workflow:

```bash
# Extract all wasot snippets from syntax tests
wasot extract-tests tests/wasot_syntax_tests.md -o test_cases/

# Generate expected S-expressions
for f in test_cases/*.wasot; do
    wasot "$f" -t sexp -o "${f%.wasot}.sexp"
done

# Run tree-sitter comparison (future)
wasot-ts-compare test_cases/
```

## Implementation Details

### Consumer Architecture

The `ToSexpConsumer` implements the `Consumer` trait to process events:

```rust
impl<W: Write + Send + Sync + 'static> Consumer for ToSexpConsumer<W> {
    fn handle(&mut self, event: Event) {
        match event {
            Event::BlockStarted { meta_line } => self.write_block_start(&meta_line),
            Event::TextFinished { lines, links, footnotes } => self.write_text(...),
            Event::BlockFinished => self.write_block_end(),
            // ...
        }
    }
}
```

See `src/consumers/to_sexp_consumer.rs` for the complete implementation.

### Data Model Preservation

The S-expression format preserves the complete wasot data model:

- **Quoted flags**: Whether strings were originally quoted
  ```lisp
  (arg content: "value" quoted: true)
  (arg content: value quoted: false)
  ```

- **Link target types**: `uri`, `id`, `footnote`, `id_in_document`
  ```lisp
  (link target_type: uri target: "https://example.com" ...)
  (link target_type: id target: "section-1" ...)
  ```

- **Embedded links in QuotedStrings**: Links within argument values
  ```lisp
  (value content: "see docs" quoted: true
    (link target_type: uri target: "https://docs.example.com" ...)
  )
  ```

- **Position information**: For precise source mapping
  ```lisp
  (link start: 5 length: 10 ...)
  (block line_index: 42 ...)
  ```

## Related Documentation

- [`architecture.md`](architecture.md) - Event-driven architecture overview
- [`ast.md`](ast.md) - Abstract syntax tree definition
- [`syntax_testing.md`](syntax_testing.md) - Syntax test format and workflow
- [`testing.md`](testing.md) - Testing infrastructure
- [`wasot_and_xml.md`](wasot_and_xml.md) - Wasot data model and XML comparison

## Running the Tests

The tree-sitter grammar is validated against the Rust parser using automated test scripts. The Rust parser's S-expression output is the **single source of truth**.

### Quick Start

```bash
# Run all tree-sitter tests
./scripts/test_treesitter.sh

# Run with verbose output
./scripts/test_treesitter.sh -v

# Run a specific test by name
./scripts/test_treesitter.sh --test "heading"
```

### Test Scripts

| Script | Purpose |
|--------|---------|
| `scripts/test_treesitter.sh` | Runs both test suites in sequence |
| `scripts/test_treesitter_parser_progressive.py` | Tests against `progressive_complexity/` files |
| `scripts/test_treesitter_parser.py` | Tests against `tests/wasot_syntax_tests.md` |

### Test Case Sources

#### 1. Progressive Complexity Tests (`progressive_complexity/`)

87 standalone `.wasot` files ordered by complexity:

```
progressive_complexity/
├── 0000_empty.wasot              # Empty file
├── 0001_single_block.wasot       # .block
├── 0002_block_with_text.wasot    # Block + text content
├── ...
├── 0027_empty_type.wasot         # Edge case: just "."
├── 0067_unclosed_quote.wasot     # Edge case: ."unclosed
├── ...
├── 0086_everything.wasot         # Complex realistic document
```

These files are designed for incremental grammar development - start with simple cases and work up.

#### 2. Syntax Tests (`tests/wasot_syntax_tests.md`)

286 test cases embedded in markdown, organized by feature:

```markdown
**Simple paragraph**

Test a basic single-level block with text content.

```wasot
.paragraph
This is a simple paragraph.
```

```yaml
- block-started:
    type: paragraph
    level: 1
- text-finished:
    lines: ["This is a simple paragraph."]
- block-finished
```
```

These are extracted automatically by `scripts/parse_syntax_file.py`.

### How Testing Works

```
┌─────────────────────────────────────────────────────────────────┐
│                    TEST COMPARISON FLOW                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  .wasot file ─────┬────► wasot -t sexp ────► S-expression       │
│       │           │         (Rust parser)    (ground truth)     │
│       │           │                               │              │
│       │           │                               ▼              │
│       │           │                         Extract elements:    │
│       │           │                         - block count        │
│       │           │                         - nesting levels     │
│       │           │                         - void markers       │
│       │           │                               │              │
│       │           │                               ▼              │
│       └───────────┴────► npx tree-sitter ──► Compare ──► PASS/FAIL
│                              parse                               │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

The test scripts compare structural elements between parsers:
- Block counts and nesting levels
- Void element detection
- Context line detection
- Basic structural equivalence

### Debugging Failures

```bash
# Show S-expressions from both parsers
python scripts/test_treesitter_parser_progressive.py --test 0027 --show-sexp

# Test a specific file manually
echo '.block' | ./target/debug/wasot -t sexp           # Rust parser
cd tree-sitter-wasot && echo '.block' > /tmp/t.wasot && npx tree-sitter parse /tmp/t.wasot
```

## Grammar Development

### File Locations

```
tree-sitter-wasot/
├── grammar.js          # Grammar definition (edit this)
├── src/
│   ├── parser.c        # Generated C parser (don't edit)
│   └── grammar.json    # Generated grammar spec
├── queries/
│   └── highlights.scm  # Syntax highlighting queries
└── test/
    └── corpus/         # Tree-sitter native tests
        └── basics.txt
```

### Development Workflow

1. **Edit the grammar**:
   ```bash
   cd tree-sitter-wasot
   vim grammar.js
   ```

2. **Regenerate the parser**:
   ```bash
   npx tree-sitter generate
   ```

3. **Test manually**:
   ```bash
   echo '.block arg=value' | npx tree-sitter parse /dev/stdin
   ```

4. **Run the test suite**:
   ```bash
   cd ..
   ./scripts/test_treesitter.sh
   ```

5. **Iterate** until all tests pass

### Grammar Structure

The grammar (`tree-sitter-wasot/grammar.js`) defines:

```javascript
module.exports = grammar({
  name: "wasot",
  rules: {
    source_file: ($) => repeat(choice($.meta_line, $.text_line)),

    meta_line: ($) => choice(
      // Context line: ./something
      seq($.dots, $.context_marker, optional($.context_content), ...),
      // Void element: .type:
      seq($.dots, $.block_type, $.void_marker, ...),
      // Regular block: .type
      seq($.dots, $.block_type, optional($.block_content), ...),
      // Empty block (just dots)
      seq($.dots, optional("\n"))
    ),

    dots: ($) => /\.+/,           // Nesting level
    block_type: ($) => ...,       // Type identifier
    void_marker: ($) => ...,      // The colon in .type:
    // ...
  }
});
```

Key concepts:
- `token.immediate()` - Match without whitespace between tokens
- `prec()` - Set precedence for ambiguous matches
- `choice()` - Alternative patterns (order matters for fallbacks)

### Common Issues

**"Block count mismatch"**: The tree-sitter grammar isn't recognizing a meta-line as a block. Check:
- Is `block_type` matching the pattern?
- Is the fallback case (empty dots) being hit correctly?

**"Tree-sitter errors: ERROR"**: Parse error in tree-sitter. Run manually to see the full tree:
```bash
npx tree-sitter parse problematic_file.wasot
```

**"Void count mismatch"**: Check that `void_marker` (`:`) is being detected after `block_type`.

## Feature Flag

The S-expression functionality is gated behind the `sexp` feature:

```toml
# Cargo.toml
[features]
sexp = ["tree-sitter", "tree-sitter-scheme"]
```

Enable when building:

```bash
cargo build --features sexp
cargo test --features sexp
```

The feature is included in the default feature set for development convenience.
