# Wasot Syntax Highlighting Analysis

A comprehensive analysis of Wasot language elements for syntax highlighting and Developer Experience (DX) considerations.

---

## Executive Summary

This report analyzes the Wasot syntax from three authoritative sources:
1. **Tree-sitter grammar** (`tree-sitter-wasot/grammar.js`) - The current tree-sitter implementation
2. **Syntax tests** (`tests/wasot_syntax_tests.md`) - Comprehensive test cases
3. **Metaline parser** (`src/parsers/wasot/parser.rs`) - The canonical Rust implementation

The goal is to identify all syntactic elements requiring highlighting and provide DX recommendations.

---

## Part 1: Wasot Language Elements Requiring Highlighting

### 1.1 Line-Level Classification

Wasot has two fundamental line types:

| Line Type | Detection | Highlighting Need |
|-----------|-----------|-------------------|
| **Meta-line** | Starts with `.` | Multiple sub-elements (see below) |
| **Text-line** | Does not start with `.` | Default text color or minimal styling |

**Tree-sitter rule:**
```js
source_file: ($) => repeat(choice($.meta_line, $.text_line)),
text_line: ($) => token(prec(-1, /[^.\n][^\n]*\n?|\n/)),
```

### 1.2 Meta-Line Components

A meta-line contains multiple distinct elements that need separate highlighting:

#### 1.2.1 Dots (Nesting Indicator)
- **Pattern:** One or more leading dots (`.`, `..`, `...`, etc.)
- **Semantic:** Indicates nesting level
- **Tree-sitter:** `dots: ($) => /\.+/`
- **Highlighting:** Structural punctuation color (muted/gray often works well)
- **Examples:**
  ```wasot
  .block          # level 1
  ..nested        # level 2
  ...deep         # level 3
  ```

#### 1.2.2 Block Type
The block type immediately follows the dots and can take several forms:

| Form | Pattern | Example | Notes |
|------|---------|---------|-------|
| **Simple identifier** | `[^\s:\/\."]+` | `.div`, `.paragraph` | Most common |
| **Quoted type** | `"..."` | `."code block"` | Allows spaces/special chars |
| **Heading** | `#+` | `.#`, `.##`, `.###` | Self-closing blocks |
| **Empty type** | (nothing) | `.`, `..` | Creates anonymous/heading blocks |
| **Unicode type** | Any Unicode | `.日本語`, `.émojis` | Full Unicode support |
| **Numeric prefix** | Starts with digit | `.123class` | Valid |

**Tree-sitter:**
```js
block_type: ($) =>
  token.immediate(
    choice(
      /#+/,                           // Heading: #, ##, ###
      /"([^"\\]|\\.)*"/,              // Quoted type
      /"[^\s]+/,                      // Unclosed quoted type
      /[^\s:\/\."]+/                  // Simple type
    )
  ),
```

**Highlighting recommendation:**
- Simple types: Keyword/tag color (e.g., blue)
- Headings (`#`, `##`): Different style, possibly bold or heading-specific color
- Quoted types: String color for quotes, tag color for content

#### 1.2.3 Void Marker
- **Pattern:** Trailing `:` after block type (e.g., `.hr:`, `.br:`, `.img:`)
- **Semantic:** Marks self-closing elements (cannot have children)
- **Tree-sitter:** `void_marker: ($) => token.immediate(/:/)`
- **Highlighting:** Same as structural punctuation
- **Examples:**
  ```wasot
  .hr:                    # Horizontal rule, void
  .img: alt text here     # Image with alt text, void
  ."page break":          # Quoted void type
  ```

#### 1.2.4 Context Marker
- **Pattern:** `/` immediately after dots
- **Semantic:** Creates context lines (`./tag`, `../tag`)
- **Tree-sitter:** `context_marker: ($) => token.immediate(/\//)`
- **Highlighting:** Distinct punctuation color
- **Examples:**
  ```wasot
  ./close-tag
  ../nested-context
  ```

#### 1.2.5 Inline Text
Text between block type and parameter separator (`..`):
- **Location:** After block type, before `..` separator
- **Highlighting:** String/text color, different from block type
- **Examples:**
  ```wasot
  .heading This is inline text
  .div Hello World
  .p "Quoted inline text"
  ```

#### 1.2.6 Parameter Separator
- **Pattern:** `..` (two dots)
- **Semantic:** Separates inline text from arguments
- **Tree-sitter:** `_dot_dot: ($) => token(prec(10, /\.\./))`
- **Highlighting:** Structural punctuation (distinct from nesting dots)
- **Examples:**
  ```wasot
  .div text .. class=container
  .block .. arg1 arg2 key=value
  ```

#### 1.2.7 Block ID
- **Pattern:** `:identifier` (colon followed by identifier)
- **Location:** Typically at end of meta-line
- **Characters allowed:** `[A-Za-z0-9_\-\u0080-\uFFFF]+` and colons for namespacing
- **Tree-sitter:** `block_id: ($) => token(prec(5, /:[a-zA-Z0-9_\-\u0080-\uFFFF]+/))`
- **Highlighting:** Special identifier color (e.g., purple, or italicized)
- **Examples:**
  ```wasot
  .div :main-section
  .p some text :paragraph-1
  .section :page:header:logo    # Namespaced ID with colons
  .div :日本語id                 # Unicode ID
  ```

#### 1.2.8 Positional Arguments
Arguments without `=` sign in the parameter section:

- **Unquoted:** Simple words
- **Quoted:** Strings with spaces
- **Tree-sitter:** `word: ($) => token(prec(1, /[^\s="]+/))`
- **Highlighting:** Value/argument color
- **Examples:**
  ```wasot
  .button .. primary large rounded
  .widget .. "argument with spaces" simple
  ```

#### 1.2.9 Keyword Arguments (kwargs)
Key-value pairs in the parameter section:

| Component | Pattern | Highlighting |
|-----------|---------|--------------|
| Key | `identifier` or `"quoted"` | Parameter name color |
| Equals | `=` | Operator/punctuation color |
| Value | `word` or `"quoted string"` | String/value color |

**Tree-sitter:**
```js
kwarg: ($) =>
  token(
    prec(5,
      choice(
        /"([^"\\]|\\.)*"="([^"\\]|\\.)*"/,   // "key"="value"
        /"([^"\\]|\\.)*"=[^\s]+/,            // "key"=value
        /[a-zA-Z_][a-zA-Z0-9_\-]*="([^"\\]|\\.)*"/,  // key="value"
        /[a-zA-Z_][a-zA-Z0-9_\-]*=[^\s]+/    // key=value
      )
    )
  ),
```

**Examples:**
```wasot
.div .. class="container" id="main"
.widget .. "my key"="my value"
.form .. submit= disabled=true
```

#### 1.2.10 Quoted Strings
String literals in arguments:

- **Pattern:** `"..."` with escape support (`\"`, `\\`)
- **Tree-sitter:** `quoted_string: ($) => token(prec(3, /"([^"\\]|\\.)*"/))`
- **Highlighting:** Standard string color (often green or orange)
- **Escape sequences** within strings should have distinct highlighting

**Examples:**
```wasot
."block type with spaces"
.div .. key="value with spaces"
.text .. content="escaped \"quotes\" here"
```

### 1.3 Text Line Elements

#### 1.3.1 Links
Links appear within text content and use the `((...))` syntax:

| Link Type | Pattern | Example |
|-----------|---------|---------|
| **URL link** | `((url))` | `((https://wasot.org))` |
| **Named link** | `((name\|url))` | `((Docs\|https://docs.wasot.org))` |
| **ID reference** | `((:id))` | `((:introduction))` |
| **Link with args** | `((name .. args))` | `((submit .. class="btn"))` |

**Highlighting recommendation:**
- Link delimiters `((` and `))`: Punctuation color
- Link target: URL/reference color (often blue/underlined)
- Link name: String color
- Internal args: Same as meta-line args

#### 1.3.2 Regular Text
- Any line not starting with `.`
- Should have minimal styling (default foreground)
- Preserve whitespace handling

---

## Part 2: Complete Token Type Summary

| Token Type | Grammar Rule | Suggested Scope | Priority |
|------------|--------------|-----------------|----------|
| Dots (nesting) | `dots` | `punctuation.definition.block` | High |
| Block type (simple) | `block_type` | `entity.name.tag` | High |
| Block type (heading) | `block_type` (`#+`) | `markup.heading` | High |
| Block type (quoted) | `block_type` (`"..."`) | `string.quoted.double` | High |
| Void marker | `void_marker` | `punctuation.definition.void` | Medium |
| Context marker | `context_marker` | `punctuation.definition.context` | Medium |
| Parameter separator | `_dot_dot` | `punctuation.separator.parameters` | Medium |
| Block ID | `block_id` | `entity.name.label` / `variable.other.id` | High |
| Positional arg | `word` | `variable.parameter` | Medium |
| Kwarg key | (part of `kwarg`) | `variable.parameter` | Medium |
| Equals sign | (part of `kwarg`) | `keyword.operator.assignment` | Low |
| Kwarg value | (part of `kwarg`) | `string.unquoted` | Medium |
| Quoted string | `quoted_string` | `string.quoted.double` | High |
| Escape sequences | (within strings) | `constant.character.escape` | Medium |
| Text line | `text_line` | `text.plain` | Low |
| Link delimiters | (in text) | `punctuation.definition.link` | Medium |
| Link target | (in text) | `markup.underline.link` | Medium |

---

## Part 3: Developer Experience (DX) Considerations

### 3.1 Visual Hierarchy

**Problem:** Wasot's dot-based nesting can be visually confusing, especially with deep nesting.

**Recommendations:**
1. **Rainbow brackets for dots:** Use different colors for different nesting levels
   - Level 1 (`.`): Base color
   - Level 2 (`..`): Second color
   - Level 3+ (`...`): Rotating palette

2. **Indentation guides:** Consider adding virtual indentation guides based on dot count

3. **Heading prominence:** Make headings (`#`, `##`) visually stand out more than regular blocks

### 3.2 Error Recovery and Tolerance

The Wasot parser is "honey-badger" style (fault-tolerant). Syntax highlighting should match:

1. **Unclosed quotes:** Highlight up to end of line in string color
   ```wasot
   ."unclosed      # Should highlight "unclosed as a (malformed) string
   ```

2. **Malformed kwargs:** Still parse as separate tokens
   ```wasot
   .div .. key=    # Empty value is valid
   .div .. =value  # Empty key is valid
   ```

3. **Invalid nesting:** Don't break highlighting
   ```wasot
   ....deep-block  # Many dots - still valid
   ```

### 3.3 Semantic Highlighting Opportunities

Beyond syntax highlighting, consider semantic tokens:

| Semantic Type | When to Apply |
|---------------|---------------|
| **Heading levels** | Differentiate `.#` vs `.##` vs `.###` |
| **Void vs regular blocks** | Visual distinction for self-closing elements |
| **Context lines** | Distinguish `./` from regular blocks |
| **ID references** | Links to other blocks could show definition lookup |

### 3.4 IDE Feature Recommendations

1. **Breadcrumbs:** Show current block hierarchy based on dot levels
2. **Outline view:** Show block structure (types and IDs)
3. **Go to definition:** For ID references (`((:id))` and `:id`)
4. **Hover info:** Show block type documentation if available
5. **Folding:** Fold nested blocks (based on nesting level changes)
6. **Auto-close:** When typing `.` at line start, suggest completing with type

### 3.5 Color Scheme Recommendations

For dark themes:
- Dots: Gray (#888)
- Block types: Cyan/Blue (#4EC9B0)
- Headings: Orange/Gold (#CE9178)
- IDs: Purple (#C586C0)
- Strings: Green (#6A9955)
- Arguments: Light blue (#9CDCFE)
- Text: White/Default (#D4D4D4)

For light themes:
- Dots: Gray (#666)
- Block types: Blue (#0000FF)
- Headings: Dark orange (#A31515)
- IDs: Purple (#800080)
- Strings: Dark green (#008000)
- Arguments: Dark blue (#001080)
- Text: Black/Default (#000000)

---

## Part 4: Tree-sitter Grammar Analysis

### 4.1 Current Grammar Strengths

1. **Proper token precedence:** Uses `prec()` to disambiguate overlapping patterns
2. **Immediate tokens:** Uses `token.immediate()` to prevent whitespace insertion
3. **Escape handling:** Properly handles `\"` and `\\` in strings
4. **Unicode support:** Accepts Unicode in block types and IDs

### 4.2 Potential Improvements

1. **Separate heading rule:** Currently headings are part of `block_type`. Consider:
   ```js
   heading: ($) => token.immediate(/#+/),
   ```

2. **Explicit escape sequences:** Currently escapes are within regex. Consider:
   ```js
   escape_sequence: ($) => /\\["\\]/,
   ```

3. **Link syntax in text:** Currently `text_line` is opaque. Consider parsing links:
   ```js
   link: ($) => seq('((', $.link_content, '))'),
   ```

4. **Inline text rule:** Currently `block_content` is a single token. Breaking it up would allow:
   - Separate highlighting for inline text vs parameters
   - Better error recovery

### 4.3 Query File Recommendations

For highlighting queries (`queries/highlights.scm`):

```scheme
; Structural elements
(dots) @punctuation.delimiter
(void_marker) @punctuation.special
(context_marker) @punctuation.special
(param_separator) @punctuation.separator

; Block types
(block_type) @tag
(block_type) @markup.heading (#match? @markup.heading "^#+$")

; Identifiers
(block_id) @variable.other.member

; Arguments
(word) @variable.parameter
(kwarg) @variable.parameter
(quoted_string) @string

; Text
(text_line) @text
(block_content) @string
```

---

## Part 5: Discrepancies Between Sources

### 5.1 Grammar vs Parser Differences

| Feature | Tree-sitter Grammar | Rust Parser | Notes |
|---------|---------------------|-------------|-------|
| Void elements | `void_marker` rule | `is_void` field | Grammar explicit, parser uses field |
| Heading type | Part of `block_type` | Empty `type_name` + level | Parser converts to `##` style |
| Links | Not parsed | Parsed in `QuotedString` | Grammar treats as text |
| ID extraction | Inline in `content_parts` | Regex extraction | Parser more sophisticated |

### 5.2 Recommendations for Alignment

1. The Rust parser should be considered canonical
2. Tree-sitter grammar should be enhanced to match parser capabilities
3. Consider adding link parsing to tree-sitter for better highlighting

---

## Part 6: Summary and Action Items

### Essential Highlighting Elements (Must Have)
1. Nesting dots (`.`, `..`, `...`)
2. Block types (simple, quoted, heading)
3. Block IDs (`:identifier`)
4. Parameter separator (`..`)
5. Quoted strings (`"..."`)
6. Keyword arguments (`key=value`)

### Enhanced Highlighting Elements (Nice to Have)
1. Void marker (`:`)
2. Context marker (`/`)
3. Escape sequences within strings
4. Link syntax in text content
5. Rainbow nesting levels

### DX Features (Future Work)
1. Semantic tokens for heading levels
2. Breadcrumb support
3. Outline view
4. Go to definition for IDs
5. Folding ranges

---

*Report generated: 2026-02-13*
*Sources: tree-sitter-wasot/grammar.js, tests/wasot_syntax_tests.md, src/parsers/wasot/parser.rs*
