# HTML Support for Wasot - Comprehensive Analysis

## Executive Summary

This report analyzes the requirements for adding HTML input and output support to the Wasot CLI, building on the existing XML support infrastructure. The goal is to enable:
- **HTML Input**: Parse HTML files and convert them to Wasot events (`HtmlEmitter`)
- **HTML Output**: Convert Wasot events to HTML format (`HtmlConsumer` or `ToHtmlConsumer`)

## 1. Current Architecture Overview

### 1.1 Event-Driven Streaming Model

Wasot uses an event-driven architecture with two core traits defined in `src/events/events.rs`:

```rust
/// An `Emitter` is a source of `Event`s.
pub trait Emitter: Send + Sync {
    type Reader: Iterator<Item = Event> + Send;
    fn subscribe(&self) -> Self::Reader;
    fn run(self);
    fn run_with_context(self, context: &mut Context);
}

/// A `Consumer` processes `Event`s from an event stream.
pub trait Consumer: Send + Sync {
    fn on_start(&mut self) {}
    fn on_finish(&mut self) {}
    fn handle(&mut self, event: Event) {}
    fn handle_with_context(&mut self, event: Event, context: &mut Context);
    fn run<I>(&mut self, events: I) where I: Iterator<Item = Event> + Send;
}
```

### 1.2 Event Types

The system uses five event types (from `src/events/events.rs:56-86`):

```rust
pub enum Event {
    BlockStarted { meta_line: MetaLine },
    BlockFinished,
    TextFinished { lines: Vec<String>, links: Vec<Vec<Link>>, footnotes: Vec<Footnote> },
    ContextLine { meta_line: MetaLine },
    VoidElement { meta_line: MetaLine },  // For self-closing HTML elements like <br>, <img>
}
```

### 1.3 Data Flow Pattern

```
Emitter → Events → Consumer
│                    │
├─ WasotFileEmitter  ├─ XmlListener (stdout)
├─ JsonlEmitter      ├─ ToXmlConsumer (writer)
├─ MarkdownEmitter   ├─ ToJsonlConsumer
├─ SqliteEmitter     ├─ WasotReconstructor
└─ [NEW: HtmlEmitter] └─ [NEW: ToHtmlConsumer]
```

## 2. Current XML Implementation Analysis

### 2.1 XML Output - Two Implementations

#### XmlListener (`src/consumers/xml_listener.rs`)
- Simple consumer that prints XML to stdout
- Uses `print!` macros for direct output
- Tracks indentation level for pretty-printing
- ~620 lines including tests

#### ToXmlConsumer (`src/consumers/to_xml_consumer.rs`)
- Uses `xml-rs` crate's `EventWriter` for proper XML serialization
- Supports arbitrary `Write` implementations (files, buffers)
- Maintains element stack for nesting
- Handles XML declaration, attributes, and proper escaping
- ~1230 lines including comprehensive tests

### 2.2 XML Input - Not Yet Implemented

The `Format` enum in `main.rs:143-158` includes `Xml` as an input format, but there's **no `XmlEmitter` yet**. Content detection exists at `main.rs:317`:

```rust
match first_char {
    '<' => Format::Xml,         // XML tag
    '{' | '[' => Format::Jsonl, // JSON object or array
    _ => Format::Wasot,         // Everything else is wasot
}
```

**Verified**: Running `wasot -f xml -t wasot` produces:
```
wasot: Conversion from Xml to Wasot is not yet implemented
```

This means **XML input (XmlEmitter) is also pending implementation**, providing an opportunity to implement both XmlEmitter and HtmlEmitter with shared patterns.

## 3. CLI Integration Points

### 3.1 Format Enum (`main.rs:143-158`)

```rust
#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)]
enum Format {
    Wasot,
    Xml,
    Binary,
    Jsonl,
    Sql,
    #[cfg(feature = "sqlite")]
    Sqlite,
    // [NEW] Html,
}
```

### 3.2 Conversion Routing (`main.rs:382-437`)

The `process_default()` function routes format conversions:
```rust
match (input_format, output_format) {
    (Format::Wasot, Format::Wasot) => process_pretty(...),
    (Format::Wasot, Format::Xml) => process_to_xml(...),
    // ... other conversions
    // [NEW] (Format::Wasot, Format::Html) => process_to_html(...),
    // [NEW] (Format::Html, Format::Wasot) => process_from_html(...),
}
```

### 3.3 Format Detection (`main.rs:330-341`)

```rust
fn detect_format_from_extension(path: &str) -> Option<Format> {
    match ext {
        "wasot" => Some(Format::Wasot),
        "xml" => Some(Format::Xml),
        "jsonl" => Some(Format::Jsonl),
        "wasotb" => Some(Format::Binary),
        "sql" => Some(Format::Sql),
        // [NEW] "html" | "htm" => Some(Format::Html),
        _ => None,
    }
}
```

## 4. Void Elements - Critical for HTML Support

Wasot already has void element support (see `docs/void_elements.md`), which maps directly to HTML void elements.

### 4.1 Wasot Void Element Syntax

```wasot
.hr:                                    # Basic void element
.img: alt text here                     # Void element with inline text
.img: alt text .. src="photo.jpg"       # Void element with arguments
.br:                                    # Line break
```

### 4.2 HTML5 Void Elements

The following HTML elements are void (cannot have content):

| Element | Description | Wasot Equivalent |
|---------|-------------|------------------|
| `<area>` | Image map area | `.area:` |
| `<base>` | Document base URL | `.base:` |
| `<br>` | Line break | `.br:` |
| `<col>` | Table column | `.col:` |
| `<embed>` | External content | `.embed:` |
| `<hr>` | Horizontal rule | `.hr:` |
| `<img>` | Image | `.img:` |
| `<input>` | Form input | `.input:` |
| `<link>` | External resource link | `.link:` |
| `<meta>` | Metadata | `.meta:` |
| `<source>` | Media source | `.source:` |
| `<track>` | Text track | `.track:` |
| `<wbr>` | Word break opportunity | `.wbr:` |

### 4.3 Event Mapping

Wasot's `Event::VoidElement` maps directly to HTML void elements:
- Input: HTML `<br>` → `Event::VoidElement { meta_line: { type_name: "br", is_void: true } }`
- Output: `Event::VoidElement` → HTML `<br>` (no closing tag in HTML5)

## 5. Existing HTML-Related Code

### 5.1 Old Python Implementation (`.old-implementation/apps/from_html/main.py`)

A legacy Python script exists that:
1. Uses BeautifulSoup for HTML parsing
2. Uses markdownify for HTML-to-Markdown conversion
3. Implements content extraction heuristics:
   - Removes `<nav>`, `<button>` elements
   - Removes base64 images
   - Finds main content using selectors: `role="main"`, `class="content"`, `<main>`, `<article>`

Key functions:
- `extract_main_content(html)` - Extracts main content from HTML
- `remove_excessive_blank_lines(text, max_consecutive_blanks=3)` - Cleans output

### 5.2 MarkdownEmitter Plugin (`src/plugins/markdown_emitter.rs`)

The MarkdownEmitter provides a useful template for implementing HtmlEmitter:
- Parses Markdown syntax line-by-line
- Maintains state for code blocks, quotes
- Emits appropriate events for headings, paragraphs, code blocks
- ~950 lines including tests

## 6. Recommended Rust Crates for HTML

### 6.1 For HTML Parsing (HtmlEmitter)

| Crate | Description | Pros | Cons |
|-------|-------------|------|------|
| **scraper** | CSS selector-based HTML parsing | Easy API, good for content extraction | Higher-level, may miss structure |
| **html5ever** | Full HTML5 parser (Servo project) | Standards-compliant, streaming | Complex API |
| **kuchikiki** | DOM tree on top of html5ever | Intuitive DOM traversal | Additional dependency |
| **select.rs** | jQuery-like selectors | Simple, familiar API | Less actively maintained |

**Recommendation**: Use `scraper` for initial implementation due to its simpler API, with option to move to `html5ever` for more complex parsing needs.

### 6.2 For HTML Output (ToHtmlConsumer)

| Crate | Description | Pros | Cons |
|-------|-------------|------|------|
| **html_escape** | HTML entity escaping | Lightweight, focused | Only escaping |
| **markup** | HTML macro DSL | Compile-time checks | Less flexible |
| **maud** | Compile-time HTML templates | Type-safe | Different paradigm |

**Recommendation**: Implement HTML output manually (like `XmlListener`) using `html_escape` for entity escaping. This provides maximum control over the output format and keeps the approach consistent with the existing `xml-rs` based XML output.

## 7. Implementation Plan

### 7.1 Phase 1: HTML Output (`ToHtmlConsumer`)

**New Files:**
- `src/consumers/to_html_consumer.rs` - Main implementation
- `src/consumers/html_listener.rs` - Simple stdout variant (optional)

**Implementation Strategy:**

```rust
pub struct ToHtmlConsumer<W: Write> {
    writer: W,
    element_stack: Vec<String>,
    indent_level: usize,
    pretty_print: bool,
}
```

**Key Considerations:**
1. **HTML5 vs XHTML**: Use HTML5 syntax (no self-closing tags except for void elements)
2. **Void Elements**: Handle `<br>`, `<hr>`, `<img>`, `<input>`, `<meta>`, `<link>` properly
3. **Attribute Escaping**: Escape `"`, `&`, `<`, `>` in attribute values
4. **Content Escaping**: Escape `&`, `<`, `>` in text content
5. **Block Type Mapping**: Map Wasot block types to HTML elements

**Wasot to HTML Mapping:**

| Wasot | HTML |
|-------|------|
| `.paragraph` or `.p` | `<p>` |
| `.heading` or `.h1`-`.h6` or `#`-`######` | `<h1>`-`<h6>` |
| `.section` | `<section>` |
| `.article` | `<article>` |
| `.div` | `<div>` |
| `.quote` or `.blockquote` | `<blockquote>` |
| `.code` | `<pre><code>` |
| `.list` | `<ul>` or `<ol>` |
| `.item` or `.li` | `<li>` |
| Generic `.xyz` | `<div class="xyz">` or `<xyz>` (configurable) |

### 7.2 Phase 2: HTML Input (`HtmlEmitter`)

**New Files:**
- `src/emitters/html_emitter.rs`

**Implementation Strategy:**

```rust
pub struct HtmlEmitter {
    file_path: Option<String>,
    reader: Option<Box<dyn Read + Send + Sync>>,
    bus: Arc<Mutex<Bus<Event>>>,
    next_auto_id: Arc<Mutex<u64>>,
    // Configuration options
    extract_main_content: bool,
    strip_nav: bool,
    strip_scripts: bool,
}
```

**Parsing Approach:**
1. Use `scraper` to parse HTML into a DOM tree
2. Optionally extract main content (like the old Python implementation)
3. Walk the DOM tree depth-first
4. Emit events for each element:
   - `BlockStarted` when entering an element
   - `TextFinished` when encountering text nodes
   - `BlockFinished` when leaving an element
   - `VoidElement` for self-closing elements

**HTML to Wasot Mapping:**

| HTML | Wasot |
|------|-------|
| `<p>` | `.p` |
| `<h1>`-`<h6>` | `.h1`-`.h6` or `..`-`#######` headings |
| `<div>` | `.div` |
| `<section>` | `.section` |
| `<article>` | `.article` |
| `<blockquote>` | `.blockquote` |
| `<pre><code>` | `.code` |
| `<ul>`, `<ol>` | `.list` |
| `<li>` | `.item` |
| `<a>` | Inline links in text |
| `<img>` | `.img:` (void element) |
| `<br>` | `.br:` (void element) |
| `<hr>` | `.hr:` (void element) |

### 7.3 Phase 3: CLI Integration

**Changes to `main.rs`:**

1. Add `Html` variant to `Format` enum
2. Add `.html` and `.htm` to `detect_format_from_extension()`
3. Add HTML detection to `detect_format_from_content()` (look for `<!DOCTYPE html` or `<html`)
4. Add conversion routes:
   - `(Format::Wasot, Format::Html) => process_to_html(...)`
   - `(Format::Html, Format::Wasot) => process_from_html(...)`
   - `(Format::Html, Format::Xml)` etc.

**New `process_to_html()` function (following `process_to_xml` pattern at line 1359):**

```rust
fn process_to_html(file_path: Option<&str>, pretty: bool) -> Result<(), Box<dyn std::error::Error>> {
    use std::path::Path;
    use std::thread;

    let emitter = match file_path {
        Some(path) => {
            if !Path::new(path).exists() {
                return Err(format!("File not found: {path}").into());
            }
            WasotFileEmitter::from_file(path)
        }
        None => {
            let mut buffer = String::new();
            io::stdin().read_to_string(&mut buffer)?;
            WasotFileEmitter::from_string(buffer)
        }
    };

    let receiver = emitter.subscribe();

    let emitter_handle = thread::spawn(move || {
        emitter.run();
    });

    let mut html_listener = HtmlListener::new(pretty);
    html_listener.run(receiver);

    emitter_handle
        .join()
        .map_err(|_| "Emitter thread panicked")?;

    Ok(())
}
```

**Changes to `src/lib.rs`:**
- Export new emitter/consumer types
- Add to public API

### 7.4 Phase 4: Testing

**Unit Tests:**
- Test each event type mapping
- Test attribute and content escaping
- Test void element handling
- Test nested structures

**Integration Tests:**
- Roundtrip: HTML → Wasot → HTML (structural equivalence)
- Roundtrip: Wasot → HTML → Wasot (content preservation)
- Compare with expected output files

## 8. Dependencies to Add

```toml
# Cargo.toml additions
[dependencies]
scraper = "0.20"        # For HTML parsing (input)
html_escape = "0.2"     # For HTML entity escaping (output)
```

## 9. Example Usage After Implementation

```bash
# Convert Wasot to HTML
wasot doc.wasot -t html
wasot doc.wasot -o doc.html

# Convert HTML to Wasot
wasot page.html -t wasot
wasot -f html page.html -o page.wasot

# Pipe support
curl https://example.com | wasot -f html -t wasot > example.wasot

# With selector
wasot page.html --select "article" -t wasot
```

## 10. Estimated Effort

| Component | Complexity | Estimated Lines | Time Estimate |
|-----------|------------|-----------------|---------------|
| `ToHtmlConsumer` | Medium | 400-600 | 2-3 days |
| `HtmlListener` (stdout) | Low | 200-300 | 1 day |
| `HtmlEmitter` | High | 500-800 | 3-4 days |
| CLI Integration | Low | 100-150 | 0.5 day |
| Tests | Medium | 500-800 | 2-3 days |
| Documentation | Low | 100-200 | 0.5 day |
| **Total** | | **1800-2850** | **9-12 days** |

## 11. Open Questions

1. **Generic block handling**: Should `.custom-element` map to `<div class="custom-element">` or `<custom-element>`?
   - Recommendation: Default to `<div class="...">` with configurable custom element mode

2. **Attribute preservation**: Should all kwargs become HTML attributes?
   - Recommendation: Yes, with `class` having special handling

3. **Link handling**: How to handle Wasot inline links `[[url|text]]` in HTML output?
   - Recommendation: Convert to `<a href="url">text</a>`

4. **Content extraction for input**: Should we implement the same heuristics as the old Python tool?
   - Recommendation: Make it optional with `--extract-content` flag

5. **Whitespace handling**: How strict should we be about preserving whitespace?
   - Recommendation: Normalize by default, with `--preserve-whitespace` option

## 12. Files to Create/Modify

### New Files
- `src/consumers/to_html_consumer.rs`
- `src/consumers/html_listener.rs` (optional)
- `src/emitters/html_emitter.rs`

### Files to Modify
- `src/main.rs` - Add Format::Html, routing
- `src/consumers/mod.rs` - Export new consumers
- `src/emitters/mod.rs` - Export new emitter
- `src/lib.rs` - Public API exports
- `Cargo.toml` - Add dependencies

## 13. References

- Current XML implementation: `src/consumers/xml_listener.rs`, `src/consumers/to_xml_consumer.rs`
- MarkdownEmitter pattern: `src/plugins/markdown_emitter.rs`
- Event system: `src/events/events.rs`
- CLI structure: `src/main.rs` (see `process_to_xml()` at line 1359 for output pattern)
- Old Python HTML tool: `.old-implementation/apps/from_html/main.py`
- Wasot and XML documentation: `docs/wasot_and_xml.md`
- **Void elements documentation: `docs/void_elements.md`** (critical for HTML void element mapping)

---

*Report generated: 2026-02-06*
*Based on codebase analysis of wasot-compiler v0.1.0*
