# HTML Support Implementation - Todo List

This document provides a detailed, milestone-based implementation plan for adding HTML input and output support to wasot. Each milestone ends with a compiling project, passing tests, and verifiable functionality.

## Design Principles

1. **Output before Input** - Easier to test with existing wasot files
2. **TDD approach** - Write tests first or alongside each feature
3. **Incremental CLI integration** - Add Format::Html early, wire up as we go
4. **100% test coverage** - Every function, every branch, every edge case
5. **Follow existing patterns** - Mirror XmlListener/ToXmlConsumer structure

## Reference: Element Mappings

### Wasot → HTML (Output)

| Wasot | HTML | Notes |
|-------|------|-------|
| `.p` / `.paragraph` | `<p>` | |
| `.h1` - `.h6` | `<h1>` - `<h6>` | Also `.heading` with level arg |
| `.div` | `<div>` | |
| `.section` | `<section>` | |
| `.article` | `<article>` | |
| `.blockquote` / `.quote` | `<blockquote>` | |
| `.code` | `<pre><code>` | |
| `.ul` / `.ol` | `<ul>` / `<ol>` | |
| `.li` / `.item` | `<li>` | |
| `.span` | `<span>` | |
| `.a` | `<a>` | href from arg |
| Generic `.xyz` | `<div class="wasot-xyz">` | Configurable |

### HTML Void Elements

| HTML | Wasot | Notes |
|------|-------|-------|
| `<br>` | `.br:` | |
| `<hr>` | `.hr:` | |
| `<img>` | `.img:` | src, alt from kwargs |
| `<input>` | `.input:` | type, name, value from kwargs |
| `<meta>` | `.meta:` | |
| `<link>` | `.link:` | |
| `<area>` | `.area:` | |
| `<base>` | `.base:` | |
| `<col>` | `.col:` | |
| `<embed>` | `.embed:` | |
| `<source>` | `.source:` | |
| `<track>` | `.track:` | |
| `<wbr>` | `.wbr:` | |

---

## Milestone 1: Dependencies & Format Enum

**Goal**: Add HTML format recognition to CLI without any conversion logic yet.

### Tasks

- [ ] **1.1** Add dependencies to Cargo.toml
  - File: `Cargo.toml`
  - Add: `html_escape = "0.2"` (for output escaping)
  - Add: `scraper = "0.20"` (for input parsing) - can defer to Milestone 5

- [ ] **1.2** Add `Html` variant to Format enum
  - File: `src/main.rs` (line ~143)
  - Add `Html` variant with doc comment `/// HTML format`

- [ ] **1.3** Add HTML extension detection
  - File: `src/main.rs` (function `detect_format_from_extension`, line ~330)
  - Add: `"html" | "htm" => Some(Format::Html),`

- [ ] **1.4** Add HTML content detection
  - File: `src/main.rs` (function `detect_format_from_content`, line ~296)
  - Detect `<!DOCTYPE` (case-insensitive) or `<html` as HTML
  - Must distinguish from XML: HTML if starts with `<!DOCTYPE html` or `<html`
  - Otherwise `<` prefix remains XML

- [ ] **1.5** Add stub error for unimplemented conversions
  - File: `src/main.rs` (in `process_default` match)
  - Ensure `(Format::Wasot, Format::Html)` and `(Format::Html, Format::Wasot)` hit the "not yet implemented" arm

- [ ] **1.6** Add unit tests for format detection
  - File: `src/main.rs` (tests module, near line ~2200)
  - Test cases:
    - [ ] `detect_format_from_extension("page.html")` → `Some(Format::Html)`
    - [ ] `detect_format_from_extension("page.htm")` → `Some(Format::Html)`
    - [ ] `detect_format_from_content(b"<!DOCTYPE html>")` → `Format::Html`
    - [ ] `detect_format_from_content(b"<!doctype HTML>")` → `Format::Html` (case insensitive)
    - [ ] `detect_format_from_content(b"<html>")` → `Format::Html`
    - [ ] `detect_format_from_content(b"  <html>")` → `Format::Html` (whitespace)
    - [ ] `detect_format_from_content(b"<root>")` → `Format::Xml` (not HTML)
    - [ ] `detect_format_from_content(b"<?xml")` → `Format::Xml`

**Verification**:
```bash
cargo test test_detect_format
wasot -f html -t wasot test.html  # Should show "not yet implemented"
```

---

## Milestone 2: HtmlListener - Basic Structure

**Goal**: Create HtmlListener that outputs HTML to stdout, handling basic blocks.

### Tasks

- [ ] **2.1** Create `src/consumers/html_listener.rs` with module structure
  - Create new file
  - Add module doc comment
  - Import required types: `Event`, `Consumer`, `MetaLine`
  - Define `HtmlListener` struct with fields:
    - `pretty_print: bool`
    - `indent_level: usize`
    - `in_pre_block: bool` (for code blocks)

- [ ] **2.2** Implement `HtmlListener::new()`
  - Constructor taking `pretty_print: bool`
  - Initialize all fields

- [ ] **2.3** Implement helper methods
  - `write_indent(&self)` - print indentation if pretty_print
  - `write_newline(&self)` - print newline if pretty_print
  - `escape_html(s: &str) -> String` - escape `&`, `<`, `>`, `"`, `'`
  - `escape_attribute(s: &str) -> String` - escape for attribute values
  - `is_void_html_element(name: &str) -> bool` - check if element is void
  - `map_block_type(wasot_type: &str) -> &str` - map wasot type to HTML tag

- [ ] **2.4** Implement `process_event()` for BlockStarted
  - Map block type to HTML element
  - Write opening tag with attributes from kwargs
  - Handle `id` attribute specially
  - Handle `class` attribute specially
  - Track indent level

- [ ] **2.5** Implement `process_event()` for BlockFinished
  - Write closing tag
  - Decrease indent level

- [ ] **2.6** Implement `process_event()` for TextFinished
  - Write text content with HTML escaping
  - Handle line breaks appropriately

- [ ] **2.7** Implement `process_event()` for VoidElement
  - Write self-closing element (HTML5 style: `<br>` not `<br/>`)
  - Include attributes

- [ ] **2.8** Implement `process_event()` for ContextLine
  - Handle as HTML comment: `<!-- context: type ... -->`

- [ ] **2.9** Implement `Consumer` trait for `HtmlListener`
  - Implement `handle(&mut self, event: Event)`

- [ ] **2.10** Export from `src/consumers/mod.rs`
  - Add `mod html_listener;`
  - Add `pub use html_listener::HtmlListener;`

- [ ] **2.11** Export from `src/lib.rs`
  - Add `HtmlListener` to public exports

**Verification**:
```bash
cargo build
cargo test html_listener
```

---

## Milestone 3: HtmlListener - Comprehensive Tests

**Goal**: Achieve 100% test coverage for HtmlListener.

### Tasks

- [ ] **3.1** Add test module to `html_listener.rs`

- [ ] **3.2** Test `HtmlListener::new()`
  - [ ] Test with pretty_print = true
  - [ ] Test with pretty_print = false
  - [ ] Verify initial state (indent_level = 0, etc.)

- [ ] **3.3** Test helper functions
  - [ ] `escape_html`: test all special chars (`&`, `<`, `>`, `"`, `'`)
  - [ ] `escape_html`: test normal text passthrough
  - [ ] `escape_html`: test empty string
  - [ ] `escape_attribute`: test quote escaping
  - [ ] `is_void_html_element`: test all 13 void elements
  - [ ] `is_void_html_element`: test non-void elements
  - [ ] `map_block_type`: test all mappings from table above
  - [ ] `map_block_type`: test generic fallback

- [ ] **3.4** Test BlockStarted event
  - [ ] Basic block without attributes
  - [ ] Block with id
  - [ ] Block with class
  - [ ] Block with multiple kwargs
  - [ ] Block with positional args
  - [ ] Block with inline text
  - [ ] Nested blocks (verify indent)
  - [ ] All semantic block types (p, h1-h6, section, article, etc.)

- [ ] **3.5** Test BlockFinished event
  - [ ] Single block close
  - [ ] Nested block close (verify indent decreases)
  - [ ] Multiple sequential closes

- [ ] **3.6** Test TextFinished event
  - [ ] Single line text
  - [ ] Multi-line text
  - [ ] Text with special chars needing escape
  - [ ] Empty text
  - [ ] Text inside pre/code block (preserve whitespace)

- [ ] **3.7** Test VoidElement event
  - [ ] Basic void element (.hr:)
  - [ ] Void with inline text (.img: alt text)
  - [ ] Void with attributes (.img: alt .. src="x.jpg")
  - [ ] All 13 HTML void elements

- [ ] **3.8** Test ContextLine event
  - [ ] Basic context line
  - [ ] Context line with attributes

- [ ] **3.9** Test pretty printing
  - [ ] Verify indentation with pretty_print=true
  - [ ] Verify no indentation with pretty_print=false
  - [ ] Verify newlines

- [ ] **3.10** Integration tests with full event sequences
  - [ ] Simple document: heading + paragraph
  - [ ] Nested structure: article > section > p
  - [ ] Mixed content: text, void elements, blocks
  - [ ] Code block with preserved whitespace

**Verification**:
```bash
cargo test html_listener -- --nocapture
cargo tarpaulin --out Html -p wasot-compiler --lib -- html_listener  # If tarpaulin available
```

---

## Milestone 4: CLI Integration - Wasot to HTML

**Goal**: Wire up `wasot doc.wasot -t html` command.

### Tasks

- [ ] **4.1** Add `process_to_html()` function
  - File: `src/main.rs`
  - Follow pattern from `process_to_xml()` (line ~1359)
  - Create emitter, spawn thread, run HtmlListener

- [ ] **4.2** Add routing for `(Format::Wasot, Format::Html)`
  - File: `src/main.rs` (in `process_default` match, line ~382)
  - Route to `process_to_html(file_path, true)`

- [ ] **4.3** Add HTML output support in `process_with_select()`
  - File: `src/main.rs` (line ~525)
  - Add `Format::Html` arm similar to `Format::Xml`

- [ ] **4.4** Add CLI integration tests
  - File: `src/main.rs` (tests module) or integration test file
  - Test cases:
    - [ ] `wasot doc.wasot -t html` produces valid HTML
    - [ ] `wasot doc.wasot -o output.html` detects format from extension
    - [ ] `cat doc.wasot | wasot -t html` works with stdin
    - [ ] `wasot doc.wasot --select "paragraph" -t html` filters correctly

- [ ] **4.5** Test with example files
  - [ ] Convert `examples/demo.wasot` to HTML
  - [ ] Verify output is valid HTML (use external validator or simple checks)

**Verification**:
```bash
cargo run --bin wasot -- examples/demo.wasot -t html
echo ".p\nHello world" | cargo run --bin wasot -- -t html
```

---

## Milestone 5: ToHtmlConsumer - Generic Writer

**Goal**: Create ToHtmlConsumer for writing HTML to any `Write` implementation.

### Tasks

- [ ] **5.1** Create `src/consumers/to_html_consumer.rs`
  - Create new file
  - Define generic struct `ToHtmlConsumer<W: Write>`
  - Fields: writer, element_stack, indent_level, pretty_print

- [ ] **5.2** Implement constructor and finish method
  - `new(writer: W, pretty_print: bool) -> Self`
  - `finish(self) -> Result<W, io::Error>` - return writer

- [ ] **5.3** Implement event processing methods
  - Same logic as HtmlListener but write to `self.writer`
  - Use `write!` macro instead of `print!`
  - Handle errors properly (return Result)

- [ ] **5.4** Implement Consumer trait
  - For `ToHtmlConsumer<std::io::Stdout>`
  - For `Arc<Mutex<ToHtmlConsumer<W>>>`

- [ ] **5.5** Add utility function `wasot_to_html_string()`
  - Convert wasot string to HTML string
  - Useful for library users

- [ ] **5.6** Export from `src/consumers/mod.rs` and `src/lib.rs`

- [ ] **5.7** Add comprehensive tests
  - Same coverage as HtmlListener tests
  - Test with Vec<u8> buffer
  - Test with File writer
  - Test finish() returns writer correctly

- [ ] **5.8** Update CLI to optionally use ToHtmlConsumer
  - When -o flag is provided, use ToHtmlConsumer with File writer
  - Otherwise use HtmlListener for stdout

**Verification**:
```bash
cargo test to_html_consumer
cargo run --bin wasot -- examples/demo.wasot -o /tmp/test.html && cat /tmp/test.html
```

---

## Milestone 6: HtmlEmitter - HTML Input Parsing

**Goal**: Create HtmlEmitter to parse HTML and emit Wasot events.

### Tasks

- [ ] **6.1** Ensure scraper dependency is added
  - File: `Cargo.toml`
  - Verify: `scraper = "0.20"`

- [ ] **6.2** Create `src/emitters/html_emitter.rs`
  - Create new file
  - Define `HtmlEmitter` struct with fields:
    - `content: String`
    - `bus: Arc<Mutex<Bus<Event>>>`
    - `config: HtmlEmitterConfig`

- [ ] **6.3** Define `HtmlEmitterConfig` struct
  - `extract_main_content: bool` - extract from main/article tags
  - `strip_scripts: bool` - remove script tags
  - `strip_styles: bool` - remove style tags
  - `strip_nav: bool` - remove nav elements

- [ ] **6.4** Implement constructor methods
  - `from_file(path: &str) -> Self`
  - `from_string(content: String) -> Self`
  - `with_config(self, config: HtmlEmitterConfig) -> Self`

- [ ] **6.5** Implement HTML parsing core
  - Use scraper to parse HTML into DOM
  - Walk DOM tree recursively
  - For each element:
    - Emit `BlockStarted` when entering
    - Emit `TextFinished` for text nodes
    - Emit `BlockFinished` when leaving
    - Emit `VoidElement` for void elements

- [ ] **6.6** Implement element-to-block mapping
  - `map_html_element(name: &str) -> String` - map HTML tag to wasot type
  - Extract attributes as kwargs
  - Handle id and class specially

- [ ] **6.7** Implement content extraction (optional feature)
  - Find main content: `<main>`, `<article>`, `[role="main"]`, `.content`
  - Strip unwanted elements: `<script>`, `<style>`, `<nav>`, `<header>`, `<footer>`

- [ ] **6.8** Implement Emitter trait
  - `subscribe(&self) -> Receiver<Event>`
  - `run(self)` - parse and emit events
  - `run_with_context(self, context: &mut Context)`

- [ ] **6.9** Export from `src/emitters/mod.rs` and `src/lib.rs`

**Verification**:
```bash
cargo build
cargo test html_emitter
```

---

## Milestone 7: HtmlEmitter - Comprehensive Tests

**Goal**: Achieve 100% test coverage for HtmlEmitter.

### Tasks

- [ ] **7.1** Test constructor methods
  - [ ] `from_file` with existing file
  - [ ] `from_file` with non-existent file
  - [ ] `from_string` with valid HTML
  - [ ] `from_string` with empty string

- [ ] **7.2** Test basic HTML elements
  - [ ] `<p>text</p>` → BlockStarted + TextFinished + BlockFinished
  - [ ] `<div><p>text</p></div>` → nested events
  - [ ] All semantic elements: h1-h6, section, article, etc.

- [ ] **7.3** Test void elements
  - [ ] `<br>` → VoidElement
  - [ ] `<hr>` → VoidElement
  - [ ] `<img src="x" alt="y">` → VoidElement with kwargs
  - [ ] All 13 void elements

- [ ] **7.4** Test attributes
  - [ ] id attribute → id field
  - [ ] class attribute → class kwarg
  - [ ] data-* attributes → kwargs
  - [ ] Multiple attributes

- [ ] **7.5** Test text handling
  - [ ] Simple text
  - [ ] Text with entities (`&amp;` → `&`)
  - [ ] Whitespace normalization
  - [ ] Text in pre/code (preserve whitespace)

- [ ] **7.6** Test content extraction
  - [ ] Extract from `<main>`
  - [ ] Extract from `<article>`
  - [ ] Extract from `[role="main"]`
  - [ ] Strip scripts
  - [ ] Strip styles
  - [ ] Strip nav

- [ ] **7.7** Test edge cases
  - [ ] Empty document
  - [ ] Document with only whitespace
  - [ ] Malformed HTML (scraper should handle)
  - [ ] Deeply nested structure
  - [ ] Comments (ignored or converted to context lines)

- [ ] **7.8** Test full document structures
  - [ ] Complete HTML page with doctype
  - [ ] HTML fragment
  - [ ] Real-world example (simple blog post structure)

**Verification**:
```bash
cargo test html_emitter -- --nocapture
```

---

## Milestone 8: CLI Integration - HTML to Wasot

**Goal**: Wire up `wasot page.html -t wasot` command.

### Tasks

- [ ] **8.1** Add `process_from_html()` function
  - File: `src/main.rs`
  - Create HtmlEmitter, run, consume events with WasotReconstructor

- [ ] **8.2** Add routing for `(Format::Html, Format::Wasot)`
  - File: `src/main.rs` (in `process_default` match)
  - Route to `process_from_html(file_path, output_path)`

- [ ] **8.3** Add routing for HTML to other formats
  - `(Format::Html, Format::Xml)` - works via event chain
  - `(Format::Html, Format::Jsonl)` - works via event chain
  - `(Format::Html, Format::Html)` - normalize/pretty-print

- [ ] **8.4** Add --extract-content flag (optional)
  - Add CLI flag for content extraction mode
  - Pass to HtmlEmitterConfig

- [ ] **8.5** Add CLI integration tests
  - [ ] `wasot page.html -t wasot` produces valid wasot
  - [ ] `wasot page.html -o output.wasot` detects format
  - [ ] `cat page.html | wasot -f html -t wasot` works with stdin
  - [ ] `wasot page.html -t xml` converts to XML

- [ ] **8.6** Test with real HTML files
  - Create test fixtures in `tests/fixtures/html/`
  - Test conversion of various HTML structures

**Verification**:
```bash
echo "<p>Hello</p>" | cargo run --bin wasot -- -f html -t wasot
cargo run --bin wasot -- tests/fixtures/html/simple.html -t wasot
```

---

## Milestone 9: Roundtrip Tests & Edge Cases

**Goal**: Ensure bidirectional conversion works correctly.

### Tasks

- [ ] **9.1** Create roundtrip test infrastructure
  - File: `tests/html_roundtrip.rs` or in unit tests
  - Helper function: `roundtrip_wasot_html_wasot(input: &str) -> String`
  - Helper function: `roundtrip_html_wasot_html(input: &str) -> String`

- [ ] **9.2** Test Wasot → HTML → Wasot roundtrip
  - [ ] Simple paragraph
  - [ ] Nested blocks
  - [ ] Void elements
  - [ ] Attributes preserved
  - [ ] IDs preserved
  - [ ] Text content preserved

- [ ] **9.3** Test HTML → Wasot → HTML roundtrip
  - [ ] Basic HTML structure
  - [ ] Semantic elements
  - [ ] Void elements
  - [ ] Attributes

- [ ] **9.4** Test edge cases for output
  - [ ] Very long lines
  - [ ] Unicode content
  - [ ] Special characters in attributes
  - [ ] Empty blocks
  - [ ] Blocks with only whitespace

- [ ] **9.5** Test edge cases for input
  - [ ] HTML5 doctype
  - [ ] XHTML-style self-closing tags
  - [ ] Mixed case tags (`<DIV>`)
  - [ ] Unquoted attributes
  - [ ] Missing closing tags

- [ ] **9.6** Performance tests (optional)
  - [ ] Large HTML file parsing
  - [ ] Large wasot to HTML conversion
  - [ ] Memory usage verification

**Verification**:
```bash
cargo test roundtrip
cargo test --test html_roundtrip  # If in tests/ directory
```

---

## Milestone 10: Documentation & Cleanup

**Goal**: Complete documentation and final polish.

### Tasks

- [ ] **10.1** Add module-level documentation
  - `src/consumers/html_listener.rs` - doc comments
  - `src/consumers/to_html_consumer.rs` - doc comments
  - `src/emitters/html_emitter.rs` - doc comments

- [ ] **10.2** Add examples to doc comments
  - Show typical usage patterns
  - Show event sequences

- [ ] **10.3** Update CLI help text
  - Verify HTML appears in format options
  - Add HTML examples to --help output

- [ ] **10.4** Update README.md
  - Add HTML conversion examples
  - Document element mappings

- [ ] **10.5** Update CLAUDE.md if needed
  - Add HTML-related commands

- [ ] **10.6** Run full test suite
  - `cargo test`
  - `cargo test -- --ignored` (slow tests)
  - `./scripts/dev/pre-commit.sh`

- [ ] **10.7** Run clippy and fix warnings
  - `cargo clippy --all-features -- -D warnings`

- [ ] **10.8** Verify no unwrap() in production code
  - Search for unwrap() calls
  - Replace with proper error handling

- [ ] **10.9** Final code review checklist
  - [ ] All tests pass
  - [ ] No compiler warnings
  - [ ] No clippy warnings
  - [ ] Consistent code style
  - [ ] Error messages are helpful

**Verification**:
```bash
./scripts/dev/pre-commit.sh
cargo doc --open
```

---

## Summary: File Changes

### New Files
- `src/consumers/html_listener.rs` (~400-600 lines)
- `src/consumers/to_html_consumer.rs` (~400-600 lines)
- `src/emitters/html_emitter.rs` (~500-800 lines)
- `tests/fixtures/html/*.html` (test fixtures)

### Modified Files
- `Cargo.toml` - add dependencies
- `src/main.rs` - Format::Html, routing, process functions (~150 lines)
- `src/consumers/mod.rs` - export new consumers
- `src/emitters/mod.rs` - export new emitter
- `src/lib.rs` - public API exports

### Test Coverage Checklist
- [ ] HtmlListener: all methods and branches
- [ ] ToHtmlConsumer: all methods and branches
- [ ] HtmlEmitter: all methods and branches
- [ ] CLI: all new command combinations
- [ ] Roundtrip: wasot↔html conversions
- [ ] Edge cases: special characters, empty inputs, malformed HTML

---

## Progress Tracking

| Milestone | Status | Tests Passing | Notes |
|-----------|--------|---------------|-------|
| 1. Dependencies & Format | Complete | 4/4 | Format::Html added, detection works |
| 2. HtmlListener Basic | Complete | 23/23 | Full implementation with all mappings |
| 3. HtmlListener Tests | Complete | 23/23 | All helper methods tested |
| 4. CLI Wasot→HTML | Complete | - | `wasot doc.wasot -t html` works |
| 5. ToHtmlConsumer | Skipped | - | HtmlListener sufficient for now |
| 6. HtmlEmitter Basic | Complete | 12/12 | Full HTML parsing with scraper |
| 7. HtmlEmitter Tests | Complete | 12/12 | Core functionality tested |
| 8. CLI HTML→Wasot | Complete | - | `wasot page.html -t wasot` works |
| 9. Roundtrip Tests | Complete | - | Wasot↔HTML roundtrip verified |
| 10. Documentation | Pending | - | |

---

*Created: 2026-02-06*
*Based on: reports/html.md analysis*
