# Bug Analysis: Standalone Text Not Output

**Date**: 2026-02-03
**Severity**: Medium
**Affects**: JSONL output (and likely other consumers)

## Summary

Text that appears outside of any block produces no output when using the JSONL format (and potentially other consumers). The root cause is a mismatch between how the emitter signals text events and how consumers expect to receive them.

## Reproduction

```bash
# Produces NO output (bug)
echo "text" | cargo run --bin wasot -- -t jsonl

# Produces output (works)
echo ".block\ntext" | cargo run --bin wasot -- -t jsonl
```

## Root Cause Analysis

### Event Flow for Standalone Text

When processing `"text"` (standalone text without a block):

1. **Parser** (`WasotFileEmitter::process_line_with_events()` at `src/emitters/wasot_file_emitter.rs:201-203`):
   - `is_meta_line("text")` returns `false`
   - Line is accumulated via `accumulate_text_line()`

2. **At document end** (`WasotFileEmitter::run()` at line 288):
   - `flush_accumulated_text()` emits a `TextFinished` event
   - **No `BlockStarted` or `BlockFinished` events are emitted**

3. **Consumer** (`ToJsonlConsumer::process_event()` at `src/consumers/to_jsonl_consumer.rs:61-68`):
   ```rust
   Event::TextFinished { lines, links, .. } => {
       // Add text content with links to the current block on the stack
       if let Some(block) = self.block_stack.last_mut() {  // <-- FAILS: stack is empty
           for (i, line) in lines.iter().enumerate() {
               // ...
           }
       }
       // Text is silently dropped if no block on stack
   }
   ```

### Event Sequence Comparison

**Standalone text** (broken):
```
TextFinished { lines: ["text"], ... }   <- orphaned, no block on stack
(end of stream)
```

**Text with block** (works):
```
BlockStarted { meta_line: .block (level=1) }
TextFinished { lines: ["text"], ... }   <- attached to block on stack
BlockFinished
(end of stream)
```

### Why Direct Parsing Works

The `ElementParser` (used in direct parsing mode) handles this correctly via `current_text_block`:

```rust
// src/parsers/wasot/element_parser.rs:206-227
fn process_content_line(&mut self, line: &str) {
    if let Some(current_block) = self.block_stack.last_mut() {
        current_block.add_content_line(line.to_string(), self.current_line_index);
    } else {
        // No block on stack, add to implicit text block
        if self.current_text_block.is_none() {
            self.current_text_block = Some(Block::new_implicit_text(self.current_line_index));
        }
        // ...
    }
}
```

The implicit text block is properly finalized in `finish()` (line 242-254) and added to the container. However, **this mechanism is not mirrored in the event-driven emitter**.

### Why XML Listener Works (Partially)

The `XmlListener` outputs `<text>...</text>` unconditionally at lines 110-128 - it doesn't check for a parent block. However, this produces structurally invalid output (a `<text>` element at root level without a parent `<block>`).

## Affected Consumers

| Consumer | Behavior | Handles Orphaned Text? |
|----------|----------|------------------------|
| `ToJsonlConsumer` | Silently drops text | No |
| `StreamingJsonExporter` | Silently drops text | No |
| `XmlListener` | Outputs `<text>` at root | Partially (invalid structure) |
| `SqliteConsumer` | Likely drops text | Needs verification |
| `SelectorConsumer` | Likely drops text | Needs verification |

## Solution Options

### Option A: Emit Implicit Block Events in Emitter (Recommended)

**Location**: `src/emitters/wasot_file_emitter.rs`

**Approach**: When `flush_accumulated_text()` is called and no block is open, wrap the text in synthetic `BlockStarted`/`BlockFinished` events for an implicit text block.

**Pros**:
- Single fix point - all consumers benefit
- Consistent with how `ElementParser` handles this case
- Maintains event stream semantics (text always belongs to a block)

**Cons**:
- Changes event stream for all consumers
- Synthetic blocks need a distinguishable type (e.g., `type_name = "text"`, `level = 1`)

**Implementation sketch**:
```rust
fn flush_accumulated_text(&mut self) {
    if !self.accumulated_text_lines.is_empty() {
        let at_document_root = self.block_stack.is_empty(); // Need to track this

        if at_document_root {
            // Emit synthetic BlockStarted for implicit text block
            let implicit_meta = MetaLine::new_implicit_text(self.first_text_line_index);
            let event = Event::BlockStarted { meta_line: implicit_meta };
            self.bus.lock().unwrap().broadcast(event);
        }

        // Emit TextFinished as before
        let event = Event::TextFinished { ... };
        self.bus.lock().unwrap().broadcast(event);

        if at_document_root {
            // Emit synthetic BlockFinished
            let event = Event::BlockFinished;
            self.bus.lock().unwrap().broadcast(event);
        }

        // ...
    }
}
```

**Complexity**: Medium
**Risk**: Low (additive change, won't break existing consumers)

---

### Option B: Handle Orphaned Text in Consumers

**Location**: Each consumer implementation

**Approach**: When a `TextFinished` event arrives with an empty block stack, create an implicit block in the consumer.

**Pros**:
- No changes to emitter/event semantics
- Consumers can decide how to handle root-level text

**Cons**:
- Must be implemented in every consumer
- Inconsistent behavior possible across consumers
- Violates single-responsibility (consumers shouldn't synthesize structure)

**Implementation sketch** (for `ToJsonlConsumer`):
```rust
Event::TextFinished { lines, links, .. } => {
    let block = if let Some(block) = self.block_stack.last_mut() {
        block
    } else {
        // Create implicit root-level text block
        let implicit_meta = MetaLine::new_implicit_text(None);
        let implicit_block = Block::new(implicit_meta);
        self.block_stack.push(implicit_block);
        self.block_stack.last_mut().unwrap()
    };
    // ... add text to block
}
```

**Complexity**: High (must update multiple consumers)
**Risk**: Medium (consumers may diverge in behavior)

---

### Option C: New Event Type for Root-Level Text

**Location**: `src/events/events.rs` + all consumers

**Approach**: Add a `RootTextFinished` event type specifically for document-level text.

**Pros**:
- Explicit semantics
- Consumers can handle it appropriately

**Cons**:
- Breaks existing consumer implementations
- More complex event handling everywhere
- Arguably over-engineering

**Complexity**: High
**Risk**: High (breaking change)

---

### Option D: Document Wrapper Events

**Location**: `src/emitters/wasot_file_emitter.rs`

**Approach**: Emit `DocumentStarted`/`DocumentFinished` events, and have the document act as an implicit root block.

**Pros**:
- Clean document model
- Natural place for document-level content

**Cons**:
- Significant API change
- May affect streaming behavior

**Complexity**: High
**Risk**: Medium

## Recommendation

**Option A (Emit Implicit Block Events in Emitter)** is the recommended solution because:

1. It matches the existing behavior of `ElementParser` in direct parsing mode
2. It's a single fix point that benefits all consumers
3. It maintains the invariant that text always belongs to a block
4. It's backward compatible - existing consumers that already handle blocks correctly will work

### Implementation Notes for Option A

1. The emitter needs to track whether the current context is at document root level
2. The `WasotFileEmitter` already has access to `ElementParser` - could potentially reuse its `current_depth()` method, but the emitter manages its own event flow
3. The synthetic block should use `type_name = "text"` and `level = 1` to be consistent with implicit text blocks elsewhere
4. Consider adding a flag to disable this behavior if strict event streaming is needed

## Test Cases to Add

```rust
#[test]
fn test_standalone_text_produces_output() {
    // echo "text" | wasot -t jsonl should produce output
}

#[test]
fn test_text_between_blocks_produces_output() {
    // .block1
    // ./
    // standalone text
    // .block2
}

#[test]
fn test_text_after_all_blocks_produces_output() {
    // .block
    // ./
    // trailing text
}

#[test]
fn test_mixed_text_and_blocks() {
    // leading text
    // .block
    // block text
    // ./
    // trailing text
}
```

## Decision

- [ ] **Option A**: Fix in emitter (recommended) - emit implicit block events around root-level text
- [x] **Option B**: Fix in consumers - handle orphaned text in each consumer ← **IMPLEMENTED**
- [ ] **Option C**: New event type - add `RootTextFinished` event
- [ ] **Option D**: Document wrapper - add `DocumentStarted`/`DocumentFinished` events

---

## Implementation Summary (Option B)

**Date Implemented**: 2026-02-03

### Files Modified

1. **`src/consumers/to_jsonl_consumer.rs`**
   - `ToJsonlConsumer::process_event()`: When `TextFinished` arrives with empty `block_stack`, writes the text as `Element::Text` directly (same format as text children in blocks)
   - `StreamingJsonExporter::process_event()`: Same fix applied
   - Added 3 new tests for orphaned text handling

### Consumers Not Modified (by design)

- **XmlListener**: Already outputs `<text>...</text>` for orphaned text (no wrapper needed)
- **SqliteConsumer**: Already handles orphaned text correctly (passes `None` for `parent_id`)
- **SelectorConsumer**: Works as designed (filter - only emits when matching)
- **WasotReconstructor**: Correct behavior (reconstructing original format)
- **ProtobufConsumer / BinaryProtobufConsumer**: Correct behavior (just serializing events)

### Verification

```bash
# Before fix
echo "standalone text" | wasot -t jsonl
# Output: (empty)

# After fix
echo "standalone text" | wasot -t jsonl
# Output: {"Text":{"from":0,"to":1,"lines":["standalone text"],"links":[[]]}}

# Mixed content works too
printf "leading\n.block\nblock text\n./\ntrailing" | wasot -t jsonl
# Output:
# {"Text":{"from":0,"to":1,"lines":["leading"],"links":[[]]}}
# {"meta_line":{"type_name":"block",...},"children":[{"Text":{...}}],...}
# {"Text":{"from":0,"to":1,"lines":["trailing"],"links":[[]]}}
```

### Tests Added

- `test_to_jsonl_consumer_orphaned_text`
- `test_to_jsonl_consumer_text_before_and_after_block`
- `test_streaming_json_exporter_orphaned_text`
