# Wasot CLI Design

This document describes the design principles for the [CLI](./cli.md). For shell completion setup, see [shell completions](./shell_completions.md).

Wasot is a filter. Like `jq` for JSON or `xmllint` for XML, `wasot` transforms structured documents (see [data model](./wasot_and_xml.md)). It reads from stdin or files, writes to stdout or files, and composes with pipes.

```bash
wasot < in.wasot > out.wasot
cat doc.wasot | wasot -t xml | xmllint --format -
wasot -t jsonl doc.wasot | jq -s 'group_by(.type)'
find . -name "*.wasot" | xargs wasot check
```

---

## Design Principles

1. **Simple over complex** — Easy things should be easy
2. **Discoverable** — `wasot --help` teaches you the tool
3. **Composable** — Works with pipes and standard Unix tools
4. **Predictable** — Same input + same options = same output

---

## Interface

```
wasot [OPTIONS] [FILE]...        Convert/transform (default)
wasot check [FILE]...            Validate syntax
wasot info [FILE]                Show document statistics
wasot tools <COMMAND>            Document processing tools
wasot completions <SHELL>        Generate shell completions
```

If FILE is `-` or omitted, reads from stdin.

### Files vs Subcommands

If a file has the same name as a subcommand (e.g., a file named `tools` or `check`), use one of these approaches:

```bash
wasot ./tools                    # Relative path
wasot /path/to/tools             # Absolute path
wasot -- tools                   # Use -- to stop command parsing
```

The `--` separator tells the CLI that everything after it is a file argument, not a subcommand.

### Conversion Options

These options apply only to the default conversion command, not to subcommands:

| Short | Long | Description |
|-------|------|-------------|
| `-f` | `--from FORMAT` | Input format (auto-detected if omitted) |
| `-t` | `--to FORMAT` | Output format (default: wasot) |
| `-o` | `--output FILE` | Output file (default: stdout) |
| `-q` | `--quiet` | Suppress warnings |
| `-v` | `--verbose` | Verbose output to stderr |
| `-h` | `--help` | Show help |
| `-V` | `--version` | Show version |
| | `--color WHEN` | Color mode: auto, always, never |
| | `--select SELECTOR` | Filter blocks matching CSS-like [selector](./selectors.md) |

### Formats

| Format | Extension | Description |
|--------|-----------|-------------|
| `wasot` | `.wasot` | Wasot markup (default) |
| `xml` | `.xml` | XML representation |
| `jsonl` | `.jsonl` | JSON Lines (one event per line) |
| `binary` | `.wasotb` | Protobuf binary encoding |
| `sqlite` | — | SQLite database (requires `--features sqlite`) |

---

## Subcommands

### Default (Convert/Normalize)

```bash
wasot doc.wasot                  # Normalize (pretty-print)
wasot doc.wasot -t xml           # Convert to XML
wasot doc.wasot -o doc.xml       # Format inferred from extension
wasot -f binary data.wasotb      # Explicit input format
```

### check

Validate syntax. Exit 0 if valid, 1 if invalid.

```bash
wasot check doc.wasot            # Single file
wasot check *.wasot              # Multiple files
wasot check                      # Read from stdin
```

### info

Show document statistics: block counts by type, max depth, size.

```bash
$ wasot info doc.wasot
blocks: 1234
  document: 1
  section: 45
  paragraph: 892
depth: 6
lines: 8456
bytes: 239821
```

### completions

Generate shell completion scripts.

```bash
wasot completions bash > /etc/bash_completion.d/wasot
wasot completions zsh > ~/.zsh/completions/_wasot
wasot completions fish > ~/.config/fish/completions/wasot.fish
```

### tools

Document processing tools with nested subcommands.

#### tools document-id

Extract document IDs from Wasot files or directories. Recursively finds all `.wasot` files in directories. Files are processed in parallel for speed, and output is sorted alphabetically by path.

**Output format:** `<document-id> <relative-path>`

Files without an ID produce no output (not an error).

```bash
wasot tools document-id doc.wasot           # Single file
wasot tools document-id docs/               # All .wasot files in directory
wasot tools document-id src/ lib/ file.wasot  # Multiple paths
wasot tools document-id . | grep "^abc"     # Find IDs starting with "abc"
```

Exit codes:
- 0: Success (even if no IDs found)
- 2: Path not found

Example:
```bash
$ wasot tools document-id examples/
abc123123 examples/document-id.wasot

$ wasot tools document-id myproject/
doc-001 myproject/intro.wasot
doc-002 myproject/chapter1.wasot
doc-003 myproject/appendix/notes.wasot
```

Example document with ID:
```wasot
./id my-unique-doc-123
.title My Document ..
Content here.
```

---

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | Success |
| 1 | Invalid input (parse error, validation failure) |
| 2 | I/O error (file not found, permission denied) |
| 3 | Invalid arguments (unknown format, bad selector) |

---

## Stdout vs Stderr

| Stream | Content |
|--------|---------|
| stdout | Converted output, info output, selected blocks |
| stderr | Errors, warnings, verbose messages |

---

## Format Auto-Detection

### Input Format Detection

When `-f` is omitted, input format is detected:

**By extension:**
```
.wasot  → wasot
.xml    → xml
.jsonl  → jsonl
.wasotb → binary
```

**By content (for stdin or unknown extension):**

1. Read first ~256 bytes
2. If not valid UTF-8 → `binary`
3. Skip whitespace (spaces, tabs, newlines)
4. Check first non-whitespace character:
   - `.` or `#` → `wasot`
   - `<` → `xml`
   - `{` or `[` → `jsonl`
   - Letter → `wasot` (text content)
   - Otherwise → `binary`

### Output Format Detection

When `-t` is omitted but `-o` is given, format is detected from output file extension.

When both `-t` and `-o` are omitted, output defaults to wasot (normalization).

---

## Selector Filtering

The `--select` flag filters blocks matching a CSS-like selector:

```bash
wasot --select "paragraph" doc.wasot              # Select paragraphs
wasot --select "code" doc.wasot -t jsonl          # Select code, output as JSONL
wasot --select "section" doc.wasot -t xml         # Select sections, output as XML
wasot --select "footer" doc.wasot -o out.wasot    # Select footer, write to file
```

Selectors work with all output formats (wasot, xml, jsonl, binary).

---

## Examples

### Basic Conversion

```bash
# Normalize wasot (parse and re-emit)
wasot doc.wasot

# Convert to different formats
wasot doc.wasot -t xml
wasot doc.wasot -t jsonl
wasot doc.wasot -t binary -o doc.wasotb

# Format from output extension
wasot doc.wasot -o doc.xml
wasot doc.wasot -o doc.jsonl
```

### Pipelines

```bash
# Stream to jq
wasot doc.wasot -t jsonl | jq 'select(.meta_line.type_name == "code")'

# Transform with XSLT
wasot doc.wasot -t xml | xsltproc transform.xsl - > out.html

# Chain with other tools
cat file.wasot | wasot -t xml | xmllint --format -
```

### Validation

```bash
# Check single file
wasot check doc.wasot && echo "valid"

# Check multiple files
wasot check *.wasot

# In CI/scripts
find . -name "*.wasot" -exec wasot check {} +
```

### Roundtrip

```bash
# wasot → binary → wasot
wasot doc.wasot -t binary | wasot

# wasot → jsonl → wasot
wasot doc.wasot -t jsonl | wasot -f jsonl
```

### Batch Processing

```bash
# Convert all wasot files to XML
for f in *.wasot; do wasot "$f" -o "${f%.wasot}.xml"; done

# Parallel conversion with GNU parallel
find . -name "*.wasot" | parallel wasot {} -o {.}.xml

# Validate all files
find . -name "*.wasot" | xargs wasot check
```

---

## Testing

### Running CLI Tests

The project includes 102 integration tests for the CLI in `tests/cli_tests.rs`.

**Run all CLI tests:**

```bash
./scripts/test_cli.sh
```

**Run tests matching a filter:**

```bash
./scripts/test_cli.sh selector     # Only selector tests
./scripts/test_cli.sh check        # Only check subcommand tests
./scripts/test_cli.sh basic        # Only basic operation tests
```

### Testing a Pre-built Binary

To test a specific binary instead of building with cargo:

```bash
./scripts/test_cli_binary.sh ./target/release/wasot
./scripts/test_cli_binary.sh /usr/local/bin/wasot
./scripts/test_cli_binary.sh ./target/release/wasot selector  # With filter
```

The binary version is significantly faster (~0.2s vs ~2s) since it skips compilation.

### Environment Variable

Tests can be configured to use a custom binary via environment variable:

```bash
WASOT_TEST_BINARY=/path/to/wasot cargo test --test cli_tests
```

### Test Categories

| Category | Count | Description |
|----------|-------|-------------|
| Basic Operations | 12 | Normalize, convert, output |
| Stdin/Stdout | 4 | Pipe handling |
| Format Conversion | 5 | Format matrix |
| Auto-Detection | 6 | Extension and content detection |
| Check Subcommand | 7 | Validation |
| Info Subcommand | 4 | Statistics |
| Selector | 8 | `--select` filtering |
| Exit Codes | 6 | Return code verification |
| Edge Cases | 10 | Empty files, deep nesting |
| Help/Version | 6 | CLI interface |
| Completions | 3 | Shell completions |
| Error Handling | 5 | Error messages, stderr |
| Idempotency | 2 | Normalize twice = same result |
| Combinations | 4 | Multiple features together |
| Tools (document-id) | 21 | Document ID extraction |

### Manual Testing

```bash
# Basic functionality
wasot --help
wasot examples/demo.wasot
wasot examples/demo.wasot -t xml
wasot examples/demo.wasot -t jsonl | jq .

# Subcommands
wasot check examples/*.wasot
wasot info examples/demo.wasot

# Selectors
wasot --select "paragraph" examples/demo.wasot
wasot --select "footer" examples/demo.wasot -t jsonl

# Exit codes
wasot nonexistent.wasot; echo "Exit: $?"           # Should be 2
wasot examples/demo.wasot > /dev/null; echo "Exit: $?"  # Should be 0

# Tools
wasot tools --help
wasot tools document-id examples/
wasot tools document-id examples/ src/
```

---

## Implementation Status

| Feature | Status |
|---------|--------|
| Basic conversion (`-t`, `-o`) | ✅ Implemented |
| Auto-detect from extension | ✅ Implemented |
| Auto-detect from content | ✅ Implemented |
| `wasot check` | ✅ Implemented |
| `wasot info` | ✅ Implemented |
| `--select` | ✅ Implemented |
| `--select` + `-t` combination | ✅ Implemented |
| Exit codes | ✅ Implemented |
| SIGPIPE handling | ✅ Implemented |
| `--quiet` flag | ⚠️ Parsed but not used |
| `--verbose` flag | ⚠️ Parsed but not used |
| `--color` flag | ⚠️ Parsed but not used |
| SQLite support | ✅ With `--features sqlite` |

---

## Non-Goals

- **Watch mode** — Use external tools (entr, watchexec)
- **In-place editing** — Use shell: `wasot f > tmp && mv tmp f`
- **Recursive directory processing** — Use find/xargs/parallel
- **Config files** — Behavior determined entirely by arguments
- **Plugins** — Use pipes to external tools
