# Wasot architecture

Wasot is a file format for humans and tools to collaborate, following the [design principles](./principles.md).
Thus it needs to easily readable and writeable for both.

To make our life easier, there are multiple different representations of the same file.
There is the default `.wasot` file, which implements the Wasot-syntax (see [data model](./wasot_and_xml.md)).
To make working on it easier, there is also a Sqlite version of the same file.
This enables tools to use SQL on the files.
It is possible to read Wasot files, that are bigger than available memory (see [memory analysis](./memory_analysis.md)).
This works as Wasot is a line-based language, operating on one line at a time.

## Wasot is Event-Based

From a group of lines, an event is created and sent.
For example when a new block is started or the content of a block is finished (see [AST](./ast.md) for the data structures and [events](./events.md) for the complete event reference).
Thus Wasot is an event based system.

The core event types are:
- **BlockStarted** - A new block begins (with MetaLine)
- **BlockFinished** - A block ends
- **TextFinished** - Text content is complete (with lines, links, footnotes)
- **ContextLine** - Metadata that doesn't participate in content hierarchy
- **VoidElement** - Self-closing element like `<hr>` or `<img>`

To enable faster subsequent reads, Wasot also supports a cache format, like languages like python do.
These are just files storing the events sent by the Wasot parser.
To safe time, you can just read and trigger them instead of parsing the Wasot file again.
Therefore, you can think of a Wasot file just as a list of events, which are strictly
sorted by the line number from which the event is emitted.

## Emitters and Consumers

So Wasot is a series of events. These are emitted by different types of emitters.
One is the Wasot-file parser, which reads a file line by line, collects them into blocks and
emits events (BlockStarted, TextFinished, BlockFinished, ContextLine, VoidElement).
We can collect these events into a cache-file and replay them from there to get faster.
This happens via another emitter. Emitters are things, that produce the events defining a Wasot file.

Consumers are the target of these events.
When we write a tool, there is one emitter, getting the events from somewhere and sending them
to one or more consumers which do stuff with them.
They can either produce a output-file in another format, e.g. the cache file or a Sqlite-Database,
or do other things with the events.

Emitters and Consumers are coupled loosely using broadcast-channels.
An emitter creates it's channel on creation, and the user (for example a cli-tool) takes this
channel and passes it to the constructor of a consumer.
Adding Emitters and Consumers is easy. Emitters just have to provide a event-stream, Consumers only
have to be able to handle one.

## Transformers: Event Stream Processing

While emitters produce events and consumers process them, transformers act as intermediate processors in the event pipeline. A transformer implements both the Consumer and Emitter traits, allowing it to:

1. **Consume** events from an upstream emitter
2. **Process** or filter those events based on internal logic
3. **Re-emit** selected or modified events to downstream consumers

### Conceptual Model

```
Emitter → Transformer → Consumer
   ↓          ↓↑          ↓
events    consume    events
         transform
         re-emit
```

Transformers enable powerful composition patterns:
- **Filtering**: Only pass through events matching certain criteria
- **Enrichment**: Add metadata or context to events before re-emitting
- **Aggregation**: Collect multiple events and emit computed results
- **Routing**: Direct events to different downstream paths based on content

### Example: Selector Transformer

A selector transformer maintains an internal stack representation of the document structure. As it consumes events:
1. Updates its stack when blocks start/end
2. Checks if the current stack matches any configured selectors
3. Only re-emits events when inside matching elements

This allows downstream consumers to receive only the subset of events they care about, without needing to implement selector logic themselves.

### Transformer Characteristics

- **Stateful**: May maintain internal state between events
- **Composable**: Can chain multiple transformers together
- **Selective**: Choose which events to pass through
- **Memory-bounded**: Process events in streaming fashion

Transformers make Wasot's event architecture extremely flexible, allowing complex processing pipelines while maintaining the streaming, memory-efficient nature of the system.

### Aside: Emitter Composition

An important architectural principle: **an emitter combined with transformers is still an emitter**. This composition property enables powerful patterns, see [here](./emitter_composition.md). 

## Directory structure

- src
    - emitters
        - Everything which implements an emitter
        - emitter.rs -> The trait is moved here from facade.rs
        - parser
            - all files of the parser
    - consumers
        - Everything that implements a consumer
        - consumer.rs -> The trait is moved here from facade.rs
    - transformers
        - Components that are both consumers and emitters
        - transformer.rs -> Base trait combining Consumer + Emitter
        - selector_transformer.rs -> Filters events based on selectors
        - Examples of other transformers
    - events/
        - events.rs -> Contains the Event enum and Emitter/Consumer traits
        - See [events](./events.md) for complete documentation
    - bin
        - main.rs
        - playground.rs