# Unified TUI Design

## Goal

Single pattern for GEPA, RL training, and normal eval:
1. Process emits structured log records via Python `logging`
2. Logging handlers route records to JSONL files (overview + per-sample)
3. TUI tails files and renders progress
4. Completely decoupled - process doesn't know if anyone is watching

## Current state (what's wrong)

Three parallel emission paths that don't compose:

- **`on_chunk` callbacks** (in-process) -> updates `MultiProgress` directly
- **`EventEmitter`** (custom file writer) -> writes `events.jsonl`
- **Python `logging`** (standard) -> consumed by `TrainingMonitor` when JSONL-formatted

`MultiProgress` is embedded inside `evaluate()`. `EventEmitter` is a second logging
system with its own file handle, format, and context variable. The TUI monitor's
`parse_jsonl_line()` sniffs line shape to figure out which system produced it.

Result: summary dashboard only (turns, phase status). No drill-down into running
samples. No token counts. No streaming output.

## Target state

Python logging is the single source of truth. `EventEmitter` and `MultiProgress` deleted.

### Emission

`on_chunk` callback body calls `logger.info()`/`logger.debug()` with structured `extra={}`:

```python
async def on_chunk_with_sample_id(event):
    if isinstance(event, TextDelta):
        logger.debug("text_delta", extra={"sample_id": sid, "turn": turn, "text": event.text})
    elif isinstance(event, LLMCallEnd):
        logger.info("llm_call", extra={"sample_id": sid, "turn": turn,
                     "duration_ms": event.duration_ms, "tokens_in": event.tokens_in,
                     "tokens_out": event.tokens_out, "provider": event.provider})
    elif isinstance(event, ToolExecutionEnd):
        logger.info("tool_execution", extra={"sample_id": sid, "turn": turn,
                     "tool_name": event.tool_name, "duration_ms": event.duration_ms})
```

Wide events pattern: one rich record per meaningful unit of work (per turn, per tool call,
per LLM call). High-cardinality fields (`sample_id`, `turn`), high dimensionality
(tokens, duration, tool name, status).

### Log levels as detail tiers

- **INFO**: `sample_start`, `sample_end`, `turn`, `llm_call`, `tool_execution`, `modal_progress`
- **DEBUG**: `text_delta`, `thinking_delta`, `assistant_message` (full content), `tool_result` (full output)

### Routing via logging handlers

```
Root logger handlers (configured via dictConfig at eval startup):

  1. JSONLFileHandler -> {output_dir}/events.jsonl
     - Filter: INFO+ only
     - Purpose: overview dashboard

  2. SampleRoutingHandler -> {output_dir}/samples/{sample_id}.jsonl
     - Filter: must have sample_id in extra
     - Level: ALL (including DEBUG text deltas)
     - Purpose: per-sample drill-down and streaming

  3. StreamHandler(stderr) -> human-readable
     - For verbose/debug mode

  4. QueueHandler wrapping 1-3 for non-blocking writes
```

`SampleRoutingHandler` is a custom `logging.Handler` that inspects
`record.sample_id` and writes to the appropriate per-sample file.

File handles are opened lazily on first write and closed eagerly when
`sample_end` is seen. Peak open FDs = `max_concurrent` samples, not total.
The `QueueHandler` serializes all writes, so close-then-delete is safe.

```python
class SampleRoutingHandler(logging.Handler):
    def __init__(self, output_dir: Path):
        super().__init__()
        self.output_dir = output_dir
        self._files: dict[str, TextIO] = {}

    def emit(self, record: logging.LogRecord):
        sample_id = getattr(record, "sample_id", None)
        if sample_id is None:
            return
        if sample_id not in self._files:
            path = self.output_dir / "samples" / f"{sample_id}.jsonl"
            path.parent.mkdir(parents=True, exist_ok=True)
            self._files[sample_id] = open(path, "a")
        self._files[sample_id].write(self.format(record) + "\n")
        self._files[sample_id].flush()
        # Close file handle when sample is done
        if record.getMessage() == "sample_end":
            self._files[sample_id].close()
            del self._files[sample_id]

    def close(self):
        """Close all open file handles on shutdown."""
        for f in self._files.values():
            f.close()
        self._files.clear()
        super().close()
```

### Consumption

TUI tails files. Fully decoupled from eval process.

- **Overview**: tail `events.jsonl` -> dashboard with N samples, turns, tokens, current state
- **Drill-down**: select a sample -> tail `samples/{sample_id}.jsonl` -> streaming text, tool calls, results
- **Live streaming**: text deltas are DEBUG-level in per-sample files, TUI renders as streaming text
- **Replay**: same files work for post-hoc viewing

### JSONL record format

All records use Python logging's structure with `extra` fields flattened:

```jsonl
{"timestamp": "2025-01-15T10:23:45Z", "level": "INFO", "logger": "rollouts.eval", "msg": "sample_start", "sample_id": "sample_0001", "name": "Square_matmul"}
{"timestamp": "2025-01-15T10:23:46Z", "level": "INFO", "logger": "rollouts.eval", "msg": "turn", "sample_id": "sample_0001", "turn": 1, "status": "streaming"}
{"timestamp": "2025-01-15T10:23:46Z", "level": "DEBUG", "logger": "rollouts.eval", "msg": "text_delta", "sample_id": "sample_0001", "turn": 1, "text": "Let me analyze"}
{"timestamp": "2025-01-15T10:23:50Z", "level": "INFO", "logger": "rollouts.eval", "msg": "llm_call", "sample_id": "sample_0001", "turn": 1, "duration_ms": 3400, "tokens_in": 2000, "tokens_out": 1200, "provider": "anthropic", "model": "claude-sonnet-4-20250514"}
{"timestamp": "2025-01-15T10:24:35Z", "level": "INFO", "logger": "rollouts.eval", "msg": "tool_execution", "sample_id": "sample_0001", "turn": 1, "tool_name": "bash", "duration_ms": 45000, "result_summary": "exit 0"}
{"timestamp": "2025-01-15T10:24:36Z", "level": "INFO", "logger": "rollouts.eval", "msg": "modal_progress", "sample_id": "sample_0001", "phase": "compiling"}
{"timestamp": "2025-01-15T10:30:00Z", "level": "INFO", "logger": "rollouts.eval", "msg": "sample_end", "sample_id": "sample_0001", "score": 0.85, "duration_s": 120, "turns_used": 15}
```

One format. No type sniffing. `msg` field is the event type discriminator.

## Design decisions

### EventEmitter is NOT already using Python logging

Despite having `logger = logging.getLogger(__name__)` at the top of `events.py`,
`EventEmitter` writes JSON directly to its own file handle (`self._file.write(json.dumps(event))`).
It has its own context variable (`_emitter_ctx`), its own file lifecycle, and its own format
(`{"type": "sample_start", ...}` vs logging's `{"message": "...", "level": "INFO", ...}`).

`LoggingEventEmitter` exists as a subclass that also logs, but the base class is a
completely separate channel. Phase 2 is a genuine replacement, not just a consolidation.

### Phase 2 scope: what actually changes in on_chunk

The current `on_chunk_with_sample_id` (evaluation.py:1006) does three things per event:

1. Updates `MultiProgress` via `progress.update_task()` (display)
2. Calls `emit_event()` which writes to `EventEmitter`'s file handle (persistence)
3. Wraps the event with `sample_id` and forwards to `base_on_chunk` (frontend streaming)

The conversion:
- (1) is deleted entirely - no more `MultiProgress`
- (2) becomes `logger.info()`/`logger.debug()` calls with `extra={}`. The existing
  event mapping logic (`LLMCallEnd` -> `llm_call`, `ToolExecutionEnd` -> `tool_execution`,
  `TextEnd` -> `assistant_message`) is preserved, just targeting logging instead of `emit_event()`
- (3) stays for now - `base_on_chunk` is used by `TUIFrontend` for interactive single-agent
  mode. It can be removed later when the TUI reads from files instead of callbacks.
- NEW: `logger.debug("text_delta", ...)` for streaming content (not emitted today)

The `last_status` dedup logic and `current_turn` tracking stay - they're still needed
to avoid flooding the log with redundant status updates.

### TUI backward compatibility

OK to break the old `events.jsonl` format. This is an internal tool, not a user-facing API.
Old eval result directories will have the old format, but we don't need to support replaying
them - the per-sample result JSON files and trajectories are the durable record.

### Testing strategy

- **Phase 1** (SampleRoutingHandler): Unit tests. Write log records with `extra={"sample_id": "x"}`,
  assert correct files created, correct content, FD cleanup on `sample_end`.
- **Phase 2** (on_chunk conversion): Integration test. Run a small eval (1-2 samples, 2-3 turns),
  assert `events.jsonl` and `samples/*.jsonl` have expected records with expected fields.
  Compare against a snapshot of the current `emit_event` output to verify no data loss.
- **Phase 3** (TUI): Manual verification. The TUI is a visual tool - automated tests for
  ANSI rendering are brittle and low-value. Test the JSONL parsing logic (the `parse_jsonl_line`
  equivalent) with unit tests against example records.

## Existing infrastructure

Most of Phase 1 already exists in `rollouts/_logging/`:

- `json_formatter.py`: `JSONFormatter` that flattens `extra` fields onto JSONL records.
  Already strips builtin `LogRecord` attrs and includes only custom extras.
- `logging_config.py`: `setup_logging()` using `dictConfig` with `QueueHandler`+`QueueListener`
  (mCoding pattern), `RotatingFileHandler` for bounded JSONL files, `atexit` cleanup.
- `color_formatter.py`: ANSI color formatter for human-readable stderr output.

What's missing: `SampleRoutingHandler` and a `setup_eval_logging()` that wires it
into the existing `setup_logging()` config.

## Implementation plan

### Phase 1: Logging infrastructure

- Write `SampleRoutingHandler` (routes records to per-sample files based on `extra["sample_id"]`)
- Extend `setup_logging()` or write `setup_eval_logging(output_dir)` that adds the routing handler
  to the existing dictConfig alongside the overview JSONL file handler
- Tests for routing handler

### Phase 2: Replace EventEmitter in evaluate_sample

- Convert `on_chunk_with_sample_id` to use `logger.info()`/`logger.debug()` instead of `emit_event()` + `progress.update_task()`
- Convert `sample_start`/`sample_end` emissions to logging calls
- Remove `EventEmitter` creation from `evaluate()`
- Remove `MultiProgress` creation from `evaluate()`
- Call `setup_eval_logging()` at the start of `evaluate()` instead

### Phase 3: Update TUI to consume new format

- Update `TrainingMonitor.parse_jsonl_line()` to handle unified format (simplify)
- Add drill-down: selecting a sample tails its per-sample JSONL file
- Add token counts to overview display (data now available from `llm_call` events)
- Update `ProgressDisplay` or replace with new consumer

### Phase 4: Cleanup

- Delete `EventEmitter` class and gut `events.py`
- Delete `MultiProgress` class
- Delete `ProgressDisplay`
- Update eval configs that reference `show_progress`, `verbose`, `EventEmitter`

## File structure (target)

```
rollouts/
  _logging/
    json_formatter.py     # ALREADY EXISTS - JSONFormatter with extra flattening
    color_formatter.py    # ALREADY EXISTS - ANSI color formatter
    logging_config.py     # ALREADY EXISTS - setup_logging() with dictConfig + QueueHandler
    sample_handler.py     # NEW - SampleRoutingHandler
  evaluation.py           # Uses logger (no MultiProgress, no EventEmitter)
  tui/
    watch.py              # Main TUI entry point (tails JSONL files)
    monitor.py            # TrainingMonitor (simplified parse_jsonl_line)
    terminal.py           # Terminal abstraction
    traces.py             # Trace viewer (drill-down into sample)
```

## Code style principles for this refactor

From ~/research/docs/code_style/:

**Wide events** (logging_sucks.md): One rich record per meaningful unit of work. High-cardinality
fields (`sample_id`, `turn`) make records queryable. High dimensionality (tokens, duration,
tool name) means you can answer debugging questions without a second search. Don't scatter
context across 20 log lines - build the event throughout the request lifecycle, emit once.

**Python logging** (mcoding_logging_dense.md): `dictConfig` for configuration, not code.
Handlers on root logger, let propagation work. `extra={}` for structured fields. Custom
JSON formatter writes `.jsonl`. `QueueHandler` for non-blocking writes. Library code
shouldn't configure logging - the eval runner configures, rollouts framework just logs.

**Functional core, imperative shell** (favorites.md, code_philosophy_reference.md):
`SampleRoutingHandler` is imperative shell (manages file handles, does I/O).
`JSONFormatter._prepare_log_dict()` is functional core (record -> dict, pure transform).
`on_chunk` mapping events to log calls is pure mapping, no state.

**Don't reuse until 2+ examples** (casey_semantic_compression): Don't pre-abstract the
handler routing. Start with `SampleRoutingHandler` for evals. If GEPA or RL training
need similar per-entity routing, extract the pattern then.

**Classes for resources, functions for orchestration** (code_philosophy_reference.md):
`SampleRoutingHandler` is a class because it owns file handles (resource lifecycle).
`setup_eval_logging()` is a function. The `on_chunk` -> `logger.info()` mapping is a function.

**Parse at the boundary** (tiger_style): The TUI's JSONL parser is the boundary.
Internally, records are just dicts with a known schema. The `msg` field is the event
type discriminator - no polymorphic sniffing needed.

**Existing infra** (nmoe_experiment_tracking.md): We already have `QueueHandler`+`QueueListener`
(nmoe doesn't), `JSONFormatter` with extra flattening, and `RotatingFileHandler` for bounded
files. `SampleRoutingHandler` is the only new piece.

## References

- Wide events / canonical log lines: https://loggingsucks.com/
- Python logging best practices: mCoding (dictConfig, QueueHandler, JSON formatter, extras)
- Code philosophy: ~/research/docs/code_style/favorites.md, code_philosophy_reference.md
- nmoe analysis: ~/research/docs/code_style/nmoe_experiment_tracking.md
- Existing logging infra: rollouts/_logging/ (json_formatter.py, logging_config.py)
- Library code shouldn't configure logging - eval runner configures, rollouts framework just logs
