# Sidecar Architecture: Fix PyO3 Subinterpreter Incompatibility

## Problem

WeeChat Python plugins run inside a **subinterpreter**. The `eclipse-zenoh` library uses PyO3 bindings, and PyO3 does not support subinterpreters ([pyo3#576](https://github.com/PyO3/pyo3/issues/576)). This causes `import zenoh` to fail inside WeeChat:

```
ImportError: PyO3 modules do not yet support subinterpreters
```

## Solution

Split `weechat-zenoh/` into two processes communicating via stdin/stdout JSON Lines protocol.

## Components

### 1. `weechat-zenoh/weechat-zenoh.py` (WeeChat plugin)

Runs inside WeeChat subinterpreter. **Must not import zenoh.**

Responsibilities:
- Launch `zenoh_sidecar.py` via `subprocess.Popen`
  - `stdin=PIPE, stdout=PIPE`
  - `stderr` redirected to `~/.weechat/logs/zenoh_sidecar.log`
  - Script path: resolved relative to plugin's own `__file__` directory
- Monitor sidecar stdout via `weechat.hook_fd()` (non-blocking reads)
  - Maintain internal read buffer to accumulate partial data until `\n` (JSON Lines requires complete lines)
- Translate all Zenoh operations into JSON commands sent to sidecar stdin
- Receive events from sidecar stdout, enqueue into existing `msg_queue` / `presence_queue`
- `poll_queues_cb` logic unchanged — still processes queues every 50ms
- `/zenoh status` uses synchronous request-response (brief blocking read)
- On sidecar crash (EOF on stdout): print error to all buffers, offer `/zenoh reconnect`
- **Plugin maintains local `channels` and `privates` sets** — used for `/zenoh list`, reconnect state restoration, and buffer management

External API unchanged: `join()`, `leave()`, `send_message()`, buffer callbacks, WeeChat signals.

### 2. `weechat-zenoh/zenoh_sidecar.py` (standalone process)

Runs as a normal Python process. **Can freely import zenoh.**

Responsibilities:
- Read JSON commands from stdin (one per line)
- Manage Zenoh session, publishers, subscribers, liveliness tokens
- On Zenoh events: write JSON to stdout (one per line)
- `build_zenoh_config()` logic inlined here (removed from helpers.py)
- On stdin EOF: clean up Zenoh resources and exit gracefully

### 3. `weechat-zenoh/helpers.py` (unchanged, minus one function)

- **Remove** `build_zenoh_config()` (moved to sidecar)
- **Keep** `target_to_buffer_label()`, `parse_input()`, `ZENOH_DEFAULT_ENDPOINT`

## Communication Protocol

JSON Lines (one JSON object per line, `\n` delimited).

### Plugin → Sidecar (stdin)

```json
{"cmd": "init", "nick": "alice", "connect": "tcp/127.0.0.1:7447"}
{"cmd": "join_channel", "channel_id": "general"}
{"cmd": "leave_channel", "channel_id": "general"}
{"cmd": "join_private", "target_nick": "alice"}
{"cmd": "leave_private", "target_nick": "alice"}
{"cmd": "send", "pub_key": "channel:general", "type": "msg", "body": "hello"}
{"cmd": "set_nick", "nick": "newname"}
{"cmd": "status"}
```

### Sidecar → Plugin (stdout)

```json
{"event": "ready", "zid": "..."}
{"event": "message", "target": "channel:general", "msg": {"id":"...","nick":"alice","type":"msg","body":"hi","ts":1234}}
{"event": "presence", "channel_id": "general", "nick": "alice", "online": true}
{"event": "status_response", "zid": "...", "routers": [], "peers": [], "nick": "alice", "channels": 3, "privates": 1}
{"event": "error", "detail": "..."}
```

### Protocol Details

**Startup sequence:**
1. Plugin sends `init` command with nick and connect endpoint
2. Sidecar opens Zenoh session, declares global liveliness token
3. Sidecar emits `ready` event (with zid) — plugin can now send join commands
4. Plugin sends `join_channel`/`join_private` for autojoin targets

**`join_channel` behavior in sidecar:**
- Declares publisher, subscriber, liveliness token for the channel
- Queries current online members via `liveliness().get()`
- Emits `presence` events for each online member (reuses presence event type)
- Publishes "join" message on the channel

**`set_nick` behavior in sidecar:**
- Undeclares and redeclares global liveliness token with new nick
- Undeclares and redeclares all per-channel liveliness tokens
- Broadcasts `nick` type message to all joined channels
- No response event needed (nick changes arrive as regular messages)

**Message `id` and `ts`:** Generated by the sidecar (not the plugin). The `send` command only provides `pub_key`, `type`, and `body`. The sidecar adds `id` (uuid4 hex), `nick`, and `ts` (unix timestamp) before publishing.

**`/zenoh list`:** Handled entirely by the plugin using its local `channels`/`privates` sets. No protocol command needed.

## Sidecar Lifecycle

| Event | Action |
|-------|--------|
| `zc_init()` | Start sidecar, send `init`, wait for `ready` |
| `zc_deinit()` | SIGTERM + wait sidecar |
| Sidecar crash (EOF) | Print error to buffers, enable `/zenoh reconnect` |
| `/zenoh reconnect` | Restart sidecar, send `init`, re-join all channels/privates from local sets |
| Stdin EOF (plugin closes pipe) | Sidecar cleans up Zenoh resources and exits |

## Invariants

These must not change:
- Zenoh topic format (`wc/channels/{id}/messages`, etc.)
- Message JSON format (`{id, nick, type, body, ts}`)
- WeeChat signals (`zenoh_message_received`, `zenoh_message_sent`, `zenoh_presence_changed`)
- `weechat-agent.py` (interacts only via WeeChat commands)
- `helpers.py` public API (minus `build_zenoh_config`)

## Testing

### Sidecar Unit Tests

Test sidecar as a subprocess with stdin/stdout:
- Send JSON commands, assert JSON responses
- Mock zenoh session within sidecar (same pattern as existing `tests/conftest.py`)

### Subinterpreter Compatibility Tests

**All WeeChat plugin tests must run under subinterpreter mode** to match the real WeeChat environment and catch compatibility issues early.

Implementation:
- Use `importlib` or `code.InteractiveInterpreter` to simulate subinterpreter import
- Verify `weechat-zenoh.py` imports succeed without PyO3 errors
- CI step: scan plugin entry files for known PyO3-dependent imports (zenoh, pydantic, etc.) and fail if found

### Existing Tests

- `tests/unit/test_zenoh_config.py` — update: move `build_zenoh_config` tests to sidecar test suite
- `tests/unit/test_zenoh_signals.py` — unchanged (tests pure helpers)
- `tests/integration/` — unchanged (tests real Zenoh pub/sub, not plugin loading)

## Design Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| `/zenoh status` | Synchronous request-response | Simple, blocking is negligible (< 1ms local) |
| Sidecar crash | Error + `/zenoh reconnect` | User controls recovery; avoids complex auto-restart state |
| `build_zenoh_config` | Move to sidecar (inline) | Keep helpers.py as pure functions, no zenoh dependency |
| Sidecar stderr | Log file (`~/.weechat/logs/zenoh_sidecar.log`) | Debugging without cluttering WeeChat buffers |
| Plugin test mode | Subinterpreter | Match real WeeChat environment, catch PyO3 issues in CI |
| Message id/ts generation | Sidecar | Plugin doesn't need uuid/time; keeps `send` command minimal |
| State ownership | Plugin tracks channels/privates sets; sidecar tracks Zenoh resources | Plugin needs local state for `/zenoh list` and reconnect; sidecar owns network state |
