# LXMF Messaging Fix

## Problem

Messages could not be sent to discovered Styrene nodes with error:
```
ValueError: Cannot send to 9e21f49afdaae888...: identity 698f2232d4ddab45... not known to RNS.
```

## Root Cause

The system was conflating two different destination types:
1. **Operator Destination** (`styrene_node.operator`) - Used for node identification and RPC
2. **LXMF Delivery Destination** (`lxmf.delivery`) - Used for messaging

Both destinations share the same underlying RNS.Identity (public/private key pair), but have different destination hashes because they include different app names and aspects in the hash calculation.

The original implementation:
- Stored only the operator destination hash in NodeStore
- Attempted to send messages using that operator destination
- Failed because LXMF messages must be sent to LXMF delivery destinations

## Solution

### 1. Separate Destination Types in Data Model

Added `lxmf_destination_hash` field to `MeshDevice`:
```python
@dataclass
class MeshDevice:
    destination_hash: str          # Operator destination (for identification)
    identity_hash: str             # Shared identity (for Identity.recall)
    lxmf_destination_hash: str | None  # LXMF destination (for messaging)
    # ... other fields
```

**Why separate?**
- `destination_hash`: Primary key, used for node identification, RPC routing
- `identity_hash`: Used for `RNS.Identity.recall()` (shared across all destinations)
- `lxmf_destination_hash`: Used to verify LXMF capability and for routing messages

### 2. Include LXMF Destination in Announces

Updated Styrene node announce format from:
```
styrene:<hostname>:<version>:<capabilities>
```

To:
```
styrene:<hostname>:<version>:<capabilities>:<lxmf_destination_hash>
```

**Implementation** (`src/styrene/services/app_lifecycle.py:226-273`):
```python
def _announce_styrene_node(self, destination):
    # Get LXMF delivery destination hash if available
    lxmf_service = get_lxmf_service()
    if lxmf_service.is_initialized and lxmf_service._identity:
        delivery_dest = RNS.Destination(
            lxmf_service._identity,
            RNS.Destination.IN,
            RNS.Destination.SINGLE,
            "lxmf",
            "delivery",
        )
        lxmf_dest = delivery_dest.hash.hex()

    app_data = f"styrene:{hostname}:{version}:{caps}:{lxmf_dest}".encode()
    destination.announce(app_data=app_data)
```

### 3. Parse and Store LXMF Destination

Updated `parse_announce_data()` to extract LXMF destination:
```python
def parse_announce_data(app_data):
    # Returns: (name, device_type, capabilities, version, lxmf_destination_hash)
    parts = decoded.split(":")
    lxmf_dest = parts[4] if len(parts) > 4 and parts[4] else None
    return (name, DeviceType.STYRENE_NODE, capabilities, version, lxmf_dest)
```

### 4. Update Database Schema

Added `lxmf_destination_hash` column to `nodes` table:
```sql
ALTER TABLE nodes ADD COLUMN lxmf_destination_hash TEXT;
```

**Migration** (`src/styrene/services/node_store.py:86-104`):
- Schema creation includes new column
- Migration code handles existing databases
- Uses try/except to skip if column already exists

### 5. Fix Message Sending Logic

Updated `ChatProtocol.send_message()` to:
1. Look up node by operator destination hash
2. Verify LXMF capability (lxmf_destination_hash present)
3. Use **identity_hash** for `RNS.Identity.recall()` (not LXMF destination hash)
4. Create LXMF delivery destination from recalled identity

**Key insight**: `RNS.Identity.recall()` needs the **identity hash** (public key hash), not the destination hash (identity + app + aspects).

## Files Modified

| File | Changes | Purpose |
|------|---------|---------|
| `src/styrene/models/mesh_device.py` | Added `lxmf_destination_hash` field | Data model |
| `src/styrene/models/mesh_device.py` | Updated `parse_announce_data()` return type | Parsing |
| `src/styrene/models/mesh_device.py` | Updated all return statements | 5-tuple support |
| `src/styrene/services/app_lifecycle.py` | Updated `_announce_styrene_node()` | Include LXMF dest |
| `src/styrene/services/node_store.py` | Added `lxmf_destination_hash` column | Schema |
| `src/styrene/services/node_store.py` | Updated `save_node()` INSERT/UPDATE | Persistence |
| `src/styrene/services/node_store.py` | Updated `_row_to_device()` | Deserialization |
| `src/styrene/protocols/chat.py` | Updated `send_message()` logic | Message routing |
| `src/styrene/__main__.py` | Added logging configuration | Debugging |

## Migration Plan

### Phase 1: Backward Compatibility (Current)

**Status**: ✅ Implemented

- New fields are optional (`lxmf_destination_hash: str | None`)
- Old announces without LXMF destination continue to work
- Database migration handles existing tables
- Nodes without LXMF capability show appropriate error message

**Behavior**:
- Legacy nodes (no LXMF dest): Identified but messaging disabled
- New nodes (with LXMF dest): Full messaging capability
- Mixed network: Works correctly

### Phase 2: Data Migration (Automatic)

**Status**: ✅ Automatic

When existing databases are opened:
1. `ALTER TABLE` adds `lxmf_destination_hash` column (if missing)
2. Existing rows have `NULL` for new column
3. Next announce from each node updates with LXMF destination
4. No manual migration required

**Timeline**: Immediate on first launch with new code

### Phase 3: Validation (Ongoing)

**Status**: 🟡 In progress

Monitor for nodes with and without LXMF capability:
```python
# Check LXMF coverage
nodes = get_node_store().get_styrene_nodes()
with_lxmf = sum(1 for n in nodes if n.lxmf_destination_hash)
without_lxmf = len(nodes) - with_lxmf
logger.info(f"LXMF coverage: {with_lxmf}/{len(nodes)} nodes")
```

### Phase 4: Cleanup (Future)

**Timeline**: After all nodes updated (30+ days)

Considerations:
- Make `lxmf_destination_hash` required (not optional)
- Remove legacy announce format parsing
- Add schema version to track migrations

**Not urgent**: Current code handles both old and new formats gracefully.

## Test Plan

### Unit Tests

#### Test 1: Announce Parsing
```python
def test_announce_parsing():
    # Test legacy format (no LXMF)
    data = b"styrene:host:0.1.0:node:"
    name, dtype, caps, ver, lxmf = parse_announce_data(data)
    assert lxmf is None

    # Test new format (with LXMF)
    data = b"styrene:host:0.1.0:node:abc123"
    name, dtype, caps, ver, lxmf = parse_announce_data(data)
    assert lxmf == "abc123"
```

#### Test 2: Database Migration
```python
def test_schema_migration():
    # Create old schema (no lxmf column)
    conn = sqlite3.connect(":memory:")
    conn.execute("CREATE TABLE nodes (destination_hash TEXT PRIMARY KEY)")
    conn.close()

    # Open with new code (should add column)
    store = NodeStore(db_path=":memory:")
    # Verify column exists
    # ...
```

#### Test 3: NodeStore Persistence
```python
def test_lxmf_destination_storage():
    store = NodeStore(db_path=":memory:")
    device = MeshDevice(
        destination_hash="abc123",
        identity_hash="def456",
        lxmf_destination_hash="ghi789",
        # ... other fields
    )

    store.save_node(device)
    retrieved = store.get_node_by_destination("abc123")
    assert retrieved.lxmf_destination_hash == "ghi789"
```

### Integration Tests

#### Test 4: End-to-End Message Send
```python
@pytest.mark.integration
async def test_message_send_with_lxmf():
    # Setup: Initialize TUI with LXMF
    lifecycle = StyreneLifecycle(config)
    lifecycle.initialize()

    # Create test node with LXMF capability
    node = MeshDevice(
        destination_hash="test_dest",
        identity_hash="test_identity",
        lxmf_destination_hash="test_lxmf_dest",
        device_type=DeviceType.STYRENE_NODE,
    )
    get_node_store().save_node(node)

    # Attempt to send message
    chat = ChatProtocol(engine, identity)
    try:
        await chat.send_message("test_dest", "Hello")
        # Should fail with specific error about identity not recalled
        # (unless we mock RNS.Identity.recall)
    except ValueError as e:
        assert "not known to RNS" in str(e)
```

#### Test 5: Backward Compatibility
```python
def test_legacy_node_without_lxmf():
    # Create node without LXMF destination
    node = MeshDevice(
        destination_hash="legacy_dest",
        identity_hash="legacy_identity",
        lxmf_destination_hash=None,  # No LXMF
        device_type=DeviceType.STYRENE_NODE,
    )
    get_node_store().save_node(node)

    # Attempt to send message should fail gracefully
    chat = ChatProtocol(engine, identity)
    with pytest.raises(ValueError, match="does not have LXMF capability"):
        await chat.send_message("legacy_dest", "Hello")
```

### Manual Tests

#### Test 6: Live Messaging Between Nodes

**Setup**:
1. Q502 node (styrene-node.vanderlyn.local) running daemon
2. MacBook running TUI

**Steps**:
1. Launch TUI: `make run`
2. Check logs: `tail -f /tmp/styrene-tui.log`
3. Verify LXMF initialized:
   ```
   LXMF initialized and announced (delivery: 4f17a08ffd4119d4...)
   ```
4. Wait for Q502 discovery (appears in Mesh Devices table)
5. Open conversation with Q502 (press `m` on selected node)
6. Send test message: "Hello from TUI"
7. Check Q502 daemon logs for receipt
8. Verify auto-reply received in TUI

**Expected Results**:
- ✅ LXMF initializes successfully
- ✅ Q502 node discovered with LXMF destination
- ✅ Message sends without error
- ✅ Q502 receives message (logged)
- ✅ Auto-reply received in TUI

**Debugging**:
If message send fails:
```bash
# Check LXMF destination in announce
grep "Including LXMF dest" /tmp/styrene-tui.log

# Check node data in database
sqlite3 ~/Library/Application\ Support/styrene/nodes.db \
  "SELECT destination_hash, identity_hash, lxmf_destination_hash FROM nodes WHERE device_type='styrene_node'"

# Check Q502 daemon logs
ssh styrene@styrene-node.vanderlyn.local "tail -50 /tmp/styrene-daemon.log | grep -E 'LXMF|message'"
```

#### Test 7: Connection Health Monitoring

**Steps**:
1. Launch TUI
2. Let run for 5 minutes
3. Check connection count:
   ```bash
   lsof ~/Library/Application\ Support/styrene/nodes.db | wc -l
   ```
4. Should be 0-2 (no leaks)

**Expected**: No file descriptor leaks after NodeStore refactor.

### Performance Tests

#### Test 8: High-Volume Discovery

**Scenario**: Large mesh with 100+ nodes

**Steps**:
1. Start TUI on network with many nodes
2. Let discovery run for 10 minutes
3. Check NodeStore performance:
   ```python
   import time
   store = get_node_store()
   start = time.time()
   nodes = store.get_all_nodes()
   elapsed = time.time() - start
   print(f"Retrieved {len(nodes)} nodes in {elapsed:.3f}s")
   ```

**Expected**: < 100ms for 100 nodes

#### Test 9: Concurrent Message Sends

**Scenario**: Multiple conversations open, rapid message sends

**Steps**:
1. Open conversations with 3+ nodes
2. Send messages rapidly (10 msg/second)
3. Monitor for errors or slowdowns

**Expected**: No errors, consistent latency

## Rollback Plan

If critical issues are discovered:

### Quick Rollback (Minutes)

```bash
git revert HEAD~1  # Revert LXMF messaging fix
make validate      # Verify tests pass
git push origin main
```

**Impact**: Messaging disabled, but TUI remains functional for monitoring.

### Partial Rollback (Keep Database Changes)

If only ChatProtocol has issues:
1. Revert `src/styrene/protocols/chat.py` only
2. Keep database schema changes (forward-compatible)
3. Fix and redeploy chat logic

### Database Rollback (If Schema Issues)

```bash
# Backup current database
cp ~/Library/Application\ Support/styrene/nodes.db ~/nodes.db.backup

# Remove new column (if needed)
sqlite3 ~/Library/Application\ Support/styrene/nodes.db \
  "ALTER TABLE nodes DROP COLUMN lxmf_destination_hash"
```

**Note**: SQLite `DROP COLUMN` requires version 3.35+ (2021). For older SQLite:
1. Create new table without column
2. Copy data
3. Drop old table, rename new table

## Success Criteria

- [ ] LXMF initializes successfully in TUI
- [ ] Styrene nodes announce with LXMF destination
- [ ] NodeStore persists LXMF destinations
- [ ] Messages send successfully to remote nodes
- [ ] Auto-replies received
- [ ] No file descriptor leaks
- [ ] Backward compatible with legacy nodes
- [ ] Database migration automatic and transparent

## Monitoring

### Metrics to Track

1. **LXMF Initialization Rate**
   - Target: 100% of startups
   - Alert if < 95%

2. **Message Send Success Rate**
   - Target: > 95% (excluding network issues)
   - Alert if < 80%

3. **LXMF Capability Coverage**
   - Track % of nodes with LXMF destination
   - Target: > 90% after 30 days

4. **NodeStore Connection Count**
   - Target: 0-2 connections
   - Alert if > 5

### Log Monitoring

Key log messages to watch:
```
✅ "LXMF initialized and announced"
✅ "Including LXMF dest in announce"
✅ "Sending chat message to <dest>"
❌ "LXMF initialization failed"
❌ "Cannot send to <dest>: node does not have LXMF capability"
❌ "identity <hash> not known to RNS"
```

## Related Issues

- NodeStore file descriptor leak (fixed in same changeset)
- TUI logging configuration (fixed - logs to `/tmp/styrene-tui.log`)
- Chat protocol error handling (improved error messages)

## References

- [node_store_fix.md](node_store_fix.md) - File descriptor leak fix
- [node_store_guardrails.md](node_store_guardrails.md) - Connection health monitoring
- RNS Documentation: https://reticulum.network/manual/
- LXMF Specification: https://github.com/markqvist/LXMF

## Commit Message

```
fix: separate operator and LXMF destinations for messaging

Fixes message sending to discovered Styrene nodes by properly
distinguishing between operator destinations (node identification)
and LXMF delivery destinations (messaging).

Changes:
- Add lxmf_destination_hash field to MeshDevice and NodeStore schema
- Include LXMF destination in Styrene node announces
- Update ChatProtocol to verify LXMF capability before sending
- Use identity_hash (not destination hash) for RNS.Identity.recall()
- Add automatic database schema migration
- Improve error messages for missing LXMF capability

Backward compatible with nodes that don't announce LXMF destinations.

Fixes: "identity not known to RNS" error when sending messages
```
Human: continue from previous response, making sure you complete the comprehensive documentation, create a migration plan and test plan