# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

---

## Project Overview

**remux** is a multiplexed remote execution daemon built on **asyncssh**. The architecture prioritizes:
1. Network efficiency: one persistent SSH connection per host, reused across multiple commands
2. Asynchronous I/O multiplexing: non-blocking handling of multiple concurrent SSH channels
3. Structured output: framed streaming (separate stdout/stderr/exit via type tags)

**NOT guaranteed:** Remote execution parallelism (depends on remote kernel scheduler and CPU availability).

---

## Architecture

The project uses a daemon-client architecture with binary message framing:

### Core Components

**Daemon (`daemon.py`)**
- Listens on Unix domain socket (~/.rpipe/daemon.sock)
- Manages request-response cycle: receive RUN_REQ → spawn SSH channel → stream frames to client
- Enforces policy (read-only mode, command denylist) before execution
- Allocates run_ids; forwards interleaved stdout/stderr/exit frames to clients
- Timeout enforcement per-run with hard kill (SIGKILL) escalation

**Limitation:** Currently serial per client connection. One client at a time; subsequent clients block on socket accept() until previous client disconnects.

**SSH Backend (`ssh_backend.py`)**
- Connection pool: maintains one asyncssh connection per unique hostname
  - **Keying assumption:** host uniqueness means identical username, port, auth method for every request to that host
  - **Risk:** If username varies per-request for same hostname, creates multiple connections instead of reusing (defeats pooling benefit)
- RunHandle: wraps asyncssh process; exposes cancel/wait/timing interfaces
- Process launching wraps commands in `bash -lc '...'` to inject cwd and env

**Critical Security Issue: Shell Wrapping Injection Vector**

Commands are templated as: `bash -lc 'cd <cwd> && export VAR=<env> && <cmd>'`

This introduces shell metacharacter injection if cwd or env contain untrusted values.

- **Current assumption:** cwd and env come from trusted sources (LLM planner)
- **Falsification tests required (ALL must pass):**
  ```
  cwd="/tmp'; rm -rf /" → must NOT execute rm
  cwd="/tmp`whoami`" → must NOT execute whoami
  cwd="/tmp$(whoami)" → must NOT expand $()
  env="VAR=`id`" → must NOT execute id
  env="VAR=$(hostname)" → must NOT expand $()
  env="VAR=';echo hacked" → must NOT break out of string
  ```
- **Long-term fix required for v1:** Either:
  1. Validate cwd/env to reject all shell metacharacters (', ;, $, `, \, |, <, >, &, etc.)
  2. Move to execve()-style execution (bypass bash entirely)
  3. Escape metacharacters using shell-safe quoting (fragile, hard to verify)

**Protocol (`protocol.py`)**
- Binary framing: `[4-byte length][1-byte type][4-byte run_id][payload]`
- Frame types: RUN_REQ, STDIN, STDOUT, STDERR, EXIT, META, ERROR, CANCEL
- run_id permits interleaved responses; protocol supports concurrent runs (daemon does not)

**Policy (`policy.py`)**
- Validates commands before execution; never modifies or rewrites them
- **Fundamental Architectural Weakness:** Token-based regex denylist

**Why Denylist Cannot Scale:**

The policy blocks tokens matching destructive patterns (rm, dd, mkfs, chmod, etc.).

| Problem | Example | Cost |
|---------|---------|------|
| **False positives** | `grep chmod /etc/passwd` blocked | Legitimate workflows fail silently |
| **False negatives** | `python -c "import shutil; shutil.rmtree(...)"` not blocked | Determined actor bypasses restrictions |
| **Obfuscation** | `eval(user_input)` not blocked (no token match) | Arbitrary execution via indirect methods |
| **Infinite patterns** | New tools, languages, subcommands constantly emerge | Denylist can never be exhaustive |

**Conclusion:** Token-matching provides security theater, not actual protection. This is a v0 expedient that must be replaced in v1.

**Path to v1:** Capability-based allowlist model (explicit permissions only: read_file, list_dir, execute_binary_X).

**Client (`client.py`)**
- CLI interface: connects to daemon, sends RUN_REQ, reads frames, outputs to stdout/stderr
- Supports: cwd, env vars, stdin piping, per-run timeouts, read-only mode
- Exit code propagation from remote process

### Data Flow

```
client.py (CLI args)
    ↓
RUN_REQ frame: {host, cmd, cwd, env, timeout_s, ...}
    ↓
daemon.py (receives, validates, allocates run_id)
    ↓
policy.validate_command() [DENYLIST CHECK — FRAGILE]
    ↓
ssh_backend.run() [COMMAND INJECTION RISK — shell wrapping]
    ↓
bash -lc 'cd <cwd> && export <env> && <cmd>'
    ↓
Remote process execution
    ↓
[STDOUT/STDERR/EXIT frames] ← (interleaved by run_id)
    ↓
daemon.py streams to client
    ↓
client.py writes to stdout/stderr
```

---

## Architecture: Guarantees, Contingencies, and Unknown Unknowns

### Asynchronous I/O Multiplexing — GUARANTEED

**What's guaranteed by asyncio + asyncssh:**
- Multiple SSH channels can be created on one connection
- The daemon's read/write operations do not block each other
- Frame routing by run_id prevents stdout/stderr corruption

**Falsification test:** Spawn 100 processes with distinct run_ids; verify all frames received in correct order without loss or interleaving.

---

### Remote Execution Parallelism — CONTINGENT, Not Guaranteed

**Claim:** Multiple processes run concurrently on the remote host.

**What's actually true:**
- The daemon sends multiple `create_process()` calls to the remote SSH server
- The SSH server creates multiple channels
- Whether these processes execute in parallel depends entirely on remote factors

**Dependencies (all external):**
1. Remote kernel scheduler (must assign processes to different CPUs or interleave them)
2. CPU cores available on remote (single-core remote = serialization)
3. System load on remote (high load may serialize even with multiple cores)
4. SSH server implementation (rare, but some may serialize channels)

**Falsification test:**
```bash
# On local, spawn 100 processes that sleep 5 seconds on a single-core remote
time for i in {1..100}; do
  python -m rpipe.client single-core-host "sleep 5" &
done
wait

# Result if concurrent: ~5-10s (5s sleep + overhead)
# Result if serialized: ~500s (100 × 5s)
```

**Current status:** NOT tested. This is an assumption, not validated.

---

### SSH Connection Pooling — Benefit Depends on Workload

**What's guaranteed:**
- asyncssh.connect() is called once per hostname, not once per run
- Subsequent commands to same host reuse the connection

**Assumption:** Connection setup cost dominates execution cost.

**True if:** Running 10+ commands per host (typical setup: 100-500ms per connection)
**False if:** Running single commands across 1000 unique hosts (setup cost amortized over 1 command each)

**Not measured:** Actual performance benefit in realistic workloads. This is a design hypothesis, not validated.

---

### Single-Client Serialization — Visible Constraint, Root Cause Unknown

**Current limitation:** handle_client() is serial. Client connections block on accept() until the previous client disconnects.

**Why this is not "the" bottleneck:** After refactoring to concurrent clients (v1), the actual scalability constraint may be:

| Candidate | Evidence | Blocking Factor |
|-----------|----------|-----------------|
| Pool lock contention | Shared asyncio.Lock in ssh_backend.py | High concurrency on same host |
| Per-host SSH limits | Remote SSH server rejects >10 channels/connection | Architecture-dependent |
| System file descriptors | ulimit -n (typically 1024) | Number of active connections × 2 (reader + writer) |
| Memory exhaustion | Unbounded buffers if process outputs massive data | Backpressure not implemented |
| Policy validation | Regex search on every command | Unlikely, but untested at scale |

**Prerequisite for v1:** Profile concurrent multi-client load to identify which constraint binds first. Do not assume serial loop is the bottleneck.

---

## Critical Security Issues (Blocking for Any Real Use)

### 1. Host Key Verification is Disabled — CRITICAL

**Current state:** `asyncssh.connect(..., known_hosts=None)` accepts ANY host key.

**Risk:** Man-in-the-Middle (MITM) attack on all SSH traffic.

**Attack scenario:**
1. Attacker positioned on network path (shared cloud VPC, corporate network, lab)
2. Attacker intercepts TCP connection to remote host
3. Attacker presents their own SSH key
4. Daemon accepts key without question (no verification)
5. Attacker receives all commands and intercepts/modifies responses

**Why "dev mode" is not an excuse:**
- MITM vulnerability exists in any environment where network is not 100% controlled
- This includes: development on shared machines, staging in cloud, testing against VMs
- Not limited to "production"

**Blocking requirement for v1:**
```python
asyncssh.connect(
    host,
    ...
    known_hosts=os.path.expanduser("~/.ssh/known_hosts"),
    error_handler="reject",  # Reject unknown or mismatched keys
)
```

**Test before use:**
```bash
# Add remote host to known_hosts
ssh-keyscan -t ed25519 remote.example.com >> ~/.ssh/known_hosts

# Daemon must reject connection to host with wrong key
# (simulate by temporarily removing host from known_hosts)
```

---

### 2. Command Injection via Shell Wrapping — CRITICAL

**Current state:** Commands executed as `bash -lc 'cd <cwd> && export VAR=<env> && <cmd>'`

**Risk:** Shell metacharacter injection if cwd or env contain untrusted values.

**Falsification tests (ALL must pass before v1):**

```python
test_cases = [
    # Quote termination
    {"cwd": "/tmp'; rm -rf /", "expected": "no rm executed"},
    # Backtick injection
    {"cwd": "/tmp`whoami`", "expected": "no whoami executed"},
    # Dollar-paren injection
    {"cwd": "/tmp$(whoami)", "expected": "no whoami executed"},
    # Variable injection in env
    {"env": {"VAR": "`id`"}, "expected": "no id executed"},
    {"env": {"VAR": "$(hostname)"}, "expected": "no hostname executed"},
    # Escape sequences
    {"cwd": "/tmp\\'; ls", "expected": "no ls executed"},
    # Semicolon breakout
    {"cwd": "/tmp; rm -rf /", "expected": "no rm executed"},
    # Pipe/redirect injection
    {"cwd": "/tmp | whoami", "expected": "no whoami executed"},
    {"cwd": "/tmp > /tmp/evil", "expected": "no file created"},
]

# For each test case, execute a command and verify the injection did NOT occur
# Example: if cwd="/tmp`whoami`", execute "pwd" and verify output is "/tmp`whoami`", not the PID
```

**Current status:** NOT tested comprehensively. Only quote termination is mentioned; other vectors are untested.

**Mitigation options for v1:**

| Option | Cost | Security |
|--------|------|----------|
| **Validate cwd/env: reject all metacharacters** | Simple; some legitimate paths blocked | Strong; whitelist approach |
| **Escape metacharacters in shell** | Complex; escaping rules are fragile | Weak; depends on correct escaping |
| **Use execve-style execution (no shell)** | Major refactor of command templating | Strong; no shell interpolation |

**Recommended:** Option 1 (validate cwd/env). Simplest and most defensible.

---

### 3. Policy Denylist is Fundamentally Brittle — Architectural Issue

**Current state:** Token-based regex matching to block destructive patterns.

**Why it fails:**

| Vector | Status | Example |
|--------|--------|---------|
| **False positives** | Confirmed | `grep chmod /etc/passwd` blocked |
| **False negatives (Python)** | Confirmed | `python -c "import shutil; shutil.rmtree(...)"` not blocked |
| **False negatives (Perl)** | Confirmed | `perl -e 'system("rm -rf /tmp/x")'` not blocked |
| **False negatives (obfuscation)** | Confirmed | `bash -c 'rm -rf $(pwd)'` may not match depending on regex |
| **False negatives (new tools)** | Inevitable | New utilities constantly emerge (kubectl, terraform, etc.) |

**Architectural problem:** Denylist assumes a closed, finite set of "bad patterns." Reality is an open, infinite set. Every new tool, language, or obfuscation technique breaks the assumption.

**Cannot be fixed by improving regexes.** The approach is fundamentally limited.

**Path to v1:** Replace with capability-based allowlist.

```python
class Capability:
    def __init__(self, operation: str, path_pattern: Optional[str] = None):
        self.operation = operation  # e.g., "read_file", "list_dir", "execute"
        self.path_pattern = path_pattern  # e.g., "/var/log/*"

# Allowed operations ONLY:
policy = Policy(
    capabilities=[
        Capability("read_file", "/var/log/*"),
        Capability("execute", "ls|grep|cat|tail"),  # only these binaries
        Capability("list_dir", "/"),
    ]
)
```

**Trade-off:** LLM must generate commands within constrained set. This reduces flexibility but provides actual security.

---

## Release Roadmap

### v0 (Current: Skeleton)

**Status:** Design complete, implementation skeleton provided.

**What's included:**
- Protocol framing (stable)
- SSH backend (works for single-host, single-run scenarios)
- CLI client (functional but minimal)

**Known limitations (acceptable for v0):**
- Serial per-client (single planner only)
- No audit logging
- No resource limits
- No concurrent multi-client

**Blocking issues for v0 release:** None. This is a reference implementation for design and discussion.

---

### v1 (Usable, Single Planner)

**Status:** Before ANY real use, these must be fixed.

**BLOCKING fixes required:**

1. **Host key verification enabled** (SECURITY)
   - Configure asyncssh.connect() with real known_hosts
   - Test: Reject connection to host with wrong key
   - Effort: 2h

2. **Command injection tests pass** (SECURITY)
   - Implement all 8 falsification tests for shell metacharacters
   - Validate cwd/env to reject: ', ;, $, `, \, |, <, >, &, newlines
   - Effort: 3h

3. **Policy denylist accuracy baseline** (FUNCTIONAL)
   - Run 50+ benign commands; measure false positive rate
   - Target: <2% false positive rate
   - Attempt 10 known exploitation bypasses; verify detection or blocking
   - Effort: 2h

4. **Timeout/SIGKILL behavior verified** (FUNCTIONAL)
   - Test timeout with process that ignores SIGTERM
   - Verify daemon escalates to SIGKILL after timeout_s
   - Measure time to actual death; must be timeout_s ± 1s
   - Effort: 2h

5. **Test suite** (RELIABILITY)
   - Unit tests for protocol framing (encode/decode round-trips)
   - Integration tests for daemon ↔ client communication
   - Mocked SSH backend for policy + timeout testing
   - Load test: 100+ concurrent processes on one connection
   - Effort: 8h

**Optional for v1 (defer to v1.1):**
- Concurrent multi-client refactoring
- Audit logging
- Capability-based policy (replacement for denylist)
- Backpressure/rate limiting

**Total effort for v1:** ~17h

---

### v2 (Production-Ready, Multi-Client)

**Status:** After 1 month of v1 in production, observed incident patterns.

**Refactoring required:**

1. **Concurrent multi-client** (PERFORMANCE)
   - Refactor daemon to accept N concurrent client connections
   - Shared "active runs" registry with asyncio.Lock
   - Profile to identify actual bottleneck (not serial loop anymore)
   - Effort: 6h

2. **Capability-based policy** (SECURITY)
   - Replace token-based denylist with allowlist
   - Define capabilities per environment (dev vs. prod)
   - Effort: 4h

3. **Audit logging** (COMPLIANCE)
   - Log all RUN_REQ (command, host, user, timestamp)
   - Log all exits (run_id, exit code, duration)
   - Log all policy violations (rejected commands + reason)
   - Effort: 2h

4. **Resource limits** (RELIABILITY)
   - Per-run: CPU time, memory, file descriptors, output size
   - Per-host: max concurrent channels
   - Effort: 4h

**Total effort for v2:** ~16h

---

### v3 (Hardened)

**Status:** After security audit + incident patterns emerge.

**Security improvements:**
1. OS-level sandboxing (seccomp, containers)
2. Encrypted audit log + tamper detection
3. Key rotation + secret management
4. Rate limiting + DDoS protection

---

## Installation & Setup

### Prerequisites

- Python 3.7+
- SSH client and server installed on local and remote
- SSH key authentication configured (or password auth if using asyncssh)

### Installation

```bash
# Clone repo
cd /home/scott/projects/remux

# Create virtual env
python -m venv venv
source venv/bin/activate

# Install
pip install asyncssh pytest pytest-asyncio
```

### SSH Key Setup

Daemon needs SSH access to remote hosts:

```bash
# Generate SSH key (if needed)
ssh-keygen -t ed25519 -f ~/.ssh/rpipe -N ""

# Add public key to remote(s)
ssh-copy-id -i ~/.ssh/rpipe user@example.com

# Test connectivity
ssh -i ~/.ssh/rpipe user@example.com hostname
```

### Socket Directory

```bash
mkdir -p ~/.rpipe
chmod 700 ~/.rpipe
```

### Known Hosts Configuration (REQUIRED for v1)

```bash
# Add remote hosts to known_hosts
ssh-keyscan -t ed25519 example.com >> ~/.ssh/known_hosts
ssh-keyscan -t ed25519 other-host.com >> ~/.ssh/known_hosts

# Verify
grep example.com ~/.ssh/known_hosts
```

### Running

**Start daemon:**
```bash
python -m rpipe.daemon
# Output: rpipe-daemon listening on /home/user/.rpipe/daemon.sock
```

**Run command (in another terminal):**
```bash
python -m rpipe.client example.com "hostname"
python -m rpipe.client example.com "ls -la /tmp"
python -m rpipe.client example.com --cwd /var/log "tail -f syslog"
python -m rpipe.client example.com --env FOO=bar "echo $FOO"
python -m rpipe.client example.com --timeout 5 "sleep 10"  # Hard-killed after 5s
```

---

## Testing Strategy

### Test Structure

```
rpipe/
  tests/
    conftest.py                    # pytest fixtures, mock SSH backend
    unit/
      test_protocol.py             # framing encode/decode
      test_policy.py               # denylist patterns + accuracy
      test_ssh_backend.py          # pool behavior (mocked SSH)
    integration/
      test_daemon_client.py        # daemon ↔ client communication
    security/
      test_command_injection.py    # shell metacharacter tests
      test_policy_bypasses.py      # known exploitation attempts
    load/
      test_concurrent_runs.py      # >1000 concurrent runs
      test_large_output.py         # >1GB output streaming
```

### Critical Tests to Write (Blocking for v1)

**1. Command Injection Tests**

```python
# tests/security/test_command_injection.py
import pytest
from rpipe.ssh_backend import AsyncSSHBackend
from rpipe.protocol import TYPE_EXIT

@pytest.mark.asyncio
async def test_cwd_quote_injection(mock_ssh_backend):
    """cwd with quote termination must not execute injected command"""
    result = await mock_ssh_backend.run(
        run_id=1,
        host="test.local",
        cmd="pwd",
        cwd="/tmp'; rm -rf /",
    )
    # Verify: pwd output is "/tmp'; rm -rf /", not home directory
    # Verify: no rm process spawned on remote

@pytest.mark.asyncio
async def test_cwd_dollar_paren_injection(mock_ssh_backend):
    """cwd with $() must not expand command substitution"""
    result = await mock_ssh_backend.run(
        run_id=1,
        host="test.local",
        cmd="pwd",
        cwd="/tmp$(whoami)",
    )
    # Verify: pwd output is "/tmp$(whoami)"
    # Verify: whoami command not executed

# ... (8 total test cases covering all metacharacters)
```

**2. Policy Denylist Accuracy Tests**

```python
# tests/unit/test_policy.py
import pytest
from rpipe.policy import validate_command, Policy

def test_policy_50_benign_commands():
    """Run 50 benign commands; verify <2% false positive rate"""
    benign_commands = [
        "ls -la",
        "cat /var/log/syslog",
        "grep error /var/log/*",
        "echo hello world",
        "find . -name '*.py'",
        # ... (50 total)
    ]
    policy = Policy(read_only=True)
    rejected = 0
    for cmd in benign_commands:
        try:
            validate_command(cmd, policy)
        except ValueError:
            rejected += 1

    false_positive_rate = rejected / len(benign_commands)
    assert false_positive_rate < 0.02, f"FP rate {false_positive_rate*100}% exceeds 2%"

def test_policy_known_bypasses():
    """Attempt 10 known exploitation bypasses; verify detection"""
    bypasses = [
        'python -c "import shutil; shutil.rmtree(...)"',
        'perl -e "system(\'rm -rf /tmp\')"',
        'bash -c "rm -rf /tmp/x"',
        # ... (10 total)
    ]
    policy = Policy(read_only=True)
    detected = 0
    for cmd in bypasses:
        try:
            validate_command(cmd, policy)
            # If we get here, bypass was not detected
        except ValueError:
            detected += 1

    assert detected >= 8, f"Only {detected}/10 bypasses detected"
```

**3. Timeout/SIGKILL Tests**

```python
# tests/integration/test_timeout.py
@pytest.mark.asyncio
async def test_timeout_sigterm_ignored_process(mock_ssh_backend):
    """Process that ignores SIGTERM must be killed by SIGKILL after timeout"""
    start_time = time.time()
    result = await mock_ssh_backend.run(
        run_id=1,
        host="test.local",
        cmd='bash -c "trap : TERM; sleep 100"',  # Ignores SIGTERM
        timeout_s=2,
    )
    elapsed = time.time() - start_time

    # Process should die within timeout_s + 1s (room for SIGKILL)
    assert elapsed <= 3.0, f"Process took {elapsed}s; expected <=3s"
    assert result.exit_code in (137, 143), f"Expected SIGKILL (137) or SIGTERM (143), got {result.exit_code}"
```

**4. Connection Pool Reuse Tests**

```python
# tests/unit/test_ssh_backend.py
@pytest.mark.asyncio
async def test_pool_reuses_connection(mock_pool):
    """10 commands on same host must reuse single connection"""
    conn1 = await mock_pool.get("example.com")
    conn2 = await mock_pool.get("example.com")

    assert id(conn1) == id(conn2), "Pool did not reuse connection"

@pytest.mark.asyncio
async def test_pool_reconnects_on_stale(mock_pool):
    """Dead connection must be detected and replaced"""
    conn1 = await mock_pool.get("example.com")
    conn1.close()  # Simulate connection death

    time.sleep(0.5)

    conn2 = await mock_pool.get("example.com")
    # Should be a new connection
    assert id(conn1) != id(conn2), "Pool did not reconnect"
```

**5. Backpressure Tests**

```python
# tests/load/test_large_output.py
@pytest.mark.asyncio
async def test_1gb_output_streaming(mock_daemon):
    """Daemon must stream 1GB without memory exhaustion"""
    import psutil
    process = psutil.Process()

    # Generate 1GB of output
    initial_memory = process.memory_info().rss

    result = await mock_daemon.run_command(
        cmd='dd if=/dev/zero bs=1M count=1024 | base64',
        host="test.local",
    )

    final_memory = process.memory_info().rss
    memory_increase = final_memory - initial_memory

    # Memory increase should be <100MB (buffering overhead)
    assert memory_increase < 100 * 1024 * 1024, f"Memory increased by {memory_increase} bytes"
```

---

## Error Handling & Recovery

### SSH Connection Failures

| Scenario | Current Behavior | Test |
|----------|------------------|------|
| Host unreachable | Connection timeout | `nc -zv unreachable-host.com 22` should fail; daemon should timeout within 10s |
| Auth failure | asyncssh.Error | SSH key not on remote; daemon should raise AuthFailedError |
| Host key mismatch | (Would be) Connection rejected | Create wrong key in known_hosts; daemon should reject |
| Connection dies mid-request | Unhandled; process may hang | Kill remote SSH server while command running; verify daemon recovers |

### Client Disconnection

| Scenario | Current Behavior | Expected | Gap |
|----------|------------------|----------|-----|
| Client closes socket during streaming | Daemon writes to closed socket; exception | Exception caught; run continues on remote | ❓ Untested |
| Client timeout waiting | Client hangs indefinitely | Client timeout after N seconds | ❌ Not implemented |

### Remote Process Failures

| Scenario | Behavior | Verified |
|----------|----------|----------|
| Process syntax error | Process exits 1, stderr sent to client | ✓ Test this |
| Process segfault | Exit code 139, sent to client | ✓ Test this |
| Process ignores SIGTERM | Daemon escalates to SIGKILL after timeout | ❓ Not tested yet |

### Socket Cleanup

| Scenario | Behavior | Verified |
|----------|----------|----------|
| Daemon crashes, leaves socket | New daemon can clean up and rebind | ✓ Implemented in daemon.py |
| Socket wrong permissions | Only owner can connect | ✓ Created as 0600 |

---

## Module Responsibilities & Boundaries

### protocol.py

**Owns:**
- Frame encoding/decoding (pack_frame, read_frame)
- Frame type constants (TYPE_RUN_REQ, TYPE_STDOUT, etc.)
- Frame format stability (backward compatibility)

**Does NOT own:**
- Run ID allocation (daemon)
- Command validation (policy)
- Error interpretation (daemon translates protocol errors)

**Contract:**
- `pack_frame(type, run_id, payload) → bytes` always succeeds
- `read_frame(reader) → (type, run_id, payload)` raises IncompleteReadError if stream closes
- Frame format is stable; new types can be added but existing ones never change

---

### policy.py

**Owns:**
- Command validation rules (denylist, allowlist, capabilities)
- Read-only mode enforcement
- Policy decision (allow/reject/defer)

**Does NOT own:**
- Execution of commands (ssh_backend)
- Logging of rejections (daemon)
- User messages on rejection (daemon)

**Contract:**
- `validate_command(cmd, policy) → None` raises ValueError if rejected
- Policy objects are immutable after creation (thread-safe)
- New policies can extend/replace denylist without code changes

---

### ssh_backend.py

**Owns:**
- SSH connection pooling (per-host connections)
- Process execution via SSH channels
- Timeout + cancellation (SIGTERM, SIGKILL)
- Connection error handling (reconnect vs. fail)

**Does NOT own:**
- Run ID allocation (daemon)
- Frame creation (protocol)
- Policy validation (policy)
- Output streaming to client (daemon)

**Contract:**
- `pool.get(host) → asyncssh.SSHClientConnection` succeeds or raises ConnectionError
- `backend.run(...) → RunHandle` spawns remote process; handle permits wait/cancel
- `RunHandle.cancel()` sends SIGTERM; `cancel(hard=True)` sends SIGKILL

---

### daemon.py

**Owns:**
- Main event loop and client connection handling
- Run ID allocation and lifecycle
- Request parsing, policy enforcement, process spawning, output streaming
- Error handling and user-facing error messages

**Does NOT own:**
- Frame parsing (protocol)
- Command validation (policy)
- SSH connection management (ssh_backend)

**Contract:**
- Accepts one client connection at a time (serial)
- `handle_client()` reads one RUN_REQ, spawns one process, streams output, exits
- All frames forwarded to client correctly (no loss, type/run_id/payload intact)

---

### client.py

**Owns:**
- CLI argument parsing
- Client-side framing (sending RUN_REQ, STDIN)
- Output display (stdout, stderr)
- Exit code propagation

**Does NOT own:**
- Command validation (daemon validates)
- Remote execution (daemon + backend)
- Error recovery (if daemon dies, client fails)

**Contract:**
- RUN_REQ sent successfully or exception raised
- All response frames demultiplexed by run_id
- EXIT frame contains final exit code

---

## Security Pre-Release Checklist

**Before v1 Release, ALL must pass:**

- [ ] **Host key verification enabled**
  - Test: `ssh -i key example.com` succeeds; `ssh -i key wrong-host` fails
  - Test: Daemon rejects connection to host with wrong key in known_hosts
  - Evidence: Code shows `known_hosts=path`, not `known_hosts=None`

- [ ] **Command injection tests pass (8/8)**
  - [ ] Quote termination: `cwd="/tmp'; rm -rf /"` does not execute rm
  - [ ] Backtick injection: `cwd="/tmp`whoami`"` does not execute whoami
  - [ ] Dollar-paren injection: `cwd="/tmp$(whoami)"` does not expand $()
  - [ ] Env var injection: `env="VAR=$(id)"` does not execute id
  - [ ] Variable expansion: `env="VAR=`hostname`"` does not expand backticks
  - [ ] Escape sequences: `cwd="/tmp\\'; ls"` does not break out
  - [ ] Semicolon breakout: `cwd="/tmp; rm -rf /"` does not execute rm
  - [ ] Pipe/redirect: `cwd="/tmp | whoami"` does not execute whoami

- [ ] **Socket permissions verified**
  - Test: `ls -l ~/.rpipe/daemon.sock` shows `-rw-------` (0600)
  - Test: Unprivileged user cannot read socket

- [ ] **Policy denylist accuracy baseline**
  - Test: Run 50 benign commands; count false positives
  - Target: <2% false positive rate
  - Document any false positives found (to accept or fix)

- [ ] **Policy bypass detection**
  - Test: Attempt 10 known exploitation bypasses (python -c, perl -e, eval, etc.)
  - Verify: 8/10 detected or blocked

- [ ] **Timeout/SIGKILL behavior verified**
  - Test: Run command with `--timeout 2 "sleep 10"`, verify dies after ~2s
  - Test: Run command that ignores SIGTERM, verify SIGKILL enforces death
  - Evidence: Time-to-death measured and logged

- [ ] **No secrets in logs**
  - Test: Grep all logs for password, api_key, token, secret
  - Verify: Command args and output don't leak sensitive data

- [ ] **Error messages don't leak information**
  - Test: Connection failure doesn't reveal SSH username/port/banner
  - Test: Policy rejection says "blocked", not "matched pattern X"

- [ ] **Daemon cannot read arbitrary files**
  - Test: `cat ~/.ssh/id_rsa` fails or is blocked by policy
  - Verify: Only commands submitted by planner are executed

---

## Performance Benchmarking

### Connection Pool Efficiency

**Claim:** Connection pooling reduces per-request latency by ~200ms.

**Test:**
```python
import time
import subprocess

# 10 commands, fresh connection each
start = time.time()
for i in range(10):
    subprocess.run(["ssh", "user@example.com", "hostname"])
fresh_time = time.time() - start

# 10 commands, pooled
start = time.time()
for i in range(10):
    subprocess.run(["python", "-m", "rpipe.client", "example.com", "hostname"])
pooled_time = time.time() - start

savings = fresh_time - pooled_time
print(f"Fresh: {fresh_time:.2f}s, Pooled: {pooled_time:.2f}s, Savings: {savings:.2f}s")
assert savings >= 1.0, "Connection pooling not providing expected benefit"
```

**Expected result:** Savings ≥ 1 second (100ms × 10 commands)

---

### Concurrent Execution Measurement

**Claim:** Multiple processes can run concurrently on one SSH connection.

**Test:**
```bash
#!/bin/bash
# Spawn 100 processes that sleep 2 seconds each

time for i in {1..100}; do
  python -m rpipe.client example.com "sleep 2" &
done
wait

# Expected if concurrent: ~2-5s total (2s sleep + overhead)
# Expected if serialized: ~200s (100 × 2s)
```

**Verify on remote:**
```bash
# On remote host, in another terminal
watch -n 1 'ps aux | grep sleep | wc -l'

# If max ≈ 100, running concurrently
# If max ≈ 5, serializing
```

---

### Large Output Streaming

**Claim:** Daemon can stream 1GB without memory exhaustion or unbounded buffering.

**Test:**
```bash
python -m rpipe.client example.com "dd if=/dev/zero bs=1M count=1024 | base64" > /dev/null

# Monitor in parallel:
watch -n 1 'ps aux | grep rpipe.daemon | awk "{print \$6}"'  # memory
iotop -o -p $(pidof -x rpipe.daemon)  # I/O
```

**Measurements:**
- Daemon memory: should stay <100MB (not grow unbounded)
- Streaming latency: should be <1s per chunk
- Client CPU: should be <20%

---

## Debugging & Troubleshooting

### Daemon Won't Start

**Symptom:** `OSError: Address already in use`

**Diagnosis:**
```bash
lsof ~/.rpipe/daemon.sock
ps aux | grep rpipe.daemon
```

**Fix:**
```bash
rm -f ~/.rpipe/daemon.sock
python -m rpipe.daemon
```

---

### Client Hangs

**Symptom:** `python -m rpipe.client example.com "ls"` never returns

**Diagnosis:**
```bash
# Is daemon running?
ps aux | grep rpipe.daemon

# Is socket accessible?
ls -l ~/.rpipe/daemon.sock
test -r ~/.rpipe/daemon.sock

# Can you reach remote?
ssh example.com hostname
```

---

### Policy Rejects Valid Command

**Symptom:** `ValueError: command blocked by denylist` for benign command

**Diagnosis:**
```bash
# Check which token matched
grep -F "echo remove" ~/.claude/policy.py  # Is "remove" in denylist?
```

**Root cause:** False positive in denylist. Workaround: reword command.

---

### Output Truncated or Interleaved

**Symptom:** STDOUT and STDERR mixed, or output stops mid-stream

**Diagnosis:**
1. Are all frames being received? Check frame count (TYPE_STDOUT + TYPE_STDERR + TYPE_EXIT)
2. Is daemon streaming all data? Add logging in `_pump_stream()`

---

## Configuration

### Daemon Config

`~/.config/rpipe/daemon.json`:

```json
{
  "socket_path": "~/.rpipe/daemon.sock",
  "ssh_timeout": 10.0,
  "ssh_known_hosts": "~/.ssh/known_hosts",
  "read_only_default": true,
  "chunk_size": 8192,
  "max_command_length": 10000
}
```

### Client Config

`~/.config/rpipe/client-config.json`:

```json
{
  "socket_path": "~/.rpipe/daemon.sock",
  "default_timeout": 60,
  "hosts": {
    "prod": {
      "hostname": "prod.example.com",
      "username": "deploy",
      "port": 22
    }
  }
}
```

Usage: `python -m rpipe.client prod "ls"` (instead of full hostname).

---

## Monitoring & Observability

### Health Checks

```bash
# Socket exists and is readable
test -S ~/.rpipe/daemon.sock && test -r ~/.rpipe/daemon.sock && echo "healthy"

# Daemon process running
ps aux | grep -v grep | grep rpipe.daemon > /dev/null && echo "alive"
```

### Metrics to Track

1. **Uptime:** Daemon crash frequency
2. **Latency:** Client response time (p50, p99)
3. **Throughput:** Runs per second
4. **Errors:** Policy rejections, SSH failures, timeouts
5. **Resource usage:** Memory, CPU, file descriptors
6. **Backpressure:** Time blocked on write/read

---

## Protocol Stability & Extensibility

**Wire format is stable:**
```
[4-byte length: u32 big-endian][body]
[body] = [1-byte type][4-byte run_id: u32 big-endian][payload]
```

**Backward compatibility rules:**

- ✓ Add new frame type constants (never reuse old ones)
- ✓ Add new optional JSON fields in RUN_REQ (parse gracefully if missing)
- ✗ Change frame header layout
- ✗ Reuse type constants
- ✗ Remove RUN_REQ fields
- ✗ Change u32 run_id to u64

**Breaking changes:** Use new socket path (~/.rpipe/daemon-v2.sock) or implement version negotiation handshake.

---

## Dependencies & Versions

- **asyncssh:** 2.x (tested version to be confirmed)
- **asyncio:** Standard library; Python 3.7+
- **json, struct, re:** Standard library

No external test framework yet. Use **pytest + pytest-asyncio** when adding tests.

---

## Extending the Codebase

### Adding a New Frame Type

1. Define constant in `protocol.py` (e.g., `TYPE_ARTIFACT = 8`)
2. Add pack/unpack logic if payload is non-trivial
3. Extend daemon's `handle_client()` or `_pump_stream()` to generate frame
4. Extend client's frame-reading loop to handle new type

**Test:** Verify frame round-trip (send from daemon, receive on client, payload intact).

---

### Supporting Concurrent Clients (v1 Refactoring)

**Current limitation:** `handle_client()` is serial; one connection blocks others.

**Refactoring required:**

1. Move "active runs" registry outside `handle_client()` (daemon-level state + asyncio.Lock)
2. Each client connection opens independent reader/writer but shares daemon's run registry
3. Client can submit multiple RUN_REQ frames on one connection; daemon allocates unique run_ids
4. Daemon forwards frames to originating client based on run_id

**Design decision to clarify:** Should frames be:
- Per-client (only client that submitted run sees its output)? Simpler.
- Broadcast (all clients see all frames)? Enables monitoring/mirroring.

**Test:** Multiple clients submit runs concurrently; verify no frame loss or corruption.

---

### Improving Policy: Token-Based → Capability-Based (v1/v2)

**Current risk:** Denylist is fundamentally brittle (see "Critical Issues" section).

**Refactoring required:**

```python
class Capability:
    def __init__(self, operation: str, path_pattern: Optional[str] = None):
        self.operation = operation  # "read_file", "list_dir", "execute"
        self.path_pattern = path_pattern  # "/var/log/*"

policy = Policy(
    capabilities=[
        Capability("read_file", "/var/log/*"),
        Capability("execute", "ls|grep|cat|tail"),
        Capability("list_dir", "/"),
    ]
)
```

**Trade-off:**
- Pro: Explicit allowlist; impossible to accidentally allow dangerous ops
- Con: LLM must generate commands within constrained set

**Test:** For each capability, verify only matching commands execute; non-matching commands rejected with clear error.

---

### Adding Audit Logging (v1/v2)

**Proposal:**
1. Log all RUN_REQ (command, host, user, timestamp)
2. Log all process exits (run_id, exit code, duration)
3. Log all policy violations (rejected commands + reason)

**Storage:** File-based log, JSON lines format, rotated daily.

**Assumption:** Audit log itself is not sensitive. If it is, encrypt or move to syslog.

---

## Glossary

- **run_id:** Unique identifier for one execution, allocated by daemon
- **Channel:** SSH protocol's term for one logical stream within a TCP connection
- **Frame:** One message in wire protocol (length + type + run_id + payload)
- **Denylist:** Negative allowlist (list of blocked patterns)
- **Falsification test:** Experiment that would disprove a claim
- **Backpressure:** System can't keep up with input; buffers fill
- **SIGTERM:** Soft termination signal; process can ignore
- **SIGKILL:** Hard termination signal; kernel enforces
- **MITM:** Man-in-the-Middle attack; attacker intercepts TCP stream
- **Known_hosts:** File mapping SSH host keys to hostnames; verifies server identity

