# QA Review: CLI Slash Command Hook Specification v2

**Reviewer:** Claude (QA/Testing Expert)
**Date:** 2025-01-24
**Spec:** docs/specs/cli-slash-command-hook-spec.md
**Status:** CRITICAL ISSUES FOUND

---

## Executive Summary

This spec has several CRITICAL gaps that will cause production bugs. While v2 improves over v1 with event-based synchronization, there are still fundamental issues with state management, race conditions, and test coverage that must be addressed before implementation.

**Overall Assessment:** 5/10 - Needs significant revisions

---

## Issue #1: CRITICAL - Race Condition in Command System Readiness

### Location
`kollabor/application.py:319-339` - `_wait_for_command_system_ready()`

### Problem
The spec's polling-based readiness check has a race condition:

```python
while elapsed < max_wait:
    if (self.input_handler and
        self.input_handler.command_executor and
        hasattr(self.input_handler.command_executor, 'registry') and
        self.input_handler.command_registry.get_registry_stats()['total_commands'] > 0):
        return
    await asyncio.sleep(interval)
```

### Why It Fails
1. **TOCTOU (Time-of-check to time-of-use)**: Between the readiness check and actual command execution, the command registry could be modified by plugin initialization
2. **No atomic transition**: The `input_handler.command_registry` could be non-None but not fully populated yet
3. **Race with plugin loading**: Plugins may still be registering commands when the check passes

### What Could Happen
- CLI command executes with incomplete registry (command not found error)
- Registry count > 0 but specific command not yet registered
- Command executor exists but handler not yet wired

### Test Coverage Gap
**NO TEST EXISTS for this race condition.** The spec's testing section (lines 624-672) has:
- No concurrent loading tests
- No "slow plugin" simulation
- No registry modification during CLI execution test

### Fix Required
Add an explicit "ready" event flag:
```python
# In command executor initialization
self._ready = asyncio.Event()

# In wait function
await self.input_handler.command_executor._ready.wait()

# Set after ALL plugins registered
self._ready.set()
```

---

## Issue #2: CRITICAL - State Pollution After --stay Flag

### Location
`kollabor/application.py:306-316` - CLI command execution before interactive mode

### Problem
The spec shows CLI command execution transitioning to interactive mode with `--stay`:

```python
if cli_command:
    exit_code = await self._execute_cli_command(cli_command)
    if not self.args.stay:
        sys.exit(exit_code)
    # Continue to interactive mode
```

### What's Missing
1. **Terminal state not reset**: CLI commands may leave terminal in inconsistent state
2. **Plugin state pollution**: Command handlers may modify plugin state that affects interactive mode
3. **Event bus noise**: CLI command events remain in event history
4. **Permission state**: Session-scoped permissions from CLI command leak into interactive mode
5. **Message queue**: Display coordinator queue not cleared before interactive mode

### Concrete Failure Scenarios
```bash
# Scenario 1: Permission leak
kollab --permissions trust --stay
# Now in interactive mode with trust mode active - user doesn't realize!

# Scenario 2: Plugin state pollution
kollab --sub create myagent "task" --stay
# Plugin thinks creation succeeded but interactive mode starts with partial state

# Scenario 3: Terminal corruption
kollab --matrix --stay  # matrix modifies terminal settings
# Interactive mode starts with wrong terminal settings
```

### Test Coverage Gap
**NO TESTS for state cleanup:**
- No test for terminal state after --stay
- No test for permission system state after CLI execution
- No test for plugin state pollution
- No test for event bus state after CLI command

### Fix Required
Add state verification/cleanup in testing:
```python
# After CLI command, before interactive mode
if self.args.stay:
    await self._verify_clean_state()
    await self._reset_transient_state()

async def _verify_clean_state(self):
    """Verify no state pollution from CLI command."""
    # Check terminal state
    # Check permission system state
    # Check plugin state consistency
    # Clear message queue
```

---

## Issue #3: HIGH - Plugin Loading Failure Handling

### Location
`kollabor/cli.py:54-72` - Plugin discovery before CLI execution

### Problem
The spec assumes plugin loading always succeeds:

```python
# 1. Discover plugins (async, can be cached)
plugin_classes = await discover_plugins()

# 2. Instantiate plugins (async initialization)
plugins = await instantiate_plugins(plugin_classes)
```

### What's Missing
1. **Plugin discovery timeout**: What if `discover_plugins()` hangs?
2. **Plugin init failure**: What if a plugin crashes during initialization?
3. **Circular dependency**: What if plugin A needs plugin B but order is wrong?
4. **Partial registry**: CLI command executes with subset of plugins loaded

### Test Coverage Gap
**NO FAILURE MODE TESTS:**
- No test for plugin loading timeout
- No test for plugin init exception during CLI execution
- No test for partial plugin loading
- No test for corrupted plugin registry

### Real-World Failure
```bash
$ kollab --sub list
# Plugin agent_orchestrator fails to import
# What happens?
# A) Crashes with cryptic error?
# B) Shows "unknown command"?
# C) Hangs forever?
```

### Fix Required
Add graceful degradation and testing:
```python
try:
    plugins = await instantiate_plugins(plugin_classes, timeout=5.0)
except TimeoutError:
    logger.error("Plugin loading timeout")
    # Fall back to system commands only
    sys.exit(2)
except Exception as e:
    logger.error(f"Plugin loading failed: {e}")
    sys.exit(2)
```

---

## Issue #4: MEDIUM - ANSI Stripping is Incomplete

### Location
`kollabor/application.py:394-421` - `_print_command_result()`

### Problem
ANSI stripping regex is too narrow:

```python
ansi_escape = re.compile(r'\x1b\[[0-9;]*[mKH]|\x1b][^\x07]*\x07')
```

### What's Missing
1. **OSC sequences**: `\x1b]0;title\x07` (window title)
2. **CSI with many params**: `\x1b[38;2;255;0;0m` (RGB colors)
3. **APC sequences**: Application Program Command
4. **PM sequences**: Privacy Message
5. **DM/TERM**: Not handled

### Test Coverage Gap
**NO COMPREHENSIVE ANSI TESTS:**
- Test only shows `kollab --sub list > output.txt` - doesn't verify all escape codes
- No test for RGB color codes
- No test for OSC sequences
- No test for malformed ANSI (partial sequences)

### Fix Required
Use a proper ANSI stripping library or comprehensive regex:
```python
# Better approach: use existing library
import strip_ansi

message = strip_ansi.strip_ansi(message)

# Or comprehensive regex
ANSI_ESCAPE = re.compile(r'''
    \x1b  # ESC
    (?:   # 7-bit C1 controls
      \[[0-?]*[ -/]*[@-~]   # CSI sequences
      |][^\x07]*\x07         # OSC sequences
      |[PPX^_][^\x1b]*[\x1b@~\\]  # Other C1
    )
''', re.VERBOSE)
```

---

## Issue #5: MEDIUM - Acceptance Criteria Are Not Verifiable

### Location
Lines 607-622 - Acceptance Criteria

### Problem
Many acceptance criteria are subjective:

```markdown
4. ✓ kollab --matrix opens fullscreen in alt buffer
9. ✓ Commands requiring input show helpful error
```

### Why This Fails CI/CD
- "Helpful error" is subjective - can't verify in automated test
- "Opens fullscreen" - what does that mean exactly?
- No quantifiable metrics (time, state, return values)

### Examples of Bad Criteria

| Criterion | Problem |
|-----------|---------|
| `kollab --sub list` executes `/sub list` and exits | How do we verify it's the SAME as `/sub list`? |
| Commands use existing CommandDefinition | No test for this - just architectural intent |
| Exit codes: 0 success, 1 failure, 2 parse error | What about error code 3+? Not defined |

### Fix Required
Make criteria testable:
```markdown
4. ✓ `kollab --matrix` enters alternate buffer within 100ms
   - Test: `infocmp | grep -q 'screen\|tmux'` after command
9. ✓ Commands requiring input exit with code 2 and stderr contains:
   - "requires interactive mode"
   - "Start kollabor: kollab"
   - Command name in error message
```

---

## Issue #6: MEDIUM - No Tests for Signal Handling During Async

### Location
Lines 584-589 - Signal Handling

### Problem
Spec mentions signal handling but no tests:

```markdown
Ctrl+C during CLI command execution:
- Catches KeyboardInterrupt
- Returns exit code 130 (standard SIGINT)
- Cleanup runs normally
```

### Test Coverage Gap
**NO SIGNAL HANDLING TESTS:**
- No test for SIGINT during plugin loading
- No test for SIGTERM during command execution
- No test for signal during --stay transition
- No test for signal during fullscreen command

### What Could Fail
```bash
$ kollab --sub list
^C
# What if plugins partially initialized?
# What if command executor mid-execution?
# What if terminal in raw mode from fullscreen?
# Terminal state corrupted!
```

### Fix Required
Add signal handling tests:
```bash
# Test SIGINT during command
kollab --sub capture all &
PID=$!
sleep 0.5
kill -INT $PID
wait $PID
EXIT_CODE=$?
[ $EXIT_CODE -eq 130 ] || echo "FAIL: Wrong exit code"
```

---

## Issue #7: LOW - Reserved Command Check is Too Late

### Location
`kollabor/commands/registry.py:445-467` - `register_command()`

### Problem
Reserved command check happens at registration time, not definition time:

```python
def register_command(self, command_def: CommandDefinition) -> None:
    if command_def.name in self._reserved and command_def.plugin_name != "system":
        raise ValueError(...)
```

### Why This Matters
- Plugin author doesn't know command is reserved until runtime
- Error message appears during plugin discovery, confusing
- Can't detect during development/testing

### Fix Required
Add validation at CommandDefinition creation:
```python
@dataclass
class CommandDefinition:
    def __post_init__(self):
        if self.name in RESERVED_COMMANDS and self.plugin_name != "system":
            raise ValueError(
                f"Cannot define reserved command /{self.name} "
                f"in plugin '{self.plugin_name}'"
            )
```

---

## Issue #8: LOW - Thread Safety Not Addressed

### Location
Throughout spec - concurrent access to command registry

### Problem
No mention of thread safety:
- Command registry accessed by CLI parsing (main thread)
- Plugin registration happens during async initialization
- Command executor reads from registry

### Test Coverage Gap
**NO CONCURRENCY TESTS:**
- No test for parallel CLI command parsing and plugin loading
- No test for concurrent command registration
- No test for registry iteration during modification

### What Could Fail
```python
# Thread 1: CLI parsing
cmd_def = command_registry.get_command(cmd_name)  # May crash if dict resizing

# Thread 2: Plugin loading
command_registry.register_command(new_command)  # Modifies dict
```

### Fix Required
Either:
1. Make registry access thread-safe with locks
2. Ensure all registry access happens in same async context
3. Use immutable registry with copy-on-write

---

## Comparison to v1 Issues

### Fixed in v2 ✓
1. Event-based sync instead of blind sleep (but still has race condition - see Issue #1)
2. Namespace protection added (but timing is wrong - see Issue #7)

### Still Broken ✗
1. No state cleanup tests (Issue #2)
2. No failure mode tests (Issue #3)
3. Subjective acceptance criteria (Issue #5)
4. No signal handling tests (Issue #6)

### New Issues in v2 ⚠️
1. TOCTOU race in readiness check (Issue #1)
2. State pollution with --stay (Issue #2)
3. Thread safety not addressed (Issue #8)

---

## Recommended Test Plan

Before marking this spec as "ready for implementation", add these tests:

### 1. Race Condition Tests
```bash
# Test: Slow plugin during CLI execution
# Modify plugin to delay registration by 2s
# Run: kollab --sub list
# Expected: Command waits or fails gracefully, not race
```

### 2. State Cleanup Tests
```bash
# Test: Permission state after --stay
kollab --permissions trust --stay
# Verify permission mode in interactive
# Expected: Trust mode NOT active (fresh state)

# Test: Terminal state after fullscreen
kollab --matrix --stay
# Press 'q' to exit matrix
# Verify terminal restored
# Expected: Clean terminal state
```

### 3. Failure Mode Tests
```bash
# Test: Plugin loading failure
# Create broken plugin that raises exception
kollab --help
# Expected: System commands still work, clear error

# Test: Plugin timeout
# Create plugin that hangs in init
kollab --version
# Expected: Timeout after 5s, exit with code 2
```

### 4. Signal Handling Tests
```bash
# Test: SIGINT during command execution
(kollab --sub capture all &
PID=$!
sleep 0.5
kill -INT $PID
wait $PID
echo "Exit: $?") | grep -q "130"
```

### 5. ANSI Stripping Tests
```bash
# Test: All ANSI codes stripped
kollab --sub list | cat -A
# Expected: No escape sequences in output

# Test: RGB colors stripped
kollab --help | grep -P '\x1b' || echo "PASS"
```

---

## Summary

**CRITICAL (Must Fix):**
- Issue #1: Race condition in readiness check
- Issue #2: State pollution after --stay
- Issue #3: Plugin loading failure handling

**HIGH (Should Fix):**
- Issue #4: ANSI stripping incomplete

**MEDIUM (Recommended):**
- Issue #5: Acceptance criteria not verifiable
- Issue #6: No signal handling tests

**LOW (Nice to Have):**
- Issue #7: Reserved command check timing
- Issue #8: Thread safety

**Overall:** The spec is well-structured but has critical race conditions and state management issues that will cause production bugs. The test coverage is insufficient for a feature that touches core initialization paths. Recommend addressing CRITICAL and HIGH issues before implementation begins.

**Estimated Risk if Implemented As-Is:** 85% chance of production bugs in edge cases

---

## Appendix: Missing Test Categories

1. **Performance Tests**
   - No benchmark for "event-based synchronization" - is it actually faster?
   - No test for 50+ command registry performance
   - No test for plugin discovery time

2. **Security Tests**
   - No test for command injection with special characters
   - No test for shlex.quote() edge cases
   - No test for permission escalation

3. **Integration Tests**
   - No test for CLI + pipe mode interaction
   - No test for CLI + initial message conflict
   - No test for multiple CLI flags interaction

4. **Regression Tests**
   - No baseline for v1 behavior to ensure v2 doesn't break existing commands
   - No test for backward compatibility with existing plugins

---

**END OF REVIEW**
