# NodeStore Guardrails

This document describes the guardrails in place to prevent file descriptor leaks in the NodeStore.

## Quick Reference

### For Developers

**✅ DO:**
```python
# Use the context manager for all database operations
with self._connection() as conn:
    result = conn.execute("SELECT * FROM nodes").fetchall()
# Connection automatically closed here
```

**❌ DON'T:**
```python
# Don't store connections as instance variables
self._conn = sqlite3.connect(db_path)  # WRONG

# Don't call _get_conn() directly (deprecated)
conn = self._get_conn()  # WRONG - use _connection() instead
```

### For Operations

**Monitor connection health periodically (every 60s):**
```python
from styrene.services.node_store import check_connection_health

health = check_connection_health()
if not health["healthy"]:
    logger.error(health["warning"])
```

## Guardrails

### 1. Connection-Per-Operation Model

**What:** Each database operation creates a fresh connection and closes it immediately.

**Why:** Prevents accumulation of open connections that lead to file descriptor exhaustion.

**How:** The `_connection()` context manager enforces this pattern:
```python
@contextmanager
def _connection(self):
    conn = init_db(self._db_path)
    try:
        yield conn
    finally:
        conn.close()
        logger.debug("Closed database connection")
```

**Evidence:** Concurrent test shows 0 connections before and after 200 operations across 10 threads.

### 2. Automatic Connection Lifecycle Logging

**What:** Every connection creation/closure is logged at DEBUG level.

**Why:** Enables early detection of leaks by monitoring log patterns.

**How to use:**
```bash
# Enable debug logging
export RNS_LOGLEVEL=7
make run

# Watch for connection patterns in /tmp/styrene-tui.log
tail -f /tmp/styrene-tui.log | grep -E "Creating|Closed database"
```

**Expected pattern:**
```
Creating new database connection to /path/to/nodes.db
Closed database connection
```

**Warning signs:**
- Many "Creating" without matching "Closed"
- Connection messages during idle periods
- Increasing connection rate over time

### 3. Runtime Health Monitoring

**What:** `check_connection_health()` function counts open file descriptors.

**Why:** Provides runtime detection of leaks before they cause failures.

**Thresholds:**
- **0-2 connections:** ✅ Healthy (normal)
- **3-9 connections:** ⚠️ Warning (investigate)
- **10+ connections:** 🚨 Critical (leak detected)

**Integration example:**
```python
# In main application loop or background task
async def monitor_health():
    while True:
        await asyncio.sleep(60)  # Check every minute
        health = check_connection_health()
        if not health["healthy"]:
            logger.error(health["warning"])
            # Alert operators via metrics/alerting system
```

### 4. Thread-Safe by Design

**What:** Each thread gets its own connection, no shared state.

**Why:** Eliminates race conditions and connection reuse bugs.

**Details:**
- **Read operations:** Lock-free, concurrent access
- **Write operations:** Serialized via `_db_lock`
- **No shared connections:** Each operation creates/closes its own

**Testing:**
```bash
python /tmp/test_concurrent_nodestore.py
```

Expected: ✅ SUCCESS with 0 leaked connections after 200 concurrent operations.

### 5. Diagnostic Tools

#### Manual Connection Count Check
```python
from styrene.services.node_store import get_node_store

store = get_node_store()
count = store.get_connection_count()
print(f"Open connections: {count}")
```

#### System-Level Check (Unix)
```bash
# Count open file descriptors for nodes.db
lsof ~/Library/Application\ Support/styrene/nodes.db | wc -l

# Watch in real-time
watch -n 1 'lsof ~/Library/Application\ Support/styrene/nodes.db | wc -l'
```

Expected: 0-2 during normal operation

### 6. Code Review Checklist

When adding new NodeStore methods or modifying existing ones:

- [ ] Uses `with self._connection() as conn:` pattern
- [ ] Does NOT store connection in instance variable
- [ ] Write operations acquire `_db_lock`
- [ ] Read operations do NOT use lock (allow concurrency)
- [ ] Exception handling doesn't prevent connection closure
- [ ] No `conn.close()` calls (context manager handles it)

### 7. Testing Requirements

All new NodeStore methods must include:

1. **Unit test** verifying basic functionality
2. **Leak test** verifying connection is closed:
   ```python
   def test_no_connection_leak():
       store = get_node_store()
       initial = store.get_connection_count()

       # Call method under test
       result = store.some_method()

       final = store.get_connection_count()
       assert final == initial, f"Leaked {final - initial} connections"
   ```

3. **Concurrent test** (for write operations):
   ```python
   def test_concurrent_writes():
       def worker():
           for _ in range(10):
               store.save_node(device)

       threads = [threading.Thread(target=worker) for _ in range(10)]
       for t in threads: t.start()
       for t in threads: t.join()

       # Verify no leaks
       assert store.get_connection_count() == 0
   ```

## Verification

### Before the Fix
```bash
lsof ~/Library/Application\ Support/styrene/nodes.db | wc -l
# Output: 180+ (massive leak)
```

### After the Fix
```bash
lsof ~/Library/Application\ Support/styrene/nodes.db | wc -l
# Output: 0-2 (healthy)
```

### Concurrent Load Test
```bash
python /tmp/test_concurrent_nodestore.py
# Output: ✅ SUCCESS: No file descriptor leak detected!
```

## Historical Context

**Original Problem:** Process 13397 had 180+ open file descriptors to nodes.db
- Caused "database is locked" errors
- Led to process crashes when FD limit reached
- Performance degradation from excessive connection overhead

**Root Cause:** Shared connection with `check_same_thread=False` created in `_get_conn()`
- Connection never explicitly closed
- Multiple code paths could create duplicate connections
- Thread interactions caused connection accumulation

**Fix Applied:** 2026-01-25
- Replaced shared connection with connection-per-operation
- Added context manager with guaranteed cleanup
- Added logging and monitoring guardrails
- Documented threading model

## Emergency Response

If a leak is detected in production:

1. **Immediate:** Check connection count
   ```bash
   lsof /path/to/nodes.db | wc -l
   ```

2. **If > 10:** Restart application immediately
   ```bash
   pkill -9 -f "python -m styrene"
   make run
   ```

3. **Investigate:** Check logs for unclosed connections
   ```bash
   grep "Creating new database connection" /tmp/styrene-tui.log | wc -l
   grep "Closed database connection" /tmp/styrene-tui.log | wc -l
   # These should match
   ```

4. **Report:** File issue with:
   - Connection count before/after
   - Log excerpts showing creation/closure patterns
   - Operations performed when leak occurred
   - Thread dump if available

## References

- [node_store_fix.md](node_store_fix.md) - Detailed explanation of the fix
- [src/styrene/services/node_store.py](../src/styrene/services/node_store.py) - Implementation
- SQLite threading documentation: https://www.sqlite.org/threadsafe.html
- Python SQLite documentation: https://docs.python.org/3/library/sqlite3.html
