# Performance Architecture

**Version:** 3.0.25
**Last Updated:** February 2026

This document describes the performance characteristics, optimizations, and tuning decisions in SoulEyez.

---

## Overview

SoulEyez is designed to handle large-scale penetration testing engagements with hundreds of hosts and thousands of services. Key performance goals:

- Parse large nmap scans (1000+ hosts, 10MB+ XML) in under 5 seconds
- Query engagements with 1000+ hosts without lag
- Dashboard refresh under 2 seconds
- Minimal memory footprint for long-running sessions

---

## Database Performance

### SQLite Configuration

SoulEyez uses SQLite with performance-optimized settings:

```python
# Write-Ahead Logging (WAL) mode
PRAGMA journal_mode=WAL

# 30-second busy timeout for concurrent access
connection.execute("PRAGMA busy_timeout = 30000")
```

**Why WAL mode?**
- Allows concurrent reads during writes
- Better performance for read-heavy workloads
- Crash recovery without corruption risk

### Connection Management

**Singleton Pattern**: A single database connection is reused throughout the application via `get_db()`:

```python
_db_instance = None

def get_db():
    global _db_instance
    if _db_instance is None:
        _db_instance = Database()
    return _db_instance
```

**Benefits:**
- Eliminates connection overhead per query
- Prevents connection exhaustion
- Maintains consistent transaction state

### Database Indexes

20 indexes are defined on frequently-queried columns:

| Table | Indexed Columns |
|-------|-----------------|
| hosts | engagement_id, ip_address |
| services | host_id, port, service_name |
| findings | engagement_id, severity, category, host_id |
| credentials | engagement_id, host_id |
| jobs | engagement_id, status, tool_name |
| attack_paths | engagement_id |
| chain_rules | source_tool |

**Query Performance Impact:**
- Host lookups by IP: O(log n) vs O(n)
- Findings by severity: 10x faster on large datasets
- Job status queries: Instant regardless of job count

---

## Caching Strategy

### Dashboard TTL Caching

The dashboard uses time-based caching to avoid redundant database queries:

```python
class DashboardCache:
    def __init__(self, ttl_seconds=30):
        self.ttl = ttl_seconds
        self._cache = {}
        self._timestamps = {}

    def get(self, key):
        if key in self._cache:
            if time.time() - self._timestamps[key] < self.ttl:
                return self._cache[key]
        return None
```

**Cached Data:**
- Attack surface summary
- Correlation data
- Credential counts
- OSINT findings

**TTL:** 30 seconds (configurable)

### When Caching Applies

| Operation | Cached | TTL |
|-----------|--------|-----|
| Dashboard header stats | Yes | 30s |
| Host list | No | - |
| Service details | No | - |
| Finding severity counts | Yes | 30s |
| Job status | No | - |

---

## Parser Performance

### Nmap XML Parsing

Large scan parsing uses streaming XML where possible:

- **defusedxml** for secure parsing (prevents XXE attacks)
- **Incremental host processing** - hosts are processed as parsed, not loaded entirely into memory
- **Service deduplication** at parse time

**Benchmarks (typical hardware):**

| Scan Size | Hosts | Parse Time |
|-----------|-------|------------|
| Small | 10 | <50ms |
| Medium | 100 | ~200ms |
| Large | 1000 | ~2s |
| XLarge | 2000+ | ~5s |

### Memory Efficiency

- Parsed data is inserted into database immediately
- No in-memory caching of raw XML
- Generator patterns for large result sets

---

## Query Optimizations

### N+1 Query Prevention

**Problem (fixed in v3.0.20):**
```python
# BAD: N+1 queries
for host in hosts:
    services = get_services(host.id)  # Query per host!
```

**Solution:**
```python
# GOOD: Single query with JOIN
SELECT h.*, s.* FROM hosts h
LEFT JOIN services s ON h.id = s.host_id
WHERE h.engagement_id = ?
```

**Impact:** Dashboard load improved from 18s to 1.48s (12x faster)

### Batch Operations

- Bulk inserts for scan imports
- Batch updates for job status changes
- Transaction batching for finding creation

---

## Performance Benchmarks

### Running Benchmarks

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run all benchmarks
pytest tests/test_benchmarks.py --benchmark-only

# Run with comparison to previous
pytest tests/test_benchmarks.py --benchmark-only --benchmark-compare
```

### Benchmark Suite

| Benchmark | Description | Target |
|-----------|-------------|--------|
| `test_benchmark_parse_xml_small` | Parse 10-host XML | <100ms |
| `test_benchmark_parse_xml_medium` | Parse 100-host XML | <500ms |
| `test_benchmark_parse_xml_large` | Parse 1000-host XML | <3s |
| `test_benchmark_query_hosts_small` | Query 50 hosts | <50ms |
| `test_benchmark_query_hosts_large` | Query 1000 hosts | <200ms |
| `test_benchmark_query_services_with_join` | Join query 2500 services | <500ms |
| `test_parse_10mb_equivalent_xml` | Stress test >10MB XML | <10s |

---

## Known Limitations

### SQLite Constraints

- **Single writer**: Only one write transaction at a time (WAL helps with reads)
- **Large BLOBs**: Not optimized for storing large binary data
- **Concurrent users**: Not designed for multi-user access (single-user tool)

### Memory Considerations

- Very large engagements (5000+ hosts) may require increased memory
- Job log storage grows linearly with job count
- Consider archiving old engagements for long-term use

### Recommendations

1. **Archive completed engagements** to keep active database small
2. **Use SSD storage** for database file location
3. **Avoid running benchmarks during active scans** (I/O contention)

---

## Performance Tuning

### Environment Variables

```bash
# Increase SQLite cache (default: 2000 pages)
export SOULEYEZ_SQLITE_CACHE_SIZE=10000

# Adjust dashboard cache TTL (default: 30s)
export SOULEYEZ_CACHE_TTL=60
```

### Database Maintenance

```sql
-- Periodic optimization (run manually if needed)
VACUUM;
ANALYZE;
```

---

## Version History

| Version | Change |
|---------|--------|
| 3.0.20 | Fixed N+1 queries in dashboard (18s → 1.48s) |
| 3.0.20 | Added WAL mode and connection singleton |
| 3.0.21 | SQLMap deduplication to prevent job explosion |
| 3.0.25 | Added pytest-benchmark test suite |

---

## See Also

- [Architecture Overview](overview.md)
- [Database Schema](../database/SCHEMA.md)
- [Benchmark Tests](../../../tests/test_benchmarks.py)
