# Performance Optimization

Guide to optimizing AtomIngest performance for large-scale data ingestion.

## Table of Contents
- [Performance Overview](#performance-overview)
- [Batch Processing](#batch-processing)
- [Memory Management](#memory-management)
- [Database Optimization](#database-optimization)
- [Profiling](#profiling)
- [Large File Strategies](#large-file-strategies)
- [Benchmarks](#benchmarks)

## Performance Overview

AtomIngest is designed for high-performance data ingestion with these characteristics:

**Default Performance:**
- ~10,000 rows/second on standard hardware
- Streaming processing for constant memory usage
- Bulk insert operations
- Minimal validation overhead

**Factors Affecting Performance:**
1. Database backend (SQLite < PostgreSQL < MySQL)
2. Number of validation rules
3. Hook complexity
4. Normalization step count
5. Encryption overhead
6. Network latency (remote databases)

## Batch Processing

### Batch Size Configuration

Control batch size for bulk inserts:

```yaml
target_table: "transactions"
config:
  batch_size: 1000  # Insert 1000 records at a time
  enable_bulk_insert: true
```

**Recommended Batch Sizes:**
- SQLite: 500-1000
- PostgreSQL: 1000-5000
- MySQL: 1000-3000

**Trade-offs:**
- Larger batches = faster throughput
- Smaller batches = lower memory usage, faster error detection

### Streaming Processing

AtomIngest processes files in streaming mode by default:

```python
from atomingest.core.ingester import Ingester

# Processes file line-by-line, constant memory
ingester.run("pipeline.yaml", "large_file.csv")
```

**Benefits:**
- Handles files larger than available RAM
- Immediate processing start
- Low memory footprint

### Parallel Processing

Process multiple files in parallel:

```python
from concurrent.futures import ProcessPoolExecutor
from atomingest.core.ingester import Ingester
from atomsql import Database

def process_file(file_path: str):
    # Each process gets its own connection
    db = Database("postgresql://localhost/data")
    ingester = Ingester(db)
    stats = ingester.run("pipeline.yaml", file_path)
    db.close()
    return stats

files = ["file1.csv", "file2.csv", "file3.csv"]

with ProcessPoolExecutor(max_workers=4) as executor:
    results = executor.map(process_file, files)

for stats in results:
    print(f"Processed: {stats.processed}")
```

**Considerations:**
- Use separate database connections per process
- Be mindful of database connection limits
- Consider database write contention

## Memory Management

### Memory-Efficient Configuration

Optimize for low memory usage:

```yaml
target_table: "data"
config:
  batch_size: 100           # Smaller batches
  enable_streaming: true    # Default, but explicit
  max_errors: 100           # Limit DLQ size

error_handling:
  dead_letter_queue:
    format: "csv"           # CSV uses less memory than JSON
    buffer_size: 50         # Flush to disk frequently
```

### Large Row Handling

For rows with large text/binary data:

```yaml
schema:
  document:
    type: TextField
    # Avoid storing large fields in memory during validation
    validation:
      max_length: null      # Disable length check if not needed
```

### Monitor Memory Usage

```python
import psutil
import os

def memory_stats():
    process = psutil.Process(os.getpid())
    mem_info = process.memory_info()
    print(f"RSS: {mem_info.rss / 1024 / 1024:.2f} MB")
    print(f"VMS: {mem_info.vms / 1024 / 1024:.2f} MB")

# Monitor during ingestion
ingester.run("pipeline.yaml", "data.csv")
memory_stats()
```

## Database Optimization

### Connection Pooling

Use connection pooling for better performance:

```python
from atomsql import Database

# PostgreSQL with connection pool
db = Database(
    "postgresql://localhost/data",
    pool_size=10,
    max_overflow=20
)
```

### Indexes

Create indexes for frequently queried columns:

```python
from atomsql import Database

db = Database("postgresql://localhost/data")

# Create index for faster lookups
db.execute("""
    CREATE INDEX idx_transactions_date
    ON transactions(transaction_date)
""")

db.execute("""
    CREATE INDEX idx_transactions_customer
    ON transactions(customer_id)
""")
```

### Disable Constraints During Bulk Load

For very large imports, temporarily disable constraints:

```python
# PostgreSQL
db.execute("ALTER TABLE transactions DISABLE TRIGGER ALL")
ingester.run("pipeline.yaml", "large_file.csv")
db.execute("ALTER TABLE transactions ENABLE TRIGGER ALL")

# MySQL
db.execute("SET FOREIGN_KEY_CHECKS=0")
ingester.run("pipeline.yaml", "large_file.csv")
db.execute("SET FOREIGN_KEY_CHECKS=1")
```

**Warning:** Only use in controlled environments. Re-enable constraints after load.

### Database-Specific Optimizations

**PostgreSQL:**
```sql
-- Increase work memory for sorting/hashing
SET work_mem = '256MB';

-- Disable autovacuum during bulk load
ALTER TABLE transactions SET (autovacuum_enabled = false);

-- Re-enable after load
ALTER TABLE transactions SET (autovacuum_enabled = true);
VACUUM ANALYZE transactions;
```

**MySQL:**
```sql
-- Disable binary logging for faster inserts
SET sql_log_bin = 0;

-- Use bulk insert buffer
SET bulk_insert_buffer_size = 256 * 1024 * 1024;
```

**SQLite:**
```python
# Use WAL mode for better concurrency
db.execute("PRAGMA journal_mode=WAL")
db.execute("PRAGMA synchronous=NORMAL")
db.execute("PRAGMA cache_size=10000")
```

## Profiling

### Time Profiling

Identify slow components:

```python
import time
import logging

logging.basicConfig(level=logging.DEBUG)

start = time.time()
stats = ingester.run("pipeline.yaml", "data.csv")
duration = time.time() - start

print(f"Processed: {stats.processed}")
print(f"Duration: {duration:.2f}s")
print(f"Throughput: {stats.processed / duration:.0f} rows/sec")
```

### Detailed Profiling with cProfile

```python
import cProfile
import pstats
from io import StringIO

profiler = cProfile.Profile()
profiler.enable()

ingester.run("pipeline.yaml", "data.csv")

profiler.disable()

# Print stats
s = StringIO()
ps = pstats.Stats(profiler, stream=s).sort_stats('cumulative')
ps.print_stats(20)  # Top 20 functions
print(s.getvalue())
```

### Hook Performance

Profile individual hooks:

```python
from atomingest.pipeline.hooks import Hook, HookContext
import time

class ProfiledHook(Hook):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.total_time = 0
        self.call_count = 0

    def execute(self, context: HookContext) -> HookContext:
        start = time.time()
        result = self.process(context)
        duration = time.time() - start

        self.total_time += duration
        self.call_count += 1

        return result

    def process(self, context: HookContext) -> HookContext:
        # Your hook logic
        return context

    def report(self):
        avg = self.total_time / self.call_count if self.call_count > 0 else 0
        print(f"Total: {self.total_time:.2f}s")
        print(f"Calls: {self.call_count}")
        print(f"Average: {avg * 1000:.2f}ms")
```

## Large File Strategies

### Splitting Large Files

Split files for parallel processing:

```python
def split_csv(input_file: str, lines_per_file: int = 100000):
    """Split large CSV into smaller chunks."""
    with open(input_file) as f:
        header = f.readline()

        file_num = 0
        lines = []

        for line in f:
            lines.append(line)

            if len(lines) >= lines_per_file:
                output_file = f"{input_file}.part{file_num:03d}"
                with open(output_file, 'w') as out:
                    out.write(header)
                    out.writelines(lines)

                file_num += 1
                lines = []

        # Write remaining lines
        if lines:
            output_file = f"{input_file}.part{file_num:03d}"
            with open(output_file, 'w') as out:
                out.write(header)
                out.writelines(lines)

# Split and process
split_csv("huge_file.csv", lines_per_file=100000)
```

### Incremental Loading

Load data incrementally with checkpoints:

```yaml
target_table: "transactions"
config:
  checkpoint_interval: 10000  # Save checkpoint every 10k rows
  resume_on_failure: true
```

### Compression

Use compressed files to reduce I/O:

```python
import gzip

# AtomIngest can read gzipped files directly
ingester.run("pipeline.yaml", "data.csv.gz")
```

## Benchmarks

### Standard Hardware Benchmarks

**Test Configuration:**
- CPU: 8-core @ 3.0 GHz
- RAM: 16 GB
- Disk: SSD
- File: 1M rows, 10 columns, 500 MB

**Results:**

| Backend     | Throughput  | Duration | Memory |
|------------|-------------|----------|--------|
| SQLite     | 8,500 r/s   | 118s     | 150 MB |
| PostgreSQL | 15,000 r/s  | 67s      | 180 MB |
| MySQL      | 12,000 r/s  | 83s      | 170 MB |

**With Optimizations:**

| Backend     | Throughput  | Duration | Memory |
|------------|-------------|----------|--------|
| SQLite     | 15,000 r/s  | 67s      | 150 MB |
| PostgreSQL | 35,000 r/s  | 29s      | 180 MB |
| MySQL      | 25,000 r/s  | 40s      | 170 MB |

### Performance Impact of Features

| Feature               | Overhead |
|----------------------|----------|
| Basic validation     | ~5%      |
| Complex validation   | ~15%     |
| Single hook          | ~3%      |
| Multiple hooks       | ~10%     |
| Encryption (1 field) | ~20%     |
| Encryption (5 fields)| ~50%     |
| Normalization (3 steps) | ~8%   |

### Optimization Checklist

- [ ] Use appropriate batch size (1000-5000)
- [ ] Enable bulk inserts
- [ ] Minimize validation rules
- [ ] Optimize hook complexity
- [ ] Use connection pooling
- [ ] Create database indexes
- [ ] Use appropriate database backend
- [ ] Consider parallel processing for multiple files
- [ ] Profile to identify bottlenecks
- [ ] Use compression for large files

## Best Practices

### 1. Start with Profiling

Always profile before optimizing:

```bash
python -m cProfile -o profile.stats pipeline.py
python -m pstats profile.stats
# > sort cumulative
# > stats 20
```

### 2. Optimize the Slowest Component

Focus on the biggest bottleneck:
1. Database writes (most common)
2. Validation logic
3. Hook execution
4. Normalization steps
5. File I/O

### 3. Use Appropriate Hardware

**For large workloads:**
- Fast SSD for database storage
- Sufficient RAM to avoid swapping
- Multiple CPU cores for parallel processing
- Network bandwidth for remote databases

### 4. Monitor Production Performance

```python
from atomingest.observability.metrics import MetricsCollector

collector = MetricsCollector()
collector.enable()

stats = ingester.run("pipeline.yaml", "data.csv")

metrics = collector.get_metrics()
print(f"Throughput: {metrics['rows_per_second']}")
print(f"Error rate: {metrics['error_rate']}")
```

### 5. Test at Scale

Always test with production-scale data:
- Test with largest expected file size
- Test with worst-case data quality
- Test concurrent loads
- Monitor resource usage

## Troubleshooting Slow Performance

### Problem: Slow Database Inserts

**Solutions:**
1. Increase batch size
2. Use bulk insert mode
3. Add database indexes
4. Check database server resources
5. Use local database for testing

### Problem: High Memory Usage

**Solutions:**
1. Reduce batch size
2. Disable unnecessary features
3. Use CSV format for DLQ
4. Check for memory leaks in custom hooks

### Problem: Slow Validation

**Solutions:**
1. Simplify validation rules
2. Move expensive checks to hooks
3. Use database constraints instead
4. Profile validation functions

### Problem: Slow Normalization

**Solutions:**
1. Reduce number of steps
2. Combine multiple steps
3. Optimize regex patterns
4. Use vectorized operations

## Related Documentation

- [Pipeline Configuration](pipeline_configuration.md) - Batch sizes and config
- [Hooks](hooks.md) - Optimizing hooks
- [Error Handling](error_handling.md) - DLQ configuration
- [Patterns](patterns.md) - Performance patterns

## Next Steps

1. **Profile your pipeline** to identify bottlenecks
2. **Apply optimizations** based on profiling results
3. **Benchmark** before and after changes
4. **Monitor** production performance
5. **Iterate** on slow components
