# Error Handling Guide

Complete guide to error handling, dead letter queues, and debugging in AtomIngest.

## Table of Contents
- [Overview](#overview)
- [Error Types](#error-types)
- [Dead Letter Queue](#dead-letter-queue)
- [Error Recovery](#error-recovery)
- [Logging](#logging)
- [Debugging](#debugging)
- [Best Practices](#best-practices)

## Overview

AtomIngest provides robust error handling to ensure data quality while maintaining pipeline resilience.

**Error Handling Features:**
- ✅ Dead Letter Queue (DLQ) for failed records
- ✅ Configurable error thresholds
- ✅ Detailed error logging
- ✅ Error recovery mechanisms
- ✅ Batch vs. row-level error handling

## Error Types

### Validation Errors

Field validation failures.

```python
ValidationError: Invalid email format for field 'email'
```

**Causes:**
- Failed regex pattern match
- Out of range values
- Invalid format (email, URL, etc.)
- Business rule violations

**Example:**
```csv
name,email,age
Alice,invalid-email,200  # Invalid email, age out of range
```

### Type Conversion Errors

Data type casting failures.

```python
TypeError: Cannot convert 'abc' to IntegerField
```

**Causes:**
- Non-numeric string for numeric field
- Invalid date format
- Malformed JSON

**Example:**
```csv
name,age,created_at
Bob,not-a-number,invalid-date
```

### Database Errors

Database operation failures.

```python
IntegrityError: UNIQUE constraint failed: users.email
```

**Causes:**
- Unique constraint violations
- Foreign key violations
- Not null constraint violations
- Database connection issues

### Hook Execution Errors

Custom hook failures.

```python
HookExecutionError: API call failed in EnrichmentHook
```

**Causes:**
- External API failures
- Network timeouts
- Logic errors in custom code

## Dead Letter Queue

### Configuration

Enable DLQ in pipeline config:

```yaml
config:
  dead_letter_queue:
    enabled: true
    path: "./errors"
    format: "json"  # or "csv"
    include_error_details: true
    include_stack_trace: false
```

### DLQ Output Format

**JSON Format:**
```json
{
  "row_data": {
    "name": "Alice",
    "email": "invalid-email",
    "age": 200
  },
  "errors": [
    {
      "field": "email",
      "error_type": "ValidationError",
      "message": "Invalid email format"
    },
    {
      "field": "age",
      "error_type": "ValidationError",
      "message": "Age must be between 18 and 120"
    }
  ],
  "timestamp": "2026-01-20T10:30:00Z",
  "table": "users",
  "source": "users.csv",
  "run_id": "run_20260120_103000",
  "row_number": 42
}
```

**CSV Format:**
```csv
row_number,field,error_type,error_message,row_data,timestamp
42,email,ValidationError,Invalid email format,"{""name"":""Alice"",...}",2026-01-20T10:30:00Z
```

### DLQ File Structure

```
errors/
├── 20260120/
│   ├── users_run_20260120_103000_errors.json
│   ├── orders_run_20260120_104500_errors.json
│   └── summary_20260120.txt
└── 20260119/
    └── users_run_20260119_090000_errors.json
```

### Custom DLQ Hook

```python
from atomingest.pipeline.hooks import Hook, HookContext
import json
from datetime import datetime
from pathlib import Path

class CustomDLQHook(Hook):
    """Custom dead letter queue implementation."""

    def execute(self, context: HookContext) -> HookContext:
        # Get error details
        row_data = context.row_data
        error = context.error
        metadata = context.metadata

        # Create error record
        error_record = {
            "row_data": row_data,
            "error_type": type(error).__name__,
            "error_message": str(error),
            "timestamp": datetime.now().isoformat(),
            "table": metadata.get("table"),
            "source": metadata.get("source"),
            "run_id": metadata.get("run_id")
        }

        # Write to DLQ
        dlq_path = Path(self.config.get("output_dir", "./errors"))
        dlq_path.mkdir(parents=True, exist_ok=True)

        dlq_file = dlq_path / f"errors_{datetime.now():%Y%m%d}.json"

        with open(dlq_file, "a") as f:
            f.write(json.dumps(error_record) + "\n")

        return context
```

**Configuration:**
```yaml
hooks:
  on_error:
    - name: "custom_dlq"
      class: "myapp.hooks.CustomDLQHook"
      config:
        output_dir: "./errors"
```

## Error Recovery

### Processing Failed Records

```python
import json
from pathlib import Path

# Read DLQ file
dlq_file = Path("errors/users_run_20260120_103000_errors.json")

with open(dlq_file) as f:
    for line in f:
        error_record = json.loads(line)

        # Fix data
        row_data = error_record["row_data"]
        row_data["email"] = fix_email(row_data["email"])
        row_data["age"] = clamp_age(row_data["age"])

        # Re-ingest
        User.create(**row_data)
```

### Automated Retry

```python
class RetryDLQHook(Hook):
    """Automatically retry failed records."""

    def __init__(self, name=None, config=None):
        super().__init__(name, config)
        self.max_retries = config.get("max_retries", 3)
        self.retry_delay = config.get("retry_delay", 60)

    def execute(self, context: HookContext) -> HookContext:
        error = context.error
        retry_count = context.metadata.get("retry_count", 0)

        if retry_count < self.max_retries:
            # Wait and retry
            time.sleep(self.retry_delay)
            context.metadata["retry_count"] = retry_count + 1

            try:
                # Retry operation
                self.retry_operation(context)
            except Exception as e:
                # Still failed, increment retry count
                context.error = e
        else:
            # Max retries exceeded, write to DLQ
            self.write_to_dlq(context)

        return context
```

### Batch Error Handling

```yaml
config:
  error_handling:
    mode: "batch"  # or "row"
    max_errors_per_batch: 10
    batch_size: 1000
```

**Batch Mode:**
- Fails entire batch if any error
- Rolls back all changes in batch
- Falls back to row-by-row on failure

**Row Mode:**
- Continues processing on error
- Failed rows sent to DLQ
- Other rows processed normally

## Logging

### Configuration

```yaml
config:
  logging:
    level: "INFO"  # DEBUG, INFO, WARNING, ERROR, CRITICAL
    format: "json"  # or "text"
    output: "./logs/ingestion.log"
```

### Log Levels

**DEBUG:**
```
DEBUG: Processing row 1234
DEBUG: Executing hook: CalculateTotalHook
DEBUG: Database connection established
```

**INFO:**
```
INFO: Starting ingestion run_20260120_103000
INFO: Normalization completed: 1000 rows
INFO: Batch committed: 1000 records
```

**WARNING:**
```
WARNING: Duplicate header detected at line 250
WARNING: Null token '-' converted to None
WARNING: Hook EnrichmentHook took 5.2s (threshold: 1s)
```

**ERROR:**
```
ERROR: Validation failed for row 42: Invalid email
ERROR: Database connection lost, retrying...
ERROR: Hook execution failed: APIError
```

### Structured Logging (JSON)

```json
{
  "timestamp": "2026-01-20T10:30:00Z",
  "level": "ERROR",
  "message": "Validation failed",
  "context": {
    "table": "users",
    "run_id": "run_20260120_103000",
    "row_number": 42,
    "field": "email",
    "value": "invalid-email"
  }
}
```

### Enable Verbose Logging

```bash
# CLI flag
atomingest run pipeline.yaml sqlite:///db.db -s data.csv -v

# Environment variable
export ATOMINGEST_LOG_LEVEL="DEBUG"
atomingest run pipeline.yaml sqlite:///db.db -s data.csv
```

## Debugging

### Dry Run Mode

Test pipeline without database writes:

```bash
atomingest run pipeline.yaml sqlite:///test.db --dry-run
```

**Output:**
```
✅ Pipeline configuration is valid!
Target Table: users
Schema Fields: 5
Normalization Steps: 3
Validation Rules: 8
```

### Test with Sample Data

```bash
atomingest test pipeline.yaml -s data.csv -n 10 -v
```

**Output:**
```
Testing with 10 records...

Row 1: ✅ Success
Row 2: ✅ Success
Row 3: ❌ Failed - Invalid email format
Row 4: ✅ Success
...

Summary:
- Successful: 8
- Failed: 2
- Total: 10
```

### Debug Individual Row

```python
from atomingest.core.loader import SchemaLoader
from atomingest.normalization.pipeline import normalize_then_validate

# Load schema
loader = SchemaLoader()
schema = loader.load_from_yaml("pipeline.yaml")

# Test single row
test_row = {
    "name": "Alice",
    "email": "invalid-email",
    "age": 200
}

# Normalize
config = schema.get("normalization")
rows, report = normalize_then_validate([test_row], config)

# Check validation
for row in rows:
    try:
        ModelClass = loader.create_model(schema)
        instance = ModelClass(**row)
        print("✅ Row valid:", instance)
    except Exception as e:
        print("❌ Validation failed:", e)
```

### Enable Debug Logging

```python
import logging

# Enable debug logs
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("atomingest")
logger.setLevel(logging.DEBUG)

# Run ingestion
from atomingest.core.ingester import Ingester

ingester = Ingester(db)
stats = ingester.run("pipeline.yaml", "data.csv")
```

### Common Issues

#### 1. Data Not Converting

**Problem:** Numbers staying as strings.

**Debug:**
```bash
atomingest run pipeline.yaml sqlite:///test.db -s data.csv -v
```

**Solution:**
```yaml
normalization:
  steps:
    - step: TrimWhitespaceStep  # Remove spaces first
    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns: [amount, quantity]
```

#### 2. Validation Always Failing

**Problem:** All rows fail validation.

**Debug:**
```python
# Check raw data after normalization
rows, report = normalize_then_validate(source, config)
for row in rows:
    print(row)  # Inspect normalized data
```

**Solution:**
```yaml
# Ensure proper normalization order
normalization:
  steps:
    - step: ColumnNameNormalizationStep  # Standardize names
    - step: TrimWhitespaceStep  # Clean values
    - step: NullTokenNormalizationStep  # Handle nulls
    - step: BasicTypeCleanupStep  # Convert types
```

#### 3. Database Errors

**Problem:** IntegrityError: UNIQUE constraint failed.

**Debug:**
```sql
-- Check existing records
SELECT * FROM users WHERE email = 'alice@example.com';
```

**Solution:**
```yaml
config:
  business_key: [email]  # Use upsert instead of insert
  on_conflict: "update"
```

#### 4. Hook Failures

**Problem:** Hook execution errors.

**Debug:**
```python
# Test hook in isolation
from myapp.hooks import MyHook

hook = MyHook(config={"param": "value"})
context = HookContext(row_data={"test": "data"}, metadata={})

try:
    result = hook.execute(context)
    print("✅ Hook succeeded")
except Exception as e:
    print(f"❌ Hook failed: {e}")
    import traceback
    traceback.print_exc()
```

## Best Practices

### 1. Always Enable DLQ

```yaml
config:
  dead_letter_queue:
    enabled: true
    path: "./errors"
```

### 2. Set Error Thresholds

```yaml
config:
  max_errors: 100  # Stop after 100 errors
  error_rate_threshold: 0.05  # Stop if >5% fail
```

### 3. Use Structured Logging

```bash
atomingest run pipeline.yaml sqlite:///db.db -s data.csv --json-logs
```

### 4. Monitor DLQ Size

```bash
# Check error count
ls -l errors/*.json | wc -l

# Alert if DLQ growing
if [ $(ls errors/*.json | wc -l) -gt 1000 ]; then
    echo "WARNING: DLQ has over 1000 errors!"
fi
```

### 5. Implement Error Notifications

```python
class ErrorNotificationHook(Hook):
    """Send notification on error threshold."""

    def execute(self, context: HookContext) -> HookContext:
        failed_count = context.failed_count
        total_count = context.total_count
        error_rate = failed_count / total_count if total_count > 0 else 0

        threshold = self.config.get("error_rate_threshold", 0.05)

        if error_rate > threshold:
            self.send_alert(f"Error rate {error_rate:.2%} exceeds threshold {threshold:.2%}")

        return context
```

### 6. Test Error Scenarios

```python
def test_invalid_email():
    """Test handling of invalid email."""
    ingester = Ingester(db)

    # Create test file with invalid data
    with open("test_invalid.csv", "w") as f:
        f.write("name,email\n")
        f.write("Alice,invalid-email\n")

    stats = ingester.run("pipeline.yaml", "test_invalid.csv")

    assert stats.failed == 1
    assert stats.processed == 0

    # Check DLQ
    dlq_file = Path("errors/test_invalid_errors.json")
    assert dlq_file.exists()
```

## Complete Error Handling Example

```yaml
target_table: users

schema:
  name: CharField
  email: EmailField
  age: IntegerField

normalization:
  steps:
    - step: TrimWhitespaceStep
    - step: NullTokenNormalizationStep
    - step: BasicTypeCleanupStep

config:
  # Error handling
  validation_mode: lenient
  max_errors: 100
  error_rate_threshold: 0.05

  # Dead letter queue
  dead_letter_queue:
    enabled: true
    path: "./errors"
    format: "json"
    include_error_details: true

  # Batch processing
  batch_size: 1000
  batch_error_handling: "row"  # Continue on row errors

  # Logging
  logging:
    level: "INFO"
    format: "json"
    output: "./logs/ingestion.log"

hooks:
  on_error:
    - name: "dlq"
      class: "atomingest.pipeline.hooks.DeadLetterQueueHook"
      order: 10

    - name: "error_notification"
      class: "myapp.hooks.ErrorNotificationHook"
      order: 20
      config:
        error_rate_threshold: 0.05
        notification_emails:
          - admin@example.com
```

## Related Documentation

- [Pipeline Configuration](pipeline_configuration.md) - Complete pipeline setup
- [Hooks](hooks.md) - Custom error handling hooks
- [CLI Reference](cli_reference.md) - CLI debugging options

## Next Steps

1. Enable DLQ in pipeline configuration
2. Set appropriate error thresholds
3. Test error scenarios with sample data
4. Monitor DLQ files regularly
5. Implement error recovery processes
