# Hooks & Custom Logic Guide

Complete guide to extending AtomIngest pipelines with custom hooks and transformations.

## Table of Contents
- [Overview](#overview)
- [Hook Types](#hook-types)
- [Configuration](#configuration)
- [Built-in Hooks](#built-in-hooks)
- [Custom Hooks](#custom-hooks)
- [Hook Context](#hook-context)
- [Common Patterns](#common-patterns)
- [Best Practices](#best-practices)

## Overview

Hooks provide extension points in the ingestion pipeline where you can execute custom logic. They enable data enrichment, validation, side effects, and custom transformations.

**Hook Capabilities:**
- ✅ Data transformation and enrichment
- ✅ External API calls
- ✅ Custom validation logic
- ✅ Audit logging
- ✅ Side effects (notifications, etc.)
- ✅ Error handling

## Hook Types

AtomIngest supports hooks at multiple pipeline stages:

### PRE_SAVE
Execute before database insertion.

**Use Cases:**
- Data enrichment
- Derived field calculation
- Custom validation
- PII masking

### POST_SAVE
Execute after successful database insertion.

**Use Cases:**
- Notifications
- Cache invalidation
- Downstream system updates
- Audit logging

### PRE_NORMALIZATION
Execute before normalization steps.

**Use Cases:**
- Custom file validation
- Source system specific logic
- Pre-processing

### POST_NORMALIZATION
Execute after normalization, before validation.

**Use Cases:**
- Additional transformations
- Business logic
- Data enhancement

### ON_ERROR
Execute when an error occurs.

**Use Cases:**
- Error logging
- Dead letter queue writes
- Notifications
- Error recovery

### ON_COMPLETE
Execute after pipeline finishes (success or failure).

**Use Cases:**
- Cleanup
- Summary reports
- Metrics publishing
- Status updates

## Configuration

Hooks are configured in the `hooks` section of your pipeline YAML:

```yaml
target_table: my_table
schema:
  # ... schema definition

hooks:
  pre_save:
    - name: "my_hook"
      class: "mymodule.MyHook"
      order: 10
      config:
        param1: value1
        param2: value2

  post_save:
    - name: "notification_hook"
      class: "mymodule.NotificationHook"
      order: 20
```

### Hook Configuration Options

- `name` (string): Unique identifier for the hook
- `class` (string): Full Python path to hook class
- `order` (int): Execution order (lower runs first)
- `config` (dict): Configuration passed to hook

## Built-in Hooks

### MetadataInjectionHook

Automatically adds audit metadata to records.

```yaml
config:
  audit_metadata:
    enabled: true
    fields:
      source_file: true
      run_id: true
      ingestion_timestamp: true
```

**Adds Fields:**
- `source_file` (string): Source file name
- `run_id` (string): Unique ingestion run ID
- `ingestion_timestamp` (datetime): Ingestion time

**Auto-enabled when configured in pipeline config:**

```yaml
target_table: users
schema:
  name: CharField
  email: EmailField
  # Metadata fields added automatically
  source_file: CharField
  run_id: CharField
  ingestion_timestamp: DateTimeField

config:
  audit_metadata:
    enabled: true
```

### DeadLetterQueueHook

Writes failed records to error queue.

```yaml
hooks:
  on_error:
    - name: "dlq"
      class: "atomingest.pipeline.hooks.DeadLetterQueueHook"
      config:
        output_dir: "./errors"
        format: "json"  # or "csv"
        include_error_details: true
```

**DLQ Output Format (JSON):**
```json
{
  "row_data": {
    "name": "Alice",
    "email": "invalid-email"
  },
  "error": "ValidationError: Invalid email format",
  "timestamp": "2026-01-20T10:30:00Z",
  "table": "users",
  "source": "data.csv",
  "run_id": "run_20260120_103000"
}
```

## Custom Hooks

### Basic Hook Structure

```python
from atomingest.pipeline.hooks import Hook, HookContext

class MyCustomHook(Hook):
    """Custom hook description."""

    def __init__(self, name=None, config=None):
        super().__init__(name, config)
        # Initialize from config
        self.param1 = config.get('param1', default_value)

    def execute(self, context: HookContext) -> HookContext:
        """
        Execute hook logic.

        Args:
            context: HookContext with row_data, metadata, etc.

        Returns:
            Modified context (or same context)
        """
        # Access row data
        row_data = context.row_data

        # Perform custom logic
        row_data['derived_field'] = self.calculate_something(row_data)

        # Update context
        context.row_data = row_data

        return context
```

### Example: Data Enrichment Hook

```python
from atomingest.pipeline.hooks import Hook, HookContext
import requests

class GeocodeHook(Hook):
    """Enriches address data with lat/long coordinates."""

    def __init__(self, name=None, config=None):
        super().__init__(name, config)
        self.api_key = config.get('api_key')
        self.api_url = config.get('api_url', 'https://api.geocode.com')

    def execute(self, context: HookContext) -> HookContext:
        row_data = context.row_data
        address = row_data.get('address')

        if address:
            try:
                # Call geocoding API
                response = requests.get(
                    f"{self.api_url}/geocode",
                    params={'address': address, 'key': self.api_key}
                )
                data = response.json()

                # Add coordinates to row
                row_data['latitude'] = data['lat']
                row_data['longitude'] = data['lng']

            except Exception as e:
                logger.warning(f"Geocoding failed for {address}: {e}")
                row_data['latitude'] = None
                row_data['longitude'] = None

        context.row_data = row_data
        return context
```

**Configuration:**
```yaml
schema:
  address: CharField
  latitude: FloatField
  longitude: FloatField

hooks:
  pre_save:
    - name: "geocode"
      class: "myproject.hooks.GeocodeHook"
      order: 10
      config:
        api_key: "${GEOCODE_API_KEY}"
        api_url: "https://api.geocode.com"
```

### Example: Derived Field Hook

```python
from atomingest.pipeline.hooks import Hook, HookContext

class CalculateTotalHook(Hook):
    """Calculates total from quantity and price."""

    def execute(self, context: HookContext) -> HookContext:
        row_data = context.row_data

        quantity = row_data.get('quantity', 0)
        price = row_data.get('price', 0.0)

        # Calculate total
        row_data['total'] = quantity * price

        # Calculate tax
        tax_rate = self.config.get('tax_rate', 0.08)
        row_data['tax'] = row_data['total'] * tax_rate

        # Calculate grand total
        row_data['grand_total'] = row_data['total'] + row_data['tax']

        context.row_data = row_data
        return context
```

**Configuration:**
```yaml
hooks:
  pre_save:
    - name: "calculate_totals"
      class: "myproject.hooks.CalculateTotalHook"
      order: 5
      config:
        tax_rate: 0.08
```

### Example: Validation Hook

```python
from atomingest.pipeline.hooks import Hook, HookContext, HookExecutionError

class BusinessRuleValidationHook(Hook):
    """Custom business rule validation."""

    def execute(self, context: HookContext) -> HookContext:
        row_data = context.row_data

        # Business rule: discount cannot exceed 50%
        discount = row_data.get('discount_percentage', 0)
        if discount > 50:
            raise HookExecutionError(
                f"Discount {discount}% exceeds maximum of 50%",
                hook_name=self.name
            )

        # Business rule: refund must have original_order_id
        if row_data.get('transaction_type') == 'refund':
            if not row_data.get('original_order_id'):
                raise HookExecutionError(
                    "Refunds must reference original order ID",
                    hook_name=self.name
                )

        return context
```

### Example: Notification Hook

```python
from atomingest.pipeline.hooks import Hook, HookContext
import smtplib
from email.mime.text import MIMEText

class EmailNotificationHook(Hook):
    """Sends email notification after successful ingestion."""

    def __init__(self, name=None, config=None):
        super().__init__(name, config)
        self.smtp_host = config.get('smtp_host')
        self.smtp_port = config.get('smtp_port', 587)
        self.from_email = config.get('from_email')
        self.to_emails = config.get('to_emails', [])

    def execute(self, context: HookContext) -> HookContext:
        # Get metadata
        metadata = context.metadata
        processed_count = metadata.get('processed_count', 0)

        # Send notification
        subject = f"Ingestion Complete: {processed_count} records"
        body = f"""
        Ingestion completed successfully.

        Records processed: {processed_count}
        Table: {metadata.get('table')}
        Source: {metadata.get('source')}
        Run ID: {metadata.get('run_id')}
        """

        self.send_email(subject, body)

        return context

    def send_email(self, subject, body):
        msg = MIMEText(body)
        msg['Subject'] = subject
        msg['From'] = self.from_email
        msg['To'] = ', '.join(self.to_emails)

        with smtplib.SMTP(self.smtp_host, self.smtp_port) as server:
            server.starttls()
            server.send_message(msg)
```

**Configuration:**
```yaml
hooks:
  on_complete:
    - name: "email_notification"
      class: "myproject.hooks.EmailNotificationHook"
      order: 100
      config:
        smtp_host: "smtp.gmail.com"
        smtp_port: 587
        from_email: "noreply@example.com"
        to_emails:
          - "admin@example.com"
          - "team@example.com"
```

## Hook Context

The `HookContext` object provides access to row data and metadata.

### Context Properties

```python
@dataclass
class HookContext:
    row_data: Dict[str, Any]           # Current row being processed
    instance: Optional[Model]          # Model instance (after save)
    metadata: Dict[str, Any]           # Pipeline metadata
    error: Optional[Exception]         # Error (in ON_ERROR hooks)
    processed_count: int               # Records processed so far
    failed_count: int                  # Records failed so far
    total_count: int                   # Total records
    skip_row: bool                     # Set to True to skip row
    abort: bool                        # Set to True to abort pipeline
```

### Accessing Data

```python
def execute(self, context: HookContext) -> HookContext:
    # Access row data
    name = context.row_data.get('name')
    email = context.row_data.get('email')

    # Modify row data
    context.row_data['full_name'] = f"{name} <{email}>"

    # Access metadata
    table = context.metadata.get('table')
    source = context.metadata.get('source')
    run_id = context.metadata.get('run_id')

    # Skip row conditionally
    if context.row_data.get('status') == 'deleted':
        context.skip_row = True

    return context
```

### Accessing Database

```python
def execute(self, context: HookContext) -> HookContext:
    # Get database instance from metadata
    db = context.metadata.get('db')

    # Query related data
    user_id = context.row_data.get('user_id')
    user = User.objects(db=db).get(id=user_id)

    # Enrich row with user data
    context.row_data['user_name'] = user.name
    context.row_data['user_email'] = user.email

    return context
```

## Common Patterns

### Pattern 1: Audit Trail

```python
class AuditTrailHook(Hook):
    """Records all changes to audit log."""

    def execute(self, context: HookContext) -> HookContext:
        db = context.metadata.get('db')

        # Create audit record
        AuditLog.create(
            table_name=context.metadata.get('table'),
            action='INSERT',
            record_id=context.instance.id if context.instance else None,
            data=context.row_data,
            user_id=context.metadata.get('user_id'),
            timestamp=datetime.now(),
            run_id=context.metadata.get('run_id')
        )

        return context
```

### Pattern 2: Cache Invalidation

```python
class CacheInvalidationHook(Hook):
    """Invalidates cache after record update."""

    def __init__(self, name=None, config=None):
        super().__init__(name, config)
        self.redis_client = redis.Redis(
            host=config.get('redis_host'),
            port=config.get('redis_port', 6379)
        )

    def execute(self, context: HookContext) -> HookContext:
        # Invalidate cache keys
        user_id = context.row_data.get('user_id')
        self.redis_client.delete(f"user:{user_id}")
        self.redis_client.delete(f"user:{user_id}:profile")

        return context
```

### Pattern 3: External System Sync

```python
class SyncToExternalSystemHook(Hook):
    """Syncs data to external system."""

    def execute(self, context: HookContext) -> HookContext:
        api_url = self.config.get('api_url')
        api_key = self.config.get('api_key')

        try:
            response = requests.post(
                f"{api_url}/records",
                json=context.row_data,
                headers={'Authorization': f'Bearer {api_key}'}
            )
            response.raise_for_status()

            # Store external ID
            context.row_data['external_id'] = response.json()['id']

        except Exception as e:
            logger.error(f"External sync failed: {e}")
            # Don't fail ingestion, just log error

        return context
```

### Pattern 4: Data Masking

```python
class PIIMaskingHook(Hook):
    """Masks PII data based on user role."""

    def execute(self, context: HookContext) -> HookContext:
        user_role = context.metadata.get('user_role')

        if user_role != 'admin':
            # Mask sensitive fields
            if 'ssn' in context.row_data:
                context.row_data['ssn'] = self.mask_ssn(context.row_data['ssn'])

            if 'credit_card' in context.row_data:
                context.row_data['credit_card'] = self.mask_cc(context.row_data['credit_card'])

        return context

    def mask_ssn(self, ssn):
        return f"***-**-{ssn[-4:]}"

    def mask_cc(self, cc):
        return f"****-****-****-{cc[-4:]}"
```

## Best Practices

### 1. Keep Hooks Focused

```python
# Good - single responsibility
class CalculateTaxHook(Hook):
    def execute(self, context):
        context.row_data['tax'] = context.row_data['amount'] * 0.08
        return context

# Avoid - too many responsibilities
class DoEverythingHook(Hook):
    def execute(self, context):
        # Calculate tax
        # Send email
        # Update cache
        # Log audit
        # Call API
        return context
```

### 2. Handle Errors Gracefully

```python
def execute(self, context: HookContext) -> HookContext:
    try:
        # Risky operation
        result = self.call_external_api(context.row_data)
        context.row_data['api_result'] = result
    except Exception as e:
        logger.error(f"Hook {self.name} failed: {e}")
        # Decide: fail hard or continue
        if self.config.get('fail_on_error', False):
            raise
        else:
            context.row_data['api_result'] = None

    return context
```

### 3. Use Configuration

```python
# Good - configurable
class NotificationHook(Hook):
    def __init__(self, name=None, config=None):
        super().__init__(name, config)
        self.threshold = config.get('threshold', 1000)
        self.emails = config.get('emails', [])

# Avoid - hardcoded
class NotificationHook(Hook):
    def execute(self, context):
        if context.processed_count > 1000:
            send_email("admin@example.com", "Alert!")
```

### 4. Order Hooks Appropriately

```yaml
hooks:
  pre_save:
    - name: "validate"
      order: 10  # Run first

    - name: "enrich"
      order: 20  # Run second

    - name: "calculate"
      order: 30  # Run third
```

### 5. Test Hooks Independently

```python
import unittest

class TestCalculateTotalHook(unittest.TestCase):
    def test_calculate_total(self):
        hook = CalculateTotalHook(config={'tax_rate': 0.08})

        context = HookContext(
            row_data={'quantity': 10, 'price': 5.0},
            metadata={}
        )

        result = hook.execute(context)

        self.assertEqual(result.row_data['total'], 50.0)
        self.assertEqual(result.row_data['tax'], 4.0)
        self.assertEqual(result.row_data['grand_total'], 54.0)
```

## Related Documentation

- [Pipeline Configuration](pipeline_configuration.md) - Complete pipeline setup
- [Validation](validation.md) - Data validation rules
- [Error Handling](error_handling.md) - DLQ and error recovery
- [Security](security.md) - PII encryption and audit

## Next Steps

1. Identify extension points in your pipeline
2. Create custom hooks for business logic
3. Configure hooks in pipeline YAML
4. Test hooks with sample data
5. Monitor hook execution and errors
