# Common Patterns & Recipes

Real-world ingestion patterns and best practices.

## Table of Contents
- [CSV Ingestion Patterns](#csv-ingestion-patterns)
- [Financial Data Patterns](#financial-data-patterns)
- [E-commerce Patterns](#e-commerce-patterns)
- [API Integration Patterns](#api-integration-patterns)
- [Data Quality Patterns](#data-quality-patterns)
- [Performance Patterns](#performance-patterns)

## CSV Ingestion Patterns

### Basic CSV with Headers

**Use Case:** Standard CSV file with clean headers.

```yaml
target_table: users

schema:
  name: CharField
  email: EmailField
  age: IntegerField

normalization:
  steps:
    - step: TrimWhitespaceStep
    - step: NullTokenNormalizationStep
```

### CSV with Metadata Headers

**Use Case:** Files with title blocks, metadata rows before actual data.

```yaml
normalization:
  steps:
    - step: SkipUntilHeaderStep
      config:
        expected_columns: ["Name", "Email", "Age"]
        max_skip: 20

    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"
```

### CSV with Trailer/Footer

**Use Case:** Files with summary rows at the end.

```yaml
normalization:
  steps:
    - step: DropTrailerStep
      config:
        trailer_marker: "^TOTAL_RECORDS:"
```

### Concatenated CSV Files

**Use Case:** Multiple CSV files merged with repeated headers.

```yaml
normalization:
  steps:
    - step: RemoveBOMStep
      config:
        check_all_lines: true

    - step: DropRepeatedHeaderRowsStep
```

### CSV with Inconsistent Delimiters

**Use Case:** Mixed delimiter formats.

```yaml
normalization:
  steps:
    - step: CustomDelimiterStep
      config:
        delimiter: "|"  # or "\t" for tabs
```

## Financial Data Patterns

### Stock Market Data (NSE Bhavcopy)

**Use Case:** Daily stock market price data.

```yaml
target_table: nse_bhavcopy

schema:
  symbol:
    type: CharField
    max_length: 50
  series:
    type: CharField
    max_length: 10
  date:
    type: DateField
  open_price:
    type: FloatField
  high_price:
    type: FloatField
  low_price:
    type: FloatField
  close_price:
    type: FloatField
  volume:
    type: IntegerField

normalization:
  steps:
    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"

    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true

    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["-", "NA"]

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns:
          - open_price
          - high_price
          - low_price
          - close_price
          - volume
        date_columns:
          - date
        date_formats:
          - "%d-%b-%Y"

config:
  batch_size: 5000
  business_key: [symbol, date]
  on_conflict: "update"
```

### Bank Transactions

**Use Case:** Financial transaction records with PII.

```yaml
target_table: transactions

schema:
  transaction_id:
    type: UUIDField
    unique: true

  account_number:
    type: EncryptedCharField
    max_length: 255
    key_id: "account_encryption"

  amount:
    type: FloatField
    validation:
      - type: range
        min: 0.01

  transaction_type:
    type: CharField
    validation:
      - type: in_choices
        choices: ["debit", "credit", "transfer"]

  timestamp:
    type: DateTimeTZField

normalization:
  steps:
    - step: RemoveBOMStep
    - step: TrimWhitespaceStep
    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        date_columns: [timestamp]
        date_formats:
          - "%Y-%m-%d %H:%M:%S"

hooks:
  pre_save:
    - name: "fraud_detection"
      class: "myapp.hooks.FraudDetectionHook"
```

### Payment Card Data (PCI-DSS)

**Use Case:** Payment card processing with compliance.

```yaml
target_table: payment_cards

schema:
  card_id: UUIDField

  card_number:
    type: EncryptedCharField
    max_length: 255
    key_id: "payment_encryption"
    validation:
      - type: luhn

  cvv:
    type: EncryptedCharField
    max_length: 255
    key_id: "payment_encryption"

  expiry_date:
    type: DateField

hooks:
  post_save:
    - name: "purge_cvv"
      class: "myapp.hooks.PurgeCVVHook"
      config:
        after_seconds: 300  # Delete CVV after 5 minutes
```

## E-commerce Patterns

### Product Catalog

**Use Case:** Product listings with variants and pricing.

```yaml
target_table: products

schema:
  sku:
    type: CharField
    max_length: 50
    unique: true

  name: CharField
  description: TextField

  category:
    type: CharField
    validation:
      - type: in_choices
        choices: ["electronics", "clothing", "books", "home"]

  price:
    type: FloatField
    validation:
      - type: range
        min: 0.01

  stock:
    type: IntegerField
    default: 0

  attributes:
    type: JSONField

config:
  business_key: [sku]
  on_conflict: "update"

hooks:
  pre_save:
    - name: "price_validation"
      class: "myapp.hooks.PriceValidationHook"

    - name: "inventory_alert"
      class: "myapp.hooks.LowStockAlertHook"
      config:
        threshold: 10
```

### Customer Orders

**Use Case:** Order processing with line items.

```yaml
target_table: orders

schema:
  order_id:
    type: CharField
    unique: true
    validation:
      - type: regex
        pattern: "^ORD-[0-9]{10}$"

  customer_email:
    type: EncryptedEmailField
    key_id: "customer_pii"

  items:
    type: JSONField

  total_amount:
    type: FloatField

  order_date:
    type: DateTimeField

hooks:
  pre_save:
    - name: "calculate_total"
      class: "myapp.hooks.CalculateOrderTotalHook"

  post_save:
    - name: "send_confirmation"
      class: "myapp.hooks.OrderConfirmationEmailHook"

    - name: "update_inventory"
      class: "myapp.hooks.UpdateInventoryHook"
```

## API Integration Patterns

### Data Enrichment

**Use Case:** Enrich records with external API data.

```yaml
target_table: customers

schema:
  email: EmailField
  address: CharField

  # Enriched fields
  latitude: FloatField
  longitude: FloatField
  timezone: CharField

hooks:
  pre_save:
    - name: "geocode"
      class: "myapp.hooks.GeocodeHook"
      order: 10
      config:
        api_key: "${GEOCODE_API_KEY}"
        api_url: "https://api.geocode.com"

    - name: "timezone_lookup"
      class: "myapp.hooks.TimezoneLookupHook"
      order: 20
```

### External System Sync

**Use Case:** Sync data to external systems.

```yaml
hooks:
  post_save:
    - name: "sync_to_crm"
      class: "myapp.hooks.CRMSyncHook"
      config:
        api_url: "https://api.crm.com"
        api_key: "${CRM_API_KEY}"
        batch_sync: true
        batch_size: 100
```

## Data Quality Patterns

### Deduplication

**Use Case:** Prevent duplicate records.

```yaml
config:
  business_key: [email]
  on_conflict: "update"

  deduplication:
    enabled: true
    strategy: "keep_latest"
    compare_fields: [email, phone]
```

### Data Validation

**Use Case:** Comprehensive validation rules.

```yaml
schema:
  email:
    type: EmailField
    validation:
      - type: not_empty
      - type: email_format
      - type: regex
        pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"

  age:
    type: IntegerField
    validation:
      - type: range
        min: 18
        max: 120

  phone:
    type: CharField
    validation:
      - type: regex
        pattern: "^\\+?1?\\d{10,14}$"
```

### Data Completeness

**Use Case:** Ensure required fields are present.

```yaml
hooks:
  pre_save:
    - name: "completeness_check"
      class: "myapp.hooks.CompletenessCheckHook"
      config:
        required_fields: [name, email, phone]
        fail_on_missing: true
```

## Performance Patterns

### Large File Processing

**Use Case:** Optimize for large CSV files (millions of rows).

```yaml
config:
  batch_size: 10000
  parallel_processing: true
  workers: 4

  memory_optimization:
    enabled: true
    chunk_size: 100000

normalization:
  steps:
    - step: StreamingParserStep
      config:
        buffer_size: 65536
```

### Bulk Upsert

**Use Case:** Fast updates of existing records.

```yaml
config:
  business_key: [id]
  on_conflict: "update"

  batch_insert:
    enabled: true
    batch_size: 5000
    use_copy: true  # PostgreSQL COPY for speed
```

### Incremental Loading

**Use Case:** Only load changed/new records.

```yaml
schema:
  id: IntegerField
  updated_at: DateTimeField

hooks:
  pre_normalization:
    - name: "incremental_filter"
      class: "myapp.hooks.IncrementalFilterHook"
      config:
        timestamp_field: "updated_at"
        last_run_file: "./state/last_run.txt"
```

## Complete Real-World Example

### Daily Stock Market Ingestion

```yaml
target_table: stock_prices

schema:
  symbol:
    type: CharField
    max_length: 20
    nullable: false

  date:
    type: DateField
    nullable: false

  open_price:
    type: FloatField
    validation:
      - type: range
        min: 0

  high_price:
    type: FloatField
    validation:
      - type: range
        min: 0

  low_price:
    type: FloatField
    validation:
      - type: range
        min: 0

  close_price:
    type: FloatField
    validation:
      - type: range
        min: 0

  volume:
    type: IntegerField
    validation:
      - type: range
        min: 0

  source_file: CharField
  run_id: CharField
  ingestion_timestamp: DateTimeField

normalization:
  steps:
    - step: RemoveBOMStep

    - step: SkipUntilHeaderStep
      config:
        expected_columns: ["SYMBOL", "DATE"]
        max_skip: 50

    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"

    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true

    - step: DropRepeatedHeaderRowsStep

    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["-", "NA", "NULL", ""]

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns:
          - open_price
          - high_price
          - low_price
          - close_price
          - volume
        date_columns:
          - date
        date_formats:
          - "%d-%b-%Y"
          - "%Y-%m-%d"

config:
  batch_size: 5000
  business_key: [symbol, date]
  on_conflict: "update"

  audit_metadata:
    enabled: true

  dead_letter_queue:
    enabled: true
    path: "./errors"
    format: "json"

  max_errors: 100

hooks:
  pre_save:
    - name: "price_validation"
      class: "myapp.hooks.PriceValidationHook"
      order: 10
      config:
        check_ohlc: true  # Ensure Open <= High, Low <= Close

    - name: "calculate_metrics"
      class: "myapp.hooks.CalculateMetricsHook"
      order: 20
      config:
        add_change_percent: true

  post_save:
    - name: "cache_invalidation"
      class: "myapp.hooks.CacheInvalidationHook"
      order: 30
      config:
        redis_url: "${REDIS_URL}"

  on_complete:
    - name: "send_report"
      class: "myapp.hooks.SendReportHook"
      order: 100
      config:
        email_to: ["admin@example.com"]
```

**Run Command:**
```bash
atomingest run \
  config/stock_prices.yaml \
  postgresql://localhost/finance_db \
  -s data/bhavcopy_$(date +%Y%m%d).csv \
  -b 5000 \
  -v
```

## Related Documentation

- [Pipeline Configuration](pipeline_configuration.md) - Complete reference
- [Normalization](normalization.md) - Data cleaning steps
- [Validation](validation.md) - Validation rules
- [Hooks](hooks.md) - Custom logic
- [Security](security.md) - PII encryption

## Next Steps

1. Choose pattern matching your use case
2. Customize schema and normalization
3. Test with sample data
4. Add hooks for business logic
5. Monitor and optimize performance
