# Comprehensive User Guide for AtomIngest

A complete guide to building production-ready data ingestion pipelines with AtomIngest, covering everything from basic concepts to advanced fintech implementations.

## Table of Contents

1. [Getting Started](#getting-started)
2. [Core Concepts](#core-concepts)
3. [Pipeline Development](#pipeline-development)
4. [Advanced Features](#advanced-features)
5. [Fintech Use Cases](#fintech-use-cases)
6. [Production Deployment](#production-deployment)
7. [Best Practices](#best-practices)
8. [Troubleshooting](#troubleshooting)

## Getting Started

### What is AtomIngest?

AtomIngest is a metadata-driven data ingestion framework that eliminates boilerplate code by using YAML configuration files to define:
- Data schemas and validation rules
- Transformation and normalization steps
- Security and compliance requirements
- Performance optimizations
- Error handling and monitoring

**Current Database Support**: PostgreSQL and SQLite
**Planned**: TimescaleDB, ClickHouse, InfluxDB, Redis

### Why Choose AtomIngest?

**For Development Teams:**
- 🚀 **Rapid Development**: Define pipelines in YAML, not code
- 🔒 **Built-in Security**: PII encryption, field-level access control
- 📊 **Data Quality**: Comprehensive validation and cleansing
- 🎯 **Type Safety**: Dynamic ORM model generation
- 🔧 **Extensibility**: Custom hooks and validation rules

**For Fintech Organizations:**
- ✅ **Regulatory Compliance**: SOX, GDPR, PCI-DSS, Basel III support
- 🛡️ **Risk Management**: Real-time validation and anomaly detection
- 📈 **High Performance**: Optimized for financial data volumes
- 📋 **Audit Trails**: Immutable compliance logs
- 🏦 **Industry Patterns**: Pre-built fintech data models

### Installation

```bash
# Install AtomIngest
pip install atomingest

# Verify installation
atomingest --version

# Get help
atomingest --help
```

## Core Concepts

### 1. Pipeline Configuration

Every AtomIngest pipeline is defined by a YAML configuration file:

```yaml
# Basic pipeline structure
target_table: customer_data

schema:
  # Field definitions

normalization:
  # Data cleansing steps

validation:
  # Business rules

hooks:
  # Custom logic
```

### 2. Schema Definition

Define your data model with rich field types and validation:

```yaml
schema:
  customer_id:
    type: CharField
    max_length: 50
    unique: true
    nullable: false

  email:
    type: EmailField
    validation:
      - type: "email_format"
      - type: "domain_blacklist"
        config:
          blocked_domains: ["temp-mail.org"]

  balance:
    type: DecimalField
    precision: 15
    scale: 2
    validation:
      min: 0
      business_rules:
        - rule: "balance_limit_check"
          params:
            max_balance: 1000000.00
```

### 3. Data Types

AtomIngest supports comprehensive data types:

| Type | Description | Example |
|------|-------------|---------|
| `CharField` | Text fields with length limits | Names, IDs |
| `EmailField` | Email validation built-in | Email addresses |
| `DecimalField` | High precision numbers | Financial amounts |
| `DateTimeField` | Timestamps with timezone support | Transaction times |
| `JSONField` | Structured data | Addresses, metadata |
| `EncryptedField` | Automatic PII encryption | SSNs, card numbers |
| `BigIntegerField` | Large numbers | Transaction volumes |

### 4. Normalization Steps

Clean and transform data with built-in steps:

```yaml
normalization:
  steps:
    - step: "TrimWhitespaceStep"
      config:
        fields: ["name", "address"]

    - step: "ColumnNameNormalizationStep"
      config:
        case: "lower"
        replace_spaces: "_"

    - step: "CurrencyNormalizationStep"
      config:
        base_currency: "USD"
        fx_service: "xe.com"
```

### 5. Validation Framework

Implement business rules and data quality checks:

```yaml
validation:
  business_rules:
    - rule: "credit_score_validation"
      description: "Validate credit score range"
      config:
        min_score: 300
        max_score: 850

    - rule: "duplicate_customer_check"
      description: "Prevent duplicate customer records"
      config:
        match_fields: ["email", "phone"]
        action: "reject"
```

### 6. Security Features

Built-in security for sensitive data:

```yaml
security:
  encryption:
    enabled: true
    algorithm: "AES-256-GCM"
    key_management: "aws_kms"

  field_classification:
    pii_fields: ["ssn", "email", "phone"]
    financial_fields: ["account_number", "balance"]

  access_control:
    role_based: true
    field_level_permissions: true
```

## Pipeline Development

### Development Workflow

1. **Design Schema**: Define your data model
2. **Configure Pipeline**: Write YAML configuration
3. **Test Locally**: Use sample data
4. **Validate Quality**: Run data quality checks
5. **Deploy**: Move to production environment

### Sample Pipeline Development

Let's build a customer onboarding pipeline:

#### Step 1: Define Schema

```yaml
target_table: customers

schema:
  customer_id:
    type: CharField
    max_length: 50
    unique: true
    nullable: false

  first_name:
    type: CharField
    max_length: 100
    nullable: false
    security:
      classification: "PII"
    validation:
      - type: "name_format"

  last_name:
    type: CharField
    max_length: 100
    nullable: false
    security:
      classification: "PII"

  email:
    type: EmailField
    unique: true
    nullable: false
    validation:
      - type: "email_format"
      - type: "disposable_email_check"

  phone:
    type: CharField
    max_length: 20
    nullable: true
    security:
      classification: "PII"
    validation:
      - type: "phone_format"
      - type: "phone_number_validation"

  date_of_birth:
    type: DateField
    nullable: false
    security:
      classification: "PII"
    validation:
      - type: "age_validation"
        config:
          min_age: 18
          max_age: 120

  credit_score:
    type: IntegerField
    nullable: true
    validation:
      min: 300
      max: 850

  annual_income:
    type: DecimalField
    precision: 12
    scale: 2
    nullable: true
    validation:
      min: 0
      max: 10000000.00

  address:
    type: JSONField
    nullable: false
    security:
      classification: "PII"
    validation:
      - type: "address_validation"
        config:
          required_fields: ["street", "city", "state", "zip"]
          verify_with_usps: true
```

#### Step 2: Add Normalization

```yaml
normalization:
  steps:
    - step: "NameStandardizationStep"
      config:
        title_case: true
        remove_extra_spaces: true

    - step: "PhoneNormalizationStep"
      config:
        format: "E.164"
        default_country: "US"

    - step: "AddressNormalizationStep"
      config:
        standardize_abbreviations: true
        verify_existence: true

    - step: "EmailNormalizationStep"
      config:
        lowercase: true
        remove_dots_gmail: true
```

#### Step 3: Add Business Rules

```yaml
validation:
  business_rules:
    - rule: "duplicate_customer_check"
      description: "Prevent duplicate customers"
      config:
        match_fields: ["email", "phone"]
        fuzzy_name_matching: true
        action: "flag_for_review"

    - rule: "credit_worthiness_check"
      description: "Basic credit assessment"
      config:
        min_credit_score: 600
        min_income: 30000
        debt_to_income_ratio: 0.4

    - rule: "regulatory_compliance"
      description: "KYC/AML compliance checks"
      config:
        sanctions_list_check: true
        pep_screening: true
        identity_verification: true
```

#### Step 4: Add Security

```yaml
security:
  pii_protection:
    enabled: true
    encryption_fields: ["ssn", "phone", "address"]
    tokenization_fields: ["customer_id"]

  audit_logging:
    enabled: true
    log_level: "detailed"
    retention_days: 2555  # 7 years

  access_control:
    rbac_enabled: true
    field_level_permissions:
      pii_data:
        roles: ["compliance_officer", "customer_service_manager"]
      financial_data:
        roles: ["underwriter", "loan_officer"]
```

#### Step 5: Add Hooks

```yaml
hooks:
  pre_insert:
    - hook: "identity_verification"
      config:
        verification_service: "jumio"
        required_documents: ["drivers_license", "passport"]

    - hook: "credit_bureau_check"
      config:
        bureaus: ["experian", "equifax", "transunion"]
        soft_pull: true

  post_insert:
    - hook: "welcome_email"
      config:
        template: "customer_welcome"
        send_delay_minutes: 5

    - hook: "crm_integration"
      config:
        crm_system: "salesforce"
        create_opportunity: true
```

### Testing Your Pipeline

```bash
# Test with sample data
atomingest run customer_pipeline.yaml --input sample_customers.csv --validate-only

# Run full pipeline
atomingest run customer_pipeline.yaml --input sample_customers.csv

# Monitor results
atomingest status customer_pipeline.yaml
```

## Advanced Features

### Performance Optimization

#### High-Throughput Processing

```yaml
performance:
  mode: "high_throughput"
  batch_size: 10000
  parallel_workers: 8
  memory_buffer_mb: 512

  database:
    bulk_insert: true
    connection_pool_size: 20
    disable_foreign_keys: true  # Re-enable after load
```

#### Streaming Processing

```yaml
processing:
  mode: "streaming"
  max_latency_ms: 100
  checkpoint_interval: 1000

streaming:
  source: "kafka"
  topics: ["customer.events"]
  consumer_group: "atomingest.customers"
```

### Error Handling

#### Retry Logic

```yaml
error_handling:
  retry_policy:
    max_attempts: 3
    backoff_strategy: "exponential"
    base_delay_ms: 100
    max_delay_ms: 5000

  circuit_breaker:
    error_rate_threshold: 0.1
    open_duration_seconds: 60
```

#### Dead Letter Queue

```yaml
error_handling:
  dead_letter_queue:
    enabled: true
    table: "failed_customers"
    retention_days: 30
    max_retry_attempts: 5
```

### Monitoring & Observability

```yaml
monitoring:
  metrics:
    - name: "records_processed"
      type: "counter"
      labels: ["pipeline", "status"]

    - name: "processing_latency"
      type: "histogram"
      buckets: [10, 50, 100, 500, 1000, 5000]

  alerts:
    - condition: "error_rate > 0.05"
      action: "send_notification"
      recipients: ["data-team@company.com"]

  dashboards:
    grafana:
      enabled: true
      dashboard_url: "https://grafana.company.com/customers"
```

## Fintech Use Cases

AtomIngest provides specialized support for fintech applications:

### 1. Payment Processing

High-frequency payment transaction processing with fraud detection:

```yaml
# See full example in examples/fintech_samples/payment_processing/
target_table: transactions
processing:
  mode: "real_time"
  max_latency_ms: 100

security:
  pci_compliance: true
  tokenization: true

validation:
  fraud_detection: true
  velocity_checks: true
```

### 2. Market Data Ingestion

Ultra-high frequency market data processing:

```yaml
# See full example in examples/fintech_samples/market_data/
target_table: market_data
processing:
  mode: "streaming"
  throughput_target: 100000  # msg/sec

database:
  type: "timescaledb"
  hypertables: true

normalization:
  technical_indicators: true
```

### 3. Customer KYC

Customer onboarding with regulatory compliance:

```yaml
# See full example in examples/fintech_samples/customer_kyc/
target_table: customer_kyc

security:
  pii_encryption: true
  gdpr_compliance: true

validation:
  identity_verification: true
  aml_screening: true
  sanctions_screening: true
```

### 4. Risk Reporting

Regulatory risk reporting and calculations:

```yaml
# See full example in examples/fintech_samples/risk_reporting/
target_table: risk_reports

schedule:
  cron: "0 1 * * *"  # Daily at 1 AM

normalization:
  risk_calculations: true
  stress_testing: true

compliance:
  regulatory_submission: true
```

## Production Deployment

### Environment Setup

#### Development

```bash
# Local development with Docker
docker-compose up -d postgres redis

# Environment variables
export ATOMINGEST_ENV=development
export DB_URL=postgresql://user:pass@localhost:5432/dev
```

#### Staging

```bash
# Staging environment
export ATOMINGEST_ENV=staging
export DB_URL=postgresql://user:pass@staging-db:5432/staging
export REDIS_URL=redis://staging-redis:6379/0
```

#### Production

```bash
# Production environment
export ATOMINGEST_ENV=production
export DB_URL=postgresql://user:pass@prod-db:5432/prod
# Note: Redis integration planned for future versions

# Security
export ENCRYPTION_KEY_ID=arn:aws:kms:us-east-1:123:key/abc-123
export AUDIT_LOG_LEVEL=detailed
```

### Kubernetes Deployment

```yaml
# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: atomingest-pipeline
spec:
  replicas: 3
  selector:
    matchLabels:
      app: atomingest-pipeline
  template:
    metadata:
      labels:
        app: atomingest-pipeline
    spec:
      containers:
      - name: atomingest
        image: atomingest:latest
        env:
        - name: DB_URL
          valueFrom:
            secretKeyRef:
              name: database-secret
              key: url
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
```

### Scaling Considerations

#### Horizontal Scaling

```yaml
# Multiple workers
processing:
  parallel_workers: 8
  worker_distribution: "round_robin"

# Database sharding
database:
  sharding:
    enabled: true
    shard_key: "customer_id"
    num_shards: 4
```

#### Vertical Scaling

```yaml
# Resource optimization
performance:
  memory_buffer_mb: 2048
  cpu_intensive_operations: true

database:
  connection_pool_size: 50
  max_connections: 200
```

## Best Practices

### 1. Schema Design

- **Use appropriate data types**: Choose precise types for data integrity
- **Add validation early**: Catch issues at ingestion time
- **Plan for evolution**: Design schemas that can evolve
- **Security by design**: Classify sensitive fields from the start

### 2. Performance Optimization

- **Batch processing**: Use appropriate batch sizes
- **Parallel processing**: Leverage multiple workers
- **Database tuning**: Optimize for your workload
- **Memory management**: Monitor and tune memory usage

### 3. Error Handling

- **Fail fast**: Catch errors early in the pipeline
- **Comprehensive logging**: Log all errors with context
- **Dead letter queues**: Handle permanent failures
- **Alerting**: Set up proactive monitoring

### 4. Security

- **Encrypt sensitive data**: Use field-level encryption
- **Access controls**: Implement role-based permissions
- **Audit logging**: Track all data access
- **Regular reviews**: Audit security configurations

### 5. Testing

- **Unit tests**: Test individual components
- **Integration tests**: Test full pipeline flow
- **Data quality tests**: Validate business rules
- **Performance tests**: Ensure scalability

## Troubleshooting

### Common Issues

#### Performance Problems

**Symptom**: Slow processing
**Solutions**:
- Increase batch size
- Add more parallel workers
- Optimize database queries
- Review normalization steps

**Symptom**: High memory usage
**Solutions**:
- Reduce batch size
- Enable streaming mode
- Check for memory leaks
- Optimize data structures

#### Data Quality Issues

**Symptom**: Validation failures
**Solutions**:
- Review validation rules
- Check data sources
- Add data cleansing steps
- Improve error messages

**Symptom**: Duplicate records
**Solutions**:
- Add uniqueness constraints
- Implement deduplication
- Review business rules
- Check data sources

#### Security Concerns

**Symptom**: PII in logs
**Solutions**:
- Enable data masking
- Review logging configuration
- Update field classifications
- Audit log outputs

### Debugging Tools

```bash
# Check pipeline status
atomingest status my_pipeline.yaml

# Validate configuration
atomingest validate my_pipeline.yaml

# Test with sample data
atomingest test my_pipeline.yaml --input test_data.csv

# Debug mode
atomingest run my_pipeline.yaml --debug --log-level DEBUG
```

### Performance Monitoring

```bash
# Monitor metrics
atomingest metrics my_pipeline.yaml

# Performance profiling
atomingest profile my_pipeline.yaml --duration 60s

# Resource usage
atomingest resources my_pipeline.yaml
```

## Conclusion

AtomIngest provides a powerful, flexible framework for building production-ready data ingestion pipelines. By following this guide and leveraging the provided examples, you can:

- Build secure, compliant data pipelines quickly
- Handle complex fintech requirements with ease
- Scale to handle high-volume data processing
- Maintain data quality and governance
- Monitor and troubleshoot effectively

For specific use cases, refer to the detailed examples in the `examples/fintech_samples/` directory, and consult the [API Reference](api_reference.md) for advanced configuration options.

## Next Steps

1. **Try the Examples**: Start with the provided fintech samples
2. **Build Your Pipeline**: Follow the development workflow
3. **Join the Community**: Contribute to the AtomIngest project
4. **Get Support**: Use GitHub issues for questions and feedback

Happy ingesting! 🚀
