# AtomIngest Architecture

This document provides visual diagrams of the AtomIngest architecture using Mermaid.

---

## 1. High-Level Architecture

```mermaid
graph TB
    subgraph "Data Sources"
        CSV[CSV Files]
        JSON[JSON Files]
        EXCEL[Excel Files]
        PARQUET[Parquet Files]
    end

    subgraph "AtomIngest Framework"
        CLI[CLI Interface]
        INGESTER[Ingester]
        
        subgraph "Core Components"
            LOADER[Schema Loader]
            FACTORY[Model Factory]
            RESOLVER[Dependency Resolver]
        end
        
        subgraph "Normalization Pipeline"
            NORM_PIPE[Normalization Pipeline]
            STEPS[Normalization Steps]
            REPORT[Normalization Report]
        end
        
        subgraph "Validation & Schema"
            VALIDATOR[Validator]
            DRIFT[Drift Detector]
            DECORATORS[Field Decorators]
        end
        
        subgraph "Pipeline & Hooks"
            HOOKS[Hook Registry]
            AUDIT[Audit Metadata]
            DLQ[Dead Letter Queue]
        end
        
        subgraph "Security"
            KMS[Key Manager]
            ENCRYPT[Field Encryption]
        end
        
        subgraph "Observability"
            METRICS[Metrics Collector]
            LOGGING[JSON Logging]
        end
    end

    subgraph "AtomSQL ORM"
        DB[Database]
        MODELS[Dynamic Models]
        POOL[Connection Pool]
    end

    subgraph "Target Database"
        POSTGRES[(PostgreSQL)]
        SQLITE[(SQLite)]
        MYSQL[(MySQL)]
    end

    CSV --> CLI
    JSON --> CLI
    EXCEL --> CLI
    PARQUET --> CLI
    
    CLI --> INGESTER
    INGESTER --> LOADER
    LOADER --> FACTORY
    FACTORY --> MODELS
    
    INGESTER --> NORM_PIPE
    NORM_PIPE --> STEPS
    STEPS --> REPORT
    
    INGESTER --> VALIDATOR
    VALIDATOR --> DECORATORS
    
    INGESTER --> HOOKS
    HOOKS --> AUDIT
    HOOKS --> DLQ
    
    INGESTER --> KMS
    KMS --> ENCRYPT
    
    INGESTER --> METRICS
    INGESTER --> LOGGING
    
    INGESTER --> RESOLVER
    
    MODELS --> DB
    DB --> POOL
    POOL --> POSTGRES
    POOL --> SQLITE
    POOL --> MYSQL
    
    DRIFT -.-> MODELS
    DRIFT -.-> DB

    style INGESTER fill:#4CAF50,stroke:#2E7D32,color:#fff
    style CLI fill:#2196F3,stroke:#1565C0,color:#fff
    style DB fill:#FF9800,stroke:#E65100,color:#fff
```

---

## 2. Data Flow Pipeline

```mermaid
flowchart TD
    START([Start Ingestion]) --> READ_CONFIG[Read YAML Config]
    READ_CONFIG --> LOAD_SCHEMA[Load Schema Definition]
    LOAD_SCHEMA --> CREATE_MODEL[Create Dynamic Model]
    CREATE_MODEL --> REGISTER[Register Model with DB]
    REGISTER --> READ_SOURCE[Read Source Data]
    
    READ_SOURCE --> NORMALIZE{Normalization<br/>Enabled?}
    NORMALIZE -->|Yes| NORM_STEPS[Run Normalization Steps]
    NORMALIZE -->|No| VALIDATE
    
    NORM_STEPS --> REMOVE_BOM[Remove BOM]
    REMOVE_BOM --> SKIP_HEADER[Skip Until Header]
    SKIP_HEADER --> DROP_REPEATED[Drop Repeated Headers]
    DROP_REPEATED --> TRIM_WS[Trim Whitespace]
    TRIM_WS --> NULL_TOKENS[Normalize Null Tokens]
    NULL_TOKENS --> COL_NAMES[Normalize Column Names]
    COL_NAMES --> TYPE_CLEANUP[Basic Type Cleanup]
    TYPE_CLEANUP --> DROP_TRAILER[Drop Trailer Lines]
    
    DROP_TRAILER --> VALIDATE[Validate Data]
    VALIDATE --> PRE_SAVE[Execute Pre-Save Hooks]
    
    PRE_SAVE --> BATCH{Batch<br/>Processing?}
    BATCH -->|Yes| BATCH_INSERT[Batch Insert/Upsert]
    BATCH -->|No| SINGLE_INSERT[Single Insert/Upsert]
    
    BATCH_INSERT --> POST_SAVE[Execute Post-Save Hooks]
    SINGLE_INSERT --> POST_SAVE
    
    POST_SAVE --> ERROR{Errors?}
    ERROR -->|Yes| DLQ_WRITE[Write to DLQ]
    ERROR -->|No| SUCCESS[Record Success]
    
    DLQ_WRITE --> METRICS[Update Metrics]
    SUCCESS --> METRICS
    
    METRICS --> MORE{More<br/>Rows?}
    MORE -->|Yes| READ_SOURCE
    MORE -->|No| COMPLETE[Execute On-Complete Hooks]
    
    COMPLETE --> REPORT[Generate Report]
    REPORT --> END([End Ingestion])
    
    style START fill:#4CAF50,stroke:#2E7D32,color:#fff
    style END fill:#4CAF50,stroke:#2E7D32,color:#fff
    style ERROR fill:#F44336,stroke:#C62828,color:#fff
    style DLQ_WRITE fill:#FF9800,stroke:#E65100,color:#fff
    style NORMALIZE fill:#2196F3,stroke:#1565C0,color:#fff
    style BATCH fill:#2196F3,stroke:#1565C0,color:#fff
```

---

## 3. Component Relationships

```mermaid
classDiagram
    class Ingester {
        +Database db
        +SchemaLoader schema_loader
        +HookRegistry hook_registry
        +run(config, source) IngestionStats
        +run_async(config, source) IngestionStats
    }
    
    class SchemaLoader {
        +load_from_yaml(path) dict
        +create_model(schema) Model
        +resolve_field_type(type_str) Field
    }
    
    class NormalizationPipeline {
        +List~Step~ steps
        +NormalizationConfig config
        +normalize(rows) tuple
        +add_step(step)
    }
    
    class HookRegistry {
        +Dict hooks
        +register(type, hook)
        +execute_hooks(type, context)
    }
    
    class Validator {
        +validate(value, rules) bool
        +add_validator(name, func)
    }
    
    class KeyManager {
        +generate_key()
        +encrypt(data, key)
        +decrypt(data, key)
        +crypto_shred(key_id)
    }
    
    class MetricsCollector {
        +track_normalization(stats)
        +track_hook_execution(hook, duration)
        +record_dlq_entry(error)
    }
    
    class DependencyResolver {
        +resolve_order(configs) List
        +build_dag(configs) Graph
    }
    
    class SchemaDriftDetector {
        +detect_drift(schema, db) Report
        +compare_schemas(yaml, db) List
    }
    
    Ingester --> SchemaLoader : uses
    Ingester --> NormalizationPipeline : uses
    Ingester --> HookRegistry : uses
    Ingester --> Validator : uses
    Ingester --> MetricsCollector : uses
    Ingester --> DependencyResolver : uses
    SchemaLoader --> Validator : creates
    HookRegistry --> MetricsCollector : tracks
    SchemaDriftDetector --> SchemaLoader : validates
```

---

## 4. Hook System Architecture

```mermaid
sequenceDiagram
    participant User
    participant Ingester
    participant HookRegistry
    participant PreSaveHook
    participant TransformHook
    participant PostSaveHook
    participant OnErrorHook
    participant OnCompleteHook
    
    User->>Ingester: run(config, source)
    Ingester->>HookRegistry: load hooks from config
    
    loop For each row
        Ingester->>HookRegistry: execute_hooks(PRE_SAVE)
        HookRegistry->>PreSaveHook: execute(context)
        PreSaveHook-->>HookRegistry: modified_context
        
        Ingester->>HookRegistry: execute_hooks(TRANSFORM)
        HookRegistry->>TransformHook: execute(context)
        TransformHook-->>HookRegistry: transformed_data
        
        alt Save Successful
            Ingester->>HookRegistry: execute_hooks(POST_SAVE)
            HookRegistry->>PostSaveHook: execute(context)
            PostSaveHook-->>HookRegistry: success
        else Save Failed
            Ingester->>HookRegistry: execute_hooks(ON_ERROR)
            HookRegistry->>OnErrorHook: execute(error_context)
            OnErrorHook-->>HookRegistry: error_handled
        end
    end
    
    Ingester->>HookRegistry: execute_hooks(ON_COMPLETE)
    HookRegistry->>OnCompleteHook: execute(stats)
    OnCompleteHook-->>HookRegistry: complete
    
    Ingester-->>User: IngestionStats
```

---

## 5. Normalization Pipeline Flow

```mermaid
stateDiagram-v2
    [*] --> ReadRawData
    ReadRawData --> RemoveBOM: Raw CSV/JSON
    
    RemoveBOM --> SkipUntilHeader: BOM Removed
    SkipUntilHeader --> DropRepeatedHeaders: Header Found
    DropRepeatedHeaders --> DropTrailerLines: Duplicates Removed
    DropTrailerLines --> TrimWhitespace: Trailers Removed
    
    TrimWhitespace --> NullTokenNormalization: Whitespace Cleaned
    NullTokenNormalization --> ColumnNameNormalization: Nulls Standardized
    ColumnNameNormalization --> BasicTypeCleanup: Columns Normalized
    
    BasicTypeCleanup --> ValidationCheck: Types Cleaned
    
    ValidationCheck --> Success: Valid
    ValidationCheck --> Warning: Warnings
    ValidationCheck --> Error: Invalid
    
    Success --> [*]
    Warning --> [*]
    Error --> [*]
    
    note right of RemoveBOM
        Removes UTF-8 BOM
        characters
    end note
    
    note right of SkipUntilHeader
        Skips metadata lines
        until header found
    end note
    
    note right of BasicTypeCleanup
        Parses dates, numbers,
        booleans
    end note
```

---

## 6. Security & Encryption Flow

```mermaid
graph LR
    subgraph "Data Input"
        RAW[Raw PII Data]
    end
    
    subgraph "Key Management"
        KMS[Key Manager]
        MASTER[Master Key]
        DEK[Data Encryption Keys]
    end
    
    subgraph "Encryption Layer"
        ENCRYPT[Field Encryption]
        DECRYPT[Field Decryption]
    end
    
    subgraph "Storage"
        DB[(Database)]
        VAULT[PII Vault]
    end
    
    subgraph "Compliance"
        AUDIT[Audit Log]
        SHRED[Crypto Shredding]
    end
    
    RAW --> ENCRYPT
    KMS --> MASTER
    MASTER --> DEK
    DEK --> ENCRYPT
    ENCRYPT --> VAULT
    VAULT --> DB
    
    DB --> DECRYPT
    DEK --> DECRYPT
    DECRYPT --> OUTPUT[Decrypted Data]
    
    ENCRYPT --> AUDIT
    DECRYPT --> AUDIT
    
    SHRED -.-> DEK
    SHRED -.-> VAULT
    
    style RAW fill:#F44336,stroke:#C62828,color:#fff
    style ENCRYPT fill:#4CAF50,stroke:#2E7D32,color:#fff
    style VAULT fill:#9C27B0,stroke:#6A1B9A,color:#fff
    style SHRED fill:#FF9800,stroke:#E65100,color:#fff
```

---

## 7. Module Dependencies

```mermaid
graph TD
    subgraph "atomingest"
        INIT[__init__.py]
        
        subgraph "core"
            INGESTER_MOD[ingester.py]
            LOADER_MOD[loader.py]
            FACTORY_MOD[factory.py]
            RESOLVER_MOD[dependency_resolver.py]
            MIXINS_MOD[mixins.py]
        end
        
        subgraph "cli"
            MAIN_CLI[main.py]
        end
        
        subgraph "normalization"
            NORM_CORE[core.py]
            NORM_PIPE[pipeline.py]
            NORM_STEPS[steps.py]
            NORM_STD[standard_steps.py]
            NORM_IO[io.py]
            NORM_INT[integration.py]
        end
        
        subgraph "schema"
            VALIDATORS[validators.py]
            DECORATORS[decorators.py]
            DRIFT_DET[drift_detector.py]
            ENCRYPTION[encryption.py]
        end
        
        subgraph "pipeline"
            HOOKS_MOD[hooks.py]
            AUDIT_MOD[audit_metadata.py]
            DLQ_MOD[dlq.py]
        end
        
        subgraph "security"
            KMS_MOD[kms.py]
            VAULT_MOD[vault.py]
        end
        
        subgraph "observability"
            METRICS_MOD[metrics.py]
            LOGGING_MOD[json_logging.py]
        end
    end
    
    subgraph "atomsql"
        ATOMSQL_DB[Database]
        ATOMSQL_MODEL[Model]
        ATOMSQL_FIELD[Field]
    end
    
    INIT --> INGESTER_MOD
    INIT --> LOADER_MOD
    INIT --> NORM_CORE
    INIT --> VALIDATORS
    INIT --> RESOLVER_MOD
    INIT --> KMS_MOD
    INIT --> HOOKS_MOD
    INIT --> METRICS_MOD
    
    MAIN_CLI --> INGESTER_MOD
    MAIN_CLI --> LOADER_MOD
    
    INGESTER_MOD --> LOADER_MOD
    INGESTER_MOD --> NORM_INT
    INGESTER_MOD --> HOOKS_MOD
    INGESTER_MOD --> AUDIT_MOD
    INGESTER_MOD --> METRICS_MOD
    
    LOADER_MOD --> FACTORY_MOD
    LOADER_MOD --> VALIDATORS
    LOADER_MOD --> ATOMSQL_FIELD
    
    FACTORY_MOD --> ATOMSQL_MODEL
    FACTORY_MOD --> MIXINS_MOD
    
    NORM_INT --> NORM_PIPE
    NORM_PIPE --> NORM_STEPS
    NORM_PIPE --> NORM_STD
    NORM_PIPE --> NORM_IO
    
    VALIDATORS --> DECORATORS
    
    HOOKS_MOD --> AUDIT_MOD
    HOOKS_MOD --> DLQ_MOD
    
    KMS_MOD --> ENCRYPTION
    ENCRYPTION --> VAULT_MOD
    
    INGESTER_MOD --> ATOMSQL_DB
    
    style INIT fill:#4CAF50,stroke:#2E7D32,color:#fff
    style INGESTER_MOD fill:#2196F3,stroke:#1565C0,color:#fff
    style ATOMSQL_DB fill:#FF9800,stroke:#E65100,color:#fff
```

---

## 8. CLI Command Flow

```mermaid
flowchart TD
    START([atomingest CLI]) --> CMD{Command}
    
    CMD -->|run| RUN[Run Pipeline]
    CMD -->|validate| VAL[Validate Config]
    CMD -->|drift-detect| DRIFT[Detect Schema Drift]
    CMD -->|normalize| NORM[Normalize Data]
    CMD -->|info| INFO[Show Info]
    
    RUN --> PARSE_ARGS[Parse Arguments]
    PARSE_ARGS --> LOAD_DB[Load Database]
    LOAD_DB --> CREATE_ING[Create Ingester]
    CREATE_ING --> EXEC[Execute Pipeline]
    EXEC --> STATS[Display Statistics]
    STATS --> EXIT_RUN([Exit])
    
    VAL --> LOAD_YAML[Load YAML Config]
    LOAD_YAML --> CHECK_SCHEMA[Check Schema]
    CHECK_SCHEMA --> CHECK_VALID{Valid?}
    CHECK_VALID -->|Yes| VALID[✓ Config Valid]
    CHECK_VALID -->|No| INVALID[✗ Config Invalid]
    VALID --> EXIT_VAL([Exit])
    INVALID --> EXIT_VAL
    
    DRIFT --> LOAD_YAML2[Load YAML Config]
    LOAD_YAML2 --> CONNECT_DB[Connect to Database]
    CONNECT_DB --> COMPARE[Compare Schemas]
    COMPARE --> REPORT_DRIFT[Generate Drift Report]
    REPORT_DRIFT --> EXIT_DRIFT([Exit])
    
    NORM --> READ_FILE[Read Data File]
    READ_FILE --> APPLY_NORM[Apply Normalization]
    APPLY_NORM --> WRITE_OUT[Write Output]
    WRITE_OUT --> EXIT_NORM([Exit])
    
    INFO --> SHOW_VER[Show Version]
    SHOW_VER --> SHOW_CONFIG[Show Configuration]
    SHOW_CONFIG --> EXIT_INFO([Exit])
    
    style START fill:#4CAF50,stroke:#2E7D32,color:#fff
    style RUN fill:#2196F3,stroke:#1565C0,color:#fff
    style VAL fill:#9C27B0,stroke:#6A1B9A,color:#fff
    style DRIFT fill:#FF9800,stroke:#E65100,color:#fff
    style NORM fill:#00BCD4,stroke:#006064,color:#fff
```

---

## 9. Error Handling & DLQ

```mermaid
sequenceDiagram
    participant Source as Data Source
    participant Norm as Normalizer
    participant Val as Validator
    participant Ing as Ingester
    participant DB as Database
    participant DLQ as Dead Letter Queue
    participant Metrics as Metrics
    
    Source->>Norm: Raw Data Row
    
    alt Normalization Success
        Norm->>Val: Normalized Row
        
        alt Validation Success
            Val->>Ing: Valid Row
            
            alt Database Insert Success
                Ing->>DB: Insert/Upsert
                DB-->>Ing: Success
                Ing->>Metrics: Record Success
            else Database Error
                DB-->>Ing: Error
                Ing->>DLQ: Write Failed Row
                Ing->>Metrics: Record Failure
            end
            
        else Validation Failed
            Val->>DLQ: Invalid Row + Errors
            Val->>Metrics: Record Validation Failure
        end
        
    else Normalization Failed
        Norm->>DLQ: Malformed Row + Errors
        Norm->>Metrics: Record Normalization Failure
    end
    
    Note over DLQ: DLQ Table Structure:<br/>- original_data<br/>- error_message<br/>- error_type<br/>- timestamp<br/>- run_id
```

---

## 10. Deployment Architecture

```mermaid
C4Context
    title AtomIngest Deployment Architecture
    
    Person(user, "Data Engineer", "Configures and runs ingestion pipelines")
    
    System_Boundary(atomingest, "AtomIngest System") {
        Container(cli, "CLI Tool", "Python", "Command-line interface for pipeline execution")
        Container(core, "Core Engine", "Python", "Ingestion, normalization, validation")
        Container(hooks, "Hook System", "Python", "Pre/post processing hooks")
        Container(security, "Security Layer", "Python", "Encryption and key management")
        Container(observability, "Observability", "Python", "Metrics and logging")
    }
    
    System_Ext(sources, "Data Sources", "CSV, JSON, Excel, Parquet files")
    System_Ext(db, "Target Database", "PostgreSQL, MySQL, SQLite")
    System_Ext(prometheus, "Prometheus", "Metrics collection")
    System_Ext(vault, "HashiCorp Vault", "External key storage (optional)")
    
    Rel(user, cli, "Runs pipelines", "CLI")
    Rel(cli, core, "Executes")
    Rel(core, sources, "Reads from")
    Rel(core, hooks, "Triggers")
    Rel(core, security, "Encrypts data")
    Rel(core, db, "Writes to", "SQL")
    Rel(core, observability, "Reports to")
    Rel(observability, prometheus, "Exports metrics")
    Rel(security, vault, "Stores keys", "HTTPS")
```

---

## Component Descriptions

### Core Components
- **Ingester**: Main orchestrator that coordinates the entire ingestion pipeline
- **SchemaLoader**: Loads YAML configurations and creates dynamic models
- **DependencyResolver**: Resolves pipeline execution order based on dependencies

### Normalization
- **NormalizationPipeline**: Manages the sequence of normalization steps
- **Standard Steps**: Built-in steps for common data cleaning tasks
- **Custom Steps**: User-defined normalization logic

### Validation & Schema
- **Validator**: Validates data against defined rules
- **DriftDetector**: Detects schema changes between YAML and database
- **Decorators**: Field-level validation decorators

### Pipeline & Hooks
- **HookRegistry**: Manages and executes pipeline hooks
- **Audit Metadata**: Tracks ingestion runs and metadata
- **DLQ**: Handles failed records

### Security
- **KeyManager**: Manages encryption keys and crypto-shredding
- **Field Encryption**: Transparent field-level encryption

### Observability
- **MetricsCollector**: Collects and exports Prometheus metrics
- **JSON Logging**: Structured logging for audit and debugging

---

## Key Design Patterns

1. **Metadata-Driven**: All configuration in YAML, zero boilerplate code
2. **Plugin Architecture**: Extensible through hooks and custom steps
3. **Fail-Safe**: DLQ ensures no data loss on errors
4. **Security by Default**: Built-in encryption and compliance features
5. **Observable**: Comprehensive metrics and logging
6. **Idempotent**: Upsert strategies prevent duplicates

---

**Last Updated**: February 10, 2026
