# Core Concepts

Understanding the fundamental concepts behind AtomIngest will help you build robust data ingestion pipelines.

## Architecture Overview

```
┌─────────────┐      ┌──────────────┐      ┌──────────────┐
│   Source    │─────▶│  AtomIngest  │─────▶│   Database   │
│  (CSV/JSON) │      │   Pipeline   │      │  (AtomSQL)   │
└─────────────┘      └──────────────┘      └──────────────┘
                            │
                            ├─ Schema Definition
                            ├─ Normalization
                            ├─ Validation
                            ├─ Transformation
                            └─ Error Handling
```

## The Ingestion Flow

### 1. Configuration Loading

The pipeline starts by loading a YAML configuration file:

```yaml
target_table: customers
schema:
  # Field definitions
normalization:
  # Data transformation steps
```

AtomIngest parses this YAML and builds an execution plan.

### 2. Dynamic Model Generation

Based on the schema, AtomIngest **dynamically creates** an AtomSQL ORM model:

```yaml
# YAML Configuration
schema:
  name:
    type: CharField
    max_length: 255
  email:
    type: EmailField
```

Becomes:

```python
# Generated at Runtime
class Customer(Model):
    _table_name = "customers"
    name = CharField(max_length=255)
    email = EmailField()
```

**Key Benefit**: No manual model code needed!

### 3. Source Reading

AtomIngest reads data from various sources:

- **CSV files** (most common)
- **JSON files** (coming soon)
- **APIs** (via hooks)
- **Streaming sources** (future)

### 4. Normalization Pipeline

Each record flows through normalization steps:

```
Raw Record → Step 1 → Step 2 → Step 3 → Clean Record
```

Example steps:
- Trim whitespace
- Standardize column names
- Convert data types
- Handle null values

### 5. Validation

Validates data against schema and business rules:

```yaml
schema:
  amount:
    type: DecimalField
    validation:
      min: 0
      max: 1000000
```

Invalid records are sent to the Dead Letter Queue (DLQ).

### 6. Transformation Hooks

Custom Python functions for complex logic:

```python
def enrich_record(record):
    record['risk_score'] = calculate_risk(record)
    return record
```

### 7. Database Operations

Finally, data is persisted using AtomSQL:

- **Insert**: Add new records
- **Update**: Modify existing records
- **Upsert**: Insert or update based on business keys

## Key Components

### 1. Pipeline Configuration

The YAML file that defines everything:

```yaml
# What table to populate
target_table: trades

# How to handle duplicates
business_key: [trade_id]

# What the data looks like
schema:
  trade_id:
    type: CharField
    max_length: 50

# How to clean it
normalization:
  steps:
    - step: TrimWhitespaceStep

# Custom logic
hooks:
  pre_save: "hooks.add_metadata"
```

### 2. Schema Definition

Maps CSV columns to database fields:

```yaml
schema:
  # column_name:
  #   type: FieldType
  #   constraints...

  customer_id:
    type: CharField
    max_length: 50
    unique: true
    nullable: false

  balance:
    type: DecimalField
    nullable: false
    validation:
      min: 0
```

**Field Types** map directly to AtomSQL:
- `CharField` → VARCHAR
- `IntegerField` → INTEGER
- `DecimalField` → FLOAT/NUMERIC
- `DateField` → DATE
- `DateTimeField` → TIMESTAMP
- `BooleanField` → BOOLEAN
- `EmailField` → VARCHAR (with validation)

See [Field Types Reference](field_types.md) for complete list.

### 3. Normalization Steps

Chainable transformations:

```yaml
normalization:
  steps:
    # 1. Fix column names
    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"

    # 2. Clean whitespace
    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true

    # 3. Handle nulls
    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["-", "NA", "NULL", ""]

    # 4. Convert types
    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns: [amount, quantity]
        date_columns: [trade_date]
        date_formats: ["%Y-%m-%d"]
```

**Built-in Steps**:
- `ColumnNameNormalizationStep` - Standardize column names
- `TrimWhitespaceStep` - Remove spaces and quotes
- `NullTokenNormalizationStep` - Handle null values
- `BasicTypeCleanupStep` - Type conversion
- Custom steps via Python classes

### 4. Validation

Two levels of validation:

**Schema-level** (automatic):
```yaml
schema:
  email:
    type: EmailField  # Auto-validates email format
    nullable: false   # Auto-validates not null
```

**Field-level** (custom):
```yaml
schema:
  amount:
    type: DecimalField
    validation:
      min: 0.01
      max: 1000000

  account_number:
    type: CharField
    validation:
      regex: "^ACC[0-9]{10}$"
```

### 5. Ingestion Strategies

How to handle existing data:

#### Insert Only
```yaml
```
- Adds new records only
- Fails if duplicate key exists
- Use for: append-only logs, audit trails

#### Update Only
```yaml
business_key: [customer_id]
```
- Updates existing records only
- Ignores new records
- Use for: data corrections, status updates

#### Upsert (Insert or Update)
```yaml
business_key: [trade_id, date]
```
- Updates if record exists (based on business keys)
- Inserts if record is new
- Use for: daily feeds, incremental loads

### 6. Hooks

Extend pipelines with custom Python code:

```yaml
hooks:
  pre_read: "hooks.fetch_from_api"
  post_normalize: "hooks.enrich_data"
  pre_save: "hooks.add_metadata"
  post_save: "hooks.send_notification"
```

Hook function signature:

```python
def my_hook(record: dict) -> dict:
    """
    Process a single record.

    Args:
        record: Dictionary of field values

    Returns:
        Modified record dictionary
    """
    # Your logic here
    record['processed_at'] = datetime.now()
    return record
```

## Metadata-Driven Philosophy

AtomIngest embraces **"Configuration as Code"**:

### Traditional Approach ❌
```python
# Manual model definition
class Trade(Model):
    trade_id = CharField(max_length=50)
    symbol = CharField(max_length=10)
    quantity = IntegerField()
    price = DecimalField()

# Manual CSV parsing
def ingest_trades(file_path):
    with open(file_path) as f:
        reader = csv.DictReader(f)
        for row in reader:
            # Manual validation
            if not row['trade_id']:
                continue
            # Manual type conversion
            quantity = int(row['quantity'])
            price = float(row['price'])
            # Manual insertion
            trade = Trade.create(...)
```

### AtomIngest Approach ✅
```yaml
target_table: trades
schema:
  trade_id:
    type: CharField
    max_length: 50
  symbol:
    type: CharField
    max_length: 10
  quantity:
    type: IntegerField
  price:
    type: DecimalField

normalization:
  steps:
    - step: BasicTypeCleanupStep
```

```bash
atomingest run trades.yaml --source trades.csv
```

**Benefits**:
- ✅ No boilerplate code
- ✅ Easy to modify (just edit YAML)
- ✅ Version controlled
- ✅ Self-documenting
- ✅ Testable

## Error Handling

AtomIngest provides robust error handling:

### Dead Letter Queue (DLQ)

Failed records are automatically saved:

```python
stats = ingester.run(config="pipeline.yaml", source="data.csv")

print(f"Failed: {stats.failed} records")

# Failed records are in the DLQ table
# Query them for analysis
```

### Error Categories

1. **Validation Errors**
   - Type mismatches
   - Constraint violations
   - Regex failures

2. **Transformation Errors**
   - Date parsing failures
   - Number conversion errors
   - Custom hook exceptions

3. **Database Errors**
   - Duplicate key violations
   - Foreign key violations
   - Connection issues

## Performance Considerations

### Batching

Process records in chunks:

```python
ingester = Ingester(db, batch_size=1000)
```

Default: 500 records per batch

### Transactions

Each batch is a transaction - rollback on error:

```python
with db.transaction():
    # All operations are atomic
    ingester.run(config, source)
```

### Indexing

Add indexes to business keys for faster upserts:

```sql
CREATE INDEX idx_trade_id ON trades(trade_id);
```

## Security & Compliance

### PII Protection

Encrypt sensitive fields:

```yaml
schema:
  ssn:
    type: CharField
    max_length: 11
    encrypted: true  # Transparent encryption
```

### Audit Trails

Automatic metadata tracking:

```yaml
audit:
  enabled: true
  track_changes: true
```

Adds:
- `created_at` timestamp
- `updated_at` timestamp
- `created_by` user info

### Immutable Ledgers

Prevent modifications:

```yaml
table_options:
  immutable: true
```

## Testing Pipelines

### Dry Run Mode

Test without database writes:

```bash
atomingest run pipeline.yaml --source data.csv --dry-run
```

### Validation Only

Check configuration:

```bash
atomingest validate pipeline.yaml
```

### Unit Testing

Test pipelines in Python:

```python
from atomingest.core import Ingester
from atomsql import Database

def test_customer_pipeline():
    db = Database("sqlite:///:memory:")
    ingester = Ingester(db)

    stats = ingester.run(
        config="customers.yaml",
        source="test_data.csv"
    )

    assert stats.processed == 3
    assert stats.failed == 0
```

## Design Principles

AtomIngest follows these principles:

1. **Declarative over Imperative**
   - Define *what* you want, not *how* to do it

2. **Convention over Configuration**
   - Sensible defaults, customize when needed

3. **Fail Fast**
   - Validate early, fail clearly

4. **Composability**
   - Mix and match normalization steps

5. **Extensibility**
   - Hooks for custom logic

6. **Type Safety**
   - Strong typing via AtomSQL fields

## Next Steps

Now that you understand the concepts:

1. **[Pipeline Configuration](pipeline_configuration.md)** - Master YAML syntax
2. **[Schema Definition](schema_definition.md)** - All field types and options
3. **[Normalization](normalization.md)** - Deep dive into data transformation
4. **[Validation](validation.md)** - Custom validation rules
5. **[Hooks](hooks.md)** - Extend with custom logic
