# Pipeline Configuration Reference

Complete guide to YAML pipeline configuration in AtomIngest.

## Basic Structure

```yaml
# Required: target table name in the database
target_table: my_table

# Required: field definitions
schema:
  field_name:
    type: FieldType
    # field options...

# Optional: idempotent upsert — omit for append-only inserts
business_key:
  - field_name

# Optional: data normalisation pipeline
normalization:
  steps: []

# Optional: pipeline-level settings (batch size, audit metadata, etc.)
config:
  batch_size: 100
  audit_metadata:
    enabled: true

# Optional: custom hooks
hooks: []
```

## target_table

**Required**. The database table name where data will be stored.

```yaml
target_table: customers
```

**Rules**:
- Must be a valid SQL identifier
- Case-sensitive on some databases
- Cannot be a SQL reserved keyword
- Will be created automatically if it doesn't exist

**Examples**:
```yaml
# Simple table name
target_table: trades

# Namespace/schema (PostgreSQL)
target_table: public.daily_trades

# Descriptive names
target_table: nse_bhavcopy_2026
```

## schema

**Required**. Defines the table structure and field types.

### Basic Field Definition

```yaml
schema:
  field_name:
    type: FieldType
    nullable: true
    unique: false
    default: null
```

### Field Properties

#### type (Required)

The AtomSQL field type. Available types:

**String Types**:
```yaml
# Variable-length string with max length
name:
  type: CharField
  max_length: 255

# Unlimited text
description:
  type: TextField

# Simple string
code:
  type: StringField
```

**Numeric Types**:
```yaml
# Integer
quantity:
  type: IntegerField

# Decimal/Float
price:
  type: DecimalField

# Unix timestamp
created_timestamp:
  type: TimeStampField
```

**Date/Time Types**:
```yaml
# Date only (YYYY-MM-DD)
birth_date:
  type: DateField

# Date and time (naive)
created_at:
  type: DateTimeField

# Date and time (timezone-aware)
executed_at:
  type: DateTimeTZField
```

**Special Types**:
```yaml
# Boolean true/false
is_active:
  type: BooleanField

# Email with validation
email:
  type: EmailField

# URL with validation
website:
  type: URLField

# JSON data
metadata:
  type: JSONField

# UUID
user_id:
  type: UUIDField
  auto_add: true
```

#### nullable

Allow NULL values:

```yaml
schema:
  required_field:
    type: CharField
    nullable: false  # Must have a value

  optional_field:
    type: CharField
    nullable: true   # Can be NULL
```

**Default**: `true`

#### unique

Enforce uniqueness:

```yaml
schema:
  email:
    type: EmailField
    unique: true  # No duplicates allowed

  trade_id:
    type: CharField
    max_length: 50
    unique: true
```

Creates a UNIQUE constraint in the database.

#### default

Default value if not provided:

```yaml
schema:
  status:
    type: CharField
    max_length: 20
    default: "pending"

  created_at:
    type: DateTimeField
    default: now  # Special: current timestamp

  is_active:
    type: BooleanField
    default: true
```

#### validation

Field-level validation rules:

```yaml
schema:
  # Regex validation
  account_number:
    type: CharField
    max_length: 20
    validation:
      regex: "^ACC[0-9]{10}$"

  # Range validation
  amount:
    type: DecimalField
    validation:
      min: 0.01
      max: 1000000

  # Length validation
  password:
    type: CharField
    max_length: 255
    validation:
      min_length: 8
      max_length: 128

  # Custom validation
  symbol:
    type: CharField
    max_length: 10
    validation:
      - type: not_empty
      - type: regex
        pattern: "^[A-Z]{3,5}$"
```

### Complete Schema Example

```yaml
schema:
  # Primary key (auto-generated)
  # id: IntegerField (automatic)

  # Business identifier
  customer_id:
    type: CharField
    max_length: 50
    unique: true
    nullable: false

  # Personal info
  first_name:
    type: CharField
    max_length: 100
    nullable: false

  last_name:
    type: CharField
    max_length: 100
    nullable: false

  email:
    type: EmailField
    unique: true
    nullable: false

  phone:
    type: CharField
    max_length: 20
    nullable: true

  # Financial data
  balance:
    type: DecimalField
    nullable: false
    default: 0.0
    validation:
      min: 0

  credit_limit:
    type: DecimalField
    nullable: true
    validation:
      min: 0
      max: 1000000

  # Status
  account_status:
    type: CharField
    max_length: 20
    default: "active"
    validation:
      regex: "^(active|suspended|closed)$"

  is_verified:
    type: BooleanField
    nullable: false
    default: false

  # Timestamps
  created_at:
    type: DateTimeField
    nullable: false

  last_login:
    type: DateTimeField
    nullable: true

  # Metadata
  preferences:
    type: JSONField
    nullable: true
```

## business_key

**Optional**. A list of field names that uniquely identify a record. When set, AtomIngest performs an **upsert** (update-or-create) instead of a plain insert.

- **Without `business_key`** — every row is appended as a new record (append-only / audit-log mode).
- **With `business_key`** — if a record already exists with matching key values it is updated; otherwise a new record is inserted. This makes ingestion **idempotent**: running the same file twice produces the same result.

> **Important**: `business_key` is a top-level YAML key, not nested under `config:`.

### Single business key

```yaml
target_table: customers

business_key:
  - email          # upsert on email — re-ingesting the same email updates the record

schema:
  email:
    type: EmailField
    unique: true
  name:
    type: StringField
  status:
    type: CharField
    max_length: 20
```

### Composite business key

Use multiple fields together as the unique identifier:

```yaml
target_table: daily_prices

business_key:
  - symbol
  - trade_date     # (symbol, trade_date) pair must be unique

schema:
  symbol:
    type: CharField
    max_length: 10
  trade_date:
    type: DateField
  close_price:
    type: FloatField
  volume:
    type: IntegerField
```

### Behavior

| Scenario | Result |
|---|---|
| No `business_key` | Always INSERT — duplicates allowed |
| `business_key` set, record not found | INSERT new record |
| `business_key` set, record found | UPDATE existing record with new values |

### Rules

- Every field listed in `business_key` must be defined in `schema`.
- Business key fields should have `unique: true` or be combined-unique to match your data contract.
- Do **not** include AtomSQL's auto-generated `id` column in `business_key` — use your natural keys instead.
- `business_key` is **singular** in the YAML (not `business_keys`).

## normalization

**Optional**. Data transformation pipeline.

```yaml
normalization:
  steps:
    - step: StepClassName
      config:
        option: value
```

### Built-in Steps

#### ColumnNameNormalizationStep

Standardize column names:

```yaml
- step: ColumnNameNormalizationStep
  config:
    case: lower           # or 'upper', 'title'
    replace_spaces: "_"   # Replace spaces with underscore
    remove_special: true  # Remove special characters
```

**Example**:
- `Customer Name` → `customer_name`
- `TRADE-ID` → `trade_id`

#### TrimWhitespaceStep

Remove extra whitespace:

```yaml
- step: TrimWhitespaceStep
  config:
    trim_quotes: true      # Remove surrounding quotes
    collapse_spaces: true  # Multiple spaces → single space
    strip_newlines: true   # Remove \n and \r
```

**Example**:
- `"  John  "` → `John`
- `"ABC"` → `ABC`

#### NullTokenNormalizationStep

Convert various null representations to actual NULL:

```yaml
- step: NullTokenNormalizationStep
  config:
    null_tokens:
      - "-"
      - "NA"
      - "N/A"
      - "NULL"
      - "null"
      - "nan"
      - "NaN"
      - ""
```

**Example**:
- `"-"` → `NULL`
- `"N/A"` → `NULL`

#### BasicTypeCleanupStep

Convert data types:

```yaml
- step: BasicTypeCleanupStep
  config:
    convert_to_number: true

    # Specify numeric columns
    numeric_columns:
      - quantity
      - price
      - amount

    # Parse dates
    date_columns:
      - trade_date
      - created_at

    # Date formats to try (in order)
    date_formats:
      - "%Y-%m-%d"
      - "%d-%m-%Y"
      - "%d-%b-%Y"
      - "%Y-%m-%d %H:%M:%S"

    # Output format (optional)
    output_date_format: "%Y-%m-%d"
```

### Step Execution Order

Steps execute sequentially:

```yaml
normalization:
  steps:
    # 1. First: Standardize column names
    - step: ColumnNameNormalizationStep
      config:
        case: lower

    # 2. Then: Clean whitespace
    - step: TrimWhitespaceStep
      config:
        trim_quotes: true

    # 3. Then: Handle nulls
    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["-", "NA"]

    # 4. Finally: Convert types
    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
```

## hooks

**Optional**. Custom Python functions for complex logic.

```yaml
hooks:
  pre_read: "module.function"
  post_normalize: "module.function"
  pre_save: "module.function"
  post_save: "module.function"
```

### Hook Points

#### pre_read

Runs before reading source data:

```yaml
hooks:
  pre_read: "hooks.fetch_latest_file"
```

```python
def fetch_latest_file(context):
    """Download file from API before reading"""
    # Your logic
    return file_path
```

#### post_normalize

Runs after normalization, before validation:

```yaml
hooks:
  post_normalize: "hooks.enrich_data"
```

```python
def enrich_data(record):
    """Add derived fields"""
    record['full_name'] = f"{record['first_name']} {record['last_name']}"
    record['risk_score'] = calculate_risk(record)
    return record
```

#### pre_save

Runs before database insertion:

```yaml
hooks:
  pre_save: "hooks.add_metadata"
```

```python
def add_metadata(record):
    """Add audit fields"""
    from datetime import datetime
    record['processed_at'] = datetime.now()
    record['processed_by'] = os.getenv('USER')
    return record
```

#### post_save

Runs after successful insertion:

```yaml
hooks:
  post_save: "hooks.send_notification"
```

```python
def send_notification(record, stats):
    """Send alert after save"""
    if stats.processed % 1000 == 0:
        print(f"Processed {stats.processed} records")
    return record
```

## error_handling

**Optional**. Configure error behavior.

```yaml
error_handling:
  on_error: continue    # or 'stop'
  dlq_enabled: true
  dlq_table: dead_letter_queue
  max_errors: 100
  error_threshold: 0.05  # 5%
```

### Options

- **on_error**: `continue` (default) or `stop`
- **dlq_enabled**: Save failed records to DLQ table
- **dlq_table**: Custom DLQ table name
- **max_errors**: Stop after N errors
- **error_threshold**: Stop if error rate exceeds %

## performance

**Optional**. Performance tuning.

```yaml
performance:
  batch_size: 500        # Records per batch
  async_enabled: false   # Async processing (experimental)
  commit_frequency: 10   # Commits per N batches
```

## audit

**Optional**. Audit trail configuration.

```yaml
audit:
  enabled: true
  track_changes: true
  created_by_field: created_by
  updated_by_field: updated_by
```

Automatically adds:
- `created_at`: Timestamp of insertion
- `updated_at`: Timestamp of last update
- `created_by`: User who created record
- `updated_by`: User who last updated record

## Complete Example

```yaml
target_table: daily_trades

# Upsert on trade_id — re-running the same file won't create duplicates
business_key:
  - trade_id

schema:
  trade_id:
    type: CharField
    max_length: 50
    unique: true
    nullable: false

  symbol:
    type: CharField
    max_length: 10
    nullable: false
    validation:
      regex: "^[A-Z]{3,5}$"

  quantity:
    type: IntegerField
    nullable: false
    validation:
      min: 1

  price:
    type: FloatField
    nullable: false
    validation:
      min: 0.01

  trade_date:
    type: DateField
    nullable: false

  executed_at:
    type: DateTimeTZField
    nullable: false

  is_settled:
    type: BooleanField
    nullable: false

normalization:
  steps:
    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"

    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true

    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["-", "NA", "NULL", ""]

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns: [quantity, price]
        date_columns: [trade_date, executed_at]
        date_formats:
          - "%Y-%m-%d"
          - "%d-%b-%Y"
          - "%Y-%m-%d %H:%M:%S"

config:
  batch_size: 1000
  audit_metadata:
    enabled: true
```

## Next Steps

- **[Schema Definition](schema_definition.md)** - Deep dive into field types
- **[Normalization](normalization.md)** - Detailed normalization guide
- **[Hooks](hooks.md)** - Custom logic patterns
- **[Validation](validation.md)** - Advanced validation
