# Schema Definition Guide

Complete guide to defining data models and field types in AtomIngest.

## Table of Contents
- [Overview](#overview)
- [Basic Schema Structure](#basic-schema-structure)
- [Field Types](#field-types)
- [Field Options](#field-options)
- [Validation Rules](#validation-rules)
- [Encrypted Fields](#encrypted-fields)
- [Complex Examples](#complex-examples)

## Overview

AtomIngest schemas are defined in YAML and automatically generate AtomSQL ORM models. The framework handles type conversion, validation, and database operations.

**Key Benefits:**
- ✅ Declarative schema definition
- ✅ Automatic type conversion
- ✅ Built-in validation
- ✅ PII encryption support
- ✅ No boilerplate code

## Basic Schema Structure

Every schema requires two components:

```yaml
target_table: my_table_name

schema:
  field_name:
    type: FieldType
    nullable: true
    # ... other options
```

### Minimal Example

```yaml
target_table: users

schema:
  name: CharField
  email: EmailField
  age: IntegerField
```

### Expanded Example

```yaml
target_table: users

schema:
  name:
    type: CharField
    max_length: 255
    nullable: false
    validation:
      - type: not_empty
  email:
    type: EmailField
    unique: true
    validation:
      - type: email_format
  age:
    type: IntegerField
    nullable: true
    validation:
      - type: range
        min: 0
        max: 150
```

## Field Types

### String Fields

#### CharField
Fixed-length character field with max_length.

```yaml
username:
  type: CharField
  max_length: 50
  nullable: false
```

**Options:**
- `max_length` (int, required): Maximum string length
- `nullable` (bool): Allow None values (default: true)
- `unique` (bool): Enforce uniqueness (default: false)
- `default` (str): Default value

#### StringField
Variable-length string (alias for CharField).

```yaml
title:
  type: StringField
  nullable: false
```

#### TextField
Unlimited-length text field.

```yaml
description:
  type: TextField
  nullable: true
```

**Use Cases:**
- Long content (articles, descriptions)
- JSON data (when not using JSONField)
- Comments and notes

#### EmailField
Email address with validation.

```yaml
email:
  type: EmailField
  unique: true
  validation:
    - type: email_format
```

**Built-in Validation:**
- Regex pattern matching
- Domain validation
- Basic format checking

#### URLField
URL with validation.

```yaml
website:
  type: URLField
  nullable: true
```

### Numeric Fields

#### IntegerField
Whole numbers.

```yaml
quantity:
  type: IntegerField
  default: 0
  validation:
    - type: range
      min: 0
      max: 1000000
```

**Use Cases:**
- Counts and quantities
- IDs and references
- Age, year, etc.

#### FloatField
Decimal numbers with floating-point precision.

```yaml
price:
  type: FloatField
  nullable: false
  validation:
    - type: range
      min: 0.0
```

**Note:** For financial calculations requiring exact precision, consider storing values as integers (cents) or using string fields with custom validation.

### Date & Time Fields

#### DateField
Date without time component.

```yaml
birth_date:
  type: DateField
  nullable: true
```

**Format:** YYYY-MM-DD

#### DateTimeField
Date and time without timezone.

```yaml
created_at:
  type: DateTimeField
  nullable: false
```

**Format:** YYYY-MM-DD HH:MM:SS

#### DateTimeTZField
Date and time with timezone awareness.

```yaml
transaction_time:
  type: DateTimeTZField
  nullable: false
```

**Format:** YYYY-MM-DD HH:MM:SS+TZ

#### TimeStampField
Unix timestamp (seconds since epoch).

```yaml
event_timestamp:
  type: TimestampField
  nullable: false
```

### Boolean Field

#### BooleanField
True/False values.

```yaml
is_active:
  type: BooleanField
  default: true
```

**Accepted Values:**
- Python: `True`, `False`
- CSV/JSON: `"true"`, `"false"`, `"1"`, `"0"`, `"yes"`, `"no"`

### Structured Data Fields

#### JSONField
Stores JSON objects.

```yaml
metadata:
  type: JSONField
  nullable: true
```

**Use Cases:**
- Flexible attributes
- Configuration data
- API responses
- Nested structures

#### UUIDField
Universally unique identifiers.

```yaml
transaction_id:
  type: UUIDField
  unique: true
```

## Field Options

### Common Options

All fields support these options:

```yaml
field_name:
  type: FieldType
  nullable: true        # Allow None values (default: true)
  unique: false         # Enforce uniqueness (default: false)
  default: value        # Default value if not provided
```

### Nullable vs Required

```yaml
# Nullable field (optional)
middle_name:
  type: CharField
  max_length: 50
  nullable: true

# Required field (must be provided)
first_name:
  type: CharField
  max_length: 50
  nullable: false
```

### Unique Constraints

```yaml
# Single unique field
email:
  type: EmailField
  unique: true

# Multiple fields can be unique
username:
  type: CharField
  max_length: 50
  unique: true
```

### Default Values

```yaml
# Static default
status:
  type: CharField
  max_length: 20
  default: "pending"

# Numeric default
quantity:
  type: IntegerField
  default: 0

# Boolean default
is_active:
  type: BooleanField
  default: true
```

## Validation Rules

### Built-in Validators

#### not_empty
Ensures field is not empty or whitespace.

```yaml
validation:
  - type: not_empty
    message: "Field cannot be empty"
```

#### range
Validates numeric ranges.

```yaml
validation:
  - type: range
    min: 0
    max: 100
    message: "Value must be between 0 and 100"
```

#### length
Validates string length.

```yaml
validation:
  - type: length
    min: 8
    max: 50
    message: "Must be between 8 and 50 characters"
```

#### regex
Pattern matching validation.

```yaml
validation:
  - type: regex
    pattern: "^[A-Z]{2}[0-9]{6}$"
    message: "Must be 2 uppercase letters followed by 6 digits"
```

#### email_format
Email address validation.

```yaml
validation:
  - type: email_format
    message: "Invalid email address"
```

#### url_format
URL validation.

```yaml
validation:
  - type: url_format
    message: "Invalid URL"
```

#### in_choices
Validates value is in allowed list.

```yaml
validation:
  - type: in_choices
    choices: ["active", "inactive", "pending"]
    message: "Invalid status"
```

### Multiple Validators

Chain multiple validators:

```yaml
password:
  type: CharField
  max_length: 128
  nullable: false
  validation:
    - type: not_empty
    - type: length
      min: 8
      max: 128
    - type: regex
      pattern: "^(?=.*[A-Z])(?=.*[a-z])(?=.*[0-9])(?=.*[!@#$%^&*])"
      message: "Must contain uppercase, lowercase, number, and special character"
```

## Encrypted Fields

For PII (Personally Identifiable Information) and sensitive data, use encrypted field types.

### EncryptedStringField

```yaml
ssn:
  type: EncryptedStringField
  nullable: false
  key_id: "pii_encryption_key"
```

### EncryptedCharField

```yaml
credit_card:
  type: EncryptedCharField
  max_length: 255
  key_id: "payment_encryption_key"
```

### EncryptedTextField

```yaml
medical_notes:
  type: EncryptedTextField
  key_id: "medical_data_key"
```

### EncryptedEmailField

```yaml
private_email:
  type: EncryptedEmailField
  key_id: "email_encryption_key"
```

### EncryptedJSONField

```yaml
sensitive_metadata:
  type: EncryptedJSONField
  key_id: "metadata_encryption_key"
```

**Requirements:**
- Key management system configured
- `key_id` must reference valid encryption key
- See [Security Guide](security.md) for setup

## Complex Examples

### E-commerce Order Schema

```yaml
target_table: orders

schema:
  order_id:
    type: CharField
    max_length: 50
    unique: true
    nullable: false
    validation:
      - type: regex
        pattern: "^ORD-[0-9]{10}$"

  customer_email:
    type: EmailField
    nullable: false
    validation:
      - type: email_format

  total_amount:
    type: FloatField
    nullable: false
    validation:
      - type: range
        min: 0.01

  currency:
    type: CharField
    max_length: 3
    default: "USD"
    validation:
      - type: in_choices
        choices: ["USD", "EUR", "GBP", "JPY"]

  status:
    type: CharField
    max_length: 20
    default: "pending"
    validation:
      - type: in_choices
        choices: ["pending", "processing", "shipped", "delivered", "cancelled"]

  order_date:
    type: DateTimeField
    nullable: false

  items:
    type: JSONField
    nullable: false

  shipping_address:
    type: EncryptedTextField
    key_id: "address_encryption"
```

### Financial Transaction Schema

```yaml
target_table: transactions

schema:
  transaction_id:
    type: UUIDField
    unique: true
    nullable: false

  account_number:
    type: EncryptedCharField
    max_length: 255
    key_id: "account_encryption"
    nullable: false

  amount:
    type: FloatField
    nullable: false
    validation:
      - type: range
        min: 0.01
        max: 1000000.00

  transaction_type:
    type: CharField
    max_length: 20
    validation:
      - type: in_choices
        choices: ["deposit", "withdrawal", "transfer", "payment"]

  timestamp:
    type: DateTimeTZField
    nullable: false

  description:
    type: TextField
    nullable: true

  metadata:
    type: JSONField
    nullable: true

  is_flagged:
    type: BooleanField
    default: false
```

### User Profile Schema

```yaml
target_table: user_profiles

schema:
  user_id:
    type: UUIDField
    unique: true
    nullable: false

  username:
    type: CharField
    max_length: 50
    unique: true
    nullable: false
    validation:
      - type: regex
        pattern: "^[a-zA-Z0-9_-]{3,50}$"
        message: "Username must be 3-50 alphanumeric characters"

  email:
    type: EncryptedEmailField
    key_id: "user_pii_key"
    unique: true
    nullable: false

  full_name:
    type: EncryptedCharField
    max_length: 255
    key_id: "user_pii_key"

  birth_date:
    type: DateField
    nullable: true

  phone_number:
    type: EncryptedCharField
    max_length: 255
    key_id: "user_pii_key"
    nullable: true

  preferences:
    type: JSONField
    nullable: true

  created_at:
    type: DateTimeField
    nullable: false

  last_login:
    type: DateTimeTZField
    nullable: true

  is_verified:
    type: BooleanField
    default: false

  profile_url:
    type: URLField
    nullable: true
```

## Best Practices

### 1. Always Define Nullable Explicitly

```yaml
# Good
field_name:
  type: CharField
  max_length: 50
  nullable: false

# Avoid relying on defaults
field_name:
  type: CharField
  max_length: 50
```

### 2. Use Appropriate Field Types

```yaml
# Good - semantic types
email: EmailField
website: URLField
price: FloatField

# Avoid - generic types
email: CharField
website: CharField
price: CharField
```

### 3. Add Validation for Business Rules

```yaml
age:
  type: IntegerField
  validation:
    - type: range
      min: 18
      max: 120
      message: "Age must be between 18 and 120"
```

### 4. Encrypt Sensitive Data

```yaml
# Always encrypt PII
ssn:
  type: EncryptedStringField
  key_id: "pii_encryption_key"

# Don't store sensitive data unencrypted
ssn: CharField  # BAD!
```

### 5. Use Unique Constraints

```yaml
# Enforce business uniqueness
email:
  type: EmailField
  unique: true

order_id:
  type: CharField
  max_length: 50
  unique: true
```

### 6. Provide Meaningful Validation Messages

```yaml
validation:
  - type: regex
    pattern: "^[A-Z]{3}[0-9]{6}$"
    message: "Product code must be 3 uppercase letters followed by 6 digits (e.g., ABC123456)"
```

## Related Documentation

- [Field Types Reference](field_types.md) - Complete field type documentation
- [Validation Guide](validation.md) - Detailed validation rules
- [Security Features](security.md) - Encryption and PII protection
- [Pipeline Configuration](pipeline_configuration.md) - Full pipeline setup

## Next Steps

1. Define your schema following the patterns above
2. Add validation rules for data quality
3. Configure encryption for sensitive fields
4. Test with sample data using `--dry-run`
5. See [Pipeline Configuration](pipeline_configuration.md) for complete setup
