# Data Validation Guide

Complete guide to data quality validation and business rules in AtomIngest.

## Table of Contents
- [Overview](#overview)
- [Configuration Basics](#configuration-basics)
- [Built-in Validators](#built-in-validators)
- [Field-Level Validation](#field-level-validation)
- [Custom Validators](#custom-validators)
- [Validation Modes](#validation-modes)
- [Error Handling](#error-handling)
- [Best Practices](#best-practices)

## Overview

Validation ensures data meets business rules and quality standards before database insertion. AtomIngest provides declarative validation that integrates seamlessly with your schema.

**Why Validation?**
- ✅ Enforce business rules
- ✅ Ensure data quality
- ✅ Catch errors early
- ✅ Generate quality reports
- ✅ Prevent bad data in database

## Configuration Basics

Validation rules are defined in the `schema` section alongside field definitions:

```yaml
schema:
  field_name:
    type: FieldType
    nullable: false
    validation:
      - type: validator_name
        param1: value1
        param2: value2
        message: "Custom error message"
```

### Simple Example

```yaml
schema:
  email:
    type: EmailField
    nullable: false
    validation:
      - type: email_format
      - type: not_empty
```

## Built-in Validators

### not_empty

Ensures field is not empty or whitespace-only.

```yaml
validation:
  - type: not_empty
    message: "Field cannot be empty"
```

**Use Cases:**
- Required text fields
- Names, descriptions
- Any non-optional data

**Example:**
```yaml
username:
  type: CharField
  max_length: 50
  nullable: false
  validation:
    - type: not_empty
      message: "Username is required"
```

### range

Validates numeric values are within a range.

```yaml
validation:
  - type: range
    min: 0
    max: 100
    message: "Value must be between 0 and 100"
```

**Parameters:**
- `min` (number, optional): Minimum value (inclusive)
- `max` (number, optional): Maximum value (inclusive)
- `message` (string): Custom error message

**Examples:**

```yaml
# Age validation
age:
  type: IntegerField
  validation:
    - type: range
      min: 18
      max: 120
      message: "Age must be between 18 and 120"

# Percentage validation
discount_rate:
  type: FloatField
  validation:
    - type: range
      min: 0.0
      max: 100.0
      message: "Discount rate must be between 0 and 100"

# Positive numbers only
quantity:
  type: IntegerField
  validation:
    - type: range
      min: 1
      message: "Quantity must be at least 1"
```

### length

Validates string length.

```yaml
validation:
  - type: length
    min: 8
    max: 50
    message: "Must be between 8 and 50 characters"
```

**Parameters:**
- `min` (int, optional): Minimum length
- `max` (int, optional): Maximum length
- `message` (string): Custom error message

**Examples:**

```yaml
# Password length
password:
  type: CharField
  max_length: 128
  validation:
    - type: length
      min: 8
      max: 128
      message: "Password must be 8-128 characters"

# Exact length
zipcode:
  type: CharField
  max_length: 5
  validation:
    - type: length
      min: 5
      max: 5
      message: "ZIP code must be exactly 5 digits"

# Minimum length only
description:
  type: TextField
  validation:
    - type: length
      min: 10
      message: "Description must be at least 10 characters"
```

### regex

Pattern matching validation.

```yaml
validation:
  - type: regex
    pattern: "^[A-Z]{2}[0-9]{6}$"
    flags: 0  # Optional regex flags
    message: "Must be 2 uppercase letters followed by 6 digits"
```

**Parameters:**
- `pattern` (string): Regular expression pattern
- `flags` (int, optional): Regex flags (e.g., re.IGNORECASE)
- `message` (string): Custom error message

**Examples:**

```yaml
# Phone number (US format)
phone:
  type: CharField
  max_length: 15
  validation:
    - type: regex
      pattern: "^\\+?1?\\d{10,14}$"
      message: "Invalid phone number format"

# Product code
product_code:
  type: CharField
  max_length: 10
  validation:
    - type: regex
      pattern: "^[A-Z]{3}[0-9]{6}$"
      message: "Product code must be 3 letters + 6 digits (e.g., ABC123456)"

# Alphanumeric only
username:
  type: CharField
  max_length: 50
  validation:
    - type: regex
      pattern: "^[a-zA-Z0-9_-]+$"
      message: "Username can only contain letters, numbers, underscores, and hyphens"

# Email (custom pattern)
email:
  type: CharField
  max_length: 255
  validation:
    - type: regex
      pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
      message: "Invalid email format"
```

### email_format

Email address validation.

```yaml
validation:
  - type: email_format
    message: "Invalid email address"
```

**Built-in checks:**
- Basic format validation
- Domain validation
- Special character handling

**Example:**
```yaml
email:
  type: EmailField
  unique: true
  nullable: false
  validation:
    - type: email_format
      message: "Please provide a valid email address"
```

### url_format

URL validation.

```yaml
validation:
  - type: url_format
    message: "Invalid URL"
```

**Validates:**
- Protocol (http/https)
- Domain format
- Port numbers
- Path structure

**Example:**
```yaml
website:
  type: URLField
  nullable: true
  validation:
    - type: url_format
      message: "Website must be a valid URL (e.g., https://example.com)"
```

### in_choices

Validates value is in allowed list.

```yaml
validation:
  - type: in_choices
    choices: ["option1", "option2", "option3"]
    message: "Value must be one of: option1, option2, option3"
```

**Parameters:**
- `choices` (list): List of allowed values
- `message` (string): Custom error message

**Examples:**

```yaml
# Status field
status:
  type: CharField
  max_length: 20
  validation:
    - type: in_choices
      choices: ["pending", "processing", "completed", "failed"]
      message: "Invalid status"

# Country codes
country:
  type: CharField
  max_length: 2
  validation:
    - type: in_choices
      choices: ["US", "UK", "CA", "AU", "DE", "FR"]
      message: "Unsupported country code"

# Priority levels
priority:
  type: CharField
  max_length: 10
  validation:
    - type: in_choices
      choices: ["low", "medium", "high", "urgent"]
      message: "Priority must be: low, medium, high, or urgent"
```

### date_range

Validates date is within a range.

```yaml
validation:
  - type: date_range
    min: "2020-01-01"
    max: "2030-12-31"
    message: "Date must be between 2020 and 2030"
```

**Parameters:**
- `min` (string|date, optional): Minimum date
- `max` (string|date, optional): Maximum date
- `message` (string): Custom error message

**Examples:**

```yaml
# Birth date validation
birth_date:
  type: DateField
  validation:
    - type: date_range
      min: "1920-01-01"
      max: "2010-12-31"
      message: "Birth date must be between 1920 and 2010"

# Future dates only
appointment_date:
  type: DateField
  validation:
    - type: date_range
      min: "today"
      message: "Appointment must be in the future"

# Transaction date validation
transaction_date:
  type: DateField
  validation:
    - type: date_range
      min: "2025-01-01"
      max: "2026-12-31"
      message: "Transaction date must be in fiscal year 2025-2026"
```

## Field-Level Validation

### Single Validator

```yaml
email:
  type: EmailField
  validation:
    - type: email_format
```

### Multiple Validators

Validators are executed in order. All must pass.

```yaml
password:
  type: CharField
  max_length: 128
  nullable: false
  validation:
    - type: not_empty
      message: "Password is required"

    - type: length
      min: 8
      max: 128
      message: "Password must be 8-128 characters"

    - type: regex
      pattern: "^(?=.*[A-Z])(?=.*[a-z])(?=.*[0-9])(?=.*[!@#$%^&*])"
      message: "Password must contain uppercase, lowercase, number, and special character"
```

### Complex Validation Example

```yaml
credit_card_number:
  type: EncryptedCharField
  max_length: 255
  key_id: "payment_encryption_key"
  nullable: false
  validation:
    - type: not_empty
      message: "Credit card number is required"

    - type: regex
      pattern: "^[0-9]{13,19}$"
      message: "Credit card must be 13-19 digits"

    # Luhn algorithm validation would be custom validator
```

## Custom Validators

Create custom validators for complex business rules.

### Basic Custom Validator

```python
from atomingest.schema.validators import Validator, ValidationError

class CustomValidator(Validator):
    def __init__(self, param1, param2, message=None):
        super().__init__(message)
        self.param1 = param1
        self.param2 = param2

    def __call__(self, value, field_name=None):
        if not self.is_valid(value):
            raise ValidationError(
                self.get_error_message(f"Validation failed for {field_name}"),
                field_name=field_name,
                value=value
            )

    def is_valid(self, value):
        # Your validation logic
        return True
```

### Example: Luhn Algorithm Validator

```python
from atomingest.schema.validators import Validator, ValidationError

class LuhnValidator(Validator):
    """Validates credit card numbers using Luhn algorithm."""

    def __call__(self, value, field_name=None):
        if value is None:
            return

        # Convert to string and remove spaces
        digits = str(value).replace(" ", "")

        if not digits.isdigit():
            raise ValidationError(
                "Credit card must contain only digits",
                field_name=field_name,
                value=value
            )

        # Luhn algorithm
        total = 0
        for i, digit in enumerate(reversed(digits)):
            n = int(digit)
            if i % 2 == 1:
                n *= 2
                if n > 9:
                    n -= 9
            total += n

        if total % 10 != 0:
            raise ValidationError(
                self.get_error_message("Invalid credit card number"),
                field_name=field_name,
                value=value
            )
```

### Example: Business Hours Validator

```python
from datetime import time
from atomingest.schema.validators import Validator, ValidationError

class BusinessHoursValidator(Validator):
    """Validates time is within business hours."""

    def __init__(self, start_hour=9, end_hour=17, message=None):
        super().__init__(message)
        self.start_time = time(start_hour, 0)
        self.end_time = time(end_hour, 0)

    def __call__(self, value, field_name=None):
        if value is None:
            return

        # Extract time from datetime
        if hasattr(value, 'time'):
            check_time = value.time()
        else:
            check_time = value

        if not (self.start_time <= check_time <= self.end_time):
            raise ValidationError(
                self.get_error_message(
                    f"Time must be between {self.start_time} and {self.end_time}"
                ),
                field_name=field_name,
                value=value
            )
```

### Registering Custom Validators

```python
from atomingest.schema.validators import VALIDATOR_REGISTRY

VALIDATOR_REGISTRY['luhn'] = LuhnValidator
VALIDATOR_REGISTRY['business_hours'] = BusinessHoursValidator
```

### Using Custom Validators

```yaml
credit_card:
  type: CharField
  max_length: 19
  validation:
    - type: luhn
      message: "Invalid credit card number"

appointment_time:
  type: DateTimeField
  validation:
    - type: business_hours
      start_hour: 9
      end_hour: 17
      message: "Appointments must be during business hours (9 AM - 5 PM)"
```

## Validation Modes

### Strict Mode

Fails entire ingestion on first validation error.

```yaml
config:
  validation_mode: strict
```

**Use When:**
- Data quality is critical
- Cannot tolerate any bad records
- Need all-or-nothing guarantees

### Lenient Mode

Continues processing, sends invalid records to DLQ.

```yaml
config:
  validation_mode: lenient
  dead_letter_queue:
    enabled: true
    path: "./errors"
```

**Use When:**
- Want to process as much as possible
- Can review/fix errors later
- Need high throughput

## Error Handling

### Validation Errors in DLQ

Invalid records are written to dead letter queue:

```json
{
  "row_data": {
    "email": "invalid-email",
    "age": 200
  },
  "errors": [
    {
      "field": "email",
      "validator": "email_format",
      "message": "Invalid email address"
    },
    {
      "field": "age",
      "validator": "range",
      "message": "Age must be between 18 and 120"
    }
  ],
  "timestamp": "2026-01-20T10:30:00Z"
}
```

### Error Reports

AtomIngest generates validation reports:

```
=== Validation Report ===
Total Records: 1000
Valid Records: 985
Invalid Records: 15

Error Breakdown:
- email_format: 8 errors
- range: 5 errors
- not_empty: 2 errors

Failed Fields:
- email: 8 errors
- age: 5 errors
- username: 2 errors
```

## Best Practices

### 1. Validate Early and Often

```yaml
# Good - validate at ingestion
email:
  type: EmailField
  validation:
    - type: email_format
    - type: not_empty

# Don't rely on database constraints alone
```

### 2. Provide Helpful Error Messages

```yaml
# Good
validation:
  - type: regex
    pattern: "^[A-Z]{3}[0-9]{6}$"
    message: "Product code must be 3 uppercase letters followed by 6 digits (e.g., ABC123456)"

# Avoid
validation:
  - type: regex
    pattern: "^[A-Z]{3}[0-9]{6}$"
    message: "Invalid format"
```

### 3. Order Validators Logically

```yaml
# Good - check emptiness first
validation:
  - type: not_empty
  - type: length
    min: 8
  - type: regex
    pattern: "..."

# Avoid - regex on empty string fails ungracefully
validation:
  - type: regex
    pattern: "..."
  - type: not_empty
```

### 4. Use Appropriate Validators

```yaml
# Good - use semantic validators
email:
  type: EmailField
  validation:
    - type: email_format

# Avoid - complex regex when built-in exists
email:
  type: CharField
  validation:
    - type: regex
      pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
```

### 5. Combine with Normalization

```yaml
# Normalize first
normalization:
  steps:
    - step: TrimWhitespaceStep
    - step: NullTokenNormalizationStep

# Then validate
schema:
  email:
    type: EmailField
    validation:
      - type: email_format
```

### 6. Test Validation Rules

```bash
# Test with sample data
atomingest run config.yaml sqlite:///test.db -s test_data.csv --dry-run

# Check validation reports
atomingest run config.yaml sqlite:///test.db -s data.csv -v
```

## Complete Examples

### User Registration Schema

```yaml
target_table: users

schema:
  username:
    type: CharField
    max_length: 50
    unique: true
    nullable: false
    validation:
      - type: not_empty
        message: "Username is required"
      - type: length
        min: 3
        max: 50
        message: "Username must be 3-50 characters"
      - type: regex
        pattern: "^[a-zA-Z0-9_-]+$"
        message: "Username can only contain letters, numbers, underscores, and hyphens"

  email:
    type: EmailField
    unique: true
    nullable: false
    validation:
      - type: not_empty
        message: "Email is required"
      - type: email_format
        message: "Invalid email address"

  password:
    type: CharField
    max_length: 128
    nullable: false
    validation:
      - type: not_empty
        message: "Password is required"
      - type: length
        min: 8
        max: 128
        message: "Password must be 8-128 characters"
      - type: regex
        pattern: "^(?=.*[A-Z])(?=.*[a-z])(?=.*[0-9])(?=.*[!@#$%^&*])"
        message: "Password must contain uppercase, lowercase, number, and special character"

  age:
    type: IntegerField
    nullable: true
    validation:
      - type: range
        min: 18
        max: 120
        message: "Age must be between 18 and 120"
```

### Financial Transaction Schema

```yaml
target_table: transactions

schema:
  transaction_id:
    type: UUIDField
    unique: true
    nullable: false
    validation:
      - type: not_empty

  amount:
    type: FloatField
    nullable: false
    validation:
      - type: range
        min: 0.01
        max: 1000000.00
        message: "Amount must be between $0.01 and $1,000,000"

  currency:
    type: CharField
    max_length: 3
    nullable: false
    validation:
      - type: in_choices
        choices: ["USD", "EUR", "GBP", "JPY", "CHF"]
        message: "Unsupported currency"

  transaction_type:
    type: CharField
    max_length: 20
    nullable: false
    validation:
      - type: in_choices
        choices: ["deposit", "withdrawal", "transfer", "payment"]
        message: "Invalid transaction type"

  account_number:
    type: EncryptedCharField
    max_length: 255
    key_id: "account_encryption"
    nullable: false
    validation:
      - type: not_empty
      - type: regex
        pattern: "^[0-9]{10,12}$"
        message: "Account number must be 10-12 digits"

  transaction_date:
    type: DateTimeField
    nullable: false
    validation:
      - type: date_range
        min: "2020-01-01"
        max: "2030-12-31"
        message: "Transaction date must be between 2020 and 2030"
```

## Related Documentation

- [Schema Definition](schema_definition.md) - Define field types
- [Normalization](normalization.md) - Data cleaning
- [Error Handling](error_handling.md) - DLQ and error recovery
- [Hooks](hooks.md) - Custom validation logic

## Next Steps

1. Define validation rules for your schema
2. Test with sample data using `--dry-run`
3. Review validation error reports
4. Configure DLQ for invalid records
5. Create custom validators for business rules
