# Data Normalization Guide

Complete guide to data cleaning and transformation with AtomIngest normalization steps.

## Table of Contents
- [Overview](#overview)
- [Configuration Basics](#configuration-basics)
- [File-Level Steps](#file-level-steps)
- [Row-Level Steps](#row-level-steps)
- [Type Conversion Steps](#type-conversion-steps)
- [Step Ordering](#step-ordering)
- [Custom Steps](#custom-steps)
- [Common Patterns](#common-patterns)

## Overview

Normalization is the process of cleaning and transforming raw data before it enters your database. AtomIngest provides a powerful pipeline of normalization steps that can be chained together.

**Why Normalization?**
- ✅ Handle messy real-world data
- ✅ Remove metadata and headers
- ✅ Standardize formats
- ✅ Convert types safely
- ✅ Handle null values consistently

## Configuration Basics

Normalization is configured in the `normalization` section of your YAML pipeline:

```yaml
target_table: my_table
schema:
  # ... schema definition

normalization:
  steps:
    - step: StepName
      config:
        option1: value1
        option2: value2
    - step: AnotherStep
      config:
        # ... config
```

### Minimal Example

```yaml
normalization:
  steps:
    - step: TrimWhitespaceStep
    - step: NullTokenNormalizationStep
    - step: BasicTypeCleanupStep
```

## File-Level Steps

These steps process lines before CSV parsing.

### RemoveBOMStep

Removes UTF-8 Byte Order Mark (BOM) from files.

```yaml
- step: RemoveBOMStep
  config:
    check_all_lines: false  # If true, checks every line
```

**When to Use:**
- Files from Windows applications
- Excel exports
- Concatenated files

**Example:**
```yaml
normalization:
  steps:
    - step: RemoveBOMStep
      config:
        check_all_lines: false
```

### SkipUntilHeaderStep

Skip metadata/garbage lines until valid header is found.

```yaml
- step: SkipUntilHeaderStep
  config:
    header_regex: "^SYMBOL.*"           # Regex to match header
    expected_columns: ["SYMBOL", "DATE"] # Or list of expected columns
    separator: ","                       # Delimiter
    max_skip: 100                        # Max lines to skip
```

**When to Use:**
- Files with metadata headers
- Reports with title blocks
- Files with summary sections

**Example - NSE Bhavcopy:**
```yaml
- step: SkipUntilHeaderStep
  config:
    expected_columns: ["SYMBOL", "SERIES", "DATE1"]
    separator: ","
    max_skip: 50
```

### DropTrailerStep

Remove footer/trailer lines from the end of files.

```yaml
- step: DropTrailerStep
  config:
    trailer_marker: "TOTAL_RECORDS"  # Regex to identify trailer
    drop_lines_after: 0               # Lines to drop after marker
```

**When to Use:**
- Files with record counts at end
- Files with checksums
- Files with summary footers

## Row-Level Steps

These steps process individual rows after CSV parsing.

### TrimWhitespaceStep

Remove leading/trailing whitespace from all values.

```yaml
- step: TrimWhitespaceStep
  config:
    trim_quotes: true      # Remove surrounding quotes
    collapse_spaces: true  # Replace multiple spaces with single space
```

**Essential for:**
- CSV files with extra spaces
- Files with quoted values
- Data from legacy systems

**Example:**
```yaml
- step: TrimWhitespaceStep
  config:
    trim_quotes: true
    collapse_spaces: true
```

**Before:**
```
"  Alice  ", " Bob "
```

**After:**
```
"Alice", "Bob"
```

### NullTokenNormalizationStep

Convert null tokens to None/NULL.

```yaml
- step: NullTokenNormalizationStep
  config:
    null_tokens: ["-", "NA", "NULL", "nan", "", "N/A"]
```

**Default Tokens:** `["", "NA", "NULL", "nan", "-"]`

**When to Use:**
- CSV files with inconsistent null representations
- Files from different systems
- Data with placeholders for missing values

**Example:**
```yaml
- step: NullTokenNormalizationStep
  config:
    null_tokens: ["-", "NA", "NULL", "nan", "", "N/A", "None"]
```

**Before:**
```csv
name,age,email
Alice,-,alice@example.com
Bob,NA,
Charlie,30,N/A
```

**After:**
```csv
name,age,email
Alice,NULL,alice@example.com
Bob,NULL,NULL
Charlie,30,NULL
```

### ColumnNameNormalizationStep

Standardize column names.

```yaml
- step: ColumnNameNormalizationStep
  config:
    case: lower                        # 'lower', 'upper', or null
    replace_spaces: "_"                # Replace spaces with underscore
    remove_non_alphanumeric: true      # Remove special chars
    alias_map:                         # Rename specific columns
      "Customer Name": "customer_name"
      "Order ID": "order_id"
```

**When to Use:**
- Inconsistent column naming
- Spaces in column names
- Special characters in headers

**Example:**
```yaml
- step: ColumnNameNormalizationStep
  config:
    case: lower
    replace_spaces: "_"
    remove_non_alphanumeric: true
```

**Before:**
```
Customer Name, Order ID, Total Amount ($)
```

**After:**
```
customer_name, order_id, total_amount
```

### DropRepeatedHeaderRowsStep

Remove duplicate header rows in concatenated files.

```yaml
- step: DropRepeatedHeaderRowsStep
```

**When to Use:**
- Concatenated CSV files
- Files from batch exports
- Multi-file aggregations

**Example:**
```yaml
- step: DropRepeatedHeaderRowsStep
```

**Before:**
```csv
name,age
Alice,30
name,age  <-- duplicate header
Bob,25
```

**After:**
```csv
name,age
Alice,30
Bob,25
```

## Type Conversion Steps

### BasicTypeCleanupStep

Convert strings to appropriate types and parse dates.

```yaml
- step: BasicTypeCleanupStep
  config:
    convert_to_number: true
    numeric_columns:
      - price
      - quantity
      - total
    date_columns:
      - order_date
      - created_at
    date_formats:
      - "%Y-%m-%d"
      - "%d-%m-%Y"
      - "%d-%b-%Y"  # 19-Jan-2026
    output_date_format: "%Y-%m-%d"  # Or null to keep as datetime
```

**Options:**
- `convert_to_number` (bool): Try to convert numeric strings
- `numeric_columns` (list): Columns to parse as numbers
- `date_columns` (list): Columns to parse as dates
- `date_formats` (list): Date format patterns to try
- `output_date_format` (str|null): Output format (null = datetime object)

**Example - Financial Data:**
```yaml
- step: BasicTypeCleanupStep
  config:
    convert_to_number: true
    numeric_columns:
      - amount
      - balance
      - interest_rate
    date_columns:
      - transaction_date
      - settlement_date
    date_formats:
      - "%Y-%m-%d"
      - "%m/%d/%Y"
```

**Before:**
```csv
amount,balance,transaction_date
"1,234.56","10,000.00","19-01-2026"
```

**After:**
```python
{
  "amount": 1234.56,
  "balance": 10000.0,
  "transaction_date": datetime(2026, 1, 19)
}
```

## Step Ordering

**Order matters!** Steps are executed sequentially.

### Recommended Order

1. **File-level cleaning**
   - RemoveBOMStep
   - SkipUntilHeaderStep

2. **Column name standardization**
   - ColumnNameNormalizationStep

3. **Value cleaning**
   - TrimWhitespaceStep
   - DropRepeatedHeaderRowsStep

4. **Null handling**
   - NullTokenNormalizationStep

5. **Type conversion**
   - BasicTypeCleanupStep

### Complete Example

```yaml
normalization:
  steps:
    # 1. File-level
    - step: RemoveBOMStep
      config:
        check_all_lines: false

    - step: SkipUntilHeaderStep
      config:
        expected_columns: ["SYMBOL", "DATE"]
        max_skip: 50

    # 2. Column names
    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"

    # 3. Value cleaning
    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true

    - step: DropRepeatedHeaderRowsStep

    # 4. Null handling
    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["-", "NA", "NULL", "nan", ""]

    # 5. Type conversion
    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns:
          - price
          - quantity
        date_columns:
          - date
        date_formats:
          - "%d-%b-%Y"
```

## Custom Steps

You can create custom normalization steps by extending `BaseNormalizerStep`.

### File-Level Custom Step

```python
from atomingest.normalization.steps import BaseNormalizerStep, register_step
from typing import Iterator

@register_step("CustomFileStep")
class CustomFileStep(BaseNormalizerStep):
    def process_lines(self, lines: Iterator[str]) -> Iterator[str]:
        for line in lines:
            # Your custom logic
            processed_line = line.upper()
            yield processed_line
```

### Row-Level Custom Step

```python
from atomingest.normalization.steps import BaseNormalizerStep, register_step
from typing import Dict, Any, Optional

@register_step("CustomRowStep")
class CustomRowStep(BaseNormalizerStep):
    def process_row(self, row: Dict[str, Any]) -> Optional[Dict[str, Any]]:
        # Your custom logic
        row['calculated_field'] = row['a'] + row['b']
        return row
```

### Using Custom Steps

```yaml
normalization:
  steps:
    - step: CustomFileStep
      config:
        option: value

    - step: CustomRowStep
```

## Common Patterns

### NSE Bhavcopy Files

```yaml
normalization:
  steps:
    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"

    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true

    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["-", "NA", "NULL"]

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns:
          - prev_close
          - open_price
          - high_price
          - low_price
          - close_price
        date_columns:
          - date1
        date_formats:
          - "%d-%b-%Y"
```

### Bank Statement CSV

```yaml
normalization:
  steps:
    - step: RemoveBOMStep

    - step: SkipUntilHeaderStep
      config:
        expected_columns: ["Date", "Description", "Amount"]
        max_skip: 20

    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"

    - step: TrimWhitespaceStep
      config:
        trim_quotes: true

    - step: NullTokenNormalizationStep

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns: [amount, balance]
        date_columns: [date]
        date_formats:
          - "%m/%d/%Y"
          - "%Y-%m-%d"
```

### Excel Export with Metadata

```yaml
normalization:
  steps:
    - step: RemoveBOMStep
      config:
        check_all_lines: false

    - step: SkipUntilHeaderStep
      config:
        header_regex: "^Product.*"
        max_skip: 100

    - step: DropTrailerStep
      config:
        trailer_marker: "^Total Records:"

    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true

    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["", "-", "N/A", "#N/A"]

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        date_formats:
          - "%m/%d/%Y"
```

### API Response (JSON Lines)

```yaml
normalization:
  steps:
    - step: TrimWhitespaceStep

    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["null", "NULL", ""]

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        date_columns:
          - timestamp
          - created_at
        date_formats:
          - "%Y-%m-%dT%H:%M:%S"
          - "%Y-%m-%dT%H:%M:%S.%fZ"
```

## Best Practices

### 1. Always Trim Whitespace

```yaml
- step: TrimWhitespaceStep
  config:
    trim_quotes: true
    collapse_spaces: true
```

### 2. Standardize Column Names Early

```yaml
- step: ColumnNameNormalizationStep
  config:
    case: lower
    replace_spaces: "_"
```

### 3. Handle Nulls Consistently

```yaml
- step: NullTokenNormalizationStep
  config:
    null_tokens: ["-", "NA", "NULL", "nan", "", "N/A", "None"]
```

### 4. Specify Date Formats Explicitly

```yaml
date_formats:
  - "%Y-%m-%d"      # ISO format
  - "%d-%b-%Y"      # 19-Jan-2026
  - "%m/%d/%Y"      # US format
  - "%d/%m/%Y"      # European format
```

### 5. Test with Sample Data

Use `--dry-run` to validate normalization:

```bash
atomingest run config.yaml sqlite:///test.db -s sample.csv --dry-run
```

### 6. Log Normalization Metrics

Enable verbose logging to see normalization statistics:

```bash
atomingest run config.yaml sqlite:///test.db -s data.csv -v
```

## Troubleshooting

### Data Not Converting to Numbers

**Problem:** Numeric strings staying as strings.

**Solution:**
```yaml
- step: TrimWhitespaceStep  # Remove spaces first
- step: BasicTypeCleanupStep
  config:
    convert_to_number: true
    numeric_columns: [amount]  # Specify columns explicitly
```

### Dates Not Parsing

**Problem:** Date strings not converting to dates.

**Solution:**
```yaml
- step: BasicTypeCleanupStep
  config:
    date_columns: [date]
    date_formats:
      - "%d-%b-%Y"  # Add your exact format
      - "%Y-%m-%d"
```

### Headers Appearing in Data

**Problem:** Duplicate headers from concatenated files.

**Solution:**
```yaml
- step: DropRepeatedHeaderRowsStep
```

### Column Names Not Matching Schema

**Problem:** Spaces or special characters in column names.

**Solution:**
```yaml
- step: ColumnNameNormalizationStep
  config:
    case: lower
    replace_spaces: "_"
    remove_non_alphanumeric: true
```

## Related Documentation

- [Schema Definition](schema_definition.md) - Define data types
- [Validation](validation.md) - Data quality rules
- [Pipeline Configuration](pipeline_configuration.md) - Complete pipeline setup
- [Common Patterns](patterns.md) - Real-world examples

## Next Steps

1. Identify data quality issues in your source files
2. Configure normalization steps in order
3. Test with `--dry-run` flag
4. Review normalization reports
5. Add validation rules for data quality
