## 🔧 Normalization Pipeline

AtomIngest includes a powerful normalization pipeline to clean and standardize your raw data before ingestion. The pipeline supports various data sources (CSV, JSON, Excel, Parquet) and provides a comprehensive set of normalization steps.

### Quick Example

```python
from atomingest.normalization import normalize_csv_file, NormalizationStepConfig

# Define normalization steps
steps = [
    NormalizationStepConfig(step="RemoveBOMStep", config={}, enabled=True),
    NormalizationStepConfig(step="TrimWhitespaceStep", config={"trim_quotes": True}, enabled=True),
    NormalizationStepConfig(step="NullTokenNormalizationStep", config={"null_tokens": ["", "NA", "NULL"]}, enabled=True),
]

# Normalize CSV file
rows, report = normalize_csv_file("data.csv", steps=steps)

print(f"Processed {report.rows_emitted} rows")
print(f"Warnings: {len(report.warnings)}")
```

### Using the Pipeline Directly

```python
from atomingest.normalization import NormalizationPipeline, NormalizationConfig

# Create configuration
config = NormalizationConfig.from_dict({
    "strict": False,
    "source_type": "csv",
    "source_config": {"delimiter": ",", "encoding": "utf-8"},
    "steps": [
        {"step": "ColumnNameNormalizationStep", "config": {"case": "lower"}, "enabled": True},
        {"step": "BasicTypeCleanupStep", "config": {"numeric_columns": ["amount", "price"]}, "enabled": True}
    ]
})

# Run pipeline
pipeline = NormalizationPipeline(config)
rows, report = pipeline.run("data.csv")

# Process results
for row in rows:
    print(row)

print(report.summary())
```

### Integration with AtomIngest Pipeline

You can embed normalization directly in your AtomIngest YAML configuration:

```yaml
target_table: products

# Normalization configuration
normalization:
  strict: false
  source_type: csv
  source_config:
    delimiter: ","
    encoding: "utf-8"
  steps:
    - step: RemoveBOMStep
      config: {}
      enabled: true
    - step: SkipUntilHeaderStep
      config:
        expected_columns: ["id", "name", "price"]
      enabled: true
    - step: TrimWhitespaceStep
      config:
        trim_quotes: true
        collapse_spaces: true
      enabled: true
    - step: NullTokenNormalizationStep
      config:
        null_tokens: ["", "NA", "NULL", "nan", "-"]
      enabled: true
    - step: ColumnNameNormalizationStep
      config:
        case: lower
        replace_spaces: "_"
        remove_non_alphanumeric: true
      enabled: true
    - step: BasicTypeCleanupStep
      config:
        numeric_columns: ["price", "quantity"]
        convert_to_number: true
        date_columns: ["created_at"]
        date_formats: ["%Y-%m-%d", "%m/%d/%Y"]
      enabled: true

schema:
  id: {type: IntegerField}
  name: {type: CharField, max_length: 100}
  price: {type: DecimalField, precision: 10, scale: 2}
  quantity: {type: IntegerField}
  created_at: {type: DateField}
```

### Available Normalization Steps

| Step | Purpose | Key Config Options |
|------|---------|-------------------|
| `RemoveBOMStep` | Remove UTF-8 BOM from files | `check_all_lines` |
| `SkipUntilHeaderStep` | Skip metadata lines until header found | `header_regex`, `expected_columns` |
| `DropRepeatedHeaderRowsStep` | Remove duplicate headers in concatenated files | None |
| `TrimWhitespaceStep` | Trim whitespace and quotes | `trim_quotes`, `collapse_spaces` |
| `NullTokenNormalizationStep` | Convert null tokens to None | `null_tokens` |
| `ColumnNameNormalizationStep` | Standardize column names | `case`, `replace_spaces`, `alias_map` |
| `BasicTypeCleanupStep` | Parse and clean data types | `numeric_columns`, `boolean_columns`, `date_columns` |
| `DropTrailerLinesStep` | Remove trailer/footer lines | `trailer_regex`, `row_count_regex` |

### Expected Report Structure

The normalization pipeline returns a detailed report with processing metrics:

```python
# Sample report output
{
    "rows_read": 1000,
    "rows_emitted": 995,
    "metadata_skipped": 2,
    "trailer_skipped": 1,
    "duplicate_headers": 0,
    "warnings": [
        "UTF-8 BOM detected and removed.",
        "Failed to parse date value '2023-13-45' in column 'date'"
    ],
    "trailer_row_count": 998,
    "validation_errors": []
}
```
