# AtomIngest Normalization Examples

This directory contains comprehensive examples demonstrating the AtomIngest normalization pipeline capabilities.

## Files Overview

| File | Description |
|------|-------------|
| `simple_dataset.yaml` | Basic normalization configuration for clean customer data |
| `complex_dataset.yaml` | Advanced configuration handling messy financial data with metadata/trailers |
| `simple_customers.csv` | Sample customer data with basic formatting issues |
| `complex_transactions.csv` | Sample financial data with headers, trailers, and various data quality issues |
| `demo_normalization.py` | Python script demonstrating programmatic usage of normalization pipeline |
| `report_examples.md` | Detailed documentation of report structure and expected outputs |

## Quick Start

### 1. Run the Demo Script

```bash
cd examples/normalization/
python demo_normalization.py
```

This will demonstrate:
- Simple normalization with basic steps
- Complex normalization with advanced features
- YAML configuration loading (if PyYAML is installed)

### 2. Try the Simple Example

```python
from atomingest.normalization import normalize_csv_file, NormalizationStepConfig

steps = [
    NormalizationStepConfig(step="TrimWhitespaceStep", config={"trim_quotes": True}, enabled=True),
    NormalizationStepConfig(step="ColumnNameNormalizationStep", config={"case": "lower"}, enabled=True),
]

rows, report = normalize_csv_file("simple_customers.csv", steps=steps)
print(f"Processed {report.rows_emitted} rows with {len(report.warnings)} warnings")
```

### 3. Use YAML Configuration

```python
import yaml
from atomingest.normalization import NormalizationConfig, NormalizationPipeline

# Load configuration from YAML
with open("simple_dataset.yaml", "r") as f:
    config_data = yaml.safe_load(f)

# Create pipeline
config = NormalizationConfig.from_dict(config_data["normalization"])
pipeline = NormalizationPipeline(config)

# Process data
rows, report = pipeline.run("simple_customers.csv")
```

## Normalization Steps Demonstrated

### Basic Steps (Simple Example)
- **RemoveBOMStep**: Remove UTF-8 BOM characters
- **TrimWhitespaceStep**: Clean whitespace and quotes
- **NullTokenNormalizationStep**: Convert null tokens to None
- **ColumnNameNormalizationStep**: Standardize column names
- **BasicTypeCleanupStep**: Parse and clean data types

### Advanced Steps (Complex Example)
- **SkipUntilHeaderStep**: Skip metadata until header is found
- **DropRepeatedHeaderRowsStep**: Remove duplicate headers
- **DropTrailerLinesStep**: Remove trailer/summary lines
- **Advanced column mapping**: Handle legacy column names with aliases
- **Multiple date formats**: Parse various international date formats
- **Row count validation**: Validate against trailer row counts

## Expected Outcomes

### Simple Dataset Processing
```
Input:  "1,001", " John ", " Doe ",john.doe@example.com,25,yes,2023-01-15
Output: {"customer_id": 1001, "first_name": "John", "last_name": "Doe", ...}
```

### Complex Dataset Processing
```
Input:  Messy file with metadata, duplicate headers, trailers
Output: Clean normalized records with detailed processing report
```

## Configuration Keys Reference

### Core Configuration
- `strict`: Enable strict mode (fail on warnings)
- `source_type`: Data source type ("csv", "json", "excel", "parquet")
- `source_config`: Source-specific configuration (delimiter, encoding, etc.)
- `steps`: Array of normalization steps to apply

### Step Configuration Structure
```yaml
steps:
  - step: "StepClassName"           # Required: Step class name
    config: {}                     # Required: Step-specific configuration
    enabled: true                  # Required: Enable/disable step
```

### Common Step Configurations

#### TrimWhitespaceStep
```yaml
config:
  trim_quotes: true       # Remove surrounding quotes
  collapse_spaces: true   # Replace multiple spaces with single space
```

#### ColumnNameNormalizationStep
```yaml
config:
  case: "lower"                    # "lower", "upper", or null
  replace_spaces: "_"              # Replace spaces with underscores
  remove_non_alphanumeric: true    # Remove special characters
  alias_map:                       # Map old names to new names
    "Old Name": "new_name"
```

#### BasicTypeCleanupStep
```yaml
config:
  numeric_columns: ["amount", "price"]     # Columns to parse as numbers
  convert_to_number: true                  # Convert to actual numbers vs strings
  boolean_columns: ["is_active"]           # Columns to parse as boolean
  date_columns: ["created_date"]           # Columns to parse as dates
  date_formats: ["%Y-%m-%d", "%m/%d/%Y"]   # Date parsing formats to try
  output_date_format: "%Y-%m-%d"           # Standardized output format
```

## Integration Patterns

### With AtomIngest Pipeline
```yaml
target_table: customers
normalization:
  # ... normalization config here
schema:
  # ... schema definition here
```

### Standalone Usage
```python
from atomingest.normalization import normalize_then_validate

rows, report = normalize_then_validate(
    source="data.csv",
    steps=normalization_steps,
    strict=False
)
```

### Custom Step Development
```python
from atomingest.normalization.steps import BaseNormalizerStep, register_step

@register_step("CustomCleanupStep")
class CustomCleanupStep(BaseNormalizerStep):
    def process_row(self, row):
        # Custom normalization logic
        return row
```

## Troubleshooting

### Common Issues
1. **Step not found error**: Ensure step name matches exactly (case-sensitive)
2. **Date parsing failures**: Check date_formats configuration matches your data
3. **Column mapping issues**: Verify alias_map keys match actual column names
4. **Row count validation**: Ensure trailer_regex correctly identifies count lines

### Debug Tips
1. Set `strict: false` during development to see all warnings
2. Check `report.warnings` for detailed error messages
3. Use smaller test files to isolate configuration issues
4. Enable verbose logging in your application

## Performance Considerations

### For Large Files
- Use `trailer_position: "end-only"` instead of "anywhere"
- Set reasonable `max_skip` limits for header detection
- Consider processing in chunks for very large datasets

### Memory Usage
- The pipeline processes data in streaming fashion
- Only final results are kept in memory
- Use the iterator interface for large datasets:

```python
row_iterator, report = pipeline.run("large_file.csv")
for row in row_iterator:
    process_immediately(row)  # Don't accumulate in memory
```

## Next Steps

1. Review the [main normalization documentation](../../docs/normalization.md)
2. Explore the [fintech samples](../fintech_samples/) for real-world examples
3. Check the [API reference](../../docs/api_reference.md) for complete method documentation
4. See [performance tuning guide](../../docs/performance.md) for optimization tips
