# Normalization Report Output Examples

This document shows the expected report structure and sample outputs from the AtomIngest normalization pipeline.

## Report Structure

The `NormalizationReport` class tracks detailed metrics during the normalization process:

```python
class NormalizationReport:
    rows_read: int = 0              # Total input rows processed
    rows_emitted: int = 0           # Valid rows output after normalization
    metadata_skipped: int = 0       # Metadata/header lines skipped
    trailer_skipped: int = 0        # Trailer/footer lines removed
    duplicate_headers: int = 0      # Duplicate header rows removed
    trailer_row_count: Optional[int] = None  # Expected row count from trailer
    warnings: List[str] = []        # Warning messages collected
    validation_errors: List[str] = []  # Validation errors encountered
```

## Simple Dataset Report Example

### Input Data
```csv
Customer ID, First Name , Last Name,Email Address,Age,Active Status,Registration Date
"1,001", " John ", " Doe ",john.doe@example.com,25,yes,2023-01-15
2002, Jane , Smith ,jane.smith@example.com,N/A,true,01/20/2023
"3,003"," Bob "," Johnson ",bob@example.com,30,1,2023/02/15
```

### Normalization Steps Applied
- RemoveBOMStep
- TrimWhitespaceStep (trim_quotes=True)
- NullTokenNormalizationStep (null_tokens=["", "NA", "NULL"])
- ColumnNameNormalizationStep (case=lower, replace_spaces="_")
- BasicTypeCleanupStep (numeric_columns=["customer_id", "age"])

### Expected Report Output
```json
{
    "rows_read": 3,
    "rows_emitted": 3,
    "metadata_skipped": 0,
    "trailer_skipped": 0,
    "duplicate_headers": 0,
    "trailer_row_count": null,
    "warnings": [],
    "validation_errors": [],
    "summary": "Processed 3 rows successfully with 0 warnings"
}
```

### Normalized Data Output
```json
[
    {
        "customer_id": 1001,
        "first_name": "John",
        "last_name": "Doe",
        "email": "john.doe@example.com",
        "age": 25,
        "active_status": true,
        "registration_date": "2023-01-15"
    },
    {
        "customer_id": 2002,
        "first_name": "Jane",
        "last_name": "Smith",
        "email": "jane.smith@example.com",
        "age": null,
        "active_status": true,
        "registration_date": "2023-01-20"
    },
    {
        "customer_id": 3003,
        "first_name": "Bob",
        "last_name": "Johnson",
        "email": "bob@example.com",
        "age": 30,
        "active_status": true,
        "registration_date": "2023-02-15"
    }
]
```

## Complex Dataset Report Example

### Input Data (with metadata and trailers)
```
=====================================
Financial Transaction Export v2.1
Generated: 2024-01-15 09:30:00
Period: Jan 1, 2024 - Jan 15, 2024
=====================================

Transaction_ID,Account_Number,Amount,Currency,Date,Status
"TXN_001",12345678,"1,234.56",USD,2024-01-15,PROCESSED
TXN_002,"87654321","2,567.89",EUR,15/01/2024,PROCESSED
Transaction_ID,Account_Number,Amount,Currency,Date,Status
TXN_003,99887766,"invalid",USD,bad-date,FAILED

TOTAL RECORDS: 3
Record Count: 2
END OF FILE
```

### Normalization Steps Applied
- RemoveBOMStep
- SkipUntilHeaderStep (expected_columns=["Transaction_ID", "Account_Number"])
- DropRepeatedHeaderRowsStep
- DropTrailerLinesStep (trailer_regex="^(TOTAL|Record Count:|END)")
- TrimWhitespaceStep
- NullTokenNormalizationStep
- ColumnNameNormalizationStep (with alias mapping)
- BasicTypeCleanupStep (with strict validation)

### Expected Report Output
```json
{
    "rows_read": 3,
    "rows_emitted": 2,
    "metadata_skipped": 5,
    "trailer_skipped": 3,
    "duplicate_headers": 1,
    "trailer_row_count": 2,
    "warnings": [
        "Failed to parse numeric value 'invalid' in column 'amount': invalid literal for float()",
        "Failed to parse date value 'bad-date' in column 'transaction_date' with formats: ['%Y-%m-%d', '%m/%d/%Y']"
    ],
    "validation_errors": [],
    "summary": "Processed 2 of 3 rows with 2 warnings. Row count validation: PASSED (expected: 2, actual: 2)"
}
```

### Normalized Data Output
```json
[
    {
        "transaction_id": "TXN_001",
        "account_number": "12345678",
        "amount": 1234.56,
        "currency_code": "USD",
        "transaction_date": "2024-01-15",
        "status": "PROCESSED"
    },
    {
        "transaction_id": "TXN_002",
        "account_number": "87654321",
        "amount": 2567.89,
        "currency_code": "EUR",
        "transaction_date": "2024-01-15",
        "status": "PROCESSED"
    }
]
```

## Report with Validation Failures

### Input Data (strict mode)
```csv
id,email,age
1,invalid-email,150
2,john@example.com,25
3,jane@example.com,-5
```

### Strict Mode Configuration
```yaml
normalization:
  strict: true  # Fail on warnings
  steps:
    - step: BasicTypeCleanupStep
      config:
        numeric_columns: ["age"]
        convert_to_number: true
      enabled: true
```

### Expected Report Output (with errors)
```json
{
    "rows_read": 1,
    "rows_emitted": 0,
    "metadata_skipped": 0,
    "trailer_skipped": 0,
    "duplicate_headers": 0,
    "trailer_row_count": null,
    "warnings": [],
    "validation_errors": [
        "ValueError: Age value '150' exceeds maximum allowed value of 120",
        "ValueError: Age value '-5' is below minimum allowed value of 0"
    ],
    "summary": "Processing failed in strict mode. 2 validation errors encountered."
}
```

## Report Method Examples

### Basic Report Methods
```python
# Get summary string
summary = report.summary()
# Output: "Processed 100 rows successfully with 3 warnings"

# Check if processing was successful
is_valid = report.is_valid()
# Output: True (if no validation errors)

# Get warning count
warning_count = len(report.warnings)
# Output: 3

# Validate row count against trailer
validation_result = report.validate_row_count(strict=False)
# Output: "PASSED" or "FAILED"

# Get detailed metrics
metrics = {
    "success_rate": report.rows_emitted / report.rows_read * 100,
    "skip_rate": (report.metadata_skipped + report.trailer_skipped) / report.rows_read * 100,
    "error_rate": len(report.validation_errors) / report.rows_read * 100
}
```

### Conditional Processing Based on Report
```python
rows, report = normalize_csv_file("data.csv", steps=steps)

# Check if processing was successful enough
if report.rows_emitted > 0:
    print(f"Successfully processed {report.rows_emitted} rows")

    # Warning threshold check
    if len(report.warnings) > 10:
        print("Warning: High number of warnings detected")

    # Row count validation
    if report.trailer_row_count:
        if report.validate_row_count() == "PASSED":
            print("Row count validation passed")
        else:
            print("Row count mismatch detected")

    # Process the normalized data
    for row in rows:
        # Insert into database, send to queue, etc.
        process_record(row)

else:
    print("Processing failed - no valid rows produced")
    for error in report.validation_errors:
        print(f"Error: {error}")
```

## Integration with AtomIngest Pipeline

When using normalization within a full AtomIngest pipeline, the report is available in hooks and error handlers:

```python
def post_normalization_hook(rows, report, context):
    """Custom hook to handle normalization results."""

    # Log metrics
    context.logger.info(f"Normalization completed: {report.summary()}")

    # Handle warnings
    if report.warnings:
        for warning in report.warnings:
            context.logger.warning(f"Normalization warning: {warning}")

    # Quality checks
    success_rate = report.rows_emitted / report.rows_read
    if success_rate < 0.95:  # Less than 95% success
        context.logger.error(f"Low success rate: {success_rate:.2%}")
        if context.config.get("fail_on_low_quality", False):
            raise ValueError("Data quality threshold not met")

    return rows
```
