# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.2.0] - 2026-03-15

### Added

- **Dead Letter Queue (DLQ)** — full file-based NDJSON persistence for failed rows, with
  retry runner, CSV/JSON export, and status tracking (`PENDING`, `RETRYING`, `RESOLVED`,
  `EXHAUSTED`). Wired into the `ON_ERROR` hook and Prometheus metrics automatically.
- **`DropTrailerLinesStep`** normalizer step — drops trailer/footer lines by regex pattern,
  with optional row-count extraction for downstream validation. Accepts both `pattern` and
  `trailer_regex` config keys.

### Fixed

- **`business_key` dedup was silently broken** — the ingester was reading
  `pipeline_config["business_keys"]` (nested, plural) while YAML places `business_key`
  (singular) at the top level. Idempotent upserts now work correctly. The ingester also
  accepts `business_keys` (plural) as a backward-compatibility alias.
- **`id` field conflict** — passing `id` as a CSV data column caused AtomSQL's `save()`
  to run `UPDATE WHERE id=N` instead of `INSERT`, silently dropping all rows. Documented
  that `id` is reserved for AtomSQL's auto-increment primary key.
- **14 pre-existing test failures** on `develop` resolved: `_table_name` placement in
  drift-detector tests, `capture_context` arity in audit-metadata tests, nested
  `config.config` lookup for `audit_metadata`, root `__init__.py` missing `__version__`,
  and version strings in CLI tests.
- **5 skipped ingester tests** (SQLite in-memory pool isolation) — switched fixtures to
  file-based SQLite via `tmp_path`; tests now run and assert actual DB state.

### Changed

- **Accurate YAML documentation** — removed the fictional `strategy: insert/update/upsert`
  top-level field from all docs and examples (it never existed in the code). Rewrote the
  `business_key` reference with correct key name, behavior table, and working examples.
  Renamed `business_keys` → `business_key` across all 5 affected doc files.

### Test Suite

- 930 tests passing, 0 failing, 0 skipped (up from 896 passing / 14 failing / 5 skipped
  at 0.1.1)

## [0.1.1] - 2026-02-10

### Fixed
- Fixed CLI version display to use package `__version__` instead of hardcoded value
- Fixed import paths in main `__init__.py` for proper module loading

## [0.1.0] - 2026-01-29

### Added
- Initial release of AtomIngest
- Core ingestion framework with YAML-driven configuration
- Dynamic model generation using AtomSQL metaprogramming
- Comprehensive normalization pipeline with 8+ built-in steps:
  - `RemoveBOMStep` - Remove UTF-8 BOM characters
  - `SkipUntilHeaderStep` - Skip metadata until header found
  - `DropRepeatedHeaderRowsStep` - Remove duplicate headers
  - `DropTrailerLinesStep` - Remove trailer/footer lines
  - `TrimWhitespaceStep` - Clean whitespace and quotes
  - `NullTokenNormalizationStep` - Convert null tokens to None
  - `ColumnNameNormalizationStep` - Standardize column names
  - `BasicTypeCleanupStep` - Parse and clean data types
- Multi-format data source support (CSV, JSON, Excel, Parquet)
- Hook system for extensible pipeline customization
- Schema drift detection and validation
- Audit metadata injection with run tracking
- Security features including PII encryption and key management
- Comprehensive error handling and reporting
- CLI interface with `atomingest` command
- Fintech-focused examples and documentation
- Comprehensive test suite with 850+ tests

### Features
- **YAML-Driven Configuration**: Define table schemas, constraints, and pipelines declaratively
- **Dynamic Model Generation**: Uses Python metaprogramming to create AtomSQL models at runtime
- **Fintech-Grade Security**: PII encryption, immutable ledgers, crypto-shredding for GDPR
- **Robust Validation**: Regex, range checks, and custom business rules
- **High Performance**: Async batching and smart buffering
- **Reliability**: Idempotency, Dead Letter Queues, transaction management

### Documentation
- Comprehensive README with quick start guide
- Detailed API documentation
- Fintech user guide with compliance patterns
- Normalization pipeline documentation and examples
- Performance tuning guide
- Security best practices

### Examples
- Basic ingestion pipeline examples
- Fintech sample projects (payment processing, market data)
- Simple and complex normalization configurations
- Hook system usage examples
- Schema validation examples

[Unreleased]: https://github.com/prashant-fintech/atomingest/compare/v0.1.0...HEAD
[0.1.0]: https://github.com/prashant-fintech/atomingest/releases/tag/v0.1.0
