# Changelog

All notable changes to this project will be documented in this file.

## [2.1.4] - 2026-01-27
### Security Remediation & Hardening (Phase 3)
- **JSON Logging**: Implemented `JsonFormatter` for secure, structured logging.
- **Strict Error Handling**: Introduced `PipelineError` consistency and strict mode overrides.
- **Enhanced Schema**: Added `pattern` (regex) and `datetime` validation support.

## [2.1.2] - 2026-01-27
### Security Remediation (Phase 1 & 2)
- **Path Traversal Protection**: Restricted file operations to CWD.
- **Docker Hardening**: Migrated to digest-pinned base images and non-root user.
- **Memory Guardrails**: Implemented active memory monitoring and checkpoint eviction.
- **Config Validation**: Added strict validation for `PipelineConfig` at initialization.

## [2.1.1] - 2026-01-27

### 🔄 Verification & Testing
- **Stress Testing**: Verified pipeline with 5 Million rows (Product Sales Schema).
- **Benchmarking**: Confirmed linear scaling (~300k rows/sec).
- **Documentation**: Added `testlog.md` with detailed performance reports.

### 🐛 Bug Fixes
- **Outlier Removal**: Fixed index misalignment bug in `remove_outliers` that caused incorrect row filtering when used after `drop_duplicates`.

## [2.1.0] - 2026-01-26

### 🚀 New Features
- **Schema Validation**: Added `validate_schema()` method to enforce data contracts via YAML.
- **Chunking Support**: Added `process_in_chunks()` generator to handle files larger than RAM.
- **Isolation Forest**: Added ML-based outlier detection (`method='isolation_forest'`).
- **Export Formats**: Added support for Parquet (`to_parquet`) and JSON (`to_json`).
- **Configurable Rollback**: Added `max_checkpoints` setting to control memory usage.

### ⚡ Performance
- **Vectorized String Ops**: 10x-100x speedup in `standardize_text` by removing lambda functions.
- **Regex Caching**: Optimized `validate_email` by caching compiled regex patterns.

### 🛡️ Security & Robustness
- **Path Traversal Protection**: Added validation for all file I/O operations.
- **Safe Custom Functions**: Added `allow_custom_functions` flag (default: False).
- **Zero-Division Safety**: Fixed crashes in Z-score calculation for constant columns.

### 🐛 Bug Fixes
- Fixed `TypeError` in report timestamp generation (Timezone awareness).
- Fixed `int64` JSON serialization error in audit reports.
- Fixed off-by-one logic error in `rollback()` method.
- Resolved broken imports in `src/__init__.py`.

## [2.0.0] - 2026-01-01
- Initial major release.
