# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.0.7] - 2025-08-08

### Added
- **Universal Step Builder Test System** - Comprehensive testing framework for step builder validation
  - **CLI Interface** - New command-line interface for running builder tests (`cursus.cli.builder_test_cli`)
  - **Four-Level Test Architecture** - Structured testing across Interface, Specification, Path Mapping, and Integration levels
  - **Test Scoring System** - Advanced scoring mechanism with weighted evaluation across test levels
  - **Processing Step Builder Tests** - Specialized test variants for processing step builders
  - **Test Pattern Detection** - Smart pattern-based test categorization and level detection

- **Enhanced Validation Infrastructure** - Major expansion of validation capabilities
  - **Builder Test Variants** - Specialized test classes for different step builder types (Processing, Training, Transform)
  - **Interface Standard Validator** - Validation of step builder interface compliance
  - **Test Scoring and Rating** - Comprehensive scoring system with quality ratings (Excellent, Good, Satisfactory, Needs Work, Poor)
  - **Test Report Generation** - Automated generation of detailed test reports with charts and statistics

- **CLI Enhancements** - Extended command-line interface functionality
  - **Universal Builder Test CLI** - Run tests at different levels and variants for any step builder
  - **Validation CLI** - Command-line validation tools for step builders and specifications
  - **Builder Discovery** - Automatic discovery and listing of available step builder classes

### Enhanced
- **Test Coverage and Quality** - Improved test infrastructure and coverage analysis
  - **Pattern-Based Test Scoring** - Advanced scoring system with level-based weighting and importance factors
  - **Test Level Detection** - Smart detection of test levels using explicit prefixes, keywords, and fallback mapping
  - **Comprehensive Test Reports** - Detailed reports with pass rates, level scores, and failure analysis
  - **Test Execution Performance** - Optimized test execution with better error handling and reporting

- **Documentation and Naming** - Improved consistency and documentation
  - **Brand Consistency** - Updated all references from "AutoPipe" to "Cursus" throughout the codebase
  - **Package Name Standardization** - Consistent use of "cursus" instead of legacy "autopipe" references
  - **Documentation Headers** - Standardized documentation format and headers across modules

### Fixed
- **Import Path Corrections** - Fixed import errors and path issues
  - **Module Import Fixes** - Resolved import path issues in validation and CLI modules
  - **Test Module Stability** - Improved test module imports and execution reliability
  - **Processing Step Builder Tests** - Fixed test execution issues for tabular preprocessing step builders

### Technical Details
- **Test Architecture** - Four-level testing system with weighted scoring (Interface: 1.0x, Specification: 1.5x, Path Mapping: 2.0x, Integration: 2.5x)
- **Scoring Algorithm** - Importance-weighted scoring with test-specific multipliers and comprehensive rating system
- **CLI Integration** - Full command-line interface for test execution with verbose output and detailed reporting
- **Test Discovery** - Automatic discovery of step builder classes with AST parsing for missing dependencies
- **Report Generation** - JSON reports, console output, and optional chart generation with matplotlib

### Quality Assurance
- **Universal Test Coverage** - Comprehensive testing framework covering all step builder types and patterns
- **Automated Quality Assessment** - Scoring system provides objective quality metrics for step builders
- **Regression Prevention** - Enhanced test suite prevents regressions in step builder implementations
- **Development Workflow** - Improved developer experience with CLI tools and automated validation

## [1.0.6] - 2025-08-07

### Added
- **Comprehensive Test Infrastructure** - Major expansion of testing capabilities
  - **Core Test Suite** - New comprehensive test runner for all core components (`test/core/run_core_tests.py`)
  - **Test Coverage Analysis** - Advanced coverage analysis tool with detailed reporting (`test/analyze_test_coverage.py`)
  - **Test Organization** - Restructured test directory with component-based organization
  - **Performance Metrics** - Test execution timing and performance analysis
  - **Quality Assurance Reports** - Automated generation of test quality and coverage reports

- **Three-Tier Configuration Architecture** - New configuration management system
  - **ConfigFieldTierRegistry** - Registry for field tier classifications (Tier 1: Essential User Inputs, Tier 2: System Inputs, Tier 3: Derived Inputs)
  - **Field Classification System** - Systematic categorization of 50+ configuration fields across three tiers
  - **Enhanced Configuration Management** - Improved configuration merging and serialization with tier awareness

- **Validation Module** - New pipeline validation infrastructure
  - **Pipeline Validation** - Utilities for validating pipeline components, specifications, and configurations
  - **Early Error Detection** - Catch configuration errors early in the development process
  - **Compatibility Checking** - Ensure specifications and contracts are valid and compatible

### Enhanced
- **Test Coverage Metrics** - Comprehensive analysis and reporting
  - **Overall Statistics** - 602 tests across 36 test modules with 85.5% success rate
  - **Component Coverage** - Detailed coverage analysis for Assembler (100%), Compiler (100%), Base (86.2%), Config Fields (99.0%), Deps (68.2%)
  - **Function Coverage** - 50.6% function coverage (166/328 functions tested) with detailed gap analysis
  - **Redundancy Analysis** - Identification and reporting of test redundancy patterns

- **Configuration Field Management** - Major improvements to config field handling
  - **Type-Aware Serialization** - Enhanced serialization with better type preservation
  - **Configuration Merging** - Improved config merger with better error handling
  - **Circular Reference Tracking** - Enhanced circular reference detection and prevention

### Fixed
- **Import Path Corrections** - Fixed multiple import path issues across the codebase
  - **Test Module Imports** - Corrected import paths in test modules for better reliability
  - **Configuration Module Imports** - Fixed import issues in config field modules
  - **Base Class Imports** - Resolved import path issues in base classes

- **Test Infrastructure Stability** - Improved test execution reliability
  - **Mock Configuration** - Fixed mock setup issues in factory tests
  - **Test Isolation** - Improved test isolation to prevent state contamination
  - **Path Configuration** - Fixed path configuration issues in test modules

### Technical Details
- **Test Infrastructure** - 602 tests with 1.56 second execution time and 1,847 total assertions
- **Code Organization** - Restructured test directory from flat structure to component-based hierarchy
- **Quality Metrics** - 94.9% unique test names with 5.1% redundancy across components
- **Performance** - Average test duration of 2.59ms per test with efficient memory usage
- **Documentation** - Comprehensive test documentation with usage examples and troubleshooting guides

### Quality Assurance
- **Automated Reporting** - Generated comprehensive test reports in JSON and Markdown formats
- **Coverage Tracking** - Detailed function-level coverage analysis with gap identification
- **Performance Monitoring** - Test execution performance metrics and optimization recommendations
- **Regression Prevention** - Enhanced test suite to prevent future regressions

## [1.0.5] - 2025-08-06

### Fixed
- **CRITICAL: Circular Import Resolution** - Completely resolved all circular import issues in the package
  - Fixed circular dependency in `cursus.core.base.builder_base` module that was preventing 89.3% of modules from importing
  - Implemented lazy loading pattern using property decorator to break circular import chain
  - Root cause: `builder_base.py` → `step_names` → `builders` → `builder_base.py` circular dependency
  - Solution: Converted direct import to lazy property loading with graceful fallback
  - **Result**: 98.7% module import success rate (157/159 modules), up from 10.7% (17/159 modules)

### Added
- **Comprehensive Circular Import Test Suite** - New testing infrastructure to prevent future regressions
  - Created `test/circular_imports/` directory with complete test framework
  - Added 5 comprehensive test categories covering all package modules
  - Automated detection and reporting of circular import issues
  - Import order independence testing to ensure robust module loading
  - Detailed error reporting with exact circular dependency chains
  - Test output logging with timestamps and comprehensive statistics

### Changed
- **Package Architecture Improvement** - Enhanced module loading reliability
  - All core packages now import successfully without circular dependencies
  - Maintained Single Source of Truth design principle while fixing imports
  - Preserved existing API compatibility during circular import resolution
  - Improved error handling for optional dependencies

### Technical Details
- **Module Import Success Rate**: Improved from 10.7% to 98.7% (157/159 modules successful)
- **Circular Imports Eliminated**: Reduced from 142 detected circular imports to 0
- **Core Package Health**: 100% of core packages (cursus.core.*) now import cleanly
- **Test Coverage**: 159 modules tested across 15 package categories
- **Only Remaining Import Issues**: 2 modules with missing optional dependencies (expected behavior)
- **Package Categories Tested**: Core (4), API (1), Steps (7), Processing (1), Root (2) - all 100% clean

### Quality Assurance
- **Comprehensive Testing**: All 5 circular import tests now pass (previously 1/5 passing)
- **Regression Prevention**: Test suite integrated for ongoing monitoring
- **Package Health Monitoring**: Automated detection of import issues
- **Development Workflow Restored**: Normal import behavior for all development activities

## [1.0.4] - 2025-08-06

### Fixed
- **DAG Compiler Enhancement** - Fixed template state management in `PipelineDAGCompiler`
  - Added `_last_template` attribute to store template after pipeline generation
  - Fixed timing issues with template metadata population during compilation
  - Added `get_last_template()` method to access fully-populated templates
  - Added `compile_and_fill_execution_doc()` method for proper sequencing of compilation and document filling

### Changed
- **Package Redeployment** - Updated source code and repackaged for PyPI distribution
- **Version Increment** - Incremented version to 1.0.4 for new PyPI release

### Technical Details
- **Template State Management** - Templates now properly retain state after pipeline generation, enabling access to pipeline metadata for execution document generation
- **Execution Document Integration** - New method ensures proper sequencing when both compiling pipelines and filling execution documents
- Rebuilt package with latest source code changes
- Successfully uploaded to PyPI: https://pypi.org/project/cursus/1.0.4/
- All dependencies and metadata validated
- Package available for installation via `pip install cursus==1.0.4`

## [1.0.3] - 2025-08-03

### Fixed
- Fixed import error in processing module `__init__.py` for `MultiClassLabelProcessor` (was incorrectly named `MulticlassLabelProcessor`)
- Corrected class name reference in module exports

## [1.0.2] - 2025-08-03

### Added
- **Processing Module** - New `cursus.processing` module with comprehensive data processing utilities
  - **Base Processor Classes** - `Processor`, `ComposedProcessor`, `IdentityProcessor` for building processing pipelines
  - **Categorical Processing** - `CategoricalLabelProcessor`, `MulticlassLabelProcessor` for label encoding
  - **Numerical Processing** - `NumericalVariableImputationProcessor`, `NumericalBinningProcessor` for data preprocessing
  - **Text/NLP Processing** - `BertTokenizeProcessor`, `GensimTokenizeProcessor` for text tokenization
  - **Domain-Specific Processors** - `BSMProcessor`, `CSProcessor`, `RiskTableProcessor` for specialized use cases
  - **Data Loading Utilities** - `BSMDataLoader`, `BSMDatasets` for data management
  - **Processor Composition** - Support for chaining processors using `>>` operator

### Fixed
- **Import Path Corrections** - Fixed all incorrect import paths in builder_registry.py and related modules
  - Corrected circular import issues using TYPE_CHECKING pattern
  - Fixed imports from non-existent `base_script_contract` to proper `...core.base.contract_base`
  - Updated all contract files to use correct base class imports
  - Resolved dependency resolver import issues in builder_base.py
- **Registry System** - Improved stability of step builder registry initialization
- **Type Safety** - Enhanced type checking with proper runtime placeholders

### Technical Details
- **Processing Pipeline** - Processors can be used in preprocessing, inference, evaluation, and other ML pipeline steps
- **Modular Design** - Each processor is self-contained with clear interfaces and composition support
- **Optional Dependencies** - Graceful handling of optional dependencies for specialized processors
- Fixed 10+ contract files with incorrect import statements
- Implemented TYPE_CHECKING pattern to break circular dependencies
- Added runtime placeholders for optional dependencies
- Corrected relative import paths throughout the registry system

## [1.0.1] - 2025-08-01

### Fixed
- Minor bug fixes and stability improvements
- Documentation updates

## [1.0.0] - 2025-01-31

### Added
- **Initial PyPI Release** - First public release of Cursus
- **Core API** - Main pipeline compilation functionality
  - `compile_dag()` - Simple DAG compilation
  - `compile_dag_to_pipeline()` - Advanced compilation with configuration
  - `PipelineDAGCompiler` - Full-featured compiler class
  - `create_pipeline_from_dag()` - Convenience function for quick pipeline creation

- **Command Line Interface** - Complete CLI for pipeline management
  - `cursus compile` - Compile DAG files to SageMaker pipelines
  - `cursus validate` - Validate DAG structure and compatibility
  - `cursus preview` - Preview compilation results
  - `cursus list-steps` - Show available step types
  - `cursus init` - Generate new projects from templates

- **Core Architecture** - Production-ready pipeline generation system
  - **Pipeline DAG** - Mathematical framework for pipeline topology
  - **Dependency Resolution** - Intelligent matching with semantic compatibility
  - **Step Builders** - Transform specifications into executable SageMaker steps
  - **Configuration Management** - Hierarchical configuration with validation
  - **Registry System** - Component registration and discovery

- **ML Framework Support** - Optional dependencies for different use cases
  - **PyTorch** - PyTorch Lightning models with SageMaker integration
  - **XGBoost** - XGBoost training pipelines with hyperparameter tuning
  - **NLP** - Natural language processing models and utilities
  - **Processing** - Advanced data processing and transformation

- **Template System** - Project scaffolding and examples
  - XGBoost template for tabular data pipelines
  - PyTorch template for deep learning workflows
  - Basic template for simple processing pipelines

- **Quality Assurance** - Enterprise-ready validation and testing
  - Comprehensive error handling and debugging
  - Type-safe specifications with compile-time checks
  - Built-in quality gates and validation rules
  - Production deployment compatibility

### Features
- **🎯 Graph-to-Pipeline Automation** - Transform simple graphs into complete SageMaker pipelines
- **⚡ 10x Faster Development** - Minutes to working pipeline vs. weeks of manual configuration
- **🧠 Intelligent Dependency Resolution** - Automatic step connections and data flow
- **🛡️ Production Ready** - Built-in quality gates, validation, and enterprise governance
- **📈 Proven Results** - 60% average code reduction across pipeline components

### Technical Specifications
- **Python Support** - Python 3.8, 3.9, 3.10, 3.11, 3.12
- **AWS Integration** - Full SageMaker compatibility with boto3 and sagemaker SDK
- **Architecture** - Modular, extensible design with clear separation of concerns
- **Dependencies** - Minimal core dependencies with optional framework extensions
- **Testing** - Comprehensive test suite with unit and integration tests

### Documentation
- Complete API documentation with examples
- Command-line interface reference
- Architecture and design principles
- Developer guide for contributions and extensions
- Ready-to-use pipeline examples and templates

### Performance
- **Code Reduction** - 55% average reduction in pipeline code
- **Development Speed** - 95% reduction in development time
- **Lines Eliminated** - 1,650+ lines of complex SageMaker configuration code
- **Quality Improvement** - Built-in validation prevents common configuration errors

## [Unreleased]

### Planned Features
- **Enhanced Templates** - Additional pipeline templates for common ML patterns
- **Visual DAG Editor** - Web-based interface for visual pipeline construction
- **Advanced Monitoring** - Built-in pipeline monitoring and alerting
- **Multi-Cloud Support** - Extension to other cloud ML platforms
- **Auto-Optimization** - Automatic resource and cost optimization
- **Integration Plugins** - Pre-built integrations with popular ML tools

---

## Release Notes

### Version 1.0.0 - Production Ready

This initial release represents the culmination of extensive development and testing in enterprise environments. Cursus is now production-ready with:

- **98% Complete Implementation** - All core features implemented and tested
- **Enterprise Validation** - Proven in production deployments
- **Comprehensive Documentation** - Complete guides and API reference
- **Quality Assurance** - Extensive testing and validation frameworks

### Migration from Internal Version

If you're migrating from an internal or pre-release version:

1. **Update Imports** - Change from `src.pipeline_api` to `cursus.api`
2. **Install Package** - `pip install cursus[all]` for full functionality
3. **Update Configuration** - Review configuration files for any breaking changes
4. **Test Thoroughly** - Validate all existing DAGs with `cursus validate`

### Getting Started

For new users:

1. **Install** - `pip install cursus`
2. **Generate Project** - `cursus init --template xgboost --name my-project`
3. **Validate** - `cursus validate dags/main.py`
4. **Compile** - `cursus compile dags/main.py --name my-pipeline`

### Support

- **Documentation** - https://github.com/TianpeiLuke/cursus/blob/main/README.md
- **Issues** - https://github.com/TianpeiLuke/cursus/issues
- **Discussions** - https://github.com/TianpeiLuke/cursus/discussions

---

**Cursus v1.0.0** - Making SageMaker pipeline development 10x faster through intelligent automation. 🚀
