# Updated FSM Package Implementation Plan

## Overview
This updated implementation plan incorporates the design clarifications from use case analysis, particularly the data handling strategies, resource management system, and streaming capabilities. The plan prioritizes practical implementation based on validated use cases.

## Key Updates from Original Plan

### Major Additions
1. **Data Operation Modes** (COPY, REFERENCE, DIRECT)
2. **Resource Management System** as first-class component
3. **Native Streaming Support** with chunking
4. **Transaction Management** strategies
5. **Simplified API** for common use cases

### Priority Changes
- Resource management moved earlier (Phase 2)
- Streaming support moved earlier (Phase 3)
- Data modes integrated throughout phases
- Configuration simplified based on use cases

## Development Phases

### Phase 1: Core Foundation with Data Modes (Week 1-2) ✅ COMPLETED
Establish the basic FSM structure with data handling strategies from the start.

**Status**: Completed on 2025-09-05
- All components implemented and tested
- 104 tests passing with 95% code coverage

#### Checklist:
- [x] **Project Setup**
  - [x] Create package directory structure
  - [x] Configure pyproject.toml with dependencies
  - [x] Set up testing framework (pytest)
  - [x] Configure linting and formatting (ruff, black)
  - [x] Create initial __init__.py files

- [x] **Data Mode System** (`core/data_modes.py`) **[NEW]**
  - [x] Define DataMode enum (COPY, REFERENCE, DIRECT)
  - [x] Implement DataHandler abstract base class
  - [x] Implement CopyModeHandler
    - [x] Deep copy on state entry
    - [x] Isolated data modifications
    - [x] Explicit commit mechanism
  - [x] Implement ReferenceModeHandler
    - [x] Lazy loading strategy
    - [x] Optimistic locking
    - [x] Reference resolution
  - [x] Implement DirectModeHandler
    - [x] In-place modifications
    - [x] Single-threaded enforcement
  - [x] Create unit tests for each mode

- [x] **Transaction Management** (`core/transactions.py`) **[NEW]**
  - [x] Define TransactionStrategy enum
  - [x] Implement SingleTransaction strategy
  - [x] Implement BatchTransaction strategy
    - [x] Configurable batch size
    - [x] Staging support
    - [x] Commit triggers
  - [x] Implement ManualTransaction strategy
  - [x] Create transaction context manager
  - [x] Add unit tests

- [x] **Base Interfaces** (`functions/base.py`)
  - [x] Define IValidationFunction interface
  - [x] Define ITransformFunction interface
  - [x] Define IStateTestFunction interface
  - [x] Define IEndStateTestFunction interface
  - [x] Define IResource interface **[NEW]**
  - [x] Create base exception classes
  - [x] Add type hints and documentation

- [x] **Core Data Models** (`core/state.py`)
  - [x] Implement StateDefinition class
    - [x] Name and metadata properties
    - [x] Schema definition using dataknobs_data.Field
    - [x] Resource requirements **[NEW]**
    - [x] Data mode preferences **[NEW]**
    - [x] Start/end state flags
    - [x] Function references
  - [x] Implement StateInstance class
    - [x] Unique ID generation
    - [x] Data storage with mode support **[UPDATED]**
    - [x] Transaction participation **[NEW]**
    - [x] Resource access context **[NEW]**
    - [x] Status tracking
    - [x] Arc execution history
  - [x] Add validation methods
  - [x] Create unit tests

### Phase 2: Resource Management System (Week 2-3) **[MOVED UP] ✅ COMPLETED**
Implement resource management before networks to support use cases.

**Status**: ✅ Completed on 2025-09-05 with 44 tests passing

#### Checklist:
- [x] **Resource Manager Core** (`resources/manager.py`) **[NEW]**
  - [x] Implement ResourceManager class
    - [x] Resource registration
    - [x] Lifecycle management
    - [x] Health monitoring
    - [x] Connection pooling
  - [x] Define resource configuration schema
  - [x] Implement resource acquisition/release
  - [x] Add timeout and retry logic
  - [x] Create unit tests

- [x] **Database Resource** (`resources/database.py`) **[NEW]**
  - [x] Implement DatabaseResourceAdapter class
    - [x] Connection pooling (via dataknobs_data)
    - [x] Transaction support
    - [x] Query execution
    - [x] Result streaming (via dataknobs_data)
  - [x] Support multiple backends via dataknobs_data
  - [x] Add connection health checks
  - [x] Create integration tests

- [x] **File System Resource** (`resources/filesystem.py`) **[NEW]**
  - [x] Implement FileSystemResource class
    - [x] File reading/writing
    - [x] Streaming support (via file handles)
    - [x] Directory operations
    - [x] Temporary file handling
  - [x] Add file locking support (via file handles)
  - [x] Implement buffering strategies
  - [x] Create unit tests

- [x] **HTTP Service Resource** (`resources/http.py`) **[NEW]**
  - [x] Implement HTTPServiceResource class
    - [x] Session management
    - [x] Request/response handling
    - [x] Retry logic
    - [x] Rate limiting
  - [x] Add authentication support (Bearer, Basic, API Key)
  - [x] Implement circuit breaker
  - [x] Create unit tests

- [x] **LLM Resource** (`resources/llm.py`) **[NEW]**
  - [x] Implement LLMResource class
    - [x] Model configuration
    - [x] Prompt handling
    - [x] Response streaming (placeholder)
    - [x] Token management
  - [x] Support multiple providers (OpenAI, Anthropic, Ollama, HuggingFace)
  - [ ] Add caching layer (deferred to future)
  - [x] Create integration tests

- [ ] **Vector Store Resource** (`resources/vector_store.py`) **[NEW]**
  - [ ] Implement VectorStoreResource class
    - [ ] Connection management
    - [ ] Vector operations
    - [ ] Similarity search
    - [ ] Metadata handling
  - [ ] Support multiple backends
  - [ ] Add batching support
  - [ ] Create integration tests

### Phase 3: Streaming and Network Implementation (Week 3-4) ✅ **COMPLETED** (2025-09-05)
Build streaming support early to validate large data handling.

#### Checklist:
- [x] **Streaming Core** (`streaming/core.py`) **[MOVED UP]**
  - [x] Implement StreamConfig class
    - [x] Chunk size configuration
    - [x] Parallelism settings
    - [x] Memory limits
    - [x] Backpressure handling
  - [x] Define IStreamSource interface
  - [x] Define IStreamSink interface
  - [x] Create stream context manager
  - [x] Add unit tests

- [x] **File Streaming** (`streaming/file_stream.py`) **[NEW]**
  - [x] Implement FileStreamSource
    - [x] Line-based reading
    - [x] Format detection (JSON, CSV, etc.)
    - [x] Compression support
    - [x] Progress tracking
  - [x] Implement FileStreamSink
    - [x] Buffered writing
    - [x] Format serialization
    - [x] Atomic writes
  - [x] Create performance tests

- [x] **Database Streaming** (`streaming/db_stream.py`) **[NEW]**
  - [x] Implement DatabaseStreamSource
    - [x] Cursor-based iteration
    - [x] Batch fetching
    - [x] Memory management
  - [x] Implement DatabaseStreamSink
    - [x] Batch inserts
    - [x] Upsert support
    - [x] Transaction batching
  - [x] Create integration tests

- [x] **State Network** (`core/network.py`)
  - [x] Implement StateNetwork class
    - [x] Network name/label
    - [x] State collection management
    - [x] Arc collection management
    - [x] Resource requirements aggregation **[NEW]**
    - [x] Streaming support flags **[NEW]**
  - [x] Add methods for:
    - [x] Adding/removing states
    - [x] Adding/removing arcs
    - [x] Validating network consistency
    - [x] Resource dependency analysis **[NEW]**
  - [x] Create unit tests

- [x] **Arc Implementation** (`core/arc.py`)
  - [x] Implement ArcDefinition class
    - [x] Target state reference
    - [x] Pre-test function reference
    - [x] Transform function reference
    - [x] Resource requirements **[NEW]**
  - [x] Implement PushArc subclass
    - [x] Target network reference
    - [x] Return state reference
    - [x] Data isolation mode **[NEW]**
  - [x] Implement ArcExecution class
    - [x] Execute with resource context **[NEW]**
    - [x] Stream-aware execution **[NEW]**
    - [x] Transaction participation **[NEW]**
  - [x] Create unit tests

### Phase 4: Execution Engine with Modes (Week 4-5) **✅ COMPLETED**
Implement execution engine with full support for data modes and resources.

**Status:** ✅ COMPLETED - 2025-09-05

#### Checklist:
- [x] **Execution Context** (`execution/context.py`)
  - [x] Implement ExecutionContext class
    - [x] Data mode configuration **[NEW]**
    - [x] Transaction management **[NEW]**
    - [x] Resource allocation **[NEW]**
    - [x] Stream coordination **[NEW]**
    - [x] Current state tracking
    - [x] Network stack (for push operations)
  - [x] Add context isolation per mode
  - [x] Support for parallel execution paths
  - [x] Create unit tests

- [x] **Execution Engine** (`execution/engine.py`)
  - [x] Implement ExecutionEngine class
    - [x] Mode-aware execution **[NEW]**
    - [x] Resource-aware scheduling **[NEW]**
    - [x] Stream processing support **[NEW]**
    - [x] Execute single transition
    - [x] Handle branching paths
    - [x] Manage push/pop for network calls
  - [x] Add execution strategies:
    - [x] Depth-first traversal
    - [x] Breadth-first traversal
    - [x] Resource-optimized traversal **[NEW]**
    - [x] Stream-optimized traversal **[NEW]**
  - [x] Implement error handling with retry
  - [x] Add execution hooks/callbacks
  - [x] Create comprehensive tests

- [x] **Network Executor** (`execution/network.py`) **[NEW]**
  - [x] Implement NetworkExecutor class
    - [x] Network execution with context isolation
    - [x] Hierarchical network push/pop operations
    - [x] Data passing between networks
    - [x] Resource management across networks
    - [x] Parallel network execution
  - [x] Add network validation
  - [x] Implement network statistics
  - [x] Create unit tests

- [x] **Batch Executor** (`execution/batch.py`) **[NEW]**
  - [x] Implement BatchExecutor class
    - [x] Parallel record processing
    - [x] Resource pooling
    - [x] Progress tracking
    - [x] Error aggregation
  - [x] Add batching strategies
  - [x] Implement result collection
  - [x] Create performance benchmarks

- [x] **Stream Executor** (`execution/stream.py`) **[NEW]**
  - [x] Implement StreamExecutor class
    - [x] Chunk-based processing
    - [x] Pipeline coordination
    - [x] Memory management
    - [x] Backpressure handling
  - [x] Add stream transformations
  - [x] Implement progress reporting
  - [x] Create performance tests

**Tests Created:** 38 tests in `test_execution.py`

### Phase 5: Configuration and Building (Week 5-6) ✅ **COMPLETED**
Simplified configuration based on use case patterns.

**Status**: ✅ Completed on 2025-09-08 with 25 tests passing

#### Checklist:
- [x] **Configuration Schema** (`config/schema.py`)
  - [x] Define configuration schema using pydantic
    - [x] Data mode configuration **[NEW]**
    - [x] Transaction configuration **[NEW]**
    - [x] Resource definitions **[NEW]**
    - [x] Streaming configuration **[NEW]**
    - [x] FSM definition
    - [x] Network definition
    - [x] State definition
    - [x] Arc definition
  - [x] Add use case templates **[NEW]**
  - [x] Create validation rules
  - [x] Generate JSON Schema
  - [x] Create unit tests

- [x] **Configuration Loader** (`config/loader.py`)
  - [x] Implement ConfigLoader class
    - [x] Load from file
    - [x] Load from dictionary
    - [x] Environment variable expansion
    - [x] Resource reference resolution **[NEW]**
  - [x] Add template support **[NEW]**
  - [x] Support for includes/imports
  - [x] Create unit tests

- [x] **FSM Builder** (`config/builder.py`)
  - [x] Implement FSMBuilder class
    - [x] Build FSM from configuration
    - [x] Resource registration **[NEW]**
    - [x] Mode configuration **[NEW]**
    - [x] Function resolution
    - [x] Validate completeness
  - [x] Add builder patterns for common use cases **[NEW]**
  - [x] Support for custom function registration
  - [x] Create integration tests

- [x] **Built-in Functions Library** (`functions/library/`)
  - [x] Common Validators (`validators.py`)
    - [x] RequiredFieldsValidator
    - [x] SchemaValidator
    - [x] RangeValidator
    - [x] PatternValidator
    - [x] TypeValidator
    - [x] LengthValidator
    - [x] UniqueValidator
    - [x] DependencyValidator
  - [x] Common Transformers (`transformers.py`)
    - [x] FieldMapper
    - [x] ValueNormalizer
    - [x] TypeConverter
    - [x] DataEnricher
    - [x] FieldFilter
    - [x] ValueReplacer
    - [x] ArrayFlattener
    - [x] DataSplitter
  - [x] Database Functions (`database.py`) **[NEW]**
    - [x] DatabaseFetch
    - [x] DatabaseUpsert
    - [x] BatchCommit
    - [x] DatabaseQuery
  - [x] Stream Functions (`streaming.py`) **[NEW]**
    - [x] ChunkReader
    - [x] RecordParser
    - [x] FileAppender
    - [x] StreamAggregator
  - [x] LLM Functions (`llm.py`) **[NEW]**
    - [x] PromptBuilder
    - [x] LLMCaller
    - [x] ResponseValidator
    - [x] ConversationManager
  - [x] Create comprehensive tests

**Implementation Notes:**
- Integrated configuration system with existing FSM components (core FSM, ExecutionEngine, StateNetwork)
- Created wrapper FSM class that provides simplified API while maintaining full functionality
- Added missing get_data_handler() function to data_modes.py
- Fixed all component integration issues between builder, FSM, and ExecutionEngine
- All 25 configuration tests passing with proper validation and building

### Phase 6: History and Storage (Week 6-7) ✅ **COMPLETED**
Implement history tracking with mode-aware storage.

**Status**: ✅ Completed on 2025-09-08 with 20 tests passing

#### Checklist:
- [x] **History Tracking** (`execution/history.py`)
  - [x] Implement ExecutionHistory class
    - [x] Use dataknobs_structures.Tree
    - [x] Mode-aware storage **[NEW]**
    - [x] Resource usage tracking **[NEW]**
    - [x] Stream progress tracking **[NEW]**
    - [x] Track state instances
    - [x] Track arc executions
    - [x] Record timestamps and metrics
  - [x] Add methods for:
    - [x] Adding execution steps
    - [x] Querying paths
    - [x] Pruning strategies based on mode **[NEW]**
    - [x] Serializing history
    - [x] Generating reports
  - [x] Create visualization methods
  - [x] Create unit tests

- [x] **Storage Interface** (`storage/base.py`)
  - [x] Define IHistoryStorage interface
  - [x] Define storage configuration
  - [x] Add mode-specific storage strategies **[NEW]**
  - [x] Create storage factory
  - [x] Add unit tests

- [x] **Memory Storage** (`storage/memory.py`)
  - [x] Implement InMemoryStorage class
  - [x] Add size limits per mode **[NEW]**
  - [x] Implement LRU eviction
  - [x] Add compression for REFERENCE mode **[NEW]**
  - [x] Create unit tests

- [x] **Database Storage** (`storage/database.py`)
  - [x] Implement UnifiedDatabaseStorage class **[UNIFIED APPROACH]**
  - [x] Use dataknobs_data abstractions
  - [x] Add mode-specific schemas **[NEW]**
  - [x] Support for ALL dataknobs_data backends (Memory, SQLite, PostgreSQL, MongoDB, Elasticsearch, S3, File)
  - [x] Add indexing for queries
  - [x] Implement partitioning for streams **[NEW]**
  - [x] Create integration tests

- [x] **File Storage** (`storage/file.py`)
  - [x] Implement FileStorage class
  - [x] Support for JSON/YAML serialization
  - [x] Add streaming write support **[NEW]**
  - [x] Implement rotation policies
  - [x] Add compression support
  - [x] Create unit tests

**Implementation Notes:**
- Leveraged dataknobs_data's existing database backends instead of creating new storage implementations
- Created unified UnifiedDatabaseStorage that works with ANY dataknobs_data backend through AsyncDatabase interface
- Enhanced dataknobs_data's upsert method to accept just a Record, leveraging Record's built-in ID management
- Fixed API mismatches between expected and actual dataknobs_data interfaces (Query.filter, sort_by, Record.get_value)
- Memory and File storage are now thin wrappers around UnifiedDatabaseStorage for maximum code reuse
- Achieved high test coverage: 92% for history.py, 81% for database.py, 100% for memory.py and file.py

### Phase 7: API and Integration (Week 7-8) **[97% COMPLETE]**
Create simplified APIs and integration patterns.

**Status**: 97% Complete - 2025-09-09

**Remaining Tasks**:
- Fix integration pattern tests (test_patterns_integration.py)
- Fix CLI tool tests (test_cli_real.py)
- Complete API documentation
- Complete pattern guides

**Completed**:
- Simple and Advanced APIs implemented ✅
- All API tests passing (46/46 - Simple: 22, Advanced: 24) ✅
- API refactoring completed (10_API_TO_GENERAL_REFACTOR.md) ✅
  - Created ContextFactory for centralized context creation ✅
  - Created ResultFormatter for standardized result formatting ✅
  - Enhanced ExecutionContext with get_complete_path() ✅
  - Added ResourceManager factory methods ✅
  - Improved FSM core state resolution ✅
  - Zero code duplication achieved ✅
  - Test coverage: ContextFactory 83%, ResultFormatter 100% ✅
- Database ETL and File Processing patterns implemented ✅
- CLI tool implemented ✅
- Arc name filtering functionality added ✅
- Execution context fixes completed ✅
- Function registration fixes completed ✅
- Config handling enhancements completed ✅

#### Checklist:
- [x] **Simple API** (`api/simple.py`) **[NEW]**
  - [x] Implement simplified FSM class
    - [x] process() for single records
    - [x] process_batch() for collections
    - [x] process_stream() for large data
  - [x] Add convenience methods
  - [x] Create quick-start examples (process_file, validate_data, batch_process)
  - [x] All functionality tests passing ✅
  - [x] API refactoring completed ✅
  - [ ] Write API documentation

- [x] **Advanced API** (`api/advanced.py`) **[NEW]**
  - [x] Implement full-control interfaces
    - [x] Custom executors
    - [x] Resource management
    - [x] Transaction control
    - [x] Stream coordination
  - [x] Add monitoring hooks (ExecutionHook class)
  - [x] Create advanced examples (FSMDebugger, profiling, tracing)
  - [x] All functionality tests passing ✅
  - [x] API refactoring completed ✅
  - [ ] Write detailed documentation

- [x] **Integration Patterns** (`patterns/`) **[NEW]**
  - [x] Database ETL pattern
  - [x] File processing pattern
  - [ ] API orchestration pattern
  - [ ] LLM workflow pattern
  - [ ] Error recovery pattern
  - [x] Create pattern templates
  - [ ] Write pattern guides

- [x] **CLI Tool** (`cli/`) **[NEW]**
  - [x] FSM validation command
  - [x] Execution command
  - [x] History inspection
  - [x] Resource management
  - [x] Performance profiling
  - [ ] Create user documentation

**Implementation Notes:**
- Simple API provides high-level abstraction with SimpleFSM class for easy data processing
- Advanced API offers AdvancedFSM class with step-by-step execution, breakpoints, tracing, and profiling
- FSMDebugger class added for interactive debugging sessions
- Both APIs integrate with all existing FSM components (data modes, resources, streaming, storage)
- Convenience functions added for common operations (file processing, batch processing, validation)
- Database ETL pattern implemented with full ETL pipeline support (extract, transform, load)
  - Supports multiple ETL modes: FULL_REFRESH, INCREMENTAL, UPSERT, APPEND
  - Includes checkpoint/resume capability and error threshold monitoring
  - Pre-configured patterns for database sync, migration, and data warehouse loading
- File Processing pattern implemented with multi-format support
  - Auto-detects file formats (JSON, CSV, XML, Parquet, text)
  - Three processing modes: STREAM, BATCH, WHOLE
  - Built-in validation, filtering, transformation, and aggregation capabilities
  - Pre-configured patterns for CSV processing, JSON streaming, and log analysis
- CLI Tool implemented with comprehensive command-line interface
  - Five command groups: config, run, debug, history, pattern
  - Configuration management: create from templates, validate, visualize structure
  - Execution commands: single execution, batch processing, stream processing
  - Debug capabilities: breakpoints, tracing, profiling, interactive debugging
  - History management: list executions, show details, query by criteria
  - Pattern execution: ETL pipelines, file processing with built-in patterns
  - Rich terminal UI with tables, progress bars, syntax highlighting
  - Full integration with Simple and Advanced APIs
  - Registered as 'fsm' command via pyproject.toml entry point

**Recently Completed (2025-09-09):**
- Fixed all API test failures: 24/24 tests now passing ✅
- Resolved execution context state tracking issues (dual string name + StateInstance)
- Implemented arc name filtering functionality in both sync and async engines
- Fixed function registration system with inline function compilation support
- Enhanced config loader to handle legacy pre_test format transformations
- Identified API code duplication issues that need refactoring (documented in 10_API_TO_GENERAL_REFACTOR.md)
- Next step: API refactoring to eliminate code duplication before proceeding to Phase 8

### Phase 8: Testing and Documentation (Week 8-9) **[IN PROGRESS]**
Comprehensive testing with focus on modes and resources.

**Status**: 65% Complete - 2025-01-15

**Completed Today (2025-01-15):**
- ✅ Resolved all technical debt with data handling inconsistency
- ✅ Fixed OllamaProvider for Docker environments
- ✅ Enhanced LLM conversation example with real provider support
- ✅ Added environment-based configuration for all LLM providers

**Previous Completions (2025-01-12):**
- ✅ Consolidated FSM class hierarchy (merged Builder FSM wrapper into CoreFSM)
- ✅ Fixed stream mode execution with proper chunk handling
- ✅ Added properties and accessors to core classes for better encapsulation
- ✅ Created comprehensive stream mode tests
- ✅ Fixed all builder execution tests to use real FSM execution
- ✅ Created unit tests for AdvancedFSM operations
- ✅ Implemented Database ETL pipeline example

**Previous Completions (2025-09-12):**
- ✅ Fixed custom function registration in FSM
- ✅ Implemented proper error handling for state vs arc transforms
- ✅ Fixed test failures in test_network_arc.py
- ✅ Fixed deprecation warnings in test_simple_api_timeout.py
- ✅ Created and tested two comprehensive examples:
  - Data validation pipeline with custom functions
  - Large file processor with REFERENCE mode
- ✅ Updated design documentation with learnings

#### Checklist:
- [ ] **Unit Tests**
  - [ ] Test all data modes independently
  - [ ] Test resource lifecycle
  - [ ] Test streaming components
  - [ ] Test transaction strategies
  - [ ] Achieve >90% code coverage (currently at 17%)
  - [x] Test all error conditions
  - [x] Test edge cases

- [ ] **Integration Tests**
  - [ ] Database processing scenarios
  - [ ] File streaming scenarios
  - [ ] LLM workflow scenarios
  - [ ] Multi-resource coordination
  - [ ] Mode switching tests
  - [ ] Transaction rollback tests
  - [ ] End-to-end use cases

- [ ] **Performance Tests** **[NEW]**
  - [ ] Benchmark data modes
  - [ ] Measure resource overhead
  - [ ] Test streaming throughput
  - [ ] Profile memory usage
  - [ ] Load testing
  - [ ] Create performance reports

- [ ] **Example Implementations** (`examples/`) **[PARTIALLY COMPLETE]**
  - [x] Database ETL pipeline ✅
    - [x] COPY mode with batching ✅
    - [x] Transaction management ✅
    - [x] Error recovery ✅
    - [x] Async database operations ✅
    - [x] Custom cleaning/validation functions ✅
  - [x] Large file processor ✅
    - [x] REFERENCE mode streaming ✅
    - [x] Chunk processing ✅
    - [x] Progress tracking ✅
  - [x] Simple FSM Debugger ✅
    - [x] Working example with current AdvancedFSM features ✅
    - [x] Breakpoint demonstration ✅
    - [x] State inspection ✅
    - [x] Execution tracing ✅
    - [x] Performance profiling ✅
  - [x] Advanced FSM Debugger ✅
    - [x] Step-by-step execution with inspection ✅
    - [x] Interactive debugging session ✅
    - [x] Data change tracking ✅
    - [x] **Note**: Depends on AdvancedFSM improvements (see docs/active/16_ADVANCED_FSM_IMPROVEMENTS.md) ✅
  - [x] LLM conversation system ✅
    - [x] Multi-stage conversation flow with intent analysis ✅
    - [x] Template-based and LLM-based response generation ✅
    - [x] Context management and conversation history ✅
    - [x] Support for Ollama, OpenAI, Anthropic, HuggingFace ✅
    - [x] Graceful fallback to templates when LLM unavailable ✅
    - [x] Token usage tracking ✅
  - [x] Simple data validator ✅
    - [x] DIRECT mode for speed ✅
    - [x] Schema validation ✅
    - [x] Quick processing ✅
    - [x] Custom function registration ✅
    - [x] Multi-stage validation pipeline ✅
    - [x] Conditional routing based on results ✅
    - [x] Unit tests (12 tests passing) ✅
  - [ ] API Orchestration workflow **[TO IMPLEMENT]**
    - [ ] Sequential and parallel API calls
    - [ ] Rate limiting and circuit breakers
    - [ ] Request/response transformation
    - [ ] Error recovery and retry logic
    - [ ] Response aggregation
  - [ ] LLM Chain Processing **[TO IMPLEMENT]**
    - [ ] Sequential LLM chains (output feeds next input)
    - [ ] Chain-of-thought reasoning
    - [ ] RAG pattern with document retrieval
    - [ ] Prompt chaining with context preservation
    - [ ] Response validation and structured parsing

- [x] **Documentation** **[MOSTLY COMPLETE]**
  - [x] Quick Start Guide ✅
  - [x] Data Modes Guide ✅
  - [x] Resource Management Guide ✅
  - [x] Streaming Guide ✅
  - [ ] Configuration Reference
  - [x] API Reference (SimpleFSM and AdvancedFSM) ✅
  - [x] Pattern Catalog (ETL, File Processing, API, LLM, Error Recovery) ✅
  - [ ] Performance Tuning Guide
  - [ ] Migration Guide
  - [x] Custom Functions Design Document (15_CUSTOM_FUNCTIONS_DESIGN_FIX.md) ✅
  - [x] Consolidated FSM Design (14_CONSOLIDATED_FSM_DESIGN.md) ✅
  - [x] Advanced FSM Improvements Plan (16_ADVANCED_FSM_IMPROVEMENTS.md) ✅

**Next Tasks**:
1. Implement API Orchestration example (`examples/api_workflow.py`)
2. Implement LLM Chain Processing example (`examples/llm_chain.py`)
3. Complete remaining documentation (Configuration Reference, Performance Guide, Migration Guide)

### Phase 9: Optimization and Polish (Week 9-10) **[NEW PHASE]**
Performance optimization and production readiness.

#### Checklist:
- [ ] **Performance Optimizations**
  - [ ] Implement caching strategies
    - [ ] Function result caching
    - [ ] Resource connection caching
    - [ ] Validation caching
  - [ ] Optimize data mode switching
  - [ ] Improve streaming throughput
  - [ ] Reduce memory footprint
  - [ ] Add JIT compilation hooks

- [ ] **Production Features**
  - [ ] Monitoring and metrics
    - [ ] Prometheus integration
    - [ ] Custom metrics
    - [ ] Performance dashboards
  - [ ] Logging improvements
    - [ ] Structured logging
    - [ ] Log levels per component
    - [ ] Audit trail
  - [ ] Health checks
    - [ ] Resource health
    - [ ] System health
    - [ ] Readiness probes

- [ ] **Developer Experience**
  - [ ] Improved error messages
  - [ ] Debugging tools
  - [ ] Configuration validation
  - [ ] Type stub generation
  - [ ] IDE integrations

## Resource Requirements (Updated)

### Team Composition
- **Lead Developer**: Architecture, data modes, core implementation
- **Resource Developer**: Resource management system **[NEW]**
- **Streaming Developer**: Streaming implementation **[NEW]**
- **Backend Developer**: Storage and execution engine
- **Integration Developer**: DataKnobs package integration
- **Test Engineer**: Testing framework and test cases
- **Documentation Writer**: User guides and API docs

### Tools and Infrastructure
- Python 3.10+
- Development environment with all DataKnobs packages
- Test databases (PostgreSQL, MongoDB)
- Mock LLM services for testing
- Performance testing environment
- CI/CD pipeline with matrix testing
- Documentation build system

## Risk Mitigation (Updated)

### Technical Risks
1. **Mode Switching Complexity**
   - Mitigation: Clear mode boundaries, comprehensive testing
   
2. **Resource Leak Potential**
   - Mitigation: Context managers, automatic cleanup, health monitoring

3. **Streaming Memory Issues**
   - Mitigation: Bounded buffers, backpressure, monitoring

4. **Transaction Consistency**
   - Mitigation: Two-phase commit, rollback support, testing

### Schedule Risks
1. **Resource Integration Complexity**
   - Mitigation: Start with database, add others incrementally

2. **Streaming Implementation**
   - Mitigation: Use proven patterns, leverage dataknobs_data

3. **Performance Requirements**
   - Mitigation: Early benchmarking, iterative optimization

## Success Criteria (Updated)

### Functional
- [x] All three data modes working correctly
- [ ] Core resources implemented (Database, File, HTTP, LLM, Vector)
- [ ] Streaming support for large data
- [ ] Transaction management working
- [x] All use cases implementable

### Non-Functional
- [ ] COPY mode overhead <10% vs direct processing
- [ ] Streaming can handle files >10GB
- [ ] Resource pooling reduces connection overhead by >50%
- [ ] Memory usage bounded in all modes
- [ ] Code coverage >90%

### Deliverables
- [x] Source code in packages/fsm
- [x] Comprehensive test suite
- [ ] Documentation for all features
- [~] Example implementations (LLM example partially complete)
- [ ] Performance benchmarks
- [x] Pattern library

### Phase 8 Progress (Completed Items)
- [x] Created unified function management system (`functions/manager.py`)
  - Handles both sync and async functions consistently
  - Preserves async nature through proper wrapping
  - Provides interface adaptation for FSM requirements
  - Manages registration, resolution, and caching
- [x] Fixed async/sync function detection in AsyncExecutionEngine
- [x] Fixed ExecutionResult handling in base engine
- [x] Implemented LLM conversation example with:
  - Multi-stage conversation flow
  - Intent analysis and template-based responses
  - Context management
  - 27 passing tests

### Phase 8 Remaining Work
- [x] **Simplify FSM data handling** ✅ COMPLETED
  - Implemented hybrid solution that provides consistent data handling
  - Functions receive predictable data formats based on their signatures
  - No more need for defensive type checking in user functions
  - Inline lambdas work with `state.data` pattern as expected
  - Regular functions receive plain dict data
- [x] **Enhance LLM example** ✅ COMPLETED (2025-01-15)
  - Fixed OllamaProvider for Docker environments and better error handling
  - Support for actual LLM API calls via Ollama, OpenAI, Anthropic, HuggingFace
  - Environment-based configuration for all LLM providers
  - Auto-detection of Docker environment for proper Ollama URL
  - Token usage tracking and reporting
  - Graceful fallback to template-based responses
  - Model validation on initialization

## Implementation Priority (Based on Use Cases)

### Critical Path (Must Have)
1. Data modes (COPY as default)
2. Basic resource management (Database)
3. Core FSM execution
4. Configuration loading
5. Simple API

### High Priority (Should Have)
1. Streaming support
2. Transaction management
3. LLM and Vector resources
4. Batch processing
5. Error recovery

### Medium Priority (Nice to Have)
1. Advanced storage backends
2. Performance optimizations
3. CLI tool
4. Monitoring integration
5. Pattern library

### Low Priority (Future)
1. Distributed execution
2. ML integration
3. Visual designer
4. Advanced caching

## Technical Debt Identified

### Data Handling Inconsistency ✅ RESOLVED (2025-01-15)
- **Issue**: Functions receive data in different formats (dict vs SimpleNamespace vs StateNamespace)
- **Impact**: User functions need defensive type checking
- **Root Cause**: BaseEngine.prepare_state_transform wraps data in SimpleNamespace (line 174)
- **Specific Problems Found**:
  - Some functions expect dict subscript access (`data["key"]`) while others expect attribute access (`data.key`)
  - The FSM is inconsistent about when it wraps and unwraps data
  - Transform functions receive `state_obj` which is a SimpleNamespace/StateNamespace wrapper
  - If transforms return the wrapper instead of raw data, that wrapper gets stored in context.data
  - ResultFormatter returns context.data directly, so wrapped objects leak into results
  - Tests like `test_database_etl_example_simple.py` have lambdas expecting `state.data["input"]` but fail with "SimpleNamespace object is not subscriptable"

- **Solution Implemented (Hybrid Approach)**:
  1. Created `FSMData` wrapper class that supports both dict-style and attribute access
  2. Created `StateDataWrapper` for inline lambdas expecting `state.data` pattern
  3. Added `ensure_dict()` utility to convert any data format to plain dict
  4. Modified execution engines to:
     - Always store plain dict in `context.data`
     - Wrap data appropriately when calling functions
     - Use `wrap_for_lambda()` for inline lambdas
  5. Fixed double-wrapping issue in builder where InterfaceWrapper was wrapped again
  6. Enhanced InterfaceWrapper to detect function signatures and adapt calling convention
  7. Fixed inline function registration with unique naming based on source/target states
  8. Added expression vs statement detection for inline code compilation
  9. Implemented proper tuple handling for test functions returning (bool, reason)
  10. Added JSON serialization support for FSMData and ExecutionResult objects
  11. Fixed arc transforms to properly unwrap ExecutionResult objects

- **Files Modified**:
  - Created: `src/dataknobs_fsm/core/data_wrapper.py`
  - Modified: `src/dataknobs_fsm/execution/base_engine.py`
  - Modified: `src/dataknobs_fsm/execution/engine.py`
  - Modified: `src/dataknobs_fsm/execution/async_engine.py`
  - Modified: `src/dataknobs_fsm/functions/manager.py`
  - Modified: `src/dataknobs_fsm/config/builder.py`
  - Modified: `src/dataknobs_fsm/core/arc.py`
  - Created: `src/dataknobs_fsm/utils/json_encoder.py`

- **Result**: Functions now receive data in a consistent format. Inline lambdas work with `state.data` pattern while regular functions receive plain dicts. All tests are now passing.

### Function Registration Complexity
- **Issue**: Multiple paths for function registration and resolution
- **Impact**: Confusion about when functions are wrapped vs raw
- **Solution**: Unified function manager (partially addressed)
- **Refactor Completed**:
  - Created `functions/manager.py` with unified `FunctionManager` class
  - Replaced `_function_registry` and `_builtin_functions` dicts with `_function_manager`
  - All functions now wrapped in `FunctionWrapper` which preserves async nature
  - Interface adaptation through `InterfaceWrapper` for ITransformFunction, IValidationFunction, etc.
- **Breaking Changes for Tests**:
  - Tests checking `builder._function_registry` must use `builder._function_manager.has_function(name)`
  - `builder._resolve_function()` now returns wrapped functions, not raw functions
  - To access original function from wrapper: `wrapper.func` or `wrapper.wrapper.func` for InterfaceWrapper
  - AsyncExecutionEngine now checks for `transform` method on InterfaceWrapper objects
- **Migration Guide for Broken Tests**:
  ```python
  # Old way
  assert "my_func" in builder._function_registry
  assert builder._function_registry["my_func"] == custom_func

  # New way
  assert builder._function_manager.has_function("my_func")
  wrapper = builder._function_manager.get_function("my_func")
  assert wrapper.func == custom_func
  ```

### Interface Wrapper Complexity
- **Issue**: InterfaceWrapper adds another layer of indirection
- **Impact**: Harder to debug and understand function calls
- **Solution**: Consider simpler interface adaptation approach
- **Current Implementation**:
  - InterfaceWrapper wraps FunctionWrapper to provide interface methods (transform, validate, test, etc.)
  - AsyncExecutionEngine had to be modified to detect `transform` method on wrapper
  - Functions wrapped multiple times: raw → FunctionWrapper → InterfaceWrapper
  - Each layer preserves async nature through `_is_async` property checks

### Quick Fixes for Test Migration
- For KeyError issues: Check if data is wrapped in namespace, use `getattr(data, 'field', default)` instead of `data['field']`
- For state transition issues: Debug transform return values, ensure they return dicts not namespace objects
- For validation issues: Ensure validators handle both dict and namespace inputs

## Development Guidelines (Updated)

### Code Style
- Follow PEP 8 guidelines
- Use type hints throughout
- Maintain comprehensive docstrings
- Keep functions focused and testable
- Use async/await for I/O operations

### Testing Strategy
- Test each data mode independently
- Mock external resources
- Use property-based testing for complex logic
- Benchmark performance regularly
- Test failure scenarios thoroughly

### Documentation
- Document data mode selection criteria
- Provide resource configuration examples
- Include streaming best practices
- Show transaction patterns
- Update examples for each use case

## Quick Start Implementation Order

For rapid prototype and validation:

1. **Week 1**: Core + COPY mode + Database resource
2. **Week 2**: Basic execution + Simple API
3. **Week 3**: Configuration + One complete use case
4. **Week 4**: Testing + Documentation

This allows demonstrating value quickly while building toward full implementation.

## Next Steps (Updated 2025-09-15)

### Immediate Priority
1. **Implement Missing Examples**
   - [ ] `examples/api_workflow.py` - API Orchestration with rate limiting and circuit breakers
   - [ ] `examples/llm_chain.py` - Multi-step LLM chain processing with RAG pattern

### Documentation Completion
2. **Finalize Documentation**
   - [ ] Configuration Reference - Complete YAML/JSON configuration guide
   - [ ] Performance Tuning Guide - Optimization strategies for each mode
   - [ ] Migration Guide - Upgrading from other workflow systems

### Testing Enhancement
3. **Increase Test Coverage**
   - Current: ~17% overall
   - Target: >90% coverage
   - Focus on integration tests for patterns
   - Add performance benchmarks

### Phase 9 Preparation
4. **Performance Optimization**
   - Profile current implementation
   - Identify bottlenecks
   - Implement caching strategies
   - Optimize streaming throughput
