# DataKnobs FSM Package Design Document

## Executive Summary

The DataKnobs FSM (Finite State Machine) package provides a general-purpose, configurable state machine framework for processing data records through sequences of states and transitions. The FSM operates on data records, transforming them through state transitions while maintaining comprehensive execution history.

## Core Concepts

### 1. FSM Definition (Class-like)
An FSM definition acts as a template, similar to a class definition in OOP:
- Defines state networks containing states and arcs (transitions)
- Can be instantiated multiple times with different input data
- Represented as a directed graph of potential transitions

### 2. State Networks
A state network is a labeled collection of states connected by arcs:
- **Label**: String identifier for reference
- **States**: One or more states with at least one start and one end state
- **Arcs**: Transitions between states, including "push" arcs to other networks

### 3. States
Each state definition includes:
- **Name**: String identifier
- **Input Schema**: Expected data structure (using dataknobs_data.Field definitions)
- **Validity Test**: Function to validate input data meets requirements
- **State Test**: Boolean function validating if data meets state conditions
- **StateTransform Function**: Optional data transformation executed when entering the state
- **Start/End Designation**: Boolean flags for start/end states
- **End State Test**: For end states, returns one of:
  - CANNOT_END: State cannot terminate
  - CAN_END: State may continue or terminate
  - MUST_END: State must terminate
- **Outgoing Arcs**: Zero or more transitions to other states

### 4. Arcs (Transitions)
First-class objects defining transitions:
- **Target State**: Destination state reference
- **Pre-test Function**: Optional validation before transition
- **ArcTransform Function**: Optional data transformation during arc traversal
- **Push Arc Type**: Special arc that:
  - Executes another state network
  - Returns control to specified state upon completion

### 5. State Instantiation
Runtime instances of states:
- **Unique ID**: Format: `<state_name>-<counter>` (e.g., "validate-001")
- **Data**: Actual record or reference (database row, etc.)
- **Validity Status**: Result of validity test
- **Arc Execution History**: Track of attempted arcs and their results
- **Status**: One of:
  - SUCCESSFUL: Valid data, all arcs executed, reached valid end
  - FAILED: Invalid data or no valid path forward
  - PROCESSING: Status undetermined

### 6. Execution History
Maintains comprehensive tracking:
- Tree structure (not graph) representing execution paths
- Multiple successful and failing paths possible
- Configurable storage strategies for scalability

### 7. Transform Functions
The FSM supports two distinct types of data transformation functions:

#### StateTransforms
- **When**: Executed when entering a state (before state processing)
- **Purpose**: Prepare/normalize data for state operations
- **Signature**: `transform(state: State) -> Dict[str, Any]`
- **Configuration**: Via `state.functions.transform`
- **Example Use Cases**:
  - Data normalization and validation
  - Format conversion (JSON to internal format)
  - Adding computed fields
  - Data enrichment from external sources

#### ArcTransforms  
- **When**: Executed during arc traversal (transition between states)
- **Purpose**: Transform data as it flows between states
- **Signature**: `transform(data: Any, context: FunctionContext) -> Any`
- **Configuration**: Via `arc.transform`
- **Example Use Cases**:
  - Data filtering and projection
  - Conditional data modification
  - State-specific data preparation
  - Data routing and splitting

#### Execution Order
```
Input Data → StateTransform → State Processing → ArcTransform → Next State
```

#### Key Differences
| Aspect | StateTransform | ArcTransform |
|--------|----------------|--------------|
| Timing | State entry | Arc traversal |
| Input | State object | Raw data + context |
| Scope | State preparation | Inter-state data flow |
| Frequency | Once per state entry | Once per arc traversal |
| Configuration | `functions.transform` | `arc.transform` |

## Architecture

### Package Structure
```
packages/fsm/
├── src/
│   └── dataknobs_fsm/
│       ├── __init__.py
│       ├── core/
│       │   ├── __init__.py
│       │   ├── state.py          # State definition and instantiation
│       │   ├── arc.py            # Arc definition and execution
│       │   ├── network.py        # State network management
│       │   └── fsm.py            # Main FSM class
│       ├── execution/
│       │   ├── __init__.py
│       │   ├── engine.py         # Execution engine
│       │   ├── history.py        # History tracking
│       │   └── context.py        # Execution context
│       ├── functions/
│       │   ├── __init__.py
│       │   ├── base.py           # Base function interfaces
│       │   ├── validators.py     # Validation functions
│       │   └── transformers.py   # Transform functions
│       ├── config/
│       │   ├── __init__.py
│       │   ├── schema.py         # Configuration schema
│       │   ├── loader.py         # Config loading
│       │   └── builder.py        # FSM builder from config
│       ├── storage/
│       │   ├── __init__.py
│       │   ├── base.py           # Storage interface
│       │   ├── memory.py         # In-memory storage
│       │   ├── database.py       # Database storage
│       │   └── file.py           # File-based storage
│       └── streaming/
│           ├── __init__.py
│           ├── processor.py      # Stream processing
│           └── batch.py          # Batch optimizations
├── tests/
├── examples/
└── pyproject.toml
```

## Key Design Decisions

### 1. Data Handling
- **Immutable by default**: Transform functions create new data records
- **Reference support**: Can work with database references instead of full records
- **Schema validation**: Uses dataknobs_data.Field for type safety

### 2. Concurrency
- **Thread-safe FSM definitions**: Definitions are immutable after creation
- **Separate execution contexts**: Each execution maintains isolated state
- **Async support**: Optional async/await for I/O-bound operations

### 3. History Management
- **Tree structure**: Uses dataknobs_structures.Tree for execution paths
- **Pluggable storage**: Different backends via strategy pattern
- **Configurable depth**: Limits for memory management

### 4. Error Handling
- **Graceful degradation**: Failed paths don't stop other branches
- **Configurable retry**: Via decorator pattern on functions
- **Error states**: Optional error handling states in network

### 5. Configuration
- **YAML/JSON support**: Via dataknobs_config
- **Factory pattern**: For function instantiation
- **Reference resolution**: Support for referencing external modules

## Integration with DataKnobs Packages

### dataknobs_config
- FSM definitions loadable from YAML/JSON
- Support for variable substitution and references
- Factory pattern for function instantiation

### dataknobs_data
- Record abstraction for state data
- Field definitions for schema validation
- Database backend support for references
- Stream processing integration

### dataknobs_structures
- Tree structure for execution history
- RecordStore for state data caching
- Conditional dictionaries for dynamic configuration

### dataknobs_utils
- General utilities for logging, timing, etc.

### dataknobs_xization
- Text normalization for string data processing
- Tokenization support for NLP applications

## API Design

### Core Interfaces

```python
from abc import ABC, abstractmethod
from typing import Any, Dict, Optional, List, Callable
from enum import Enum

class EndStateStatus(Enum):
    CANNOT_END = "cannot_end"
    CAN_END = "can_end"
    MUST_END = "must_end"

class StateStatus(Enum):
    SUCCESSFUL = "successful"
    FAILED = "failed"
    PROCESSING = "processing"

class IValidationFunction(ABC):
    @abstractmethod
    def validate(self, data: Any) -> bool:
        """Validate data against requirements."""
        pass

class ITransformFunction(ABC):
    @abstractmethod
    def transform(self, data: Any, context: FunctionContext = None) -> Any:
        """Transform data. Used for both StateTransforms and ArcTransforms.
        
        For StateTransforms: data is a State object
        For ArcTransforms: data is the raw data being transformed
        """
        pass

class IStateTestFunction(ABC):
    @abstractmethod
    def test(self, data: Any) -> bool:
        """Test if data meets state conditions."""
        pass

class IEndStateTestFunction(ABC):
    @abstractmethod
    def test(self, data: Any) -> EndStateStatus:
        """Determine end state status."""
        pass
```

### Configuration Schema Example

```yaml
fsm:
  name: "DataProcessingFSM"
  networks:
    - name: "main"
      states:
        - name: "start"
          is_start: true
          schema:
            fields:
              - name: "input_data"
                type: "string"
                required: true
          validity_test:
            class: "dataknobs_fsm.functions.validators.RequiredFieldsValidator"
            params:
              required_fields: ["input_data"]
          arcs:
            - target: "validate"
              pre_test:
                class: "dataknobs_fsm.functions.validators.NonEmptyValidator"
              transform:
                class: "dataknobs_fsm.functions.transformers.NormalizeTransformer"
        
        - name: "validate"
          state_test:
            class: "custom.validators.DataValidator"
          arcs:
            - target: "process"
              condition: "valid"
            - target: "error"
              condition: "invalid"
        
        - name: "process"
          arcs:
            - type: "push"
              network: "enrichment"
              return_to: "complete"
        
        - name: "complete"
          is_end: true
          end_state_test:
            class: "dataknobs_fsm.functions.validators.CompletionValidator"
        
        - name: "error"
          is_end: true
          end_state_test:
            always: "must_end"
    
    - name: "enrichment"
      states:
        - name: "enrich_start"
          is_start: true
          arcs:
            - target: "add_metadata"
        
        - name: "add_metadata"
          transform:
            class: "custom.transformers.MetadataEnricher"
          arcs:
            - target: "enrich_complete"
        
        - name: "enrich_complete"
          is_end: true
```

### Transform Function Configuration Examples

```yaml
networks:
  - name: "data_processing"
    states:
      - name: "input"
        is_start: true
        arcs:
          - target: "normalize"
      
      - name: "normalize"
        # StateTransform: prepares data when entering this state
        functions:
          transform: 
            type: inline
            code: "lambda state: {'normalized': state.data['raw'].upper()}"
        arcs:
          - target: "validate"
            # ArcTransform: transforms data during transition to validate state
            transform:
              type: inline
              code: "lambda data, ctx: {'input': data['normalized'], 'timestamp': ctx.timestamp}"
      
      - name: "validate"
        # StateTransform: adds validation metadata
        functions:
          transform:
            class: "custom.transformers.ValidationPreparer"
        arcs:
          - target: "output"
            # ArcTransform: formats final output
            transform:
              class: "custom.transformers.OutputFormatter"
      
      - name: "output"
        is_end: true
```

## Implementation Considerations

### Performance Optimizations
1. **Lazy evaluation**: Defer computation until needed
2. **Caching**: Cache validated states and transformations
3. **Batch processing**: Process multiple records in parallel
4. **Memory management**: Configurable history pruning

### Extensibility Points
1. **Custom functions**: Any class implementing required interfaces
2. **Storage backends**: Pluggable history storage
3. **Network execution strategies**: Different traversal algorithms
4. **Event hooks**: For monitoring and debugging

### Testing Strategy
1. **Unit tests**: Individual components
2. **Integration tests**: With DataKnobs packages
3. **Performance tests**: Streaming and batch processing
4. **Example FSMs**: Common use case implementations

## Use Cases

### 1. Data Validation Pipeline
- Start → Schema Validation → Business Rules → Output/Error

### 2. ETL Processing
- Extract → Transform → Validate → Load → Verify

### 3. Document Processing
- Input → Parse → Extract → Enrich → Store

### 4. Workflow Automation
- Request → Approve → Execute → Notify → Complete

## Future Enhancements

1. **Visual FSM Designer**: Web-based graph editor
2. **Distributed Execution**: Multi-node processing
3. **Machine Learning Integration**: Adaptive transitions
4. **Real-time Monitoring**: Dashboard for execution tracking
5. **Version Control**: FSM definition versioning

## Dependencies

### Required DataKnobs Packages
- dataknobs-config >= 0.1.0
- dataknobs-data >= 0.2.0
- dataknobs-structures >= 1.0.0
- dataknobs-utils >= 0.1.0

### External Dependencies
- pydantic >= 2.0 (for validation)
- networkx >= 3.0 (for graph operations)
- typing-extensions >= 4.0

## Glossary

- **FSM**: Finite State Machine
- **State Network**: Collection of states and transitions
- **Arc**: Transition between states
- **Push Arc**: Special arc that executes another network
- **State Instance**: Runtime instance of a state with data
- **Execution Context**: Environment for FSM execution
- **History Tree**: Tree structure tracking execution paths