# FSM Design Clarifications

Based on practical use cases, this document clarifies key design decisions for the DataKnobs FSM package.

## Data Handling Strategies

### Data Operation Modes

The FSM will support three distinct data operation modes:

#### 1. **Copy Mode** (Default)
- Data is copied from source before processing
- All operations performed on the copy
- Original data remains unchanged until explicit commit
- Enables safe concurrent processing
- Higher memory usage but safer

```python
class DataMode(Enum):
    COPY = "copy"           # Default - work on copies
    REFERENCE = "reference"  # Work on references with locking
    DIRECT = "direct"       # Direct manipulation (no concurrency)

class ExecutionConfig:
    data_mode: DataMode = DataMode.COPY
    auto_commit: bool = False
    batch_size: Optional[int] = None
```

#### 2. **Reference Mode**
- Works with references to data (database row IDs, file offsets)
- Implements optimistic locking for database records
- Lazy loading of actual data when needed
- Lower memory footprint
- Suitable for large datasets

#### 3. **Direct Mode**
- Operates directly on source data
- No concurrency support
- Lowest memory usage
- Fastest for single-threaded operations
- Requires exclusive access

### Transaction Management

```python
class TransactionStrategy(Enum):
    SINGLE = "single"        # Commit each record immediately
    BATCH = "batch"         # Batch commits at intervals
    MANUAL = "manual"       # Explicit commit control
    TWO_PHASE = "two_phase" # Two-phase commit for distributed

class TransactionConfig:
    strategy: TransactionStrategy
    batch_size: int = 100
    temp_storage: Optional[str] = None  # For staging
    isolation_level: str = "READ_COMMITTED"
```

## Resource and Service Integration

### Resource Manager Design

Resources are external systems or services that states can interact with:

```python
from abc import ABC, abstractmethod
from typing import Any, Dict, Optional

class IResource(ABC):
    """Base interface for external resources."""
    
    @abstractmethod
    async def acquire(self) -> Any:
        """Acquire resource connection/handle."""
        pass
    
    @abstractmethod
    async def release(self) -> None:
        """Release resource."""
        pass
    
    @abstractmethod
    def validate_health(self) -> bool:
        """Check if resource is healthy."""
        pass

class ResourceManager:
    """Manages lifecycle of external resources."""
    
    def __init__(self):
        self._resources: Dict[str, IResource] = {}
        self._pools: Dict[str, Any] = {}  # Connection pools
    
    def register_resource(self, name: str, resource: IResource):
        """Register a resource for use in FSM."""
        self._resources[name] = resource
    
    async def acquire(self, name: str) -> Any:
        """Acquire a resource by name."""
        if name not in self._resources:
            raise ResourceNotFoundError(f"Resource '{name}' not registered")
        return await self._resources[name].acquire()
```

### Built-in Resource Types

```python
class DatabaseResource(IResource):
    """Database connection resource."""
    def __init__(self, connection_string: str, pool_size: int = 10):
        self.connection_string = connection_string
        self.pool = None

class FileSystemResource(IResource):
    """File system access resource."""
    def __init__(self, base_path: str, mode: str = "r"):
        self.base_path = base_path
        self.mode = mode

class HTTPServiceResource(IResource):
    """HTTP service client resource."""
    def __init__(self, base_url: str, timeout: int = 30):
        self.base_url = base_url
        self.session = None

class LLMResource(IResource):
    """LLM service resource."""
    def __init__(self, model: str, api_key: str, **kwargs):
        self.model = model
        self.client = None

class VectorStoreResource(IResource):
    """Vector database resource."""
    def __init__(self, connection_string: str, dimension: int):
        self.connection = None
        self.dimension = dimension
```

## Use Case Implementations

### Use Case 1: Database Record Processing

```yaml
fsm:
  name: "DatabaseRecordProcessor"
  
  settings:
    data_mode: "copy"
    transaction:
      strategy: "batch"
      batch_size: 100
      temp_storage: "sqlite:///tmp/staging.db"
  
  resources:
    - name: "source_db"
      type: "database"
      config:
        connection_string: "${SOURCE_DB_URL}"
        
    - name: "target_db"
      type: "database"
      config:
        connection_string: "${TARGET_DB_URL}"
  
  networks:
    - name: "main"
      states:
        - name: "fetch"
          is_start: true
          resources: ["source_db"]
          transform:
            class: "dataknobs_fsm.transforms.DatabaseFetch"
            params:
              query: "SELECT * FROM records WHERE status = 'pending'"
              
        - name: "process"
          transform:
            class: "custom.transforms.RecordEnhancer"
            params:
              enhancement_type: "enrich"
              
        - name: "stage"
          resources: ["target_db"]
          transform:
            class: "dataknobs_fsm.transforms.StageForCommit"
            params:
              table: "staging_table"
              
        - name: "commit"
          is_end: true
          resources: ["target_db"]
          transform:
            class: "dataknobs_fsm.transforms.BatchCommit"
            params:
              mode: "upsert"
              target_table: "processed_records"
```

### Use Case 2: Large File Streaming

```yaml
fsm:
  name: "FileStreamProcessor"
  
  settings:
    data_mode: "reference"
    streaming:
      enabled: true
      chunk_size: 10000
      parallel_chunks: 4
  
  resources:
    - name: "input_file"
      type: "filesystem"
      config:
        path: "${INPUT_FILE_PATH}"
        mode: "r"
        
    - name: "output_file"
      type: "filesystem"
      config:
        path: "${OUTPUT_FILE_PATH}"
        mode: "a"
  
  networks:
    - name: "main"
      states:
        - name: "stream_chunk"
          is_start: true
          resources: ["input_file"]
          transform:
            class: "dataknobs_fsm.transforms.FileChunkReader"
            params:
              format: "jsonl"
              chunk_lines: 1000
              
        - name: "parse_records"
          transform:
            class: "dataknobs_fsm.transforms.RecordParser"
            params:
              schema: "${RECORD_SCHEMA}"
              
        - name: "process_batch"
          transform:
            class: "custom.transforms.BatchProcessor"
            params:
              parallel: true
              
        - name: "write_output"
          is_end: true
          resources: ["output_file"]
          transform:
            class: "dataknobs_fsm.transforms.FileAppender"
            params:
              format: "jsonl"
              buffer_size: 10000
```

### Use Case 3: LLM Conversation Processing

```yaml
fsm:
  name: "LLMConversationProcessor"
  
  settings:
    data_mode: "copy"
    error_handling:
      max_retries: 3
      retry_delay: 2
      fallback_states: true
  
  resources:
    - name: "conversation_db"
      type: "database"
      config:
        connection_string: "${CONVERSATION_DB_URL}"
        
    - name: "vector_store"
      type: "vector_store"
      config:
        connection_string: "${VECTOR_STORE_URL}"
        dimension: 1536
        
    - name: "llm_primary"
      type: "llm"
      config:
        model: "gpt-4"
        api_key: "${OPENAI_API_KEY}"
        temperature: 0.7
        
    - name: "llm_validator"
      type: "llm"
      config:
        model: "gpt-3.5-turbo"
        api_key: "${OPENAI_API_KEY}"
        temperature: 0
  
  networks:
    - name: "main"
      states:
        - name: "load_conversation"
          is_start: true
          resources: ["conversation_db"]
          transform:
            class: "ConversationLoader"
            params:
              include_history: true
              
        - name: "fetch_template_data"
          resources: ["conversation_db"]
          transform:
            class: "TemplateDataFetcher"
            params:
              template_id_field: "conversation.template_id"
              
        - name: "select_prompt_template"
          state_test:
            class: "TopicBasedTemplateSelector"
          arcs:
            - target: "technical_prompt"
              condition: "topic == 'technical'"
            - target: "creative_prompt"
              condition: "topic == 'creative'"
            - target: "general_prompt"
              condition: "default"
              
        - name: "technical_prompt"
          transform:
            class: "PromptBuilder"
            params:
              template_name: "technical_assistant"
          arcs:
            - target: "extract_concepts"
            
        - name: "creative_prompt"
          transform:
            class: "PromptBuilder"
            params:
              template_name: "creative_assistant"
          arcs:
            - target: "extract_concepts"
            
        - name: "general_prompt"
          transform:
            class: "PromptBuilder"
            params:
              template_name: "general_assistant"
          arcs:
            - target: "extract_concepts"
              
        - name: "extract_concepts"
          transform:
            class: "ConceptExtractor"
            params:
              method: "tfidf"
              max_concepts: 5
              
        - name: "fetch_context"
          resources: ["vector_store"]
          transform:
            class: "VectorSearch"
            params:
              top_k: 10
              include_metadata: true
              
        - name: "augment_prompt"
          transform:
            class: "PromptAugmenter"
            params:
              context_field: "retrieved_documents"
              max_tokens: 2000
              
        - name: "call_llm"
          resources: ["llm_primary"]
          transform:
            class: "LLMCaller"
            params:
              max_tokens: 1000
              stream: false
          error_handler:
            retry: true
            fallback_state: "llm_error_handler"
            
        - name: "validate_response"
          resources: ["llm_validator"]
          transform:
            class: "HallucinationChecker"
            params:
              check_facts: true
              check_consistency: true
          arcs:
            - target: "regenerate"
              condition: "has_hallucinations == true"
            - target: "store_results"
              condition: "has_hallucinations == false"
              
        - name: "regenerate"
          resources: ["llm_primary"]
          transform:
            class: "RegenerateWithConstraints"
            params:
              add_constraints: true
              max_attempts: 2
          arcs:
            - target: "validate_response"
              
        - name: "store_results"
          resources: ["conversation_db"]
          transform:
            class: "ConversationStorer"
            params:
              store_all_versions: true
              include_metadata: true
              
        - name: "prepare_response"
          is_end: true
          transform:
            class: "ResponseFormatter"
            params:
              include_sources: true
              
        - name: "llm_error_handler"
          is_end: true
          transform:
            class: "ErrorResponseBuilder"
            params:
              user_message: "I'm having trouble processing your request."
```

## Refined Design Decisions

### 1. Concurrency Model
- **Decision**: Support three modes (COPY, REFERENCE, DIRECT)
- **Default**: COPY mode for safety
- **Optimization**: DIRECT mode available when concurrency not needed

### 2. Data Mutation Strategy
- **Decision**: Configurable per FSM instance
- **Copy Mode**: Immutable with explicit commits
- **Reference Mode**: Lazy loading with locking
- **Direct Mode**: In-place mutations

### 3. Resource Management
- **Decision**: First-class resource manager with lifecycle control
- **Types**: Database, FileSystem, HTTP, LLM, VectorStore
- **Pooling**: Built-in connection pooling
- **Health Checks**: Automatic health monitoring

### 4. Error Recovery
- **Decision**: Multi-level error handling
- **Retry**: Configurable exponential backoff
- **Fallback**: Alternative state paths
- **Circuit Breaker**: For external services

### 5. History Storage
- **Decision**: Pluggable with automatic pruning
- **Default**: Memory with size limits
- **Options**: Database, File, Hybrid
- **Pruning**: Time-based and size-based

### 6. Push Arc Semantics
- **Decision**: Isolated execution contexts with merge strategies
- **Data Flow**: Configurable (copy, reference, shared)
- **Return**: Explicit merge or replace

### 7. Streaming Support
- **Decision**: Native streaming with backpressure
- **Chunking**: Automatic chunk management
- **Parallelism**: Configurable parallel chunks
- **Memory**: Bounded memory usage

### 8. Configuration Format
- **Decision**: Flexible YAML/JSON with schema validation
- **Validation**: Pydantic models
- **References**: Environment variables and includes
- **Extensions**: Custom resource types

## Implementation Priorities

Based on these use cases, the implementation priority should be:

1. **Core FSM with Copy Mode** - Basic functionality
2. **Resource Manager** - External service integration
3. **Database Resource** - Most common use case
4. **Streaming Support** - Large data handling
5. **LLM/Vector Resources** - Advanced AI use cases
6. **Reference/Direct Modes** - Performance optimizations
7. **Advanced Error Handling** - Production readiness

## API Examples

### Simple Database Processing
```python
from dataknobs_fsm import FSM, ExecutionConfig, DataMode

# Load FSM from config
fsm = FSM.from_yaml("database_processor.yaml")

# Configure execution
config = ExecutionConfig(
    data_mode=DataMode.COPY,
    transaction_strategy="batch",
    batch_size=100
)

# Process records
async def process_pending_records():
    async with fsm.create_executor(config) as executor:
        # Fetch pending records
        records = await executor.fetch_input(
            source="source_db",
            query="SELECT * FROM records WHERE status='pending'"
        )
        
        # Process through FSM
        results = await executor.process_batch(records)
        
        # Results automatically committed based on config
        return results
```

### Streaming File Processing
```python
from dataknobs_fsm import FSM, StreamConfig

fsm = FSM.from_yaml("file_processor.yaml")

stream_config = StreamConfig(
    chunk_size=10000,
    parallel_chunks=4,
    memory_limit="1GB"
)

async def process_large_file(input_path: str, output_path: str):
    async with fsm.create_stream_processor(stream_config) as processor:
        # Process file in chunks
        async for result_chunk in processor.process_file(
            input_path=input_path,
            output_path=output_path
        ):
            print(f"Processed {result_chunk.record_count} records")
```

### LLM Conversation
```python
from dataknobs_fsm import FSM, ResourceManager

# Setup resources
resources = ResourceManager()
resources.register("llm", LLMResource(
    model="gpt-4",
    api_key=os.getenv("OPENAI_API_KEY")
))
resources.register("vector_store", VectorStoreResource(
    connection_string=os.getenv("VECTOR_STORE_URL")
))

# Load FSM
fsm = FSM.from_yaml("llm_processor.yaml", resources=resources)

async def process_conversation(conversation_id: str, user_input: str):
    # Create input record
    input_data = {
        "conversation_id": conversation_id,
        "user_input": user_input,
        "timestamp": datetime.now()
    }
    
    # Process through FSM
    result = await fsm.process(input_data)
    
    # Result includes final response and all intermediate data
    return result.final_response
```

This refined design addresses all the practical use cases while maintaining simplicity and flexibility.