# Checkpoint and Recovery Patterns

Best practices for implementing checkpoint-based recovery in distributed workflow engines.

## Overview

Checkpointing enables crash recovery by periodically saving execution state, allowing workflows to resume from the last successful point after failures.

## Checkpoint Types

### 1. Coordinated Checkpointing

All stages synchronize to save a consistent global state.

**Pros:** Prevents domino effect, guarantees system-wide consistency
**Cons:** Synchronization overhead

```python
async def coordinated_checkpoint(pipeline: Pipeline, context: PipelineContext):
    """Create a consistent global checkpoint."""
    # Pause all running stages
    await pause_all_stages()
    
    # Wait for in-flight operations to complete
    await drain_queues()
    
    # Save consistent state
    checkpoint = Checkpoint(
        checkpoint_type=CheckpointType.COORDINATED,
        data_snapshot=context.data.copy(),
        completed_stages=context.completed_stage_ids,
        timestamp=datetime.now(UTC),
    )
    await checkpoint_store.save(checkpoint)
    
    # Resume execution
    await resume_all_stages()
```

### 2. Uncoordinated Checkpointing

Each stage saves its own state independently.

**Pros:** Minimal synchronization overhead
**Cons:** May lead to domino effect (cascading rollbacks)

```python
async def uncoordinated_checkpoint(stage_id: str, result: dict):
    """Save stage checkpoint independently."""
    checkpoint = Checkpoint(
        checkpoint_type=CheckpointType.STAGE_COMPLETE,
        stage_id=stage_id,
        data_snapshot=result,
        timestamp=datetime.now(UTC),
    )
    await checkpoint_store.save(checkpoint)
```

### 3. Communication-Induced Checkpointing

Checkpoints triggered based on message flow patterns.

```python
class MessageLogger:
    """Log messages for replay during recovery."""
    
    async def log_send(self, from_stage: str, to_stage: str, message: dict):
        await self.store.append({
            "type": "send",
            "from": from_stage,
            "to": to_stage,
            "message": message,
            "timestamp": time.time(),
        })
    
    async def replay_messages(self, after_checkpoint: Checkpoint):
        """Replay messages sent after the checkpoint."""
        messages = await self.store.get_after(after_checkpoint.timestamp)
        for msg in messages:
            await self.deliver(msg["to"], msg["message"])
```

## Best Practices

### 1. Checkpoint After Each Stage

```python
async def _run_stage(self, stage: PipelineStage, context: PipelineContext):
    try:
        result = await self._execute_handler(stage, context)
        context.update(result)
        context.record_stage_complete(stage.id, result)
        
        # Checkpoint after successful completion
        await self._save_checkpoint(
            context, 
            CheckpointType.STAGE_COMPLETE
        )
        
    except Exception as e:
        context.record_stage_failed(stage.id, str(e))
        
        # Checkpoint failure state too
        await self._save_checkpoint(
            context,
            CheckpointType.STAGE_FAILED
        )
        raise
```

### 2. Serialize Only Essential State

```python
class PipelineContext:
    def create_checkpoint(self) -> Checkpoint:
        """Create minimal checkpoint for recovery."""
        return Checkpoint(
            execution_id=self.execution_id,
            pipeline_id=self.pipeline_id,
            
            # Essential: What stages completed
            completed_stages=self.completed_stage_ids,
            failed_stages=self.failed_stage_ids,
            
            # Essential: Data for downstream stages
            data_snapshot=self._serialize_essential_data(),
            
            # Skip: Transient state, caches, connections
        )
    
    def _serialize_essential_data(self) -> dict:
        """Filter data to only recoverable items."""
        return {
            k: v for k, v in self.data.items()
            if self._is_serializable(v) and not k.startswith("_")
        }
```

### 3. Implement Recovery Line Detection

Find the consistent point to resume from:

```python
async def find_recovery_line(execution_id: str) -> Checkpoint | None:
    """Find the last consistent checkpoint to resume from."""
    checkpoints = await checkpoint_store.list_checkpoints(execution_id)
    
    # Sort by timestamp descending
    checkpoints.sort(key=lambda c: c.created_at, reverse=True)
    
    for checkpoint in checkpoints:
        if checkpoint.checkpoint_type == CheckpointType.STAGE_COMPLETE:
            # Verify checkpoint integrity
            if await self._verify_checkpoint(checkpoint):
                return checkpoint
    
    # No valid checkpoint found - must restart
    return None

async def _verify_checkpoint(self, checkpoint: Checkpoint) -> bool:
    """Verify checkpoint data is not corrupted."""
    try:
        # Check data can be deserialized
        json.loads(checkpoint.to_json())
        
        # Verify completed stages are valid
        for stage_id in checkpoint.completed_stages:
            if stage_id not in self._known_stage_ids:
                return False
        
        return True
    except Exception:
        return False
```

### 4. Handle Partial Failures

```python
async def resume_from_checkpoint(
    self,
    pipeline: Pipeline,
    checkpoint: Checkpoint,
) -> PipelineContext:
    """Resume execution from checkpoint."""
    context = PipelineContext(
        pipeline_id=pipeline.id,
        pipeline_name=pipeline.name,
        execution_id=checkpoint.execution_id,
    )
    
    # Restore data state
    context.data = checkpoint.data_snapshot.copy()
    
    # Mark completed stages
    for stage_id in checkpoint.completed_stages:
        stage = pipeline.get_stage(stage_id)
        if stage:
            context.stage_results[stage_id] = StageResult(
                stage_id=stage_id,
                stage_name=stage.name,
                status=StageStatus.COMPLETED,
            )
    
    # Mark failed stages
    for stage_id in checkpoint.failed_stages:
        stage = pipeline.get_stage(stage_id)
        if stage:
            context.stage_results[stage_id] = StageResult(
                stage_id=stage_id,
                stage_name=stage.name if stage else stage_id,
                status=StageStatus.FAILED,
            )
    
    return context
```

### 5. Checkpoint Storage Options

**In-Memory (Testing Only):**
```python
class InMemoryCheckpointStore:
    def __init__(self):
        self._checkpoints: dict[str, list[Checkpoint]] = {}
```

**Redis (Low Latency):**
```python
class RedisCheckpointStore:
    async def save(self, checkpoint: Checkpoint) -> None:
        key = f"flow:cp:{checkpoint.execution_id}"
        await self.redis.rpush(key, checkpoint.to_json())
        await self.redis.expire(key, 86400 * 7)  # 7 day TTL
    
    async def load(self, execution_id: str) -> Checkpoint | None:
        key = f"flow:cp:{execution_id}"
        data = await self.redis.lindex(key, -1)  # Latest
        return Checkpoint.from_json(data) if data else None
```

**PostgreSQL (Durability):**
```python
class PostgresCheckpointStore:
    async def save(self, checkpoint: Checkpoint) -> None:
        await self.pool.execute("""
            INSERT INTO checkpoints 
            (id, execution_id, pipeline_id, checkpoint_type, data, created_at)
            VALUES ($1, $2, $3, $4, $5, $6)
        """, checkpoint.id, checkpoint.execution_id, checkpoint.pipeline_id,
             checkpoint.checkpoint_type.value, checkpoint.to_json(),
             checkpoint.created_at)
```

**S3/Object Storage (Large State):**
```python
class S3CheckpointStore:
    async def save(self, checkpoint: Checkpoint) -> None:
        key = f"checkpoints/{checkpoint.execution_id}/{checkpoint.id}.json"
        await self.s3.put_object(
            Bucket=self.bucket,
            Key=key,
            Body=checkpoint.to_json().encode(),
            ContentType="application/json",
        )
```

### 6. Garbage Collection

Clean up old checkpoints to prevent storage bloat:

```python
async def cleanup_old_checkpoints(
    store: CheckpointStore,
    retention_days: int = 7,
) -> int:
    """Delete checkpoints older than retention period."""
    cutoff = datetime.now(UTC) - timedelta(days=retention_days)
    
    # Keep at least the latest checkpoint per execution
    executions = await store.list_executions()
    deleted_count = 0
    
    for execution_id in executions:
        checkpoints = await store.list_checkpoints(execution_id)
        
        # Sort by time, keep most recent
        checkpoints.sort(key=lambda c: c.created_at, reverse=True)
        
        for checkpoint in checkpoints[1:]:  # Skip latest
            if checkpoint.created_at < cutoff:
                await store.delete_checkpoint(checkpoint.id)
                deleted_count += 1
    
    return deleted_count
```

### 7. Compensation (Saga Pattern)

For workflows with side effects, implement compensation:

```python
class SagaStage(PipelineStage):
    compensate_handler: str | None = None

async def run_with_compensation(
    self,
    pipeline: Pipeline,
    context: PipelineContext,
):
    """Execute with compensation on failure."""
    completed_stages: list[str] = []
    
    try:
        for stage in pipeline.topological_order():
            await self._run_stage(stage, context)
            completed_stages.append(stage.id)
    
    except Exception as e:
        # Run compensation in reverse order
        for stage_id in reversed(completed_stages):
            stage = pipeline.get_stage(stage_id)
            if stage.compensate_handler:
                try:
                    handler = await self._get_handler(stage.compensate_handler)
                    await handler(context.data)
                except Exception as comp_error:
                    logger.error(f"Compensation failed for {stage_id}: {comp_error}")
        raise
```

## Recovery Scenarios

### Scenario 1: Process Crash

```python
# On startup, check for incomplete executions
async def recover_incomplete_executions():
    incomplete = await checkpoint_store.get_incomplete_executions()
    
    for execution_id in incomplete:
        checkpoint = await checkpoint_store.load(execution_id)
        if checkpoint:
            pipeline = await pipeline_store.get(checkpoint.pipeline_id)
            await executor.execute(pipeline, resume_from=execution_id)
```

### Scenario 2: Network Partition

```python
# Use distributed locking to prevent duplicate execution
async def safe_resume(execution_id: str):
    lock_key = f"flow:lock:{execution_id}"
    
    if not await distributed_lock.acquire(lock_key, ttl=300):
        raise ConcurrentExecutionError("Another instance is resuming")
    
    try:
        await executor.execute(pipeline, resume_from=execution_id)
    finally:
        await distributed_lock.release(lock_key)
```

### Scenario 3: Stage Timeout

```python
async def _run_stage_with_timeout(self, stage, context):
    try:
        result = await asyncio.wait_for(
            self._execute_handler(stage, context),
            timeout=stage.timeout,
        )
        return result
    except asyncio.TimeoutError:
        # Save timeout checkpoint before re-raising
        await self._save_checkpoint(
            context,
            CheckpointType.STAGE_FAILED,
        )
        raise StageTimeoutError(f"Stage {stage.id} timed out")
```

## References

- [Checkpoint-Based Rollback Recovery](https://www.chriswirz.com/distributed-systems/rollback-recovery)
- [Recovery in Distributed Systems](https://www.geeksforgeeks.org/operating-systems/recovery-in-distributed-systems/)
- [Building Resilient Systems with Temporal](https://aws.amazon.com/blogs/apn/building-resilient-distributed-systems-with-temporal-and-aws/)
- [Workflow Orchestration Patterns](https://www.ijsr.net/getabstract.php?paperid=SR25629180438)
