# Palantir Event Capture - Architecture

## Overview

The `palantir-event-capture` library provides a comprehensive event capture system for data pipelines. It follows a layered architecture with clear separation of concerns.

## Architecture Layers

```
┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                         │
│              (User Functions/Pipelines)                      │
└───────────────────┬─────────────────────────────────────────┘
                    │
                    │ @capture_errors
                    │ @async_capture_errors
                    │ @retry_with_capture
                    │
┌───────────────────▼─────────────────────────────────────────┐
│                   Decorator Layer                            │
│          (decorators/capture.py)                             │
│  - Function wrapping                                         │
│  - Exception handling                                        │
│  - Context extraction                                        │
└───────────────────┬─────────────────────────────────────────┘
                    │
                    │ ErrorEvent
                    │
┌───────────────────▼─────────────────────────────────────────┐
│                  Service Layer                               │
│          (services/error_capture.py)                         │
│  - Event validation                                          │
│  - Metadata enrichment                                       │
│  - Transport routing                                         │
└───────────────────┬─────────────────────────────────────────┘
                    │
            ┌───────┴───────┐
            │               │
┌───────────▼────┐  ┌──────▼──────────┐
│  Storage Layer │  │ Transport Layer │
│   (Optional)   │  │ (Kafka/Custom)  │
│  - Local cache │  │ - Kafka         │
│  - Database    │  │ - Console       │
└────────────────┘  └─────────────────┘
```

## Core Components

### 1. Models Layer (`models/`)

**Purpose**: Define data structures and validation schemas

- `captured_models.py`: Pydantic models for events and configuration
  - `ErrorEvent`: Standardized event structure
  - `CaptureConfig`: Configuration for capture behavior
  - `PipelineComponent`: Enum of supported components (expanded list)
  - `Severity`: Event severity levels
  - `EventType`: Types of events (ERROR, WARNING, INFO, SUCCESS)

**Key Features**:

- Pydantic validation (V1 and V2 compatible)
- UUID tracking prevents duplicate events
- Timezone-aware timestamps (UTC)
- JSON serialization support

### 2. Decorators Layer (`decorators/`)

**Purpose**: Provide easy-to-use function wrappers

- `capture.py`: Core decorator implementations
  - `@capture_errors`: Sync function error capture
  - `@async_capture_errors`: Async function error capture
  - `@retry_with_capture`: Retry logic with capture
  - `@capture_method_errors`: Class-level decoration

**Responsibilities**:

- Wrap user functions transparently
- Extract execution context (function name, file, line number)
- Handle exceptions and capture stack traces
- Pass events to service layer

### 3. Services Layer (`services/`)

**Purpose**: Core business logic and coordination

- `error_capture.py`: Main capture service

  - Event creation and validation
  - Frame analysis for context extraction
  - Transport coordination
  - Singleton pattern for global access
- `transport.py`: Event transport abstraction

  - Base `Transport` interface
  - `KafkaTransport`: Kafka producer wrapper
  - `ConsoleTransport`: Console logging
  - `MultiTransport`: Combine multiple transports
  - `BackgroundTransport`: Async background worker for any transport
- `storage_service.py`: Optional event persistence

  - Local file storage
  - Database integration
  - Event query and retrieval

**Key Design Patterns**:

- Singleton: Global service instance
- Strategy: Pluggable transport backends
- Chain of Responsibility: Multiple transports

### 4. Configuration Layer (`config/`)

**Purpose**: Centralized configuration management

- `defaults.py`: Default configuration values
- `settings.py`: Configuration loading and validation
  - Environment variable support
  - Programmatic configuration
  - Config validation

**Configuration Sources** (priority order):

1. Programmatic configuration
2. Environment variables
3. Default values

### 5. Utilities Layer (`utils/`)

**Purpose**: Reusable helper functions

- `helpers.py`: Common utility functions

  - Frame analysis
  - Stack trace formatting
  - Type conversions
- `context_managers.py`: Context manager implementations

  - `capture_context`: Capture errors in code blocks
  - Automatic cleanup and error propagation
- `logging.py`: Logging setup and configuration

  - `structured_logging`: Loguru integration
  - `monitor_errors.py`: Simple Kafka event monitoring script

## Data Flow

### 1. Normal Execution Flow

```
User Function
     │
     │ Decorated with @capture_errors
     │
     ▼
Decorator intercepts execution
     │
     │ try: execute function
     │
     ▼
Function executes successfully
     │
     ▼
Return result to user
```

### 2. Error Capture Flow

```
User Function raises Exception
     │
     ▼
Decorator catches exception
     │
     ├─► Extract frame info (file, line, function)
     ├─► Extract stack trace
     └─► Create CaptureConfig
     │
     ▼
ErrorCaptureService.capture_exception()
     │
     ├─► Validate exception type
     ├─► Enrich metadata
     ├─► Create ErrorEvent (Pydantic)
     │
     ▼
For each registered transport:
     │
     ├─► ConsoleTransport.send()
     │    └─► Log to console
     │
     ├─► KafkaTransport.send()
     │    └─► Producer.send(topic, event)
     │
     └─► CustomTransport.send()
          └─► Your custom logic
     │
     ▼
Reraise exception (if configured)
```

### 3. Retry Flow

```
User Function (with @retry_with_capture)
     │
     ▼
Attempt 1: Exception raised
     │
     ├─► Capture event (LOW severity)
     ├─► Sleep delay (e.g., 1s)
     │
     ▼
Attempt 2: Exception raised
     │
     ├─► Capture event (LOW severity)
     ├─► Sleep delay * backoff (e.g., 2s)
     │
     ▼
Attempt 3 (final): Exception raised
     │
     ├─► Capture event (HIGH severity)
     ├─► Send to transport
     │
     ▼
Raise exception to caller
```

## Event Model Schema

```python
ErrorEvent:
  id: UUID                          # Unique event identifier
  event_type: EventType             # ERROR, WARNING, INFO, SUCCESS
  
  # Pipeline Context
  pipeline_name: Optional[str]      # Name of the pipeline
  engine_name: Optional[str]        # Processing engine
  
  # Component Info
  component: PipelineComponent      # KAFKA, POSTGRES, etc.
  component_name: Optional[str]     # Custom component name
  severity: Severity                # CRITICAL, HIGH, MEDIUM, LOW
  
  # Error Details
  message: str                      # Human-readable message
  timestamp: float                  # Unix timestamp
  timestamp_iso: str                # ISO 8601 format
  error_type: str                   # Exception class name
  stack_trace: Optional[str]        # Full stack trace
  
  # Execution Context
  function_name: Optional[str]      # Function where error occurred
  file_path: Optional[str]          # File path
  line_number: Optional[int]        # Line number
  metadata: Dict[str, Any]          # Custom metadata
  
  # Resolution Tracking
  resolved: bool                    # Resolution status
  resolved_at: Optional[float]      # Resolution timestamp
  resolved_by: Optional[str]        # Who resolved it
  resolution_notes: Optional[str]   # Resolution notes
```

## Transport Architecture

### Base Transport Interface

```python
class Transport(ABC):
    @abstractmethod
    def send(self, event: ErrorEvent) -> bool:
        """Send event to destination"""
        pass
  
    @abstractmethod
    def close(self) -> None:
        """Cleanup resources"""
        pass
```

### Kafka Transport

```python
KafkaTransport:
  - Lazy connection (on first send)
  - Automatic reconnection
  - Topic routing by severity
  - Batch sending support
  - Compression (optional)
```

### Multi Transport

Combine multiple transports:

```python
MultiTransport([
    ConsoleTransport(),
    BackgroundTransport(KafkaTransport()),
    CustomDatabaseTransport()
])
```

### Background Transport

Wraps any other transport to send events asynchronously using a worker thread and queue:

```python
transport = BackgroundTransport(
    base_transport=KafkaTransport(),
    max_queue_size=1000
)
```

## Configuration System

### Environment Variables

```bash
# Kafka
KAFKA_BOOTSTRAP_SERVERS_RCA=localhost:9092
KAFKA_EVENT_TOPIC_RCA=rca.events
KAFKA_ENABLED_RCA=true
KAFKA_COMPRESSION_RCA=gzip

# Behavior
LOG_LEVEL=INFO
LOG_TO_CONSOLE=true
SEND_TO_TRANSPORT=true
DEFAULT_SEVERITY=MEDIUM
ENABLE_STACK_TRACES=true
USE_BACKGROUND_TRANSPORT=false

# Retry
DEFAULT_MAX_RETRIES=3
DEFAULT_RETRY_DELAY=1.0
DEFAULT_RETRY_BACKOFF=2.0
```

### Configuration Priority

1. **Decorator arguments** (highest priority)
2. **Programmatic configuration** via `configure()`
3. **Environment variables**
4. **Default values** (lowest priority)

## Error Handling Philosophy

### 1. Fail-Safe Design

The library never causes application failures:

- Capture errors are logged, not raised
- Transport failures are silent
- Invalid configuration falls back to defaults

### 2. Zero Performance Impact

- Minimal overhead on success path
- Lazy initialization
- Async transport (optional)
- Batch sending (optional)

### 3. Rich Context

Capture maximum context without user effort:

- Automatic stack traces
- Function metadata
- Frame analysis
- Custom metadata support

## Extension Points

### 1. Custom Transport

```python
class MyTransport(Transport):
    def send(self, event: ErrorEvent) -> bool:
        # Your logic here
        return True
  
    def close(self) -> None:
        # Cleanup
        pass
```

### 2. Custom Storage

```python
class MyStorage(StorageService):
    def save(self, event: ErrorEvent) -> None:
        # Your storage logic
        pass
  
    def query(self, filters: dict) -> List[ErrorEvent]:
        # Your query logic
        pass
```

### 3. Event Enrichment

```python
def enrich_event(event: ErrorEvent) -> ErrorEvent:
    event.metadata['hostname'] = socket.gethostname()
    event.metadata['env'] = os.getenv('ENV')
    return event

service = get_capture_service()
service.add_enricher(enrich_event)
```

## Performance Considerations

### Memory

- Events are created on-demand
- UUID registry cleaned periodically
- Stack traces truncated (configurable)
- Metadata size limits (recommended)

### Latency

- Decorator overhead: ~0.1ms (no error)
- Event creation: ~0.5ms
- Kafka send (async): ~1-5ms
- Synchronous send: ~10-50ms (depends on network)

### Optimization Tips

1. Use async transports for production
2. Batch events when possible
3. Disable stack traces for low-severity events
4. Set appropriate metadata size limits
5. Use connection pooling for transports

## Testing Strategy

### Unit Tests

- Model validation
- Decorator behavior
- Service logic
- Transport implementations

### Integration Tests

- End-to-end capture flow
- Kafka integration
- Configuration loading
- Error scenarios

### Mock Support

All external dependencies can be mocked:

- Kafka producer
- File system
- Database connections

## Security Considerations

### PII Protection

- Avoid capturing sensitive data in metadata
- Sanitize stack traces (optional)
- Redact patterns (configurable)

### Authentication

- Kafka SASL support
- Custom transport auth
- Certificate-based auth

## Future Enhancements

1. **Metrics**: Prometheus/StatsD integration
2. **Tracing**: OpenTelemetry support
3. **Sampling**: Intelligent event sampling
4. **Aggregation**: Event deduplication
5. **Alerting**: Direct alerting integration
6. **Dashboard**: Real-time event dashboard
