# API Reference

Python API reference for AtomIngest programmatic usage.

## Table of Contents
- [Core Classes](#core-classes)
- [Schema Loading](#schema-loading)
- [Normalization](#normalization)
- [Hooks](#hooks)
- [Security](#security)
- [Utilities](#utilities)

## Core Classes

### Ingester

Main class for running ingestion pipelines.

```python
from atomingest.core.ingester import Ingester
from atomsql import Database

# Create database connection
db = Database("sqlite:///data.db")

# Create ingester
ingester = Ingester(db, hook_registry=None)

# Run ingestion
stats = ingester.run(
    config="pipeline.yaml",
    source="data.csv"
)

print(f"Processed: {stats.processed}")
print(f"Failed: {stats.failed}")
print(f"Total: {stats.total_rows}")
```

**Constructor:**
```python
Ingester(db: Database, hook_registry: Optional[HookRegistry] = None)
```

**Parameters:**
- `db` (Database): AtomSQL database instance
- `hook_registry` (HookRegistry, optional): Custom hook registry

**Methods:**

#### run()
Execute ingestion pipeline.

```python
run(config: Union[Path, str], source: Union[Path, str]) -> IngestionStats
```

**Parameters:**
- `config`: Path to YAML configuration file
- `source`: Path to source data file

**Returns:**
- `IngestionStats`: Statistics about the ingestion run

**Example:**
```python
stats = ingester.run("pipeline.yaml", "data.csv")
```

### IngestionStats

Statistics from ingestion run.

```python
@dataclass
class IngestionStats:
    processed: int = 0      # Successfully processed records
    failed: int = 0         # Failed records
    total_rows: int = 0     # Total rows in source
```

**Usage:**
```python
stats = ingester.run(config, source)

print(f"Success rate: {stats.processed / stats.total_rows * 100:.2f}%")
print(f"Failed: {stats.failed}")
```

## Schema Loading

### SchemaLoader

Loads and parses YAML configuration.

```python
from atomingest.core.loader import SchemaLoader

loader = SchemaLoader()

# Load YAML
schema = loader.load_from_yaml("pipeline.yaml")

# Create model class
ModelClass = loader.create_model(schema)

# Register with database
db.register(ModelClass)

# Use model
instance = ModelClass(name="Alice", email="alice@example.com")
instance.save()
```

**Constructor:**
```python
SchemaLoader()
```

**Methods:**

#### load_from_yaml()
Load and parse YAML configuration.

```python
load_from_yaml(file_path: str) -> Dict[str, Any]
```

**Parameters:**
- `file_path`: Path to YAML file

**Returns:**
- Dictionary with:
  - `target_table` (str): Table name
  - `schema` (Dict): Parsed field definitions
  - `config` (Dict): Pipeline configuration
  - `normalization` (NormalizationConfig): Normalization config
  - `hooks` (Dict): Hook configurations

**Example:**
```python
schema = loader.load_from_yaml("pipeline.yaml")
print(schema["target_table"])  # "users"
print(schema["schema"])  # Field definitions
```

#### create_model()
Create dynamic model class from schema.

```python
create_model(schema: Dict[str, Any]) -> Type[Model]
```

**Parameters:**
- `schema`: Schema dictionary from `load_from_yaml()`

**Returns:**
- Dynamically generated Model class

**Example:**
```python
ModelClass = loader.create_model(schema)
instance = ModelClass(name="Alice")
```

## Normalization

### normalize_then_validate()

Execute normalization pipeline.

```python
from atomingest.normalization.pipeline import normalize_then_validate
from atomingest.normalization.core import NormalizationConfig

# Load config
config = NormalizationConfig.from_dict(yaml_config["normalization"])

# Normalize and validate
rows, report = normalize_then_validate(
    source="data.csv",
    config=config
)

# Process rows
for row in rows:
    print(row)

# Check report
print(f"Rows emitted: {report.rows_emitted}")
print(f"Warnings: {report.warnings}")
```

**Function:**
```python
normalize_then_validate(
    source: Union[str, Path, Iterator],
    config: NormalizationConfig,
    strict: bool = False
) -> Tuple[Iterator[Dict], NormalizationReport]
```

**Parameters:**
- `source`: CSV file path or row iterator
- `config`: Normalization configuration
- `strict`: Fail on warnings if True

**Returns:**
- Tuple of (row iterator, report)

### NormalizationConfig

Configuration for normalization pipeline.

```python
from atomingest.normalization.core import NormalizationConfig

config = NormalizationConfig.from_dict({
    "steps": [
        {
            "step": "TrimWhitespaceStep",
            "config": {"trim_quotes": True}
        },
        {
            "step": "NullTokenNormalizationStep",
            "config": {"null_tokens": ["-", "NA"]}
        }
    ]
})
```

### Custom Normalization Step

Create custom normalization step:

```python
from atomingest.normalization.steps import BaseNormalizerStep, register_step
from typing import Dict, Any, Optional

@register_step("MyCustomStep")
class MyCustomStep(BaseNormalizerStep):
    """Custom normalization step."""

    def process_row(self, row: Dict[str, Any]) -> Optional[Dict[str, Any]]:
        # Transform row
        row['custom_field'] = self.transform(row)
        return row

    def transform(self, row: Dict[str, Any]) -> Any:
        # Custom logic
        return "transformed"
```

**Usage:**
```yaml
normalization:
  steps:
    - step: MyCustomStep
      config:
        param: value
```

## Hooks

### Hook Base Class

Create custom hooks:

```python
from atomingest.pipeline.hooks import Hook, HookContext

class MyHook(Hook):
    """Custom hook description."""

    def __init__(self, name: str = None, config: Dict[str, Any] = None):
        super().__init__(name, config)
        # Initialize from config
        self.param = config.get('param', 'default')

    def execute(self, context: HookContext) -> HookContext:
        """
        Execute hook logic.

        Args:
            context: Hook execution context

        Returns:
            Modified context
        """
        # Access row data
        row_data = context.row_data

        # Perform custom logic
        row_data['new_field'] = self.calculate(row_data)

        # Update context
        context.row_data = row_data

        return context
```

### HookContext

Context object passed to hooks.

```python
@dataclass
class HookContext:
    row_data: Dict[str, Any]           # Current row
    instance: Optional[Model] = None   # Model instance (after save)
    metadata: Dict[str, Any] = None    # Pipeline metadata
    error: Optional[Exception] = None  # Error (in ON_ERROR hooks)
    processed_count: int = 0           # Records processed
    failed_count: int = 0              # Records failed
    total_count: int = 0               # Total records
    skip_row: bool = False             # Skip this row
    abort: bool = False                # Abort pipeline
```

**Usage:**
```python
def execute(self, context: HookContext) -> HookContext:
    # Access data
    name = context.row_data.get('name')

    # Access metadata
    run_id = context.metadata.get('run_id')

    # Skip row conditionally
    if should_skip(context.row_data):
        context.skip_row = True

    return context
```

### HookRegistry

Manage hook execution.

```python
from atomingest.pipeline.hooks import HookRegistry, HookType

# Create registry
registry = HookRegistry()

# Register hooks
registry.register(HookType.PRE_SAVE, my_hook, order=10)
registry.register(HookType.POST_SAVE, notification_hook, order=20)

# Execute hooks
context = HookContext(row_data={"name": "Alice"}, metadata={})
context = registry.execute_hooks(HookType.PRE_SAVE, context)
```

**Methods:**

#### register()
Register a hook.

```python
register(hook_type: HookType, hook: Hook, order: int = 0)
```

#### execute_hooks()
Execute all hooks of a type.

```python
execute_hooks(hook_type: HookType, context: HookContext) -> HookContext
```

## Security

### KeyManager

Manage encryption keys.

```python
from atomingest.security.kms import KeyManager

# Create new key
key = KeyManager.create_key("user_pii")

# Get existing key (or create if not exists)
key = KeyManager.get_key("user_pii")

# Shred key (crypto-shredding)
success = KeyManager.shred_key("user_pii")
```

**Class Methods:**

#### create_key()
Generate and store new data key.

```python
create_key(key_identifier: str) -> bytes
```

**Parameters:**
- `key_identifier`: Unique identifier for key

**Returns:**
- Raw encryption key (bytes)

#### get_key()
Retrieve encryption key.

```python
get_key(key_identifier: str) -> bytes
```

**Parameters:**
- `key_identifier`: Key identifier

**Returns:**
- Raw encryption key (bytes)

**Note:** Creates new key if doesn't exist.

#### shred_key()
Permanently delete encryption key.

```python
shred_key(key_identifier: str) -> bool
```

**Parameters:**
- `key_identifier`: Key identifier to delete

**Returns:**
- True if successful

### Encrypted Fields

Use encrypted fields in models:

```python
from atomingest.schema.encryption import (
    EncryptedCharField,
    EncryptedTextField,
    EncryptedEmailField,
    EncryptedJSONField
)

class SecureUser(Model):
    _table_name = "secure_users"

    # Encrypted fields
    ssn = EncryptedCharField(max_length=255, key_id="user_pii")
    notes = EncryptedTextField(key_id="user_pii")
    email = EncryptedEmailField(key_id="user_pii")
    metadata = EncryptedJSONField(key_id="user_pii")
```

**Auto-decryption:**
```python
user = SecureUser.objects().get(id=1)
print(user.ssn)  # Automatically decrypted
```

## Utilities

### Validators

Create custom validators:

```python
from atomingest.schema.validators import Validator, ValidationError

class LuhnValidator(Validator):
    """Validate credit card using Luhn algorithm."""

    def __call__(self, value: Any, field_name: str = None) -> None:
        if not self.validate_luhn(value):
            raise ValidationError(
                "Invalid credit card number",
                field_name=field_name,
                value=value
            )

    def validate_luhn(self, value: str) -> bool:
        # Luhn algorithm implementation
        return True
```

### Decorators

Apply validation to schema:

```python
from atomingest.schema.decorators import apply_validation_to_schema

# Apply validation rules to model
apply_validation_to_schema(ModelClass, validation_configs)
```

## Complete Example

```python
from atomingest.core.ingester import Ingester
from atomingest.core.loader import SchemaLoader
from atomingest.pipeline.hooks import Hook, HookContext, HookRegistry, HookType
from atomsql import Database
import logging

# Setup logging
logging.basicConfig(level=logging.INFO)

# Custom hook
class CalculateTotalHook(Hook):
    def execute(self, context: HookContext) -> HookContext:
        row = context.row_data
        row['total'] = row['quantity'] * row['price']
        return context

# Create database
db = Database("sqlite:///data.db")

# Create hook registry
registry = HookRegistry()
registry.register(
    HookType.PRE_SAVE,
    CalculateTotalHook(name="calc_total"),
    order=10
)

# Create ingester
ingester = Ingester(db, hook_registry=registry)

# Run ingestion
stats = ingester.run(
    config="pipeline.yaml",
    source="data.csv"
)

# Check results
print(f"Processed: {stats.processed}")
print(f"Failed: {stats.failed}")

# Query data
loader = SchemaLoader()
schema = loader.load_from_yaml("pipeline.yaml")
ModelClass = loader.create_model(schema)

for record in ModelClass.objects():
    print(f"{record.name}: ${record.total}")

# Cleanup
db.close()
```

## Type Hints

AtomIngest uses type hints extensively:

```python
from typing import Dict, Any, Optional, Iterator, Type
from pathlib import Path
from atomsql.orm import Model

def my_function(
    config: Dict[str, Any],
    source: Union[str, Path],
    strict: bool = False
) -> Iterator[Dict[str, Any]]:
    """Function with type hints."""
    ...
```

## Error Handling

```python
from atomingest.core.ingester import Ingester
from atomingest.pipeline.hooks import HookExecutionError
from atomsql.utils.exceptions import ImproperlyConfigured

try:
    ingester = Ingester(db)
    stats = ingester.run("pipeline.yaml", "data.csv")
except FileNotFoundError:
    print("Pipeline file not found")
except ImproperlyConfigured as e:
    print(f"Configuration error: {e}")
except HookExecutionError as e:
    print(f"Hook failed: {e.hook_name}")
except Exception as e:
    print(f"Ingestion failed: {e}")
```

## Related Documentation

- [Schema Definition](schema_definition.md) - YAML schema reference
- [Hooks](hooks.md) - Hook development guide
- [Normalization](normalization.md) - Normalization steps
- [Security](security.md) - Encryption and security

## Python Version Support

- **Python 3.9+** required
- **Python 3.11+** recommended

## Dependencies

Core dependencies:
- `atomsql` - ORM layer
- `pyyaml` - YAML parsing
- `cryptography` - Encryption
- `click` - CLI interface

Optional dependencies:
- `redis` - Caching
- `prometheus-client` - Metrics
- `psycopg2` - PostgreSQL
- `mysqlclient` - MySQL
