# Loaders

Loaders write transformed data to destinations.

## Overview

```python
from pycharter import PostgresLoader, FileLoader, CloudStorageLoader
```

## PostgresLoader

Load data to PostgreSQL databases.

::: pycharter.etl_generator.PostgresLoader
    options:
      show_root_heading: true
      show_source: false

### Load Modes

| Mode | Description |
|------|-------------|
| `insert` | Insert all records (fails on duplicates) |
| `upsert` | Insert or update on conflict |
| `replace` | Truncate table and insert |

### Examples

```python
from pycharter import PostgresLoader

# Basic insert
loader = PostgresLoader(
    connection_string="postgresql://user:pass@localhost/db",
    table="users"
)

# Upsert (update on conflict)
loader = PostgresLoader(
    connection_string="postgresql://user:pass@localhost/db",
    table="users",
    schema="public",
    mode="upsert",
    conflict_columns=["id"]
)

# Replace (truncate and insert)
loader = PostgresLoader(
    connection_string="postgresql://user:pass@localhost/db",
    table="users",
    mode="replace"
)

# With SSH tunnel
loader = PostgresLoader(
    connection_string="postgresql://user:pass@localhost/db",
    table="users",
    ssh_tunnel={
        "host": "bastion.example.com",
        "port": 22,
        "username": "deploy",
        "key_file": "~/.ssh/id_rsa"
    }
)
```

## DatabaseLoader

Generic database loader supporting multiple databases.

```python
from pycharter import DatabaseLoader

# MySQL
loader = DatabaseLoader(
    connection_string="mysql://user:pass@localhost/db",
    table="users"
)

# SQLite
loader = DatabaseLoader(
    connection_string="sqlite:///data.db",
    table="users"
)
```

## FileLoader

Load data to local files.

::: pycharter.etl_generator.FileLoader
    options:
      show_root_heading: true
      show_source: false

### Supported Formats

| Format | Extension | Notes |
|--------|-----------|-------|
| JSON | `.json` | Pretty-printed array |
| JSON Lines | `.jsonl` | One JSON object per line |
| CSV | `.csv` | With header row |
| Parquet | `.parquet` | Columnar, compressed |

### Examples

```python
from pycharter import FileLoader

# JSON output
loader = FileLoader(path="output/users.json", file_format="json")

# JSON Lines (streaming-friendly)
loader = FileLoader(path="output/users.jsonl", file_format="jsonl")

# CSV output
loader = FileLoader(path="output/users.csv", file_format="csv")

# Parquet (compressed, columnar)
loader = FileLoader(path="output/users.parquet", file_format="parquet")

# Append mode
loader = FileLoader(
    path="output/users.jsonl",
    file_format="jsonl",
    write_mode="append"
)
```

## CloudStorageLoader

Load data to cloud storage (S3, GCS, Azure Blob).

::: pycharter.etl_generator.CloudStorageLoader
    options:
      show_root_heading: true
      show_source: false

### Examples

```python
from pycharter import CloudStorageLoader

# AWS S3
loader = CloudStorageLoader(
    provider="s3",
    bucket="my-bucket",
    path="output/users.json",
    file_format="json",
    credentials={
        "aws_access_key_id": "${AWS_KEY}",
        "aws_secret_access_key": "${AWS_SECRET}"
    }
)

# Google Cloud Storage
loader = CloudStorageLoader(
    provider="gcs",
    bucket="my-bucket",
    path="output/users.json",
    file_format="json"
)

# Azure Blob Storage
loader = CloudStorageLoader(
    provider="azure",
    container="my-container",
    path="output/users.json",
    file_format="json",
    credentials={
        "connection_string": "${AZURE_STORAGE_CONNECTION_STRING}"
    }
)

# Parquet to S3
loader = CloudStorageLoader(
    provider="s3",
    bucket="data-lake",
    path="users/date=2024-01-01/users.parquet",
    file_format="parquet"
)
```

## LoaderFactory

Create loaders from configuration:

```python
from pycharter.etl_generator.loaders import LoaderFactory

# Create from config
loader = LoaderFactory.create({
    "type": "postgres",
    "connection_string": "postgresql://...",
    "table": "users"
})

# Register custom loader
LoaderFactory.register("bigquery", BigQueryLoader)
```

## LoadResult

::: pycharter.etl_generator.LoadResult
    options:
      show_root_heading: true
      members:
        - success
        - rows_loaded
        - duration_seconds
        - error

## Config-Driven Loaders

Define loaders in YAML:

```yaml title="load.yaml (PostgreSQL)"
type: postgres
connection_string: "${DATABASE_URL}"
table: users
schema: public
mode: upsert
conflict_columns:
  - id
```

```yaml title="load.yaml (File)"
type: file
file_path: output/users.json
format: json
write_mode: overwrite
```

```yaml title="load.yaml (S3)"
type: cloud_storage
storage:
  provider: s3
  bucket: my-bucket
  path: output/users.json
format: json
credentials:
  aws_access_key_id: "${AWS_KEY}"
  aws_secret_access_key: "${AWS_SECRET}"
```

## Custom Loaders

Create custom loaders by extending `BaseLoader`:

```python
from pycharter.etl_generator.loaders import BaseLoader
from pycharter.etl_generator.result import LoadResult
from typing import List, Dict, Any

class BigQueryLoader(BaseLoader):
    def __init__(self, project: str, dataset: str, table: str):
        self.project = project
        self.dataset = dataset
        self.table = table
    
    async def load(self, data: List[Dict[str, Any]], **params) -> LoadResult:
        try:
            # Implement BigQuery loading logic
            client = bigquery.Client(project=self.project)
            table_ref = f"{self.dataset}.{self.table}"
            
            job = client.load_table_from_json(data, table_ref)
            job.result()  # Wait for completion
            
            return LoadResult(success=True, rows_loaded=len(data))
        except Exception as e:
            return LoadResult(success=False, error=str(e))

# Register
LoaderFactory.register("bigquery", BigQueryLoader)
```

## See Also

- [ETL Pipelines Tutorial](../tutorials/etl-pipelines.md)
- [Database Configuration Guide](../guides/database-config.md)
- [Pipeline](pipeline.md)
