# Pipeline

The `Pipeline` class is the main entry point for building and running ETL pipelines.

## Overview

```python
from pycharter import Pipeline, HTTPExtractor, FileLoader, Rename

# Build with pipe operator
pipeline = (
    Pipeline(HTTPExtractor(url="https://api.example.com/data"))
    | Rename({"old": "new"})
    | FileLoader(path="output.json")
)

# Run
result = await pipeline.run()
```

## API Reference

::: pycharter.etl_generator.Pipeline
    options:
      show_root_heading: true
      show_source: true
      members:
        - __init__
        - __or__
        - run
        - from_config_dir
        - from_config_files
        - from_config_file

## Factory Methods

### from_config_dir

Load pipeline from a directory containing `extract.yaml`, `transform.yaml`, and `load.yaml`:

```python
pipeline = Pipeline.from_config_dir("pipelines/users/")
```

### from_config_files

Load from explicit file paths:

```python
pipeline = Pipeline.from_config_files(
    extract="configs/extract.yaml",
    transform="configs/transform.yaml",  # Optional
    load="configs/load.yaml",
    variables={"API_KEY": "secret"}
)
```

### from_config_file

Load from a single combined config file:

```python
pipeline = Pipeline.from_config_file("pipeline.yaml")
```

## PipelineResult

::: pycharter.etl_generator.PipelineResult
    options:
      show_root_heading: true
      members:
        - success
        - rows_extracted
        - rows_transformed
        - rows_loaded
        - rows_failed
        - duration_seconds
        - errors

## Examples

### Basic Pipeline

```python
import asyncio
from pycharter import Pipeline, FileExtractor, FileLoader

pipeline = (
    Pipeline(FileExtractor(path="input.json"))
    | FileLoader(path="output.json")
)

result = asyncio.run(pipeline.run())
print(f"Loaded {result.rows_loaded} rows")
```

### With Transformations

```python
from pycharter import Pipeline, HTTPExtractor, PostgresLoader, Rename, Filter, AddField

pipeline = (
    Pipeline(HTTPExtractor(url="https://api.example.com/users"))
    | Rename({"userName": "user_name"})
    | Filter(lambda r: r.get("active"))
    | AddField("processed_at", "now()")
    | PostgresLoader(connection_string="...", table="users")
)
```

### Error Handling

```python
from pycharter.shared.errors import ErrorMode, ErrorContext

# Collect errors instead of raising
result = await pipeline.run(
    error_context=ErrorContext(mode=ErrorMode.COLLECT)
)

if result.errors:
    for error in result.errors:
        print(f"Error: {error}")
```

### With Variables

```python
pipeline = Pipeline.from_config_dir(
    "pipelines/users/",
    variables={
        "API_KEY": os.environ["API_KEY"],
        "OUTPUT_PATH": "/data/output.json"
    }
)
```

## Message Queue Acknowledgment

When a pipeline uses a messaging extractor (Kafka, RabbitMQ, SQS), the pipeline automatically acknowledges each batch after loading:

```python
from pycharter.etl_generator.extractors import KafkaExtractor

pipeline = (
    Pipeline(KafkaExtractor(topics=["orders"], consumer_group="etl"))
    | Rename({"orderId": "order_id"})
    | PostgresLoader(connection_string="...", table="orders")
)

# Pipeline calls extractor.acknowledge() after each batch load
result = await pipeline.run()

# Don't forget to close the consumer
await pipeline.extractor.close()
```

The pipeline determines success from the `LoadResult`:

| Scenario | Acknowledge call |
|----------|-----------------|
| Load succeeds | `acknowledge(batch_index, success=True)` |
| Load fails | `acknowledge(batch_index, success=False)` |
| Dry run | `acknowledge(batch_index, success=True)` |
| No loader set | `acknowledge(batch_index, success=True)` |

If `acknowledge()` raises an exception, it is logged as a warning and the pipeline continues.

## Incremental Extraction

Track a watermark field across runs so only new/updated records are extracted:

```yaml title="extract.yaml"
type: database
query: "SELECT * FROM events WHERE updated_at > '${watermark}'"
connection_string: "${DATABASE_URL}"
incremental:
  enabled: true
  watermark_field: updated_at
  initial_value: "2024-01-01"
```

```yaml title="settings.yaml"
state_store:
  backend: file      # or "sqlite"
  path: ./.state
```

The pipeline persists the highest watermark value on success and injects it into the next run.

## Testing Utilities

PyCharter provides mock classes and assertion helpers for testing pipelines without real I/O:

```python
from pycharter import MockExtractor, MockLoader, PipelineTestHarness

# Mock extractor yields fixture data
extractor = MockExtractor(data=[
    [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
])

# Mock loader captures what was loaded
loader = MockLoader()

pipeline = Pipeline(extractor) | Rename({"name": "full_name"}) | loader
result = await pipeline.run()

# Assert on captured data
from pycharter import assert_record_count, assert_fields_present
assert_record_count(loader.loaded_data, 2)
assert_fields_present(loader.loaded_data, ["id", "full_name"])
```

## See Also

- [Streaming and Messaging Guide](../guides/streaming-and-messaging.md)
- [ETL Pipelines Tutorial](../tutorials/etl-pipelines.md)
- [Extractors](extractors.md)
- [Transformers](transformers.md)
- [Loaders](loaders.md)
