# ETL Examples: CSV → Transform → CSV

Sample configs to run a simple ETL in the PyCharter UI: extract from a mock CSV, apply transforms, and load to a new CSV file.

## Files

| File | Purpose |
|------|--------|
| `input.csv` | Mock source data (products: id, name, category, price, quantity, updated_at) |
| `extract.yaml` | Extract config: read from CSV file |
| `transform.yaml` | Transform config: rename, convert types, add fields, select columns |
| `load.yaml` | Load config: write to CSV file |
| `pipeline.yaml` | Single-file pipeline (optional; for CLI/config-dir use) |
| `variables.yaml` | Variable injection: values for `${VAR}` in extract/load/transform (e.g. `INPUT_PATH`, `OUTPUT_PATH`) |
| `settings.yaml` | Shared settings: OpenLineage emission, DLQ, etc. Used when running from config dir |

## Using in the ETL UI

1. **Start the API** from the **pycharter repo root** so relative paths resolve:
   ```bash
   cd pycharter
   pycharter api
   ```
   If you run the API from another directory, use variables (step 4).

2. **Open the ETL page** in the UI and load or paste configs:
   - **Extract:** Open `data/etl_examples/extract.yaml` and paste its contents into the Extract panel (or use a template and edit path to `data/etl_examples/input.csv`).
   - **Transform:** Open `data/etl_examples/transform.yaml` and paste into the Transform panel.
   - **Load:** Open `data/etl_examples/load.yaml` and paste into the Load panel.

3. **Run the pipeline**
   - Optionally run **Dry run** first (extract + transform only, no write).
   - Click **Run** to extract, transform, and load. Output is written to `data/etl_examples/output.csv`.

4. **If the API runs from a different directory**, set variables in the UI so paths are absolute:
   - `INPUT_PATH` = full path to `input.csv` (e.g. `/Users/you/repo/pycharter/data/etl_examples/input.csv`)
   - `OUTPUT_PATH` = full path for `output.csv` (e.g. `/Users/you/repo/pycharter/data/etl_examples/output.csv`)

## Transform summary

- **Rename:** `product_id` → `id`, `product_name` → `name`, `updated_at` → `source_updated_at`
- **Convert:** `price` → float, `quantity` → int, `source_updated_at` → datetime
- **Defaults:** `category` defaults to `uncategorized` if missing
- **Add:** `processed_at` (current time), `source` (literal `etl_example`)
- **Select:** only the listed columns are written to the output CSV

## CLI (optional)

From the pycharter repo root, running from the config directory loads `variables.yaml` and `settings.yaml` automatically (so paths and OpenLineage are configured):

```bash
cd pycharter
python -c "
import asyncio
from pycharter.etl_generator import Pipeline
async def main():
    p = Pipeline.from_config_dir('data/etl_examples')
    r = await p.run()
    print(r)
asyncio.run(main())
"
```

Or run using the single-file pipeline:

```bash
pycharter etl run data/etl_examples/pipeline.yaml
```

(If your CLI supports `pycharter etl run` and a pipeline file; otherwise use the Python snippet above.)
