---
title: Introduction to ETL
---

**What you'll learn**: What ETL (Extract, Transform, Load) means, why it's important, how it relates to working with JSON data from APIs, and how `etielle`'s fluent API supports each ETL step.

## What is ETL?

**ETL** stands for **Extract, Transform, Load**---the three fundamental steps for moving data from one system to another. Understanding these steps is key to understanding how `etielle` works:

| ETL Step | What it means | How `etielle` does it |
|----------|---------------|----------------------|
| **Extract** | Navigate and pull data from a source | `goto()` and `each()` navigate nested JSON |
| **Transform** | Reshape and format the data | Transforms, `Field`, and `TempField` reshape data |
| **Load** | Insert data into the target system | `load(session).run()` persists to database |

ETL is a core concept in data engineering and has been used for decades to move data between systems. `etielle` brings declarative ETL to Python for JSON-to-relational transformations.

## ETL in Action: A Simple Example

Imagine you're building an app that tracks GitHub repositories. Here's how each ETL step works with `etielle`'s fluent API:

### 1. Extract: Navigate the JSON structure

**Goal**: Pull a subset of data from GitHub's API (returns nested JSON with repositories, contributors, commits)

**How `etielle` does it**: `goto()` and `each()` define how to walk through the JSON:

```python
from etielle import etl, Field, TempField, get

result = (
    etl(github_data)
    # Extract: Navigate to repositories array
    .goto("repositories").each()
    .map_to(table="repos", fields=[...])

    # Extract: Navigate to commits nested inside each repo
    .goto("commits").each()
    .map_to(table="commits", fields=[...])

    .run()
)
```

**Key concept**: `goto()` tells `etielle` *where* to go in the JSON, and `each()` tells it to iterate through items. They handle the **Extract** step.

Learn more: [Navigation](navigation.qmd)

### 2. Transform: Reshape data in memory

**Goal**: Extract specific fields, link parent-child records, format values, build table structures

**How `etielle` does it**: Multiple features work together to transform data:

1. **Field-level transforms** extract and modify individual values:
    ```python
    from etielle import get, get_from_parent, concat, literal

    # Transform: Extract a field from the current item
    repo_name = get("name")

    # Transform: Get a field from the parent (for relationship linking)
    parent_repo_id = get_from_parent("id")

    # Transform: Combine values (e.g. "123" -> "repo_123")
    full_id = concat(literal("repo_"), get("id"))
    ```

2. **Table-level transforms** via `map_to()` define output structure:
    ```python
    from etielle import etl, Field, TempField, get

    (
        etl(data)
        .goto("repositories").each()
        .map_to(table="repositories", fields=[
            Field("id", get("id")),          # Output column
            Field("name", get("name")),      # Output column
            Field("url", get("url")),        # Output column
            TempField("id", get("id"))       # Join key (not in output)
        ])
    )
    ```

3. **Relationship transforms** link records together with `link_to()`:
    ```python
    from etielle import etl, Field, TempField, get, get_from_parent

    (
        etl(data)
        .goto("repositories").each()
        .map_to(table=Repository, fields=[...])

        .goto("commits").each()
        .map_to(table=Commit, fields=[
            Field("message", get("message")),
            TempField("repo_id", get_from_parent("id"))
        ])
        .link_to(Repository, by={"repo_id": "id"})  # Link commits to repos
    )
    ```

**Key concept**: Transforms, fields, and relationships are all processed in-memory, before any database persistence. The fluent API chains these operations naturally.

Learn more: [Transforms](transforms.qmd), [Mapping Tables](mapping.qmd), [Relationships](relationships.qmd)

### 3. Load: Persist to database

**Goal**: Insert the transformed data into PostgreSQL

**How `etielle` does it**: `load(session).run()` persists the in-memory data:

```python
from etielle import etl, Field, TempField, get

result = (
    etl(data)
    .goto("users").each()
    .map_to(table=User, fields=[...])
    .load(session)   # Configure database session
    .run()           # Execute and persist
)

session.commit()     # You control the transaction
```

**Key concept**: The **Load** step is optional and happens via `load()`. Without it, `etielle` just transforms JSON to in-memory Python objects.

Learn more: [Database Loading](database-loading.qmd)

---

**The full ETL flow with `etielle`**:

1. **Extract** (`goto()`, `each()`): "Go to `repositories`, iterate through each one"
2. **Transform** (transforms + `Field` + `link_to()`): "Get the `name` field, format the `id`, build table rows, link children to parents"
3. **Load** (`load().run()`): "Persist the in-memory objects to PostgreSQL" (optional)

Without `etielle`, you'd write nested loops, manual field extraction, and explicit inserts. With `etielle`, you declare the mapping once, and the library handles the rest.

## The Three Pillars of `etielle` ETL

To summarize, `etielle` implements ETL through three core feature groups:

### 1. Extract: Navigation

**What it does**: Navigate nested JSON structures and iterate over items

**Key methods**:
- `goto(path)`: Navigate to a nested location
- `each()`: Iterate over list items or dict key-value pairs
- `goto_root(index)`: Switch between multiple JSON roots

**Example**:
```python
(
    etl(data)
    .goto("users").each()           # Extract: Navigate and iterate
    .goto("posts").each()           # Extract: Nested iteration
    .map_to(...)
)
```

Learn more: [Navigation](navigation.qmd)

### 2. Transform: Fields, Transforms, and Relationships

**What it does**: Reshape data in memory from nested JSON to structured tables/objects

**Key features**:

**Transforms** (value extraction):
- `get()`: Get field from current node
- `get_from_parent()`: Get field from ancestor (for relationships)
- `concat()`, `format_id()`: Format and combine values
- `coalesce()`: Provide fallback values

**Field types** (table structure):
- `Field(name, transform)`: Output column
- `TempField(name, transform)`: Join key only (not in output)
- `merge` policies: Sum, append, min/max when merging rows

**Relationships**:
- `link_to(Parent, by={...})`: Link child records to parents

**Example**:
```python
.map_to(table=Post, fields=[
    Field("title", get("title")),
    Field("user_id", get_from_parent("id")),    # Link to parent
    TempField("id", get("id"))                   # Join key
])
.link_to(User, by={"user_id": "id"})            # Bind relationship
```

Learn more: [Transforms](transforms.qmd), [Mapping Tables](mapping.qmd), [Relationships](relationships.qmd)

### 3. Load: Database Persistence

**What it does**: Persist the in-memory transformed data to a database (optional)

**Key features**:
- `load(session)`: Configure database session
- `run()`: Execute pipeline and flush to database
- One-shot flushing for performance
- You control the transaction (commit/rollback)

**Example**:
```python
result = (
    etl(data)
    .goto("users").each()
    .map_to(table=User, fields=[...])
    .load(session)
    .run()
)

session.commit()  # You control the transaction
```

**Note**: This step is optional! You can use `etielle` just for in-memory JSON transformation without database persistence.

Learn more: [Database Loading](database-loading.qmd)

## Next Steps

Now that you understand ETL and how `etielle` implements each step, you're ready to dive deeper:

1. **[Quickstart](../index.qmd)** - Jump straight into using `etielle` with a complete example
2. **[Navigation](navigation.qmd)** - Master the **Extract** step: `goto()`, `each()`, `goto_root()`
3. **[Transforms](transforms.qmd)** - Master the **Transform** step: value extraction and formatting
4. **[Mapping Tables](mapping.qmd)** - Master the **Transform** step: `Field`, `TempField`, merge policies
5. **[Relationships](relationships.qmd)** - Master the **Transform** step: `link_to()` for linking records
6. **[Database Loading](database-loading.qmd)** - Master the **Load** step: `load().run()` for persistence
