---
title: Introduction to ETL
---

**What you'll learn**: What ETL (Extract, Transform, Load) means, why it's important, how it relates to working with JSON data from APIs, and how `etielle`'s features support each ETL step.

## What is ETL?

**ETL** stands for **Extract, Transform, Load**—the three fundamental steps for moving data from one system to another. Understanding these steps is key to understanding how `etielle` works:

| ETL Step | What it means | How `etielle` does it |
|----------|---------------|----------------------|
| **Extract** | Navigate and pull data from a source | **Traversals** and **Mappings** walk through nested JSON |
| **Transform** | Reshape and format the data | **Transforms**, **Emissions**, and **Relationships** reshape data in memory |
| **Load** | Insert data into the target system | **Database adapters** (`flush_to_db()`) persist to database |

ETL is a core concept in data engineering and has been used for decades to move data between systems, especially in business intelligence and data warehousing. `etielle` brings declarative ETL to Python for JSON-to-relational transformations.

## ETL in Action: A Simple Example

Imagine you're building an app that tracks GitHub repositories. Here's how each ETL step works, with `etielle`'s corresponding concepts:

### 1. Extract: Navigate the JSON structure

**Goal**: Pull a subset of data from GitHub's API (returns nested JSON with repositories, contributors, commits)

**How `etielle` does it**: **Traversals** define how to walk through the JSON:

```python
from etielle.core import TraversalSpec

# Extract repositories
repos_traversal = TraversalSpec(
    path=["repositories"],  # Start at "repositories" key
    mode="auto",  # Iterate through the array
    emits=[...]  # What to do with each repository
)

# Extract commits nested inside each repository
commits_traversal = TraversalSpec(
    path=["repositories"],
    mode="auto",
    inner_path=["commits"],  # Nested path inside each repo
    inner_mode="auto",
    emits=[...]  # What to do with each commit
)
```

**Key concept**: Traversals tell `etielle` *where* to go in the JSON and *how* to iterate through it. They handle the **Extract** step.

Learn more: [Traversals](traversals.qmd)

### 2. Transform: Reshape data in memory

**Goal**: Extract specific fields, link parent-child records, format values, build table structures

**How `etielle` does it**: Multiple features work together to transform data:

1. **Field-level transforms** extract and modify individual values:
    ```python
    from etielle.transforms import get, get_from_parent, concat, literal

    # Transform: Extract a field from the current item
    repo_name = get("name")

    # Transform: Get a field from the parent (e.g., for relationship linking)
    parent_repo_id = get_from_parent("id")

    # Transform: Remix values into a new output value (e.g. "123" -> "repo_123")
    full_id = concat(literal("repo_"), get("id"))
    ```

2. **Table-level transforms** (Emissions) define output structure:
    ```python
    from etielle.core import TableEmit, Field

    # Transform: Define the output table structure (still in memory)
    TableEmit(
        table="repositories",  # Target table name
        join_keys=[get("id")],  # Unique identifier (like a primary key)
        fields=[
            Field("id", get("id")),
            Field("name", get("name")),
            Field("url", get("url"))
        ]
    )
    ```

3. **Relationship transforms** link records together in memory (foreign keys, ORM relationships).
    ```python
    from etielle.relationships import ManyToOneSpec

    # Transform: Define a many-to-one relationship
    relationship = ManyToOneSpec(
        child_table="commits",
        parent_table="repositories",
        # attribute name on the child instance that references the parent instance
        attr="repository",
        child_to_parent_key=[
            # transform to get the parent's join_key from the child item's context
            get("repository_id")
        ],
        required=True
    )
    ```

**Key concept**: Transforms, emissions, and relationships are all done in-memory, before database persistence. Although shown separately above, transforms, emissions, and relationship linking are typically combined into a mapping that runs all at once.

Learn more: [Transforms](transforms.qmd), [Emissions](emissions.qmd), [Relationships](relationships.qmd)

### 3. Load: Persist to database

**Goal**: Insert the transformed data into PostgreSQL

**How `etielle` does it**: **Database adapters** persist the in-memory data:

```python
from etielle.sqlalchemy_adapter import flush_to_db

# Load: Write data to the database
flush_to_db(session, mapping_results)
```

**Key concept**: The **Load** step is optional and happens via database adapters. Without this step, `etielle` just transforms JSON to in-memory Python objects.

Learn more: [Database upserts](loading-data-into-a-database.qmd), [SQLAlchemy adapter](sqlalchemy-adapter.qmd)

---

**The full ETL flow with `etielle`**:

1. **Extract** (Traversals): "Go to `repositories`, iterate through each one"
2. **Transform** (Transforms + Emissions + Relationships): "Get the `name` field, format the `id`, build table rows in memory, associate child records with parent records by parent id"
3. **Load** (Database adapters): "Persist the in-memory objects to PostgreSQL" (optional)

Without `etielle`, you'd write nested loops, manual field extraction, and explicit inserts. With `etielle`, you declare the mapping once, and the library handles the rest.

## The Three Pillars of `etielle` ETL

To summarize, `etielle` implements ETL through three core feature groups:

### 1. Extract: Traversals and Mappings

**What they do**: Navigate nested JSON structures and decide what to iterate over

**Key classes**:
- `TraversalSpec`: Defines a path through the JSON
- `MappingSpec`: Container for multiple traversals

**Example**:
```python
TraversalSpec(
    path=["users"],           # Extract: Start here
    inner_path=["posts"],     # Extract: Then go here
    mode="auto",              # Extract: How to iterate
)
```

Learn more: [Traversals](traversals.qmd)

### 2. Transform: Transforms, Emissions, and Relationships

**What they do**: Reshape data in memory from nested JSON to structured tables/objects

**Key features**:

**Field-level transforms**:
- `get()`: Get field from current node
- `get_from_parent()`: Get field from ancestor (for relationships)
- `concat()`, `format_id()`: Format and combine values
- `coalesce()`: Provide fallback values

**Table-level transforms (Emissions)**:
- `TableEmit`: Build dict tables in memory
- `InstanceEmit`: Build Pydantic/TypedDict/ORM instances in memory
- `join_keys`: Define unique row identifiers for merging

**Relationship transforms**:
- `RelationshipEmit`: Link records together (foreign keys, ORM relationships)

**Example**:
```python
# Field transform
Field("user_id", get_from_parent("id"))  # Link to parent

# Table transform
TableEmit(
    table="users",
    join_keys=[get("id")],
    fields=[Field("id", get("id")), Field("name", get("name"))]
)
```

Learn more: [Transforms](transforms.qmd), [Emissions](emissions.qmd), [Relationships](relationships.qmd)

### 3. Load: Database Adapters

**What they do**: Persist the in-memory transformed data to a database (optional)

**Key features**:
- `flush_to_db()`: Persist instances to database via SQLAlchemy/SQLModel
- One-shot flushing for performance
- Relationship binding before persistence

**Example**:
```python
from etielle.sqlalchemy_adapter import flush_to_db

# Load: Persist to database
flush_to_db(session, mapping_results)
```

**Note**: This step is optional! You can use `etielle` just for in-memory JSON transformation without database persistence.

Learn more: [Database upserts](loading-data-into-a-database.qmd), [SQLAlchemy adapter](sqlalchemy-adapter.qmd)

## Next Steps

Now that you understand ETL and how `etielle` implements each step, you're ready to dive deeper:

1. **[Quickstart](index.qmd)** - Jump straight into using `etielle` with a complete example
2. **[Traversals](traversals.qmd)** - Master the **Extract** step: navigate nested JSON
3. **[Transforms](transforms.qmd)** - Master the **Transform** step: field-level data extraction and reshaping
4. **[Emissions](emissions.qmd)** - Master the **Transform** step: table-level structure definition
5. **[Relationships](relationships.qmd)** - Master the **Transform** step: link records together
6. **[Database upserts](loading-data-into-a-database.qmd)** - Master the **Load** step: persist to database
