---
name: loony-build-pipeline
description: Guide for building a data pipeline with loony — from API to queryable views
allowed-tools: Bash(loony *), Read, Write, Edit
---

# Build a Loony Pipeline

Follow these phases in order. Complete each before moving to the next.

## Phase 0: Set Up the Service

Each service is its own directory with its own database. All CLI commands operate on the current directory.

```bash
loony init myservice    # creates myservice/ with scaffolding
cd myservice            # all commands run from here
```

If you're already in a service directory (has `loony.yaml`), skip this step.

**Checkpoint**: You're inside a directory with `loony.yaml`, `schema.yaml`, and `scripts/`.

## Phase 1: Understand the Source

Before writing any code, research the data source:
- What API endpoints are available?
- What does the response look like? (curl a sample)
- Is authentication required? What type?
- How is data paginated? (page numbers, cursors, next links)
- Is there bulk download available?
- How often does the data update?
- Can historical data change? (determines sync_mode)

**Checkpoint**: You should know the base URL, auth method, pagination style, and response shape before continuing.

## Phase 2: Write schema.yaml

Define the contract BEFORE writing scripts. Start with:
1. What raw tables will the scripts produce?
2. What staged tables clean/aggregate the raw data?
3. What materialized views will agents query?
4. What measures and dimensions does each view expose?

For each raw table, decide sync_mode:
- API returns all data each time, small dataset → `replace`
- API returns all data, data can be revised → `merge` with primary_key
- API returns only new records since last call → `append` with incremental_key

```bash
# Validate your schema.yaml
cat schema.yaml
```

**Checkpoint**: schema.yaml has all tables declared with sync_mode, primary keys, column descriptions, and measures/dimensions on views.

## Phase 3: Write the Script

Create `scripts/<source_name>.py`. Follow this template:

```python
import os
from datetime import datetime
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator

BASE_URL = "https://api.example.com"

@dlt.source
def my_source():
    client = RESTClient(
        base_url=BASE_URL,
        paginator=PageNumberPaginator(
            total_path="totalPages",    # adjust to API response
            page_param="page",
            base_page=1,
        ),
        data_selector="items",          # JSON path to data array
    )

    @dlt.resource(name="raw_<source>", write_disposition="<sync_mode>", primary_key=[...])
    def data():
        for page in client.paginate("/endpoint", params={...}):
            yield page

    return data

def run(conn):
    pipeline = dlt.pipeline(
        pipeline_name="<source>",
        destination="postgres",
        dataset_name=os.environ.get("LOONY_SCHEMA", "public"),
    )
    info = pipeline.run(my_source())
    rows = 0
    if info.loads_ids:
        for metrics in info.metrics.values():
            for m in metrics:
                rows += sum(m.get("row_counts", {}).values())
    return {"rows": rows}
```

Adjust the paginator to match the API:
- `PageNumberPaginator` — for APIs with page numbers + total pages
- `OffsetPaginator` — for APIs with offset + limit
- `JSONLinkPaginator` — for APIs with next page URL in response

**Checkpoint**: Script exists, has `run(conn)`, uses dlt, matches schema.yaml sync_mode and primary_key.

## Phase 4: Test the Script

```bash
loony test <script_name>
```

Review the output:
- Did rows load?
- Are there type mismatches? (varchar vs integer)
- Is the primary key unique?
- Are there unexpected columns?

If type mismatches exist, note the bad values — you'll handle them in transforms.

**Checkpoint**: `loony test` passes with warnings (type mismatches are OK) but no errors.

## Phase 5: Write Transforms

Create `transforms/001_staged.sql` and `transforms/002_views.sql`.

**Staged table pattern**:
```sql
CREATE TABLE IF NOT EXISTS stg_<name> (
    <columns with correct types>,
    UNIQUE (<primary_key_columns>)
);

DELETE FROM stg_<name>;

INSERT INTO stg_<name> (<columns>)
SELECT
    COALESCE(NULLIF(col::text, '-')::integer, 0),  -- handle API quirks
    ...
FROM raw_<name>
WHERE <filter_conditions>;
```

**View pattern**:
```sql
CREATE MATERIALIZED VIEW IF NOT EXISTS mv_<name> AS
SELECT
    <dimensions>,
    SUM(<measure>) AS <measure_name>,
    COUNT(*) AS <count_name>
FROM stg_<name>
GROUP BY <dimensions>;

CREATE UNIQUE INDEX IF NOT EXISTS idx_mv_<name>
ON mv_<name> (<primary_key_columns>);
```

**Checkpoint**: Transforms create staged tables and views that match schema.yaml.

## Phase 6: Full Validation

```bash
loony test --all
```

This runs all scripts + transforms + validates everything against schema.yaml. All views should have rows. All measures should resolve.

## Phase 7: Deploy

```bash
loony deploy
```

This pushes to the hosted platform. You'll get back an MCP endpoint URL.

## Phase 8: Verify

```bash
loony describe
loony query mv_<name> --measures <measure> --dimensions <dimension>
```

Confirm the data is queryable and measures return expected values.
