# Loony — Governed Data Services

Loony is a platform for building data pipelines that agents can query. You write scripts that pull data from APIs or databases, transform it in SQL, and expose it via MCP endpoints.

## How It Works

```
Source (API / DB / file)
    ↓
dlt script (scripts/*.py)        ← you write this
    ↓
Raw table (raw_*)                ← dlt creates this automatically
    ↓
Validate against schema.yaml    ← loony checks types, PKs, row counts
    ↓
SQL transforms (transforms/*.sql) ← you write this
    ↓
Staged tables (stg_*) → Materialized views (mv_*)
    ↓
MCP endpoint                     ← agents query via describe/query/search
```

## Project Structure

Each service is its own directory with its own database. Create multiple services for separate concerns:

```
loony init crm           # creates crm/
loony init analytics     # creates analytics/
```

```
crm/                              # one service = one folder = one database
├── scripts/
│   ├── my_source.py              # dlt script — exports run(conn)
│   └── my_source_test.py         # optional tests
├── transforms/
│   ├── 001_staged.sql            # raw → staged (clean, type, dedup)
│   └── 002_views.sql             # staged → materialized views
├── schema.yaml                   # contract: tables, columns, types, measures, sync modes
├── loony.yaml                    # service name, schedules
├── requirements.txt              # Python deps (dlt[postgres] pre-included)
├── .env.example                  # documents required secrets
└── .env.local                    # local secrets (gitignored)
```

All CLI commands (`deploy`, `test`, `run`, `logs`) operate on the current directory. `cd` into the service you want to work on.

## File Conventions

### Scripts (scripts/*.py)

Every script MUST export a `run(conn)` function that returns `{"rows": N}`:

```python
import os
from datetime import datetime
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator

@dlt.source
def my_source():
    client = RESTClient(base_url="https://api.example.com")

    @dlt.resource(name="raw_items", write_disposition="merge", primary_key=["id"])
    def items():
        for page in client.paginate("/items", params={"limit": 100}):
            yield page

    return items

def run(conn):
    pipeline = dlt.pipeline(
        pipeline_name="my_source",
        destination="postgres",
        dataset_name=os.environ.get("LOONY_SCHEMA", "public"),
    )
    info = pipeline.run(my_source())
    rows = 0
    if info.loads_ids:
        for metrics in info.metrics.values():
            for m in metrics:
                rows += sum(m.get("row_counts", {}).values())
    return {"rows": rows}
```

Key rules:
- Use `destination="postgres"` — dlt reads `DESTINATION__POSTGRES__CREDENTIALS` from env
- Use `dataset_name=os.environ.get("LOONY_SCHEMA", "public")`
- Name dlt resources with `raw_` prefix to match schema.yaml
- Never hardcode credentials — use `os.environ`

### Sync Modes

Every raw table in schema.yaml MUST declare a `sync_mode`:

| sync_mode | dlt write_disposition | When to use |
|---|---|---|
| `replace` | `replace` | Small reference data (<100K rows). Pull everything each run. |
| `merge` | `merge` | Data that gets revised. Pull recent window, upsert by primary key. |
| `append` | `append` | Event data (transactions, logs). Only pull new records since last run. |

For merge with lookback (data that gets revised):
```python
LOOKBACK_YEARS = 3

@dlt.resource(write_disposition="merge", primary_key=["year", "country"])
def population():
    current_year = datetime.now().year
    for year in range(current_year - LOOKBACK_YEARS, current_year + 1):
        for page in client.paginate("/data", params={"year": year}):
            yield page
```

For append with incremental cursor:
```python
@dlt.resource(write_disposition="append")
def transactions(created_at=dlt.sources.incremental("created_at")):
    yield from client.paginate("/transactions", params={"since": created_at.last_value})
```

### Transforms (transforms/*.sql)

SQL files run in alphabetical order after every script run. Must be idempotent.

**Staged tables** (`001_staged.sql`):
- Prefix: `stg_`
- Clean raw data: cast types, coalesce nulls, deduplicate
- Use `DELETE FROM stg_x; INSERT INTO stg_x SELECT ... FROM raw_x` pattern
- Handle API quirks: `COALESCE(NULLIF(column::text, '-')::integer, 0)` for APIs that return "-" for missing values

**Materialized views** (`002_views.sql`):
- Prefix: `mv_`
- Aggregate staged data for querying
- Always add unique index: `CREATE UNIQUE INDEX IF NOT EXISTS idx_mv_x ON mv_x (key_columns)` — required for `REFRESH MATERIALIZED VIEW CONCURRENTLY`
- Use `CREATE MATERIALIZED VIEW IF NOT EXISTS`

### Table Naming

| Prefix | Purpose | Example |
|---|---|---|
| `raw_` | Raw ingested data from API/DB | `raw_population` |
| `stg_` | Cleaned, typed, deduplicated | `stg_population` |
| `mv_` | Aggregated materialized views | `mv_refugees_by_host` |
| `docs_` | Document chunks + vector embeddings | `docs_footnotes` |

### Schema.yaml

Declares what the pipeline produces. Validated after each run.

```yaml
tables:
  raw_items:
    description: "Raw data from Example API"
    type: data
    sync_mode: merge
    primary_key: [id]
    columns:
      id: { type: integer, description: "Unique item ID" }
      name: { type: text, description: "Item name" }
      value: { type: numeric, description: "Item value in USD" }
      loaded_at: { type: timestamptz, description: "Load timestamp" }

  mv_items_summary:
    description: "Aggregated item metrics"
    type: data
    primary_key: [category]
    measures:
      total_value:
        column: value
        type: sum
        description: "Total value of all items"
      item_count:
        type: count
        description: "Number of items"
    dimensions:
      category:
        column: category
        type: string
        description: "Item category"
```

### loony.yaml

```yaml
name: my-service
description: "What this service does"

schedule:
  my_source: "0 */6 * * *"    # cron expression

retention: 90d                  # auto-delete raw rows older than this
```

## CLI Commands

```
loony init <name>              # scaffold project files
loony test [script]            # dry-run: load → validate → transform → report
loony deploy                   # push to hosted platform
loony run <script>             # trigger a script run
loony describe [table]         # show tables, measures, dimensions
loony query <table> ...        # structured query
loony search "text"            # vector search on doc tables
loony status                   # health + auth check
loony logs                     # view run logs
loony loads                    # run history
loony secrets set KEY=val      # store encrypted secret
loony keys create              # create API key for service
```

## Common Issues

**Type mismatches**: APIs often return numbers as strings or use "-" for null. dlt infers types from the data, so columns may be `varchar` instead of `integer`. Handle in transforms:
```sql
COALESCE(NULLIF(column_name::text, '-')::integer, 0)
```

**dlt internal tables**: dlt creates `_dlt_loads`, `_dlt_pipeline_state`, etc. Ignore these — they're dlt's internal state.

**Materialized view refresh**: Views need a unique index for `REFRESH CONCURRENTLY`. Always add one matching the primary key.

**Schema validation warnings**: After each run, loony compares actual tables to schema.yaml. Warnings about type mismatches are informational — the transforms handle the casting. Missing columns are errors.
