---
name: loony-test-validate
description: Test and validate a loony pipeline — fix type mismatches, schema drift, transform errors
allowed-tools: Bash(loony *), Read, Write, Edit
---

# Test & Validate a Loony Pipeline

## Quick Validation

```bash
# Validate schema.yaml + scripts without deploying
loony validate

# Deploy and run (validates first, then pushes and executes)
loony deploy

# Re-run without redeploying (uses latest deployment)
loony run
loony run <script_name>   # run a single script
```

## Reading Deploy Output

Logs stream in real-time during `loony deploy` and `loony run`. User `print()` statements in scripts appear prefixed with `|`.

### Pass
```
  Running population.py...
    | Fetching data for 2024...
    | Fetching data for 2025...
    population.py — 18705 rows extracted
  Running 001_staged.sql...
    001_staged.sql — OK
```

### Failure
```
  Running population.py...
    | Traceback (most recent call last):
    |   ...
    | requests.exceptions.HTTPError: 403 Forbidden
    population.py — failed: ...
```
Check the streamed error for the root cause. Common issues:
- API auth error → check `loony secrets list` and `loony secrets set KEY=val`
- Type error in transforms → fix SQL cast in transforms/*.sql
- Missing table → check script produces the expected `raw_*` table

## Common Fixes

### Type Mismatch (varchar vs integer)
**Cause**: API returns numbers as strings or uses "-" for missing values. dlt infers varchar.

**Fix in transforms**:
```sql
-- In 001_staged.sql
COALESCE(NULLIF(column_name::text, '-')::integer, 0)
```

### Missing Column
**Cause**: schema.yaml declares a column the script doesn't produce.

**Fix**: Either:
- Add the column to the script's dlt resource
- Remove it from schema.yaml
- Add it in the staged transform: `now() AS loaded_at`

### Primary Key Duplicates
**Cause**: Primary key in schema.yaml doesn't uniquely identify rows.

**Fix**: Either:
- Add more columns to the primary key
- Add a DISTINCT ON in the staged transform
- Use `write_disposition="merge"` so dlt deduplicates

### Transform SQL Error
**Cause**: Usually a type mismatch between raw and staged columns, or referencing a table that doesn't exist yet.

**Fix**:
- Check the error message — it tells you which statement and line
- Ensure raw tables exist before staged transforms reference them
- Cast types explicitly — never rely on implicit casts from API data

### Pipeline Stuck / Corrupted State
**Cause**: A previous run crashed mid-way, leaving orphaned dlt state.

**Fix**:
```bash
loony reset --yes    # clears staging schemas + pending dlt packages
loony run            # re-run clean
```

## Debugging Commands

```bash
loony logs --last              # full detail of most recent run
loony logs --failed            # only failed runs
loony describe                 # check tables + row counts
loony query 'SELECT ...'      # run SQL to inspect data
loony cancel                   # kill a stuck running deployment
loony reset --yes              # clear corrupted state
```

## Validation Checklist Before Deploy

1. `loony validate` passes
2. All raw tables declared in schema.yaml with sync_mode + primary_key
3. All columns have type + description
4. Transforms handle type mismatches from API data
5. Materialized views have unique indexes for concurrent refresh
6. schema.yaml measures reference real columns in the views
