# DAG Development Guide

This document provides guidelines for creating Airflow DAGs for GCP pipeline deployments. It covers naming conventions, template usage, pattern selection, and configuration requirements.

## Overview

Each pipeline system requires a set of four DAGs that work together:

| DAG | Purpose |
|-----|---------|
| **PubSub Trigger DAG** | Senses `.ok` file arrivals from GCS via Pub/Sub and initiates the pipeline |
| **ODP Load DAG** | Submits Dataflow jobs to load raw files into ODP tables in BigQuery |
| **FDP Transform DAG** | Runs dbt transformations to produce Foundation Data Products |
| **Error Handling DAG** | Monitors `job_control` and manages failures and retries |

---

## Naming Conventions

Consistency in naming is critical for automated monitoring and maintenance.

### System Identifier

Each system has two forms of its identifier:

| Form | Use Case | Example |
|------|----------|---------|
| `SYSTEM_ID` (uppercase) | Python constants, environment variables | `GENERIC` |
| `system_id` (lowercase) | DAG IDs, file names, GCS paths, Pub/Sub topics | `generic` |

> **Convention:** Use `<SYSTEM_ID>` and `<system_id>` as placeholders in templates. Replace both when scaffolding a new system.

### DAG IDs

Follow the pattern: `{system_id}_{dag_purpose}_dag`

```
generic_pubsub_trigger_dag
generic_odp_load_dag
generic_fdp_transform_dag
generic_error_handling_dag
```

### File Names

Match the DAG ID exactly:

```
deployments/generic-orchestration/dags/
├── generic_pubsub_trigger_dag.py
├── generic_odp_load_dag.py
├── generic_fdp_transform_dag.py
└── generic_error_handling_dag.py
```

### Task IDs

Use `snake_case`, descriptive names:

```
pull_pubsub_messages
validate_ok_file
create_job_control_record
run_dataflow_pipeline
check_entity_dependencies
run_dbt_models
mark_job_complete
```

---

## Using Templates

Templates are located in `templates/dags/`. Copy and replace placeholders for each new system.

### Manual Scaffolding

```bash
SYSTEM="myapp"

# 1. Create the orchestration folder
mkdir -p deployments/${SYSTEM}-orchestration/dags

# 2. Copy DAG templates
cp templates/dags/template_pubsub_trigger_dag.py \
   deployments/${SYSTEM}-orchestration/dags/${SYSTEM}_pubsub_trigger_dag.py

cp templates/dags/template_odp_load_dag.py \
   deployments/${SYSTEM}-orchestration/dags/${SYSTEM}_odp_load_dag.py

cp templates/dags/template_fdp_transform_dag.py \
   deployments/${SYSTEM}-orchestration/dags/${SYSTEM}_fdp_transform_dag.py

cp templates/dags/template_error_handling_dag.py \
   deployments/${SYSTEM}-orchestration/dags/${SYSTEM}_error_handling_dag.py

# 3. Replace placeholders throughout
find deployments/${SYSTEM}-orchestration/dags/ -name "*.py" \
  -exec sed -i "s/<SYSTEM_ID>/MYAPP/g; s/<system_id>/${SYSTEM}/g" {} \;

# 4. Copy the CI/CD workflow template
cp templates/cicd/template_deploy_workflow.yml \
   .github/workflows/deploy-${SYSTEM}.yml
```

After copying, open each file and set system-specific values:

| Placeholder | Replace With | Example |
|-------------|-------------|---------|
| `<SYSTEM_ID>` | Uppercase system constant | `MYAPP` |
| `<system_id>` | Lowercase system identifier | `myapp` |
| `REQUIRED_ENTITIES` | Entity list for this system | `['customers', 'accounts']` |
| `<dbt_selector>` | dbt model selector | `staging.myapp` |

---

## Pattern Selection

Choose the orchestration pattern based on your transformation requirements.

### MAP Pattern (Single Entity)

Use when a system provides **one file type** that maps directly to one ODP table and one FDP model.

Example: Applications entity → `odp_generic.applications` → `fdp_generic.applications`

```python
# In the ODP Load DAG
SYSTEM_ID = "MYAPP"
REQUIRED_ENTITIES = ["applications"]  # Only one entity

# EntityDependencyChecker resolves immediately for this single entity
dependency_met = EntityDependencyChecker(
    system_id=SYSTEM_ID,
    required_entities=REQUIRED_ENTITIES,
).check(run_id=run_id, extract_date=extract_date)
```

### JOIN Pattern (Multi-Entity Coordination)

Use when the FDP transformation **requires multiple entities** to be present before dbt can run (e.g., a JOIN across Customers + Accounts + Decision).

```python
# In the ODP Load DAG
SYSTEM_ID = "GENERIC"
REQUIRED_ENTITIES = ["customers", "accounts", "decision"]

# EntityDependencyChecker waits until all three entities are loaded
# for the same extract_date before triggering the FDP transform DAG
dependency_met = EntityDependencyChecker(
    system_id=SYSTEM_ID,
    required_entities=REQUIRED_ENTITIES,
).check(run_id=run_id, extract_date=extract_date)
```

The `ODP Load DAG` runs independently for each arriving file. The `BranchPythonOperator` only triggers `FDP Transform DAG` when the last required entity for the extract date is loaded.

---

## Pub/Sub Configuration

```python
# Topic follows this naming pattern
PUBSUB_TOPIC = f"projects/{PROJECT_ID}/topics/{system_id}-file-notifications"

# Subscription
PUBSUB_SUBSCRIPTION = f"projects/{PROJECT_ID}/subscriptions/{system_id}-file-sub"
```

Ensure the GCS landing bucket has OBJECT_FINALIZE notifications configured to publish to this topic. See [GCP Deployment Guide](./GCP_DEPLOYMENT_GUIDE.md) for infrastructure setup.

---

## Testing

### 1. DAG Parse Validation

```bash
cd deployments/{system_id}-orchestration
python dags/{system_id}_pubsub_trigger_dag.py
python dags/{system_id}_odp_load_dag.py
python dags/{system_id}_fdp_transform_dag.py
```

No output = no syntax errors.

### 2. Unit Tests

```bash
pytest deployments/{system_id}-orchestration/tests/unit/ -v
```

### 3. Manual Trigger via Airflow UI

1. Navigate to the Airflow UI (Cloud Composer or `kubectl port-forward svc/airflow-webserver 8080:8080`)
2. Trigger the PubSub DAG with a mock `dag_run.conf`:

```json
{
  "file_metadata": {
    "data_file": "gs://{PROJECT_ID}-generic-int-landing/generic/customers/test_customers_20260101.csv",
    "trigger_file": "gs://{PROJECT_ID}-generic-int-landing/generic/customers/customers.csv.ok",
    "entity": "customers",
    "extract_date": "20260101"
  }
}
```

---

## Common Placeholder Reference

| Placeholder | Form | Example |
|-------------|------|---------|
| `<SYSTEM_ID>` | Uppercase constant | `GENERIC` |
| `<system_id>` | Lowercase identifier | `generic` |
| `REQUIRED_ENTITIES` | Python list | `['customers', 'accounts', 'decision']` |
| `<dbt_selector>` | dbt model path | `staging.generic` |
| `<extract_date_nodash>` | Airflow template | `{{ ds_nodash }}` |

---

## Related Documentation

| Guide | Description |
|-------|-------------|
| [Creating a New Deployment](./CREATING_NEW_DEPLOYMENT_GUIDE.md) | End-to-end scaffolding for all three units |
| [E2E Functional Flow](./E2E_FUNCTIONAL_FLOW.md) | How DAGs coordinate ingestion → ODP → FDP |
| [Error Handling Guide](./ERROR_HANDLING_GUIDE.md) | Error classification and dead letter patterns |
| [GKE Deployment Guide](./GKE_DEPLOYMENT_GUIDE.md) | Self-hosted Airflow on GKE (alternative pattern) |
