# Standard ETL Project

## Overview


This project was scaffolded using the **ACC CLI** a developer tool that generates
a standardised Databricks project structure.

By selecting the **Standard** template during `acc init`, the CLI automatically
provisions a layered architecture (Extract → Transform → Load) built on PySpark and
Delta Lake, so you can focus on writing business logic instead of setting up boilerplate.

## Project Structure

```
{{cookiecutter.project_slug}}/
│
├── configs/
│   ├── dev.yml
│   └── prod.yml
│
├── jobs/
│   └── databricks_job.json
│
├── src/
│   ├── task/
        └── __init__.py
│   │   └── extract.py
│   │   └── transform.py
│   │   └── load.py
│   │   └── pipeline.py
│
│   ├── utils/
        └── __init__.py
│   │   └── helpers.py
│   ├── __init__.py
│   ├── main.py
│
├── tests/
│   ├── __init__.py
│   ├── test_extract.py
│   ├── test_transform.py
│   ├── test_load.py
│   ├── test_pipeline.py
│
├── .gitignore
├── __init__.py
├── Makefile
├── pyproject.toml
├── README.md
├── requirements.txt
└── requirements-dev.txt
```

## How It Works

### `configs/dev.yml` & `configs/prod.yml`
Environment-specific configuration files. Hold source paths, target paths,
and format settings for dev and prod environments.

### `jobs/databricks_job.json`
Databricks job definition. Defines the cluster config, task dependencies,
and entry point for running the pipeline on Databricks.

### `src/tasks/extract.py`
Reads raw data from the configured source path and format (parquet, csv, etc.)
using Spark. Returns a raw DataFrame for downstream processing.

### `src/tasks/transform.py`
Applies business logic and data quality rules to the extracted DataFrame —
null handling, deduplication, type casting, and custom transformations.

### `src/tasks/load.py`
Writes the transformed DataFrame to the configured Delta Lake target path
in overwrite mode.

### `src/tasks/pipeline.py`
Orchestrates the full ETL sequence — calls `extract → transform → load` in order
via `run_pipeline(spark, config)`.

### `src/utils/helpers.py`
Shared utility functions used across tasks. Includes helpers for Spark session
creation, config loading, logging setup, and common DataFrame operations.

### `src/main.py`
Main entry point for the pipeline. Loads config, initialises the Spark session,
and triggers `run_pipeline()`.

### `tests/test_extract.py`
Unit tests for `extract.py`. Validates source reading logic and schema expectations.

### `tests/test_transform.py`
Unit tests for `transform.py`. Validates transformation logic and output DataFrame structure.

### `tests/test_load.py`
Unit tests for `load.py`. Validates Delta write behaviour and target path correctness.

### `tests/test_pipeline.py`
Integration tests for `pipeline.py`. Validates the full ETL sequence end-to-end.

### `Makefile`
Convenience commands for common tasks — `make build`, `make test`, `make deploy`.

### `pyproject.toml`
Project metadata and dependency management. Defines the package name, version,
and build system configuration.

### `requirements.txt` & `requirements-dev.txt`
Runtime and development dependencies respectively. `requirements-dev.txt` includes
testing and linting libraries on top of the base requirements.

### Building a `.whl` File
To package this project as a Python wheel (`.whl`) for deployment to Databricks,
run the following command from the project root:

```bash
acc build
```

This will compile the project into a distributable `.whl` file