# Medallion Architecture Project

## Overview

This project was scaffolded using the **ACC CLI** a developer tool that generates
a standardised Databricks project structure.

By selecting the **Medallion** template during `acc init`, the CLI automatically
provisions a layered architecture (Bronze → Silver → Gold) built on PySpark and
Delta Lake, so you can focus on writing business logic instead of setting up boilerplate.

## Project Structure

```
{{cookiecutter.project_slug}}/
│
├── configs/
│   ├── dev.yml
│   └── prod.yml
│
├── jobs/
│   └── databricks_job.json
│
├── src/
│   ├── bronze/
        └── __init__.py
│   │   └── ingest.py
│   ├── silver/
        └── __init__.py
│   │   └── cleanse.py
│   ├── gold/
        └── __init__.py
│   │   └── marts.py
│   ├── utils/
        └── __init__.py
│   │   └── helpers.py
│   ├── __init__.py
│   ├── main.py
│
├── tests/
│   ├── __init__.py
│   ├── test_bronze.py
│   ├── test_silver.py
│   ├── test_gold.py
│
├── .gitignore
├── __init__.py
├── Makefile
├── pyproject.toml
├── README.md
├── requirements.txt
└── requirements-dev.txt
```
## How It Works

### `configs/dev.yml` & `configs/prod.yml`
Environment-specific configuration files. Hold source paths, target paths, and
layer-specific settings (bronze_path, silver_path, gold_path) for dev and prod environments.

### `jobs/databricks_job.json`
Databricks job definition. Defines the cluster config, task dependencies,
and entry point for running the pipeline on Databricks.

### `src/bronze/ingest.py`
Entry point for the Bronze layer. Reads raw data as-is from the source
(parquet, csv, etc.) and writes it to the Bronze table without any transformations.

### `src/silver/cleanse.py`
Processes Bronze data into the Silver layer. Applies data quality rules,
drops nulls, removes duplicates, standardizes schema and writes to the Silver Delta table.

### `src/gold/marts.py`
Aggregates Silver data into the Gold layer. Builds business-ready data marts
with grouped metrics and KPIs, writing the final output to the Gold Delta table.

### `src/utils/helpers.py`
Shared utility functions used across layers. Includes helpers for Spark session
creation, config loading, logging setup, and common DataFrame operations.

### `src/main.py`
Main entry point for the pipeline. Loads config, initialises the Spark session,
and orchestrates the Bronze → Silver → Gold execution sequence.

### `tests/test_bronze.py`
Unit tests for `ingest.py`. Validates raw ingestion logic, schema expectations,
and Bronze Delta write behaviour.

### `tests/test_silver.py`
Unit tests for `cleanse.py`. Validates null handling, deduplication logic,
and Silver layer output schema.

### `tests/test_gold.py`
Unit tests for `marts.py`. Validates aggregation logic and Gold mart output
against expected results.

### `Makefile`
Convenience commands for common tasks — `make build`, `make test`, `make deploy`.

### `pyproject.toml`
Project metadata and dependency management. Defines the package name, version,
and build system configuration.

### `requirements.txt` & `requirements-dev.txt`
Runtime and development dependencies respectively. `requirements-dev.txt` includes
testing and linting libraries on top of the base requirements.


### Building a `.whl` File
To package this project as a Python wheel (`.whl`) for deployment to Databricks,
run the following command from the project root:

```bash
acc build
```

This will compile the project into a distributable `.whl` file