# Feature Engineering Pipeline Project

## Overview

This project was scaffolded using the ACC CLI — a developer tool that generates a standardised Databricks project structure.

By selecting the **Feature Engineering** template during `acc init`, the CLI automatically provisions a step-based feature pipeline architecture (Raw Data → Validate → Transform → Save Features) built on PySpark and Delta Lake, so you can focus on feature logic instead of setting up boilerplate.

## Project Structure
```

{{cookiecutter.project_slug}}/
│
├── configs/
│   ├── dev.yml
│   └── prod.yml
│
├── jobs/
│   └── feature_engineering_job.json
│
├── src/
│   ├── feature_engineering/
│   │   ├── __init__.py
│   │   ├── data_reader.py
│   │   ├── validator.py
│   │   ├── transformer.py
│   │   └── feature_writer.py
│   ├── utils/
│   │   ├── __init__.py
│   │   └── helpers.py
│   ├── __init__.py
│   └── main.py
│
├── tests/
│   ├── __init__.py
│   ├── test_data_reader.py
│   ├── test_validator.py
│   ├── test_transformer.py
│   └── test_feature_writer.py
│
├── .gitignore
├── __init__.py
├── Makefile
├── pyproject.toml
├── README.md
├── requirements.txt
└── requirements-dev.txt
```

## How It Works

### `configs/dev.yml` & `configs/prod.yml`
Environment-specific configuration files. Hold raw source paths, feature output paths, transformation parameters, and Feature Store settings for dev and prod environments.

### `jobs/feature_engineering_job.json`
Databricks job definition. Defines the cluster config, task dependencies, and entry point for running the feature engineering pipeline on Databricks.

### `src/feature_engineering/data_reader.py`
Entry point for the feature pipeline. Reads raw data as-is from the configured source (Delta table, parquet, CSV, etc.) and returns a DataFrame ready for validation.

### `src/feature_engineering/validator.py`
Validates raw data before any transformations are applied. Checks schema expectations, null thresholds, value ranges, and referential integrity. Raises early if data does not meet quality standards.

### `src/feature_engineering/transformer.py`
Core feature transformation logic. Applies encoding, scaling, aggregations, window functions, and business-specific feature derivations to produce the final feature set ready for model consumption.

### `src/feature_engineering/feature_writer.py`
Saves the transformed feature set to the configured output target — Databricks Feature Store, Unity Catalog, or a Delta table. Handles schema enforcement, upsert logic, and feature table metadata updates.

### `src/utils/helpers.py`
Shared utility functions used across the pipeline. Includes helpers for Spark session creation, config loading, logging setup, and common DataFrame operations.

### `src/main.py`
Main entry point for the pipeline. Loads config, initialises the Spark session, and orchestrates the Raw Data → Validate → Transform → Save Features execution sequence.

### `tests/test_data_reader.py`
Unit tests for `data_reader.py`. Validates raw ingestion logic, schema expectations, and that the returned DataFrame meets downstream validation requirements.

### `tests/test_validator.py`
Unit tests for `validator.py`. Validates null handling, schema checks, range validations, and that invalid datasets are correctly caught and rejected before transformation.

### `tests/test_transformer.py`
Unit tests for `transformer.py`. Validates feature derivation logic, encoding correctness, aggregation outputs, and the final feature schema against expected results.

### `tests/test_feature_writer.py`
Unit tests for `feature_writer.py`. Validates Feature Store write behaviour, upsert logic, schema enforcement, and metadata update correctness.

### `Makefile`
Convenience commands for common tasks — `make build`, `make test`, `make lint`, `make format`.

### `pyproject.toml`
Project metadata and dependency management. Defines the package name, version, and build system configuration.

#### Optional Cloud & Databricks Dependencies
If your pipeline needs to upload or interact with Amazon S3, Azure Data Lake Storage Gen2, Azure Blob Storage, or Databricks Unity Catalog, the required dependencies can be added directly to the generated pyproject.toml file.


### `requirements.txt` & `requirements-dev.txt`
Runtime and development dependencies respectively. `requirements-dev.txt` includes testing and linting libraries on top of the base requirements.

## Building a `.whl` File

To package this project as a Python wheel (`.whl`) for deployment to Databricks, run the following command from the project root:

\`\`\`
acc build
\`\`\`

This will compile the project into a distributable `.whl` file.