# Release History

## [0.0.18] - 2023-11-14
**Added**
- Support for externally managed users and groups

## [0.0.17] - 2023-11-13
**Fixed**
- Data Event model to support timestamp from string

**Added**
- Option to exclude timestamp from data event filepath
- Selects, filter, watermark options for TableDataSource
- Support for joins in Table model
- Silver Star builder

**Breaking Change**
- Refactored Table to move all building configuration into a `TableBuilder` model

## [0.0.16] - 2023-11-08
**Added**
- UDFs property to pipeline model

**Breaking Change**
- Refactored InitScript model into the more general WorkspaceFile

## [0.0.15] - 2023-11-07
**Added**
- Support for CDC table data source
- Support for SCD table

**Updated**
- Automatic catalog and schema assignation to source table from table and pipeline

## [0.0.14] - 2023-11-06
**Fixed**
- Pyspark imports when pyspark not installed

## [0.0.13] - 2023-11-06
**Fixed**
- Column `spark_func_args` parsing
- Column `spark_func_kwargs` parsing
- Support for `_any` column type

**Added**
- `schema_flat` and `has_column` DataFrame extensions for spark connect 

- **Update**
- Table Data Source to support tables external to the DLT pipeline

## [0.0.12] - 2023-11-06
**Fixed**
- Pyspark imports when pyspark not installed


## [0.0.11] - 2023-11-05
**Fixed**
- Deployment of pipeline configuration file

**Added**
- Spark optional dependencies
- Spark unit tests
- Functions library
- Support for custom functions in silver processing

**Breaking Change**
- Changed API for table columns definition
- Removed databricks-sdk as dependency

## [0.0.10] - 2023-10-31
**Fixed**
- `df_has_column` handling of arrays when column name contains a digit

## [0.0.9] - 2023-10-27
**Updated**
- df_has_column support for column names with backticks (`)

## [0.0.8] - 2023-10-24
**Added**
- Model pulumi dump method
- Support for variables in yaml files

**Breaking Changes**
- Deprecated metadata SQL methods

## [0.0.7] - 2023-10-20

**Updated**
- Bronze template notebook to leverage configuration file
- Silver template notebook to leverage configuration file

**Added**
- compute.Cluster model
- compute.InitScript model
- compute.Job model
- compute.Notebook model
- compute.Pipeline model
- compute.Warehouse model
- secrets.Secret model
- secrets.SecretScope model
- Pipeline configuration file management

**Breaking Change**
- Refactored landing mount to landing root and changed default configuration to volumes


## [0.0.6] - 2023-10-10
**Added**
- User, service principal and users group models
- Grants models
- Pulumi resources engine for user, group, catalog, schema, volume and associated grants

**Breaking Change**
- Renamed database objects to schema to be aligned with Databricks recommendations

## [0.0.5] - 2023-09-28
**Added**
- Processing method for silver tables
- Silver DLT pipeline template notebook
- `df_schema_flat` function
- `_any` as as supported type for Spark functions

**Updated**
- Excluded fields for `DataEvent` model_dump
- `df_hascolumn` function to support Spark 3.5

**Fixed**
- Table Metadata included derived properties for the columns

## [0.0.4] - 2023-09-27
**Updated**
- Refactored dlt module to support DBR < 13 and clusters in shared access mode

**Fixed**
- model_validate_yaml() for Pipeline model
- table data insert when no data is available

**Breaking Change**
- Removed spark from required dependencies

## [0.0.3] - 2023-09-25
**Added**
- Data Event cloud storage writers (Azure, AWS and Databricks mount)

## [0.0.2] - 2023-09-24
**Added**
- Data Event class
- Data Source classes
- Pipeline class
- Support for BRONZE transformations

## [0.0.1] - 2023-07-13
**Added**
- Initial pypi release
