# Why LakeLogic? ⚔️

Choosing the right tool for your data platform depends on your specific needs. LakeLogic is designed to work alongside or as a complement to industry standards like **dbt** and **Great Expectations**.

## Comparison Table

| Feature | LakeLogic 🛡️ | dbt Tests 🧪 | Great Expectations 🦒 |
| :--- | :--- | :--- | :--- |
| **Primary Focus** | Runtime Data Contracts | Transformation & Warehouse Testing | Data Observability & Profiling |
| **Execution Point** | **During** Data Movement (ETL/ELT) | **After** Data Loading (Warehouse) | **Validation Reports** (Post-Process) |
| **Engine Support** | Polars, Spark, DuckDB, Pandas | SQL Warehouse (Snowflake, BQ, etc.) | Python-based |
| **Handling Failures** | **Quarantine**: Detours bad rows in real-time | Fails the build or logs the error | Generates comprehensive quality reports |
| **Best Workflow** | **Prevention**: Shift-left data quality | **Transformation**: Model-driven testing | **Observability**: Long-term data health |

---

## 🤝 Better Together: LakeLogic & dbt

**dbt** is the industry standard for modeling and testing data inside the warehouse. LakeLogic complements dbt by handling the **runtime validation** before the data even reaches the warehouse.

-   **Complementary Strengths**: Use LakeLogic to "clean the front door" at the ingestion and Silver layers, and use dbt for complex business logic validation and cross-table reporting in the Gold layer.
-   **The Workflow**: LakeLogic ensures your **Silver** tables are Filtered, Cleaned, Transformed, and Enriched. dbt then builds your business models on top of these already-validated tables.

## 🤝 Better Together: LakeLogic & Great Expectations (GX)

**Great Expectations** is a powerful tool for deep data profiling and detailed observability documentation.

-   **Complementary Strengths**: GX is excellent for producing detailed "Data Docs" and profiling the overall health of a dataset. LakeLogic is optimized for **runtime enforcement**—making immediate, row-level decisions to quarantine data as it flows through a pipeline.
-   **The Workflow**: Use GX to build trust with stakeholders through rich visualization of data health, and use LakeLogic as the high-performance engine that enforces those health standards during every pipeline run.

## 🌍 Unified Governance: Cross-Platform Portability

In the modern enterprise, data often lives across multiple platforms. LakeLogic provides a **unified governance layer** that works consistently across your entire stack:

-   **Microsoft Fabric & Azure Synapse**: Use LakeLogic as the quality gate for your Spark-based notebooks and pipelines.
-   **Databricks**: Spark-based execution with Unity Catalog-compatible table logging (deeper integrations planned).
### Why this matters?
If you migrate from **Synapse to Fabric**, your Data Contracts stay exactly the same. 

---

## 💰 Modern Scale: The Polars/DuckDB Advantage

One of LakeLogic's core strengths is its engine-agnostic nature, allowing you to choose the most cost-effective compute for your data volume.

### 1. Optimizing Compute Costs
While Spark is the king of petabyte-scale data, many daily ingestion and validation tasks involve 1-100GB of data. For these workloads, a single-node container running **Polars** or **DuckDB** can be:
- **High Performance**: No JVM startup or cluster orchestration overhead.
- **Cost Effective**: Runs on a small Pod in AKS or a Serverless Container in ACA, reducing the need for multi-node clusters for simpler validation tasks.

### 2. Standardized Containerized Pipelines
LakeLogic provides a unified interface for data quality across your entire enterprise:
- **Dev/Test**: Run contracts locally on **DuckDB** for instantaneous debugging.
- **Mid-Scale Production**: Deploy to **AKS (Azure Kubernetes Service)** or **ACA (Azure Container Apps)** using **Polars**.
- **Large-Scale Production**: Seamlessly transition to **Databricks**, **Fabric**, or **Synapse** using **Spark**.

### 3. Spark-Native Scaling

When running on Spark, LakeLogic uses distributed operations throughout:

- **Merge/SCD2**: Uses native DataFrame joins (or Delta Lake `MERGE INTO`) instead of collecting to driver memory.
- **Counts/Metrics**: Single-pass aggregations to avoid multiple DAG executions.
- **No pandas bottleneck**: Data stays distributed for merge and SCD2 operations at any scale.

---

LakeLogic is about **Runtime Reliability** and **Infrastructure Flexibility**. It turns your Data Contract from passive documentation into an active layer of your Medallion architecture—Ensuring your **Silver** layer is Filtered, Cleaned, Transformed, and Enriched—working in harmony with your existing modeling and observability tools to ensure data integrity at every step across **Azure** and **AWS** Spark platforms. 🛡️🏛️

---

## From the Blog

Dive deeper into how LakeLogic compares to existing tools and approaches:

- :material-swap-horizontal: [**I Built LakeLogic Because 1,847 Lines of Great Expectations Weren't Telling Me Which Rows Failed**](https://lakelogic.org/blog/great-expectations-to-lakelogic/) — The founder story: why GX's approach didn't work and what replaced it.
- :material-cash-minus: [**Data Quality Management Without the Platform Tax**](https://lakelogic.org/blog/data-quality-management/) — Enterprise DQM tools vs. YAML contracts: cost, flexibility, and version control compared.
- :material-lightning-bolt: [**Stop the Spark Tax: One Data Contract, Any Engine**](https://lakelogic.org/blog/stop-the-spark-tax/) — Why engine portability eliminates logic drift between Spark, Polars, and DuckDB.

---

## Need More?

LakeLogic is fully open source and works standalone. For enterprise teams looking for additional capabilities, [LakeLogic.org](https://lakelogic.org) extends LakeLogic with AI-powered contract generation, visual lineage mapping, and collaborative governance workflows.
