Metadata-Version: 2.4
Name: conduit-core
Version: 1.1.2
Summary: The dbt of data ingestion - declarative, reliable, and testable data pipelines
Author-email: alexandro dronnen <alex12060309@gmail.com>
License: Business Source License 1.1
        
        Licensor: Alexandro Dronnen
        Licensed Work: Conduit Core
        The Licensed Work is (c) 2025 Alexandro Dronnen
        
        Additional Use Grant:
        You may use this software free of charge for any purpose — personal, educational, or commercial — 
        including within production environments and enterprise systems.
        
        The only restriction is that you may not use this software, or any modified version of it, 
        to offer a managed service, hosted service, or Software-as-a-Service (SaaS) 
        where the primary value of the service is derived from the Licensed Work itself, 
        without obtaining a commercial license from the Licensor.
        
        Change Date: 1 January 2030
        Change License: Apache License, Version 2.0
        
        On the Change Date, the Licensor will make the Licensed Work available under the 
        Change License, as provided in the Business Source License 1.1.
        
        -------------------------------------------------------------------------------
        This Business Source License (the "License") sets forth the terms under which the 
        Licensor is providing the Licensed Work. Using the Licensed Work in any form means 
        you agree to these terms.
        
        1. License Grant
           The Licensor hereby grants you the right to copy, modify, create derivative works, 
           redistribute, and make any internal or production use of the Licensed Work, 
           subject to the limitations and conditions set forth below.
        
        2. Restrictions
           The grant above does not include the right to use the Licensed Work for 
           offering it as part of a hosted, managed, or Software-as-a-Service (SaaS) product 
           where the primary value of the service is derived from the Licensed Work itself, 
           without obtaining a commercial license from the Licensor.
        
           Example allowed uses:
           - Embedding or using Conduit Core internally in your company's data pipelines.
           - Running Conduit Core in production for your own systems or customers.
           
           Example restricted uses:
           - Hosting Conduit Core as a paid cloud service or platform.
           - Building a SaaS product that uses Conduit Core as its core engine.
        
        3. Change Date and License
           On the Change Date specified above, the Licensed Work will automatically be made 
           available under the Change License, which will then govern its use.
        
        4. Notices
           You must include a copy of this License with all copies of the Licensed Work, 
           and you must include attribution to the Licensor in any derivative works.
        
        5. Disclaimer
           THE LICENSED WORK IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS OF ANY 
           KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF 
           MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO 
           EVENT SHALL THE LICENSOR BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, 
           WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, ARISING FROM OR OUT OF 
           THE USE OR INABILITY TO USE THE LICENSED WORK.
        
        -------------------------------------------------------------------------------
        For more information on the Business Source License, see:
        https://mariadb.com/bsl11
        
Project-URL: Homepage, https://github.com/alexD1990/conduit-core
Project-URL: Documentation, https://github.com/alexD1990/conduit-core#readme
Project-URL: Repository, https://github.com/alexD1990/conduit-core
Project-URL: Issues, https://github.com/alexD1990/conduit-core/issues
Keywords: etl,data-ingestion,pipeline,dbt,data-engineering
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer[all]>=0.19.2
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: pydantic>=2.11.9
Requires-Dist: pyodbc>=5.2.0
Requires-Dist: python-dotenv>=1.1.1
Requires-Dist: pandas>=2.0.0
Requires-Dist: pyarrow>=14.0.0
Requires-Dist: boto3>=1.35.0
Requires-Dist: psycopg2-binary>=2.9.0
Requires-Dist: snowflake-connector-python>=3.0.0
Requires-Dist: google-cloud-bigquery<4.0.0,>=3.38.0
Requires-Dist: mysql-connector-python<10.0.0,>=9.5.0
Provides-Extra: dev
Requires-Dist: pytest>=8.4.2; extra == "dev"
Requires-Dist: moto[s3]>=5.0.0; extra == "dev"
Requires-Dist: psutil>=7.1.2; extra == "dev"
Dynamic: license-file

# Conduit Core

**The dbt of data ingestion** — declarative, reliable, testable data pipelines in Python.

[![PyPI](https://img.shields.io/pypi/v/conduit-core)](https://pypi.org/project/conduit-core/)
[![Tests](https://img.shields.io/badge/tests-190%2B%20passing-brightgreen)](https://github.com/alexD1990/conduit-core)
[![License](https://img.shields.io/badge/license-BSL%201.1-blue)](LICENSE)

## Install

```bash
pip install conduit-core
```

## Quick Start with Templates

```bash
# List all available templates
conduit template list

# Generate a CSV → Snowflake pipeline
conduit template csv_to_snowflake > pipeline.yml

# Edit pipeline.yml (update marked fields)

# Run your pipeline
conduit run pipeline.yml
```

That's it.

## Features

- **8 Connectors** — CSV, JSON, Parquet, S3, PostgreSQL, MySQL, Snowflake, BigQuery
- **Template System** — Zero-config YAML generation for all connector combinations
- **Schema Management** — Auto-inference, validation, evolution, drift detection
- **Atomic Operations** — All-or-nothing writes with automatic rollback
- **Checkpoint/Resume** — Pick up where you left off after failures
- **Dead Letter Queue** — Quarantine bad records without stopping pipelines
- **Quality Checks** — Validate data with not_null, regex, range, custom rules
- **Incremental Loading** — Sync only new/changed records
- **Environment Variables** — Secure credential management via `${VAR}` in YAML
- **Progress Tracking** — Real-time progress bars and batch statistics
- **Audit Trail** — Complete run history with manifest files
- **Preflight Validation** — Test connections and schemas before running
- **Retry Logic** — Automatic retries with exponential backoff
- **Type Coercion** — Smart type conversion between sources and destinations
- **190+ Tests** — Comprehensive test coverage with real cloud services

## CLI Commands

```bash
conduit template list                    # List all templates
conduit template info <name>             # Show template details
conduit template <name>                  # Generate template YAML
conduit run <pipeline.yml>               # Run a pipeline
conduit manifest --last                  # Show last run details
```

## Example: PostgreSQL → Snowflake

```bash
# Generate template
conduit template postgresql_to_snowflake > pipeline.yml

# Configure credentials in .env
echo "PG_PASSWORD=xxx" >> .env
echo "SNOWFLAKE_PASSWORD=yyy" >> .env

# Run
conduit run pipeline.yml
```

## Documentation

- **[Quick Start Guide](docs/QUICK_START.md)** — Common patterns and workflows
- **[Connector Reference](docs/CONNECTORS.md)** — Configuration for all 8 connectors

## License

Business Source License 1.1 (converts to Apache 2.0 on January 1, 2030)

## Links

- **PyPI:** https://pypi.org/project/conduit-core/
- **GitHub:** https://github.com/alexD1990/conduit-core
- **Issues:** https://github.com/alexD1990/conduit-core/issues
