Metadata-Version: 2.4
Name: conduit-core
Version: 1.0.0
Summary: The dbt of data ingestion - declarative, reliable, and testable data pipelines
Author-email: alexandro dronnen <alex12060309@gmail.com>
License: Business Source License 1.1
        
        Licensor: Alexandro Dronnen
        Licensed Work: Conduit Core
        The Licensed Work is (c) 2025 Alexandro Dronnen
        
        Additional Use Grant:
        You may use this software free of charge for any purpose — personal, educational, or commercial — 
        including within production environments and enterprise systems.
        
        The only restriction is that you may not use this software, or any modified version of it, 
        to offer a managed service, hosted service, or Software-as-a-Service (SaaS) 
        where the primary value of the service is derived from the Licensed Work itself, 
        without obtaining a commercial license from the Licensor.
        
        Change Date: 1 January 2030
        Change License: Apache License, Version 2.0
        
        On the Change Date, the Licensor will make the Licensed Work available under the 
        Change License, as provided in the Business Source License 1.1.
        
        -------------------------------------------------------------------------------
        This Business Source License (the "License") sets forth the terms under which the 
        Licensor is providing the Licensed Work. Using the Licensed Work in any form means 
        you agree to these terms.
        
        1. License Grant
           The Licensor hereby grants you the right to copy, modify, create derivative works, 
           redistribute, and make any internal or production use of the Licensed Work, 
           subject to the limitations and conditions set forth below.
        
        2. Restrictions
           The grant above does not include the right to use the Licensed Work for 
           offering it as part of a hosted, managed, or Software-as-a-Service (SaaS) product 
           where the primary value of the service is derived from the Licensed Work itself, 
           without obtaining a commercial license from the Licensor.
        
           Example allowed uses:
           - Embedding or using Conduit Core internally in your company's data pipelines.
           - Running Conduit Core in production for your own systems or customers.
           
           Example restricted uses:
           - Hosting Conduit Core as a paid cloud service or platform.
           - Building a SaaS product that uses Conduit Core as its core engine.
        
        3. Change Date and License
           On the Change Date specified above, the Licensed Work will automatically be made 
           available under the Change License, which will then govern its use.
        
        4. Notices
           You must include a copy of this License with all copies of the Licensed Work, 
           and you must include attribution to the Licensor in any derivative works.
        
        5. Disclaimer
           THE LICENSED WORK IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS OF ANY 
           KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF 
           MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO 
           EVENT SHALL THE LICENSOR BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, 
           WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, ARISING FROM OR OUT OF 
           THE USE OR INABILITY TO USE THE LICENSED WORK.
        
        -------------------------------------------------------------------------------
        For more information on the Business Source License, see:
        https://mariadb.com/bsl11
        
Project-URL: Homepage, https://github.com/alexD1990/conduit-core
Project-URL: Documentation, https://github.com/alexD1990/conduit-core#readme
Project-URL: Repository, https://github.com/alexD1990/conduit-core
Project-URL: Issues, https://github.com/alexD1990/conduit-core/issues
Keywords: etl,data-ingestion,pipeline,dbt,data-engineering
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer[all]>=0.19.2
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: pydantic>=2.11.9
Requires-Dist: pyodbc>=5.2.0
Requires-Dist: python-dotenv>=1.1.1
Requires-Dist: pandas>=2.0.0
Requires-Dist: pyarrow>=14.0.0
Requires-Dist: boto3>=1.35.0
Requires-Dist: psycopg2-binary>=2.9.0
Requires-Dist: snowflake-connector-python>=3.0.0
Requires-Dist: google-cloud-bigquery<4.0.0,>=3.38.0
Requires-Dist: mysql-connector-python<10.0.0,>=9.5.0
Provides-Extra: dev
Requires-Dist: pytest>=8.4.2; extra == "dev"
Requires-Dist: moto[s3]>=5.0.0; extra == "dev"
Requires-Dist: psutil>=7.1.2; extra == "dev"
Dynamic: license-file

# Conduit Core  
![Python](https://img.shields.io/badge/python-3.9+-blue.svg) ![License](https://img.shields.io/badge/license-BSL%201.1-blue.svg) ![Tests](https://img.shields.io/badge/tests-22%20passing-brightgreen.svg) ![Coverage](https://img.shields.io/badge/coverage-85%25-green.svg) ![Code Style](https://img.shields.io/badge/code%20style-black-black.svg)

**Declarative data ingestion framework for Python — inspired by dbt's approach to analytics engineering.**

Conduit Core is an open-source Python framework for building production-grade data pipelines. Move data between CSV, JSON, Parquet, PostgreSQL, MySQL, S3, Snowflake, and BigQuery with YAML configuration, automatic retries, checkpoints, and comprehensive testing.

## Features

- **Declarative YAML Configuration** — Define sources, destinations, and transformations without writing Python
- **8 Connectors** — CSV, JSON, Parquet, S3, PostgreSQL, MySQL, Snowflake, BigQuery
- **Production-Ready Reliability** — Atomic operations, automatic retries, dead letter queue, checkpoint/resume
- **Schema Management** — Auto-inference, validation, evolution tracking, and quality checks
- **Developer Experience** — Rich progress bars, dry-run mode, audit trail, and comprehensive testing (141+ tests)

## Quick Start

```bash
pip install conduit-core
```

Create `ingest.yml`:

```yaml
sources:
  - name: users_csv
    type: csv
    path: "./users.csv"

destinations:
  - name: users_json
    type: json
    path: "./users.json"

resources:
  - name: csv_to_json
    source: users_csv
    destination: users_json
```

Run:

```bash
conduit run csv_to_json
```

## Documentation

- **[Installation & Setup](docs/installation.md)** — Installation, environment setup, and credentials
- **[Usage Guide](docs/usage.md)** — CLI commands, pipeline configuration, and examples
- **[Connectors](docs/connectors.md)** — Configuration for all 8 connectors
- **[Features](docs/features.md)** — Reliability, checkpoints, DLQ, incremental loads
- **[Architecture](docs/architecture.md)** — Design patterns, tech stack, and structure
- **[Data Quality](docs/data-quality.md)** — Schema validation and quality checks
- **[Contributing](docs/contributing.md)** — Development setup and contribution guidelines
- **[Roadmap](docs/roadmap.md)** — Future features and connector expansion

## Example: Incremental PostgreSQL → Snowflake

```yaml
sources:
  - name: pg_source
    type: postgresql
    host: ${PG_HOST}
    database: analytics
    user: ${PG_USER}
    password: ${PG_PASSWORD}

destinations:
  - name: sf_dest
    type: snowflake
    account: ${SF_ACCOUNT}
    warehouse: COMPUTE_WH
    database: ANALYTICS
    schema: PUBLIC

resources:
  - name: incremental_load
    source: pg_source
    destination: sf_dest
    query: "SELECT * FROM events"
    incremental_column: created_at
    write_mode: append
```

Run with validation:

```bash
conduit preflight incremental_load
conduit run incremental_load
```

## Why Conduit Core?

Built for data engineers who need the simplicity of declarative configuration with the reliability of enterprise-grade systems:

- ✅ **No boilerplate** — YAML instead of Python for 90% of use cases
- ✅ **Crash-safe** — Atomic writes, transactions, and checkpoint/resume
- ✅ **Observable** — Audit trail, progress bars, and detailed error messages
- ✅ **Extensible** — Plugin architecture for custom connectors
- ✅ **Battle-tested** — 141+ tests covering edge cases and failure modes

## Community

- **Issues & Bugs**: [GitHub Issues](https://github.com/conduit-core/conduit-core/issues)
- **Feature Requests**: [GitHub Discussions](https://github.com/conduit-core/conduit-core/discussions)
- **Contributing**: See [CONTRIBUTING.md](docs/contributing.md)

## License

BSL 1.1 (Business Source License) — See [LICENSE](LICENSE) for details
