# PyCharter

<p align="center">
  <strong>Data Contract Management, ETL Pipelines, and Quality Assurance for Python</strong>
</p>

<p align="center">
  <a href="https://pypi.org/project/pycharter/"><img src="https://img.shields.io/pypi/v/pycharter.svg" alt="PyPI version"></a>
  <a href="https://pypi.org/project/pycharter/"><img src="https://img.shields.io/pypi/pyversions/pycharter.svg" alt="Python versions"></a>
  <a href="https://github.com/optophi/pycharter/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License"></a>
  <a href="https://github.com/optophi/pycharter/actions"><img src="https://github.com/optophi/pycharter/workflows/Test/badge.svg" alt="Test"></a>
</p>

---

**PyCharter** is a comprehensive **data contract management platform** for Python that enables you to define, store, version, enforce, and monitor data contracts throughout your data pipelines.

## Key Features

<div class="grid cards" markdown>

-   :material-pipe:{ .lg .middle } **ETL Pipelines**

    ---

    Build data pipelines with a fluent `|` operator. 13 built-in extractors for HTTP, files, databases, cloud storage, streaming (SSE, WebSocket), and messaging (Kafka, RabbitMQ, SQS).

    [:octicons-arrow-right-24: ETL Tutorial](tutorials/etl-pipelines.md)

-   :material-file-document-check:{ .lg .middle } **Data Contracts**

    ---

    Define formal agreements specifying data structure, quality rules, and governance policies.

    [:octicons-arrow-right-24: Contracts Tutorial](tutorials/contracts-validation.md)

-   :material-chart-line:{ .lg .middle } **Quality Assurance**

    ---

    Monitor data quality with metrics, track violations, and set threshold alerts.

    [:octicons-arrow-right-24: Quality Tutorial](tutorials/quality-monitoring.md)

-   :material-database:{ .lg .middle } **Schema Registry**

    ---

    Centralized storage for schemas with PostgreSQL, SQLite, MongoDB, or Redis backends.

    [:octicons-arrow-right-24: Metadata Tutorial](tutorials/metadata-store.md)

</div>

## Quick Example

### ETL Pipeline with `|` Operator

```python
import asyncio
from pycharter import Pipeline, HTTPExtractor, PostgresLoader, Rename, Filter

# Build pipeline with fluent syntax
pipeline = (
    Pipeline(HTTPExtractor(url="https://api.example.com/users"))
    | Rename({"user_name": "name", "user_email": "email"})
    | Filter(lambda r: r.get("active", False))
    | PostgresLoader(connection_string="postgresql://...", table="users")
)

# Run the pipeline
result = asyncio.run(pipeline.run())
print(f"Loaded {result.rows_loaded} rows")
```

### Data Validation

```python
from pycharter import Validator

# Create validator from contract file
validator = Validator.from_file("user_contract.yaml")

# Validate data
result = validator.validate({"name": "Alice", "age": 30, "email": "alice@example.com"})

if result.is_valid:
    print(f"Valid: {result.data}")
else:
    print(f"Errors: {result.errors}")
```

### Quality Check

```python
from pycharter import QualityCheck, QualityThresholds

# Run quality check with thresholds
check = QualityCheck(store=store)
report = check.run(
    schema_id="user_schema_v1",
    data=records,
    thresholds=QualityThresholds(min_overall_score=95.0)
)

print(f"Quality Score: {report.quality_score.overall_score}/100")
print(f"Passed: {report.passed}")
```

## Installation

=== "Core (validation only)"

    ```bash
    pip install pycharter
    ```

=== "With API server"

    ```bash
    pip install pycharter[api]
    ```

=== "With Web UI"

    ```bash
    pip install pycharter[ui]
    ```

=== "Full installation"

    ```bash
    pip install pycharter[api,ui,etl]
    ```

## Architecture Overview

```mermaid
graph TB
    subgraph Input["Data Sources"]
        HTTP[HTTP/API]
        Files[Files]
        DB[(Database)]
        Cloud[Cloud Storage]
        Stream[SSE / WebSocket]
        MQ[Kafka / RabbitMQ / SQS]
    end

    subgraph PyCharter["PyCharter"]
        Extract[Extractors]
        Transform[Transformers]
        Load[Loaders]
        Validate[Validator]
        Quality[Quality Check]
        Store[(Metadata Store)]
    end

    subgraph Output["Destinations"]
        PG[(PostgreSQL)]
        File[Files]
        S3[Cloud Storage]
    end

    HTTP --> Extract
    Files --> Extract
    DB --> Extract
    Cloud --> Extract
    Stream --> Extract
    MQ --> Extract

    Extract --> Transform
    Transform --> Validate
    Validate --> Load
    Validate --> Quality

    Store --> Validate
    Quality --> Store

    Load --> PG
    Load --> File
    Load --> S3
```

## Next Steps

<div class="grid cards" markdown>

-   :material-rocket-launch:{ .lg .middle } **Get Started**

    ---

    Install PyCharter and run your first pipeline in minutes.

    [:octicons-arrow-right-24: Installation](getting-started/installation.md)

-   :material-school:{ .lg .middle } **Learn**

    ---

    Follow step-by-step tutorials for each major feature.

    [:octicons-arrow-right-24: Tutorials](tutorials/index.md)

-   :material-api:{ .lg .middle } **API Reference**

    ---

    Detailed documentation for all classes and functions.

    [:octicons-arrow-right-24: API Reference](api/index.md)

-   :material-github:{ .lg .middle } **Contribute**

    ---

    Help improve PyCharter by contributing code or documentation.

    [:octicons-arrow-right-24: Contributing](contributing.md)

</div>
