# How-To Guides

Practical guides for common tasks and advanced patterns.

## Available Guides

<div class="grid cards" markdown>

-   :material-hammer:{ .lg .middle } **Building the Package**

    ---

    Build the Python package with pre-built UI static files.

    [:octicons-arrow-right-24: Read Guide](building.md)

-   :material-cog:{ .lg .middle } **Configuration & Database Setup**

    ---

    Database connection configuration, initialization, and migrations.

    [:octicons-arrow-right-24: Read Guide](configuration.md)

-   :material-map-marker-path:{ .lg .middle } **Data Journey**

    ---

    Complete workflow from contract to runtime validation.

    [:octicons-arrow-right-24: Read Guide](data-journey.md)

-   :material-table:{ .lg .middle } **Database ERD**

    ---

    Entity relationship diagrams for the PyCharter schema.

    [:octicons-arrow-right-24: Read Guide](database-erd.md)

-   :material-database-edit:{ .lg .middle } **Database Operations**

    ---

    ETL write methods: insert, upsert, replace, truncate_and_load, and more.

    [:octicons-arrow-right-24: Read Guide](database-operations.md)

-   :material-puzzle:{ .lg .middle } **Custom Extractors**

    ---

    Build extractors for custom data sources.

    [:octicons-arrow-right-24: Read Guide](custom-extractors.md)

-   :material-function:{ .lg .middle } **Custom Transformers**

    ---

    Create reusable transformation functions.

    [:octicons-arrow-right-24: Read Guide](custom-transformers.md)

-   :material-check-circle:{ .lg .middle } **Custom Validators**

    ---

    Extend validation with custom rules.

    [:octicons-arrow-right-24: Read Guide](custom-validators.md)

-   :material-sitemap:{ .lg .middle } **Domain Models and Lifecycle**

    ---

    Domain entity contracts with optional lifecycle binding for FSM engines.

    [:octicons-arrow-right-24: Read Guide](domain-models-and-lifecycle.md)

-   :material-database-cog:{ .lg .middle } **Database Configuration**

    ---

    Configure and optimize database backends.

    [:octicons-arrow-right-24: Read Guide](database-config.md)

-   :material-rocket:{ .lg .middle } **Production Deployment**

    ---

    Deploy PyCharter in production environments.

    [:octicons-arrow-right-24: Read Guide](production.md)

-   :material-robot:{ .lg .middle } **Validation Worker**

    ---

    Run validation jobs asynchronously with the database-backed worker process.

    [:octicons-arrow-right-24: Read Guide](worker.md)

-   :material-swap-horizontal:{ .lg .middle } **ETL Transformations**

    ---

    Simple operations, JSONata, and custom Python functions in your transform.yaml.

    [:octicons-arrow-right-24: Read Guide](etl-transformations.md)

-   :material-timer-sync:{ .lg .middle } **Async Execution Model**

    ---

    Run pipelines from scripts, FastAPI, Jupyter, Celery, and concurrent contexts.

    [:octicons-arrow-right-24: Read Guide](async-execution.md)

-   :material-delta:{ .lg .middle } **Incremental Extraction**

    ---

    Watermark-based state tracking with FileStateStore and SqliteStateStore.

    [:octicons-arrow-right-24: Read Guide](incremental-extraction.md)

-   :material-test-tube:{ .lg .middle } **Testing Pipelines**

    ---

    Mock extractors, mock loaders, the test harness, fixture files, and assertion helpers.

    [:octicons-arrow-right-24: Read Guide](testing-pipelines.md)

-   :material-chart-bar:{ .lg .middle } **Data Profiling**

    ---

    Statistical field analysis with DataProfiler — completeness, distributions, and drift.

    [:octicons-arrow-right-24: Read Guide](data-profiling.md)

-   :material-book-open-variant:{ .lg .middle } **Wiki & Ontology**

    ---

    Semantic field annotations, concept vocabulary, lineage, and governance workflows.

    [:octicons-arrow-right-24: Read Guide](wiki.md)

</div>

## Quick Reference

### Common Patterns

#### Quality Gate in Pipeline

```python
async def run_with_quality_gate():
    result = await pipeline.run()
    
    report = check.run(schema_id=schema_id, data=output_data, thresholds=thresholds)
    
    if not report.passed:
        raise QualityError(f"Quality gate failed: {report.threshold_breaches}")
    
    return result
```

#### Retry with Backoff

```python
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
async def run_pipeline_with_retry():
    return await pipeline.run()
```

#### Validation Decorator

```python
from pycharter import validate_input

@validate_input("contracts/user.yaml")
def create_user(data: dict) -> dict:
    # data is already validated
    return db.insert(data)
```

#### Incremental Processing

```python
# Track state between runs
last_id = load_last_processed_id()

pipeline = Pipeline.from_config_files(
    extract="extract.yaml",
    load="load.yaml",
    variables={"LAST_ID": last_id}
)

result = await pipeline.run()
save_last_processed_id(result.metadata.get("last_id"))
```

## Need More Help?

- Check the [API Reference](../api/index.md) for detailed documentation
- Search [GitHub Issues](https://github.com/optophi/pycharter/issues)
- Ask on [GitHub Discussions](https://github.com/optophi/pycharter/discussions)
