# core-etl
===============================================================================

This library provides essential components for ETL processes, offering reusable interfaces 
for seamless data extraction, transformation, and loading....

===============================================================================

.. image:: https://img.shields.io/pypi/pyversions/core-etl.svg
    :target: https://pypi.org/project/core-etl/
    :alt: Python Versions

.. image:: https://img.shields.io/badge/license-MIT-blue.svg
    :target: https://gitlab.com/bytecode-solutions/core/core-etl/-/blob/main/LICENSE
    :alt: License

.. image:: https://gitlab.com/bytecode-solutions/core/core-etl/badges/release/pipeline.svg
    :target: https://gitlab.com/bytecode-solutions/core/core-etl/-/pipelines
    :alt: Pipeline Status

.. image:: https://readthedocs.org/projects/core-etl/badge/?version=latest
    :target: https://readthedocs.org/projects/core-etl/
    :alt: Docs Status

.. image:: https://img.shields.io/badge/security-bandit-yellow.svg
    :target: https://github.com/PyCQA/bandit
    :alt: Security

|


Installation
===============================================================================

Install from PyPI using pip:

.. code-block:: bash

    pip install core-etl
    uv pip install core-etl  # Or using UV...


Features
===============================================================================

**Base ETL Framework**

* Template method pattern for ETL workflow orchestration
* Comprehensive lifecycle hooks (pre-processing, execution, post-processing, cleanup)
* Built-in error handling with detailed exception logging
* Task status tracking (CREATED, EXECUTING, SUCCESS, ERROR)
* Timezone support for date/datetime processing (defaults to UTC)
* Temporary folder management for local file operations
* Extensible resource cleanup mechanisms

**File-Based ETL (IBaseEtlFromFile)**

* Process files from various sources (SFTP, local filesystem, cloud storage)
* Iterator-based file processing with error isolation per file
* Individual file success/error callbacks for custom handling
* Batch file operations with automatic error recovery
* Extensible hooks: ``get_paths()``, ``process_file()``, ``on_success()``, ``on_error()``

**Record-Based ETL (IBaseEtlFromRecord)**

* Process records from APIs, databases, files, message queues, and data streams
* Memory-efficient batch processing with configurable batch sizes
* Built-in transformation pipeline:

  * Field removal (``attrs_to_remove``)
  * Field renaming (``name_mapper``)
  * Data type casting (``type_mapper``)

* Pre and post transformation hooks for custom business logic
* Incremental processing support with ``last_processed`` markers
* Extensible methods: ``retrieve_records()``, ``process_records()``, ``pre_transformations()``, ``post_transformations()``


Quick Start
===============================================================================

Installation
-------------------------------------------------------------------------------

Install the package:

.. code-block:: bash

    pip install core-etl
    uv pip install core-etl     # Or using UV...
    pip install -e ".[dev]"     # For development...


Setting Up Environment
-------------------------------------------------------------------------------

1. Install required libraries:

.. code-block:: bash

    pip install --upgrade pip
    pip install virtualenv

2. Create Python virtual environment:

.. code-block:: bash

    virtualenv --python=python3.12 .venv

3. Activate the virtual environment:

.. code-block:: bash

    source .venv/bin/activate

Install packages
-------------------------------------------------------------------------------

.. code-block:: bash

    pip install .
    pip install -e ".[dev]"

Check tests and coverage
-------------------------------------------------------------------------------

.. code-block:: shell

    python manager.py run-tests
    python manager.py run-coverage


Contributing
===============================================================================

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Write tests for new functionality
4. Ensure all tests pass: ``pytest -n auto``
5. Run linting: ``pylint core_etl``
6. Run security checks: ``bandit -r core_etl``
7. Submit a pull request


License
===============================================================================

This project is licensed under the MIT License. See the LICENSE file for details.


Links
===============================================================================

* **Documentation:** https://core-etl.readthedocs.io/en/latest/
* **Repository:** https://gitlab.com/bytecode-solutions/core/core-etl
* **Issues:** https://gitlab.com/bytecode-solutions/core/core-etl/-/issues
* **Changelog:** https://gitlab.com/bytecode-solutions/core/core-etl/-/blob/master/CHANGELOG.md
* **PyPI:** https://pypi.org/project/core-etl/


Support
===============================================================================

For questions or support, please open an issue on GitLab or contact the maintainers.


Authors
===============================================================================

* **Alejandro Cora González** - *Initial work* - alek.cora.glez@gmail.com
