Metadata-Version: 2.1
Name: elx
Version: 0.1.0a2
Summary: A lightweight Python interface for extracting and loading using the Singer.io spec.
Author: Jules Huisman
Author-email: jules.huisman@quantile.nl
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: all
Provides-Extra: aws
Provides-Extra: azure
Provides-Extra: gcs
Provides-Extra: http
Provides-Extra: ssh
Requires-Dist: azure-common (>=1.0.0,<2.0.0) ; extra == "azure" or extra == "all"
Requires-Dist: azure-core (>=1.0.0,<2.0.0) ; extra == "azure" or extra == "all"
Requires-Dist: azure-storage-blob (>=12.0.0,<13.0.0) ; extra == "azure" or extra == "all"
Requires-Dist: boto3 (>=1.0.0,<2.0.0) ; extra == "aws" or extra == "all"
Requires-Dist: dagster (>=1.3.13,<2.0.0)
Requires-Dist: google-cloud-storage (>=2.6.0,<3.0.0) ; extra == "gcs" or extra == "all"
Requires-Dist: paramiko (>=2.0.0,<3.0.0) ; extra == "ssh" or extra == "all"
Requires-Dist: pipx (>=1.2.0,<2.0.0)
Requires-Dist: pydantic (==1.10.11)
Requires-Dist: python-dotenv (>=1.0.0,<2.0.0)
Requires-Dist: requests (>=2.0.0,<3.0.0) ; extra == "http" or extra == "all"
Requires-Dist: smart-open (>=6.3.0,<7.0.0)
Description-Content-Type: text/markdown

# ELX

🚧 Under construction

A lightweight Python interface for extracting and loading using the Singer.io spec.

⚡ Lazy install of Singer.io taps and targets \
⚡ Stream parallelism for high performance \
⚡ Remote state management \
⚡ Tap catalog is available in Python for metadata purposes

## Installation

```bash
pip install elx --pre
```

## Usage

The most basic usage is as follows. Simply define the Tap and the Target and elx will take care of the rest.

```python
from elx import Runner, Tap, Target

runner = Runner(
  Tap("git+https://gitlab.com/meltano/tap-carbon-intensity.git"),
  Target("target-jsonl")
)

runner.run()
```

### Configuration

You can configure the tap and target by passing a `config` dictionary to the `Tap` and `Target` constructors. The config will be injected into the tap and target at runtime.

```python
from elx import Tap, Target

tap = Tap(
  "tap-foo",
  config={
    "api_key": "1234567890",
    "start_date": "2020-01-01"
  }
)

target = Target(
  "target-bar",
  config={
    "file_path": "/tmp"
  }
)
```

### State

By default, elx will store the state in the same directory as the script that is running. You can override this by passing a `StateManager` to the `Runner` constructor. Behind the scenes, elx uses [smart-open](https://github.com/RaRe-Technologies/smart_open) to be able to store the state in a variety of locations.

```python
from elx import Runner, StateManager

runner = Runner(
  tap,
  target,
  state_manager=StateManager("s3://my-bucket/my-folder")
)
```

Supported paths include:

| Path                                                   | Required Environment Variables                         | Elx Extra    |
| ------------------------------------------------------ | ------------------------------------------------------ | ------------ |
| `s3://my-bucket/my-folder`                             | `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`        | `elx[s3]`    |
| `gs://my-bucket/my-folder`                             | `GOOGLE_APPLICATION_CREDENTIALS` or `GOOGLE_API_TOKEN` | `elx[gs]`    |
| `azure://my-bucket/my-folder`                          | `AZURE_STORAGE_CONNECTION_STRING`                      | `elx[azure]` |
| `~/my-folder`                                          | `None`                                                 | `None`       |
| `/tmp/my-folder`                                       | `None`                                                 | `None`       |
| `(ssh\|scp\|sftp)://username@host//my-folder`          | `None`                                                 | `None`       |
| `(ssh\|scp\|sftp)://username:password@host//my-folder` | `None`                                                 | `None`       |

