# wynd-dataingest

High-performance Python client for wynd data ingest.

It is built for services that want:

- HTTP connection reuse through `httpx`
- one `send()` method with NDJSON as the default mode
- retries with exponential backoff and jitter
- optional gzip compression based on raw payload size
- publish-ready packaging for `pip install`

## Install

```bash
pip install wynd-dataingest
```

## Quick Start

```python
from wynd_dataingest import CompressionOptions, WyndDataIngestClient

client = WyndDataIngestClient(
    url="http://127.0.0.1:8686/events",
    connections=16,
    table_name="orders",
    table_operation="insert",
    compression=CompressionOptions(gzip=True),
    headers={
        "authorization": "Basic my-token",
    },
)

result = client.send(
    [
        {"service": "api", "level": "info", "message": "order accepted"},
        {"service": "api", "message": "batch item 1"},
        {"service": "api", "message": "batch item 2"},
    ]
)

print(result.status_code)
client.close()
```

`ndjson` is the default mode, so `send()` expects a sequence and emits newline-delimited JSON unless you set `mode="json"` or `mode="text"`.

## API

### `WyndDataIngestClient(...)`

Supported constructor options:

- `url`: full endpoint URL
- `mode`: `"json" | "ndjson" | "text"`; defaults to `"ndjson"`
- `connections`: max pooled HTTP connections; defaults to `16`
- `table_name`: base table name for the `table-name` header
- `table_operation`: `"insert" | "new" | "update" | "delete"`; defaults to `"insert"` when `table_name` is set
- `headers`: default request headers
- `timeout_ms`: per-request timeout; defaults to `30000`
- `user_agent`: override the default user agent
- `retry`: `RetryOptions`
- `compression`: `CompressionOptions`

### Sending data

The client exposes:

- `send(payload, options=None)`
- `close()`

`send()` behaves by mode:

- `mode="ndjson"`: payload must be a sequence and is serialized as newline-delimited JSON
- `mode="json"`: payload is serialized with `json.dumps`
- `mode="text"`: payload must be a string or sequence of strings
- `bytes` or `bytearray` payloads are sent as-is and can override content type with `SendOptions(content_type=...)`

### Table Routing Header

When `table_name` is configured, the client automatically adds a `table-name` header:

- `table_operation="insert"` or `"new"` -> `table-name: <table_name>`
- `table_operation="update"` -> `table-name: <table_name>-updates`
- `table_operation="delete"` -> `table-name: <table_name>-deletes`

### Compression

Enable gzip with built-in defaults:

```python
from wynd_dataingest import CompressionOptions, WyndDataIngestClient

client = WyndDataIngestClient(
    url="http://127.0.0.1:8686/events",
    compression=CompressionOptions(gzip=True),
)
```

That uses the default gzip threshold of `100KB` raw bytes.

Or customize it:

```python
from wynd_dataingest import CompressionOptions, GzipCompressionOptions, WyndDataIngestClient

client = WyndDataIngestClient(
    url="http://127.0.0.1:8686/events",
    compression=CompressionOptions(
        gzip=GzipCompressionOptions(min_bytes=4096, level=6)
    ),
)
```

## Publish To PyPI

Build locally:

```bash
python3 -m pip install --upgrade build
python3 -m build
```

Upload to PyPI:

```bash
python3 -m pip install --upgrade twine
python3 -m twine upload dist/*
```

Then users can install it with:

```bash
pip install wynd-dataingest
```
