# Database Sinks

extractor can stream extracted records directly into a database while writing JSONL output. Add `--sink <name>` to any `extractor run` command.

```bash
extractor run ./docs/ --mode chunks --sink qdrant --sink-uri http://localhost:6333 --out ./output/
```

The `--out` flag is independent — you can use a sink with or without writing JSONL to disk.

---

## Overview

### Sink registry

All sinks are registered by name. The `--sink` flag selects a sink by name; aliases are supported (e.g. `postgres` and `postgresql` both work, as do `elasticsearch` and `es`).

### BaseSink protocol

Every sink implements `BaseSink` (`src/extractor/sinks/base.py`):

```python
class BaseSink(ABC):
    def connect(self) -> None: ...     # open connection, create table/index if needed
    def write_batch(self, records: list[dict]) -> None: ...  # bulk insert
    def flush(self) -> None: ...       # flush buffered records
    def close(self) -> None: ...       # close connection
```

The CLI calls `connect()` once at startup, then `feed(record)` for every element or chunk (buffered internally), then `flush()` and `close()` after the document is finished. Sinks can also be used as context managers.

### feed / batch / flush lifecycle

Each sink maintains an internal buffer. `feed(record)` appends to the buffer; when the buffer reaches `batch_size`, `write_batch` is called automatically. `flush()` drains any remaining buffered records. This means a sink with `batch_size=1000` issues at most one write call per 1000 records.

### CLI flags

| Flag | Default | Description |
|------|---------|-------------|
| `--sink` | none | Sink name: `clickhouse`, `mongodb`, `postgres`, `elasticsearch`, `qdrant`, `weaviate`, `kafka`, `webhook` |
| `--sink-uri` | sink-specific | Connection URI or `host:port` |
| `--sink-database` | `extractor` | Database name (ClickHouse, MongoDB) |
| `--sink-table` | `elements` | Table / collection / index / topic name |
| `--sink-batch` | `1000` | Batch size for writes |

---

## Sinks

### ClickHouse

| | |
|---|---|
| Install | `pip install "coresdk-extractor[clickhouse]"` |
| Default connection | `localhost:8123` |

**Connection options**

| Env var / flag | Description |
|----------------|-------------|
| `--sink-uri host:port` | ClickHouse HTTP port (default: 8123) |
| `--sink-database` | Database name (created if absent, default: `extractor`) |
| `--sink-table` | Table name (default: `elements`) |

**Schema mapping**

The table is auto-created with `MergeTree` engine ordered by `(source_sha256, sequence_index)`, partitioned by `source_media_type`. Nested objects (`entities`, `relations`, `classification`, `structured_fields`, `table_data`) are stored as JSON strings in `String` columns.

```bash
extractor run ./docs/ --mode chunks --sink clickhouse --sink-uri localhost:8123 \
  --sink-database mydb --sink-table chunks
```

---

### MongoDB

| | |
|---|---|
| Install | `pip install "coresdk-extractor[mongodb]"` |
| Default connection | `mongodb://localhost:27017` |

**Connection options**

| Env var / flag | Description |
|----------------|-------------|
| `--sink-uri` | Full MongoDB URI (default: `mongodb://localhost:27017`) |
| `--sink-database` | Database name (default: `extractor`) |
| `--sink-table` | Collection name (default: `elements`) |

**Schema mapping**

The `id` field from extractor is stored as MongoDB `_id`. Indexes are auto-created on `source_sha256`, `element_type`, and `(source_filename, sequence_index)`. Nested objects (`entities`, `relations`, etc.) are stored as native BSON documents. Duplicate `_id` errors on re-runs are silently ignored (idempotent inserts).

```bash
extractor run ./docs/ --mode chunks --sink mongodb \
  --sink-uri mongodb://user:pass@localhost:27017 \
  --sink-database ragdb --sink-table chunks
```

---

### PostgreSQL

| | |
|---|---|
| Install | `pip install "coresdk-extractor[postgres]"` |
| Default connection | `EXTRACTOR_PG_DSN` env var, or libpq env vars |

**Connection options**

| Env var / flag | Description |
|----------------|-------------|
| `EXTRACTOR_PG_DSN` | Full DSN: `postgresql://user:pass@host:5432/db` |
| `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, `PGDATABASE` | Standard libpq env vars (used when no DSN) |
| `--sink-uri` | DSN passed directly (overrides env var) |
| `--sink-table` | Table name (default: `extractor_elements`) |

**Schema mapping**

The table is auto-created with `id TEXT PRIMARY KEY`. Nested objects (`entities`, `relations`, `classification`, `structured_fields`, `table`) are stored as `JSONB`. Indexes are auto-created on `source_sha256` and `element_type`. Upsert on conflict (`ON CONFLICT (id) DO UPDATE`) is the default; set `upsert=False` when using the Python API to skip duplicates instead.

```bash
# DSN from env var
EXTRACTOR_PG_DSN=postgresql://user:pass@localhost/mydb \
  extractor run ./docs/ --mode chunks --sink postgres --sink-table doc_chunks

# DSN as CLI flag
extractor run ./docs/ --mode chunks --sink postgres \
  --sink-uri "postgresql://user:pass@localhost/mydb"
```

---

### Elasticsearch

| | |
|---|---|
| Install | `pip install "coresdk-extractor[elasticsearch]"` |
| Default connection | `http://localhost:9200` |

**Connection options**

| Env var | Description |
|---------|-------------|
| `EXTRACTOR_ES_URL` | Elasticsearch URL (default: `http://localhost:9200`) |
| `EXTRACTOR_ES_API_KEY` | Elastic Cloud / self-managed API key |
| `EXTRACTOR_ES_USER` | Basic auth username |
| `EXTRACTOR_ES_PASSWORD` | Basic auth password |
| `EXTRACTOR_ES_CA_CERT` | Path to CA certificate file (custom TLS) |

**Schema mapping**

The index is auto-created with an explicit mapping. `text` is mapped as `text` with `standard` analyzer for full-text search. `element_type`, `source_filename`, `source_sha256`, `doc_category` are `keyword`. `entities` and `relations` are stored as `object` with indexing disabled (`"enabled": false`) — use a separate pipeline if you need to query inside them.

```bash
# Elasticsearch on localhost (no auth)
extractor run ./docs/ --mode chunks --sink elasticsearch --sink-table my_index

# Elastic Cloud with API key
EXTRACTOR_ES_API_KEY=... \
  extractor run ./docs/ --mode chunks \
  --sink elasticsearch --sink-uri https://my-cluster.es.io \
  --sink-table doc_chunks
```

---

### Qdrant

| | |
|---|---|
| Install | `pip install "coresdk-extractor[qdrant]"` |
| Default connection | `http://localhost:6333` |

**Important:** the Qdrant sink stores text payloads and metadata only. It does **not** generate embeddings. Points are inserted with zero vectors. You must replace the vectors with real embeddings using a separate embedding step or Qdrant's own vectorizer.

**Connection options**

| Env var | Description |
|---------|-------------|
| `EXTRACTOR_QDRANT_URL` | Qdrant URL (default: `http://localhost:6333`) |
| `EXTRACTOR_QDRANT_API_KEY` | Qdrant Cloud API key |

**Schema mapping**

Each record becomes a Qdrant `PointStruct`. The Qdrant point `id` is a deterministic UUID derived from the extractor `id` field. The payload includes: `extractor_id`, `element_type`, `text`, `source_filename`, `source_sha256`, `source_uri`, `source_media_type`, `page`, `token_count`, `doc_category`, `doc_category_score`, `section_path`, `entity_types`, `relation_subjects`, `relation_objects`. The collection is auto-created if absent (default `vector_size=1536`; override via the Python API).

```bash
extractor run ./docs/ --mode chunks --sink qdrant \
  --sink-uri http://localhost:6333 \
  --sink-table my_collection

# Qdrant Cloud
EXTRACTOR_QDRANT_API_KEY=... \
  extractor run ./docs/ --mode chunks \
  --sink qdrant --sink-uri https://my-cluster.qdrant.io \
  --sink-table production_chunks
```

---

### Weaviate

| | |
|---|---|
| Install | `pip install "coresdk-extractor[weaviate]"` |
| Default connection | `http://localhost:8080` |

Like Qdrant, the Weaviate sink stores payloads only — embeddings are populated separately or via a Weaviate vectorizer module configured in the Weaviate schema.

**Connection options**

| Env var | Description |
|---------|-------------|
| `EXTRACTOR_WEAVIATE_URL` | Weaviate URL (default: `http://localhost:8080`) |
| `EXTRACTOR_WEAVIATE_API_KEY` | Weaviate Cloud (WCS) API key |

**Schema mapping**

A Weaviate class is auto-created if absent. Properties stored: `extractor_id`, `element_type`, `text`, `source_filename`, `source_sha256`, `source_uri`, `doc_category`, `doc_category_score`, `page`, `token_count`, `entity_types`, `relation_subjects`, `relation_objects`. The class name is set via `--sink-table` (default: `ExtractorChunk`).

```bash
extractor run ./docs/ --mode chunks --sink weaviate \
  --sink-uri http://localhost:8080 \
  --sink-table MyDocChunks

# Weaviate Cloud
EXTRACTOR_WEAVIATE_API_KEY=... \
  extractor run ./docs/ --mode chunks \
  --sink weaviate --sink-uri https://my-cluster.weaviate.network
```

---

### Kafka

| | |
|---|---|
| Install | `pip install "coresdk-extractor[kafka]"` |
| Default brokers | `localhost:9092` |

Each record is published as a JSON-encoded message. The message key is `source_sha256` (for partition locality by document). Useful for feeding downstream stream-processing pipelines (Flink, Spark Structured Streaming, etc.).

**Connection options**

| Env var | Description |
|---------|-------------|
| `EXTRACTOR_KAFKA_BROKERS` | Comma-separated broker list (default: `localhost:9092`) |
| `EXTRACTOR_KAFKA_SECURITY_PROTOCOL` | e.g. `PLAINTEXT`, `SASL_SSL` |
| `EXTRACTOR_KAFKA_SASL_MECHANISM` | e.g. `PLAIN`, `SCRAM-SHA-256` |
| `EXTRACTOR_KAFKA_SASL_USERNAME` | SASL username |
| `EXTRACTOR_KAFKA_SASL_PASSWORD` | SASL password |

**Schema mapping**

Each Kafka message value is the full JSON-serialised element or chunk record (same fields as the JSONL output). The message key is `source_sha256` encoded as UTF-8.

```bash
# Local Kafka
extractor run ./docs/ --mode chunks --sink kafka \
  --sink-uri localhost:9092 --sink-table doc_chunks

# Confluent Cloud with SASL
EXTRACTOR_KAFKA_SECURITY_PROTOCOL=SASL_SSL \
  EXTRACTOR_KAFKA_SASL_MECHANISM=PLAIN \
  EXTRACTOR_KAFKA_SASL_USERNAME=mykey \
  EXTRACTOR_KAFKA_SASL_PASSWORD=mysecret \
  extractor run ./docs/ --mode chunks \
  --sink kafka --sink-uri pkc-xxxx.us-east-1.aws.confluent.cloud:9092 \
  --sink-table prod_chunks
```

---

### Webhook

| | |
|---|---|
| Install | None — `requests` is a core dependency |
| Aliases | `webhook`, `http_post` |

POSTs each batch as a JSON array to an HTTP endpoint. Useful for custom integrations, serverless pipelines, or any service with an HTTP ingest API.

**Connection options**

| Env var | Description |
|---------|-------------|
| `EXTRACTOR_WEBHOOK_URL` | Target URL (required if `--sink-uri` is not set) |
| `EXTRACTOR_WEBHOOK_HEADERS_JSON` | JSON dict of extra request headers (e.g. auth token) |
| `EXTRACTOR_WEBHOOK_VERIFY_SSL` | Set to `false` to skip SSL verification |
| `EXTRACTOR_WEBHOOK_TIMEOUT` | Request timeout in seconds (default: 30) |

**Schema mapping**

The request body is a JSON array of full extractor records. `Content-Type: application/json` is always set. HTTP non-2xx responses raise an exception and abort the current batch.

```bash
extractor run ./docs/ --mode chunks --sink webhook \
  --sink-uri https://my-api.example.com/ingest

# With auth header
EXTRACTOR_WEBHOOK_HEADERS_JSON='{"Authorization":"Bearer mytoken"}' \
  extractor run ./docs/ --mode chunks \
  --sink webhook --sink-uri https://my-api.example.com/ingest \
  --sink-batch 50
```

---

## Writing a custom sink plugin

Implement `BaseSink` and register your class:

```python
# my_sink.py
from typing import Any
from extractor.sinks.base import BaseSink

class MyCustomSink(BaseSink):
    def __init__(self, url: str, batch_size: int = 500) -> None:
        self.url = url
        self.batch_size = batch_size
        self._buffer: list[dict[str, Any]] = []

    def connect(self) -> None:
        # open connection, create schema if needed
        ...

    def write_batch(self, records: list[dict[str, Any]]) -> None:
        # bulk write records to your store
        ...

    def feed(self, record: dict[str, Any]) -> None:
        self._buffer.append(record)
        if len(self._buffer) >= self.batch_size:
            self.write_batch(self._buffer)
            self._buffer = []

    def flush(self) -> None:
        if self._buffer:
            self.write_batch(self._buffer)
            self._buffer = []

    def close(self) -> None:
        ...
```

Register via `entry_points` in your package's `pyproject.toml` so extractor discovers it automatically:

```toml
[project.entry-points."extractor.sinks"]
my_sink = "my_package.my_sink:MyCustomSink"
```

After installing your package, `extractor run ./docs/ --sink my_sink --sink-uri ...` will use your sink. Registered plugins are listed by `extractor info`.

You can also register programmatically:

```python
from extractor.sinks.registry import register_sink
from my_package.my_sink import MyCustomSink

register_sink("my_sink", MyCustomSink)
```
