# Source Connectors

extractor can pull documents directly from remote storage using URI routing. Pass any supported URI as the `target` argument to `extractor run` — no manual download needed.

```bash
extractor run s3://my-bucket/docs/ --out ./output/
extractor run https://example.com/report.pdf
```

---

## Overview

### URI routing

When the `target` argument contains `://` and does not start with `/`, extractor treats it as a remote URI. The scheme prefix (`s3`, `az`, `gcs`, `https`, `imap`, etc.) selects the connector. All other CLI flags (`--mode`, `--chunk-size`, `--sink`, etc.) apply normally.

Cloud connectors (`s3`, `az`, `abfs`, `gcs`, `gs`) must be explicitly enabled at startup by their optional extra. HTTP/HTTPS and IMAP/email connectors are always available.

When the URI ends with `/` or refers to a prefix, the connector lists all matching files under that prefix. Use `--source-filter "*.pdf"` to restrict by glob pattern. Downloads run in parallel — control concurrency with `--source-concurrency N` (default: 4).

### BaseSource protocol

Every connector implements the `BaseSource` protocol (`src/extractor/sources/base.py`):

```python
class BaseSource(Protocol):
    scheme: str

    def list(self, uri: str, glob_pattern: str | None = None) -> Iterator[SourceFile]: ...

    @contextlib.contextmanager
    def fetch(self, source_file: SourceFile, tmp_dir: Path | None = None) -> Iterator[Path]: ...
```

`list` yields `SourceFile` metadata (URI, filename, size, ETag, last_modified). `fetch` is a context manager that downloads the file to a temp path and cleans it up on exit.

### Security model

Every downloaded file passes through the same quarantine gate (`extractor.quarantine`) as local files — XXE, SSRF, zip-bomb, and oversized-file checks — before extraction begins. If quarantine fails the file is skipped and counted as an error; processing of other files continues.

**Auth via env vars only.** Credentials are never accepted as CLI arguments. Set the env vars described below before running.

---

## Connectors

### S3 / MinIO

| | |
|---|---|
| URI format | `s3://bucket/prefix/` or `s3://bucket/key.pdf` |
| MinIO alias | `minio://bucket/prefix/` |
| Install | `pip install "coresdk-extractor[s3]"` |

**Auth env vars**

| Env var | Description |
|---------|-------------|
| `AWS_ACCESS_KEY_ID` | AWS access key (explicit credentials) |
| `AWS_SECRET_ACCESS_KEY` | AWS secret key |
| `AWS_SESSION_TOKEN` | Temporary session token (STS / assumed role) |
| `AWS_PROFILE` | Named profile from `~/.aws/credentials` |
| `EXTRACTOR_S3_ENDPOINT_URL` | Custom endpoint for MinIO or S3-compatible stores |
| `EXTRACTOR_S3_REGION` | Region override (optional) |

When none of the explicit key env vars are set, boto3 falls back to its standard credential chain (IAM instance role, ECS task role, etc.).

**Examples**

```bash
# AWS with explicit keys
AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... \
  extractor run s3://my-bucket/papers/ --source-filter "*.pdf" --out ./output/

# AWS named profile
AWS_PROFILE=my-profile extractor run s3://my-bucket/reports/ --out ./output/

# MinIO
EXTRACTOR_S3_ENDPOINT_URL=http://minio.local:9000 \
  AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \
  extractor run s3://my-bucket/docs/ --out ./output/

# Single file
extractor run s3://my-bucket/annual-report.pdf --mode chunks --out ./output/
```

**Limitations**

- Single-file URIs (no trailing `/`) extract one file only; the `list` method returns one `SourceFile`.
- Directory markers (keys ending with `/`) are skipped automatically.
- Versioned buckets: `VersionId` is captured in `source_etag`/`source_etag` fields on output records.

---

### Azure Blob Storage / ADLS Gen2

| | |
|---|---|
| URI format | `az://container/prefix/` or `az://container/blob.pdf` |
| ADLS Gen2 alias | `abfs://container/prefix/` |
| Install | `pip install "coresdk-extractor[azure]"` |

**Auth env vars**

| Env var | Description |
|---------|-------------|
| `AZURE_STORAGE_CONNECTION_STRING` | All-in-one connection string (highest priority) |
| `AZURE_STORAGE_ACCOUNT` | Storage account name |
| `AZURE_STORAGE_KEY` | Storage account key (used with `AZURE_STORAGE_ACCOUNT`) |

When only `AZURE_STORAGE_ACCOUNT` is set (no key, no connection string), `DefaultAzureCredential` is used — this supports managed identity, service principal via environment variables, Azure CLI, and VS Code. Requires `azure-identity` (included in the `azure` extra).

**Examples**

```bash
# Connection string
AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;..." \
  extractor run az://my-container/reports/ --out ./output/

# Account + key
AZURE_STORAGE_ACCOUNT=myaccount AZURE_STORAGE_KEY=mykey \
  extractor run az://my-container/docs/ --source-filter "*.docx" --out ./output/

# Managed identity (e.g. on Azure VM or AKS)
AZURE_STORAGE_ACCOUNT=myaccount \
  extractor run az://my-container/data/ --out ./output/

# ADLS Gen2 (same auth, different scheme)
extractor run abfs://my-filesystem/folder/ --out ./output/
```

**Limitations**

- The `abfs` scheme is a functional alias for `az` — both use Azure Blob Storage APIs. True ADLS hierarchical namespace features are not used.
- Large blobs are streamed to a temp file before quarantine; ensure sufficient local disk space when processing large files.

---

### Google Cloud Storage

| | |
|---|---|
| URI format | `gcs://bucket/prefix/` or `gcs://bucket/object.pdf` |
| `gs://` alias | Supported |
| Install | `pip install "coresdk-extractor[gcs]"` |

**Auth env vars**

| Env var | Description |
|---------|-------------|
| `GOOGLE_APPLICATION_CREDENTIALS` | Path to service account JSON key file |

On GKE, Cloud Run, or Compute Engine with a service account attached, no env var is needed — the client library picks up Workload Identity / Application Default Credentials automatically.

**Examples**

```bash
# Service account key file
GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json \
  extractor run gcs://my-bucket/docs/ --out ./output/

# Workload Identity (no env var needed on GKE)
extractor run gs://my-bucket/papers/ --source-filter "*.pdf" --out ./output/
```

**Limitations**

- `gs://` and `gcs://` are identical in behaviour; both are registered.
- Object `generation` is stored as `source_etag` on output records, enabling dedup with `--incremental`.

---

### HTTP / HTTPS

| | |
|---|---|
| URI format | `https://example.com/file.pdf` or `http://internal.corp/doc.docx` |
| Install | None — `requests` is a core dependency |

The HTTP connector supports **single-file URIs only** (`list` yields one `SourceFile`). Use S3/Azure/GCS connectors for bulk downloads.

**Auth env vars**

| Env var | Description |
|---------|-------------|
| `EXTRACTOR_HTTP_HEADERS_JSON` | JSON object of request headers, e.g. `{"Authorization": "Bearer <token>"}` |
| `EXTRACTOR_HTTP_VERIFY_SSL` | Set to `false` to disable SSL verification (not recommended) |
| `EXTRACTOR_HTTP_MAX_BYTES` | Download size cap in bytes (default: 524,288,000 — 500 MB) |

**Examples**

```bash
# Public URL
extractor run https://arxiv.org/pdf/2301.00001.pdf --mode chunks --out ./output/

# Authenticated endpoint
EXTRACTOR_HTTP_HEADERS_JSON='{"Authorization":"Bearer mytoken"}' \
  extractor run https://api.example.com/docs/report.pdf

# Internal HTTP with SSL disabled
EXTRACTOR_HTTP_VERIFY_SSL=false \
  extractor run http://internal.corp/reports/q4.pdf --out ./output/
```

**Limitations**

- Only single-file downloads. `list` issues a HEAD request to read `Content-Length` and `ETag`; `fetch` streams the body.
- No redirect following beyond what `requests` does by default.
- Downloads exceeding `EXTRACTOR_HTTP_MAX_BYTES` are aborted mid-stream.

---

### IMAP / Email

| | |
|---|---|
| URI format | `imap://inbox`, `imaps://inbox`, `email://inbox` |
| Install | None — uses stdlib `imaplib` |

The email connector connects to an IMAP mailbox and yields **file attachments** from matching messages. Supported attachment types: `.pdf`, `.docx`, `.doc`, `.pptx`, `.ppt`, `.xlsx`, `.xls`, `.txt`, `.md`, `.html`, `.htm`, `.epub`, `.csv`.

**Auth env vars**

| Env var | Description |
|---------|-------------|
| `EXTRACTOR_IMAP_HOST` | IMAP server hostname, e.g. `imap.gmail.com` |
| `EXTRACTOR_IMAP_PORT` | Port (default: 993) |
| `EXTRACTOR_IMAP_USERNAME` | Login username / email address |
| `EXTRACTOR_IMAP_PASSWORD` | Password or app-specific password |
| `EXTRACTOR_IMAP_FOLDER` | Mailbox folder to search (default: `INBOX`) |
| `EXTRACTOR_IMAP_SEARCH` | IMAP search string (default: `UNSEEN`) |

For Gmail and Outlook, use an **app password** rather than your account password; both providers block regular password auth for IMAP when 2FA is enabled.

**Examples**

```bash
# Process unseen emails in INBOX (default search)
EXTRACTOR_IMAP_HOST=imap.gmail.com \
  EXTRACTOR_IMAP_USERNAME=me@example.com \
  EXTRACTOR_IMAP_PASSWORD=myapppassword \
  extractor run imap://inbox --out ./output/

# Custom folder and IMAP search string
EXTRACTOR_IMAP_HOST=mail.corp.com \
  EXTRACTOR_IMAP_USERNAME=svc@corp.com \
  EXTRACTOR_IMAP_PASSWORD=secret \
  EXTRACTOR_IMAP_FOLDER=Reports \
  EXTRACTOR_IMAP_SEARCH="FROM noreply@vendor.com" \
  extractor run imap://inbox --out ./output/

# Filter to PDF attachments only
extractor run imap://inbox --source-filter "*.pdf" --out ./output/
```

**Limitations**

- Connects using `IMAP4_SSL` (TLS). Plain IMAP (port 143) is not supported.
- `imap://`, `imaps://`, and `email://` are all aliases for the same SSL connector.
- The connector fetches full message bodies to enumerate attachments; very large mailboxes with many messages may be slow. Narrow the search with `EXTRACTOR_IMAP_SEARCH`.
- Messages are not marked as read after processing — manage that outside extractor.

---

## Source connector options (CLI)

| Flag | Default | Description |
|------|---------|-------------|
| `--source-filter` | none | Glob pattern applied to filenames, e.g. `*.pdf` |
| `--source-concurrency` | 4 | Max parallel downloads |
| `--source-tmp-dir` | system temp | Directory for temp files during download |

---

## Writing a custom source connector

Implement the `BaseSource` protocol and register your instance:

```python
# my_connector.py
import contextlib
from collections.abc import Iterator
from pathlib import Path
from extractor.sources.base import SourceFile
from extractor.sources.registry import register_source

class MySource:
    scheme = "myscheme"

    def list(self, uri: str, glob_pattern: str | None = None) -> Iterator[SourceFile]:
        # yield SourceFile objects
        ...

    @contextlib.contextmanager
    def fetch(self, source_file: SourceFile, tmp_dir: Path | None = None) -> Iterator[Path]:
        # download to tmp_path, yield it, clean up in finally
        ...

register_source(MySource())
```

Import your module before calling `extractor run`, or register it via `entry_points` in your package's `pyproject.toml`:

```toml
[project.entry-points."extractor.parsers"]
myscheme = "my_package.my_connector"
```
