# Story 5.6: HTTP Sink Batching

## Status: Complete

## Priority: Medium

## Estimated Effort: Small (1-2 days)

## Dependencies: Story 5.5 (Cloud sink examples - provides `CloudSinkBase` pattern)

## Epic: Performance Optimization

---

## Context

The current `HttpSink` and `WebhookSink` send one HTTP request per log event. Even though fapilog's core `LoggerWorker` batches events before calling sinks (default 256 events, 0.25s timeout), **each `write()` call results in a separate HTTP POST**.

```
Worker batch of 256 events → 256 HTTP requests (current behavior)
Worker batch of 256 events → 1 HTTP request   (with sink-level batching)
```

### Existing Assets

**Story 5.5** introduced `CloudSinkBase` in `examples/sinks/cloud_sink_base.py` which already implements:

- Sink-level batch accumulation with `batch_size` and `batch_timeout_seconds`
- Background flush loop
- Retry with exponential backoff
- Lock-protected accumulation

This story focuses on **bringing that pattern to the built-in `HttpSink`** with HTTP-specific enhancements.

---

## Problem Statement

Users sending logs to HTTP endpoints experience:

1. **High network overhead** - 256 events = 256 TCP connections/requests
2. **Poor throughput** - Sequential HTTP calls limit throughput
3. **Higher costs** - Cloud APIs that charge per request (e.g., $0.50/million requests)
4. **Rate limiting** - APIs may reject high request volumes

### Cost Example

| Scenario                  | Events/day | HTTP Requests | Cost @ $0.50/M |
| ------------------------- | ---------- | ------------- | -------------- |
| No sink batching          | 10M        | 10M           | $5.00/day      |
| Sink batching (100/batch) | 10M        | 100K          | $0.05/day      |

---

## Acceptance Criteria

### AC1: Batch Configuration

- [ ] `batch_size` - Maximum events per batch (default: 100)
- [ ] `batch_timeout_seconds` - Maximum wait before flush (default: 5.0)
- [ ] `batch_format` - Output format: `"array"`, `"ndjson"`, `"wrapped"` (default: `"array"`)
- [ ] `batch_wrapper_key` - Key for wrapped format (default: `"logs"`)

### AC2: Batch Flushing

- [ ] Flush when batch reaches `batch_size`
- [ ] Flush when `batch_timeout_seconds` elapsed since first event
- [ ] Flush on sink stop (drain remaining events)
- [ ] Thread-safe batch accumulation with `asyncio.Lock`

### AC3: Batch Formats

- [ ] `array` - JSON array: `[{event1}, {event2}, ...]` with `Content-Type: application/json`
- [ ] `ndjson` - Newline-delimited JSON: `{event1}\n{event2}\n` with `Content-Type: application/x-ndjson`
- [ ] `wrapped` - Custom wrapper: `{"logs": [{event1}, {event2}]}` with `Content-Type: application/json`

### AC4: Error Handling

- [ ] Failed batch retried with exponential backoff (via existing `AsyncRetrier`)
- [ ] Metrics track batch success/failure
- [ ] Contained errors - sink failures don't crash pipeline

### AC5: Backwards Compatibility

- [ ] Default behavior (`batch_size=1`) matches current per-event behavior
- [ ] Existing configurations work without changes
- [ ] `HttpSinkSettings` extended with optional batch fields

---

## Technical Design

### 1. Updated HttpSinkConfig

```python
# src/fapilog/plugins/sinks/http_client.py

from enum import Enum

class BatchFormat(str, Enum):
    ARRAY = "array"
    NDJSON = "ndjson"
    WRAPPED = "wrapped"


class HttpSinkConfig:
    """Configuration for HTTP sink with optional batching."""

    def __init__(
        self,
        *,
        endpoint: str,
        headers: Mapping[str, str] | None = None,
        retry: RetryConfig | None = None,
        timeout_seconds: float = 5.0,
        # NEW: Batching configuration
        batch_size: int = 1,  # 1 = no batching (backwards compatible)
        batch_timeout_seconds: float = 5.0,
        batch_format: str | BatchFormat = BatchFormat.ARRAY,
        batch_wrapper_key: str = "logs",
    ) -> None:
        self.endpoint = endpoint
        self.headers = dict(headers or {})
        self.retry = retry
        self.timeout_seconds = timeout_seconds
        self.batch_size = batch_size
        self.batch_timeout_seconds = batch_timeout_seconds
        self.batch_format = BatchFormat(batch_format)
        self.batch_wrapper_key = batch_wrapper_key
```

### 2. Batching Mixin (Reusable Pattern)

Rather than duplicating `CloudSinkBase` logic, extract a reusable mixin:

```python
# src/fapilog/plugins/sinks/_batching.py

class BatchingMixin:
    """Mixin providing sink-level batching for HTTP-based sinks."""

    _batch: list[dict]
    _batch_lock: asyncio.Lock
    _batch_first_time: float | None
    _flush_task: asyncio.Task[None] | None
    _batch_size: int
    _batch_timeout_seconds: float

    def _init_batching(self, batch_size: int, batch_timeout_seconds: float) -> None:
        self._batch = []
        self._batch_lock = asyncio.Lock()
        self._batch_first_time = None
        self._flush_task = None
        self._batch_size = batch_size
        self._batch_timeout_seconds = batch_timeout_seconds

    async def _start_batching(self) -> None:
        if self._batch_size > 1:
            self._flush_task = asyncio.create_task(self._flush_loop())

    async def _stop_batching(self) -> None:
        if self._flush_task:
            self._flush_task.cancel()
            try:
                await self._flush_task
            except asyncio.CancelledError:
                pass
        await self._flush_batch()

    async def _enqueue_for_batch(self, entry: dict) -> None:
        if self._batch_size <= 1:
            await self._send_batch([entry])
            return

        flush_now = False
        async with self._batch_lock:
            if not self._batch:
                self._batch_first_time = time.monotonic()
            self._batch.append(entry)
            if len(self._batch) >= self._batch_size:
                flush_now = True

        if flush_now:
            await self._flush_batch()

    async def _flush_loop(self) -> None:
        while True:
            await asyncio.sleep(self._batch_timeout_seconds / 2)
            async with self._batch_lock:
                if not self._batch:
                    continue
                elapsed = time.monotonic() - (self._batch_first_time or 0)
                if elapsed >= self._batch_timeout_seconds:
                    batch = self._batch[:]
                    self._batch = []
                    self._batch_first_time = None
            if batch:
                await self._send_batch(batch)

    async def _flush_batch(self) -> None:
        async with self._batch_lock:
            if not self._batch:
                return
            batch = self._batch[:]
            self._batch = []
            self._batch_first_time = None
        await self._send_batch(batch)

    async def _send_batch(self, batch: list[dict]) -> None:
        """Override in subclass to send the batch."""
        raise NotImplementedError
```

### 3. Updated HttpSink

```python
class HttpSink(BatchingMixin):
    """Async HTTP sink with optional batching support."""

    name = "http"

    def __init__(
        self,
        config: HttpSinkConfig,
        *,
        metrics: Any | None = None,
        pool: HttpClientPool | None = None,
    ) -> None:
        self._config = config
        self._metrics = metrics
        self._pool = pool or HttpClientPool(
            max_size=4,
            timeout=config.timeout_seconds,
            acquire_timeout_seconds=2.0,
        )
        self._retrier = AsyncRetrier(config.retry) if config.retry else None
        self._last_status: int | None = None
        self._last_error: str | None = None

        # Initialize batching
        self._init_batching(config.batch_size, config.batch_timeout_seconds)

    async def start(self) -> None:
        await self._pool.start()
        await self._start_batching()

    async def stop(self) -> None:
        await self._stop_batching()
        await self._pool.stop()

    async def write(self, entry: dict[str, Any]) -> None:
        await self._enqueue_for_batch(entry)

    async def _send_batch(self, batch: list[dict]) -> None:
        """Send a batch of events as a single HTTP request."""
        try:
            payload, content_type = self._format_batch(batch)
            headers = dict(self._config.headers)
            headers["Content-Type"] = content_type

            async with self._pool.acquire() as client:
                async def _do_post() -> httpx.Response:
                    if isinstance(payload, bytes):
                        return await client.post(
                            self._config.endpoint,
                            content=payload,
                            headers=headers,
                        )
                    return await client.post(
                        self._config.endpoint,
                        json=payload,
                        headers=headers,
                    )

                if self._retrier:
                    response = await self._retrier.retry(_do_post)
                else:
                    response = await _do_post()

            self._last_status = response.status_code
            self._last_error = None

            if response.status_code >= 400:
                from ...core.diagnostics import warn
                warn(
                    "http-sink",
                    "batch delivery failed",
                    status_code=response.status_code,
                    batch_size=len(batch),
                )
                if self._metrics:
                    await self._metrics.record_events_dropped(len(batch))
            else:
                if self._metrics:
                    await self._metrics.record_events_processed(len(batch))

        except Exception as exc:
            self._last_error = str(exc)
            from ...core.diagnostics import warn
            warn(
                "http-sink",
                "batch delivery exception",
                error=str(exc),
                batch_size=len(batch),
            )
            if self._metrics:
                await self._metrics.record_events_dropped(len(batch))

    def _format_batch(self, batch: list[dict]) -> tuple[bytes | list | dict, str]:
        """Format batch according to configured format."""
        fmt = self._config.batch_format

        if fmt == BatchFormat.NDJSON:
            lines = [json.dumps(entry, default=str) for entry in batch]
            return "\n".join(lines).encode("utf-8"), "application/x-ndjson"

        if fmt == BatchFormat.WRAPPED:
            return {self._config.batch_wrapper_key: batch}, "application/json"

        # Default: array
        return batch, "application/json"

    async def health_check(self) -> bool:
        return (
            self._last_error is None
            and self._last_status is not None
            and self._last_status < 400
        )
```

### 4. Settings Integration

```python
# src/fapilog/core/settings.py

class HttpSinkSettings(BaseModel):
    """Configuration for the built-in HTTP sink."""

    endpoint: str | None = Field(default=None, ...)
    headers: dict[str, str] = Field(default_factory=dict, ...)
    # ... existing fields ...

    # NEW: Batching
    batch_size: int = Field(
        default=1,
        ge=1,
        description="Events per HTTP request (1 = no batching)",
    )
    batch_timeout_seconds: float = Field(
        default=5.0,
        gt=0.0,
        description="Max seconds before flushing partial batch",
    )
    batch_format: str = Field(
        default="array",
        description="Batch format: 'array', 'ndjson', or 'wrapped'",
    )
    batch_wrapper_key: str = Field(
        default="logs",
        description="Key for 'wrapped' format",
    )
```

---

## Configuration Examples

### Environment Variables

```bash
# Enable batching with 50 events per batch, NDJSON format
FAPILOG_SINK_CONFIG__HTTP__BATCH_SIZE=50
FAPILOG_SINK_CONFIG__HTTP__BATCH_TIMEOUT_SECONDS=2.0
FAPILOG_SINK_CONFIG__HTTP__BATCH_FORMAT=ndjson
```

### Settings Object

```python
from fapilog import Settings

settings = Settings(
    core__sinks=["http"],
    sink_config__http__endpoint="https://logs.example.com/ingest",
    sink_config__http__batch_size=100,
    sink_config__http__batch_timeout_seconds=5.0,
    sink_config__http__batch_format="array",
)
```

### Direct Instantiation

```python
from fapilog.plugins.sinks.http_client import HttpSink, HttpSinkConfig

sink = HttpSink(HttpSinkConfig(
    endpoint="https://logs.example.com/ingest",
    batch_size=100,
    batch_timeout_seconds=5.0,
    batch_format="ndjson",
))
```

---

## Test Plan

### Unit Tests

1. **test_http_sink_batching.py**

   - Test batch fills to `batch_size` before sending
   - Test batch flushes on timeout
   - Test batch flushes on stop (drain)
   - Test `batch_size=1` sends immediately (backwards compat)

2. **test_http_sink_formats.py**

   - Test `array` format produces JSON array
   - Test `ndjson` format produces newline-delimited JSON
   - Test `wrapped` format produces `{key: [...]}`
   - Test correct `Content-Type` headers

3. **test_http_sink_error_handling.py**
   - Test retry on failure
   - Test metrics recorded correctly
   - Test error containment

### Integration Tests

1. **test_http_sink_with_worker.py**
   - Test worker batching + sink batching work together
   - Test flush propagation from worker to sink

### Performance Tests

1. **benchmark_http_sink_batching.py**
   - Compare throughput: `batch_size=1` vs `batch_size=100`
   - Measure request count reduction

---

## Documentation Updates

1. Update `docs/plugins/sinks.md` with batching configuration
2. Add configuration examples for each batch format
3. Update API reference for `HttpSinkConfig`

---

## Rollout Plan

1. Add `BatchingMixin` to `_batching.py`
2. Update `HttpSinkConfig` with batch fields
3. Update `HttpSink` to use `BatchingMixin`
4. Update `HttpSinkSettings` in settings.py
5. Add unit tests
6. Update documentation
7. Consider applying same pattern to `WebhookSink` (follow-up)

---

## Related Stories

- **Story 5.5**: Cloud sink examples (provides `CloudSinkBase` pattern)
- **Story 5.0**: Wire processors (may compress batched payloads)
