# Story 5.14: Official AWS CloudWatch Logs Sink

## Status: Complete

## Priority: High

## Estimated Effort: Medium (3-4 days)

## Dependencies: None

## Epic: Cloud Sink Examples (Story 5.5)

---

## Context

fapilog currently has a CloudWatch sink in `examples/sinks/cloudwatch_sink.py`. This story promotes it to an official `contrib` sink with proper testing via LocalStack, comprehensive documentation, and clear patterns for users building SDK-based cloud integrations.

AWS CloudWatch represents the **SDK-based integration pattern** — one of two archetypal patterns for cloud sinks (the other being HTTP-based, covered by the Loki sink). This makes it valuable as both a functional sink and a learning resource.

---

## Goals

1. Provide a production-ready CloudWatch Logs sink
2. Demonstrate SDK-based async integration patterns
3. Enable CI testing via LocalStack
4. Document patterns for users building similar sinks (GCP, Azure SDK, Kafka)

---

## Acceptance Criteria

### AC1: Sink Implementation

- [ ] Move `examples/sinks/cloudwatch_sink.py` to `src/fapilog/plugins/sinks/contrib/cloudwatch.py`
- [ ] Implement full `BaseSink` protocol with lifecycle methods
- [ ] Support `write_serialized()` fast path
- [ ] Add circuit breaker integration for fault isolation
- [ ] Handle CloudWatch-specific error codes gracefully
- [ ] Respect CloudWatch limits (256KB max event size, 10K events per batch)

### AC2: Configuration

- [ ] `CloudWatchSinkConfig` dataclass with all options
- [ ] Environment variable support (`FAPILOG_CLOUDWATCH_*`)
- [ ] Settings integration in `sink_config.cloudwatch`
- [ ] Sensible defaults for common use cases

Configuration options:

```python
@dataclass
class CloudWatchSinkConfig:
    log_group_name: str         # Required
    log_stream_name: str        # Required or auto-generated
    region: str                 # Default: from AWS_REGION env
    create_log_group: bool      # Default: True
    create_log_stream: bool     # Default: True
    batch_size: int             # Default: 100 (max 10,000)
    batch_timeout_seconds: float # Default: 5.0
    max_retries: int            # Default: 3
    endpoint_url: str | None    # For LocalStack testing
```

### AC3: LocalStack Testing

- [ ] Integration tests run against LocalStack in CI
- [ ] Tests cover: log group creation, stream creation, batch sending, error handling
- [ ] Tests verify sequence token handling
- [ ] Tests verify payload format matches CloudWatch API spec
- [ ] GitHub Actions workflow with LocalStack service

### AC4: Error Handling

- [ ] Handle `InvalidSequenceTokenException` with automatic retry
- [ ] Handle `DataAlreadyAcceptedException` gracefully
- [ ] Handle `ResourceNotFoundException` with clear error message
- [ ] Handle `ResourceAlreadyExistsException` on creation (idempotent)
- [ ] Handle throttling (`ThrottlingException`) with backoff
- [ ] Emit diagnostics for all error conditions

### AC5: Registration and Discovery

- [ ] Register as `cloudwatch` in `fapilog.sinks` entry point group
- [ ] Loadable via `core.sinks = ["cloudwatch"]`
- [ ] `PLUGIN_METADATA` with dependencies declared
- [ ] Optional dependency: `fapilog[aws]` installs boto3

### AC6: Documentation

- [ ] `docs/plugins/sinks/cloudwatch.md` - Full reference documentation
- [ ] Configuration examples (env vars, Settings object, programmatic)
- [ ] IAM permissions required (document minimum permissions)
- [ ] Troubleshooting guide for common issues
- [ ] Link to "Building SDK-based Sinks" pattern guide

### AC7: Example Updates

- [ ] Remove `examples/sinks/cloudwatch_sink.py` (now in contrib)
- [ ] Update `examples/sinks/__init__.py`
- [ ] Add `examples/cloudwatch_logging/` with complete working example
- [ ] Example includes: FastAPI app, Docker Compose with LocalStack, README

---

## Technical Design

### 1. File Structure

```
src/fapilog/plugins/sinks/
├── contrib/
│   ├── __init__.py
│   └── cloudwatch.py       # New location
```

### 2. Implementation

```python
# src/fapilog/plugins/sinks/contrib/cloudwatch.py

from __future__ import annotations

import asyncio
import json
import time
from dataclasses import dataclass, field
from typing import Any

from ....core import diagnostics
from ....core.circuit_breaker import SinkCircuitBreaker, SinkCircuitBreakerConfig
from ....core.serialization import SerializedView
from .._batching import BatchingMixin

# Lazy import boto3 to avoid hard dependency
boto3: Any = None
ClientError: type = Exception


def _ensure_boto3() -> None:
    global boto3, ClientError
    if boto3 is None:
        import boto3 as _boto3
        from botocore.exceptions import ClientError as _ClientError
        boto3 = _boto3
        ClientError = _ClientError


# CloudWatch Limits
MAX_EVENT_SIZE_BYTES = 256 * 1024  # 256 KB
MAX_BATCH_SIZE = 10_000
MAX_BATCH_BYTES = 1_048_576  # 1 MB


@dataclass
class CloudWatchSinkConfig:
    """Configuration for AWS CloudWatch Logs sink."""

    log_group_name: str
    log_stream_name: str | None = None  # Auto-generate if None
    region: str | None = field(
        default_factory=lambda: os.getenv("AWS_REGION", os.getenv("AWS_DEFAULT_REGION"))
    )
    create_log_group: bool = True
    create_log_stream: bool = True
    batch_size: int = 100
    batch_timeout_seconds: float = 5.0
    max_retries: int = 3
    retry_base_delay: float = 0.5
    endpoint_url: str | None = None  # For LocalStack
    circuit_breaker_enabled: bool = True
    circuit_breaker_threshold: int = 5


class CloudWatchSink(BatchingMixin):
    """AWS CloudWatch Logs sink with batching and retry.

    This sink demonstrates the SDK-based integration pattern:
    - Wraps blocking boto3 calls in asyncio.to_thread()
    - Handles AWS-specific authentication via environment/IAM
    - Manages cloud resources (creates log group/stream if needed)
    - Handles provider-specific quirks (sequence tokens)

    Use this as a reference for building sinks for other SDK-based
    providers (GCP, Azure SDK, Kafka, etc.).
    """

    name = "cloudwatch"

    def __init__(
        self,
        config: CloudWatchSinkConfig | None = None,
        **kwargs: Any,
    ) -> None:
        if config is None:
            # Allow construction from kwargs for settings integration
            config = CloudWatchSinkConfig(**kwargs)

        self._config = config
        self._client: Any = None
        self._sequence_token: str | None = None
        self._circuit_breaker: SinkCircuitBreaker | None = None

        # Initialize batching
        self._init_batching(config.batch_size, config.batch_timeout_seconds)

    async def start(self) -> None:
        _ensure_boto3()

        client_kwargs: dict[str, Any] = {}
        if self._config.region:
            client_kwargs["region_name"] = self._config.region
        if self._config.endpoint_url:
            client_kwargs["endpoint_url"] = self._config.endpoint_url

        self._client = await asyncio.to_thread(
            boto3.client, "logs", **client_kwargs
        )

        if self._config.create_log_group:
            await self._ensure_log_group()
        if self._config.create_log_stream:
            await self._ensure_log_stream()

        if self._config.circuit_breaker_enabled:
            self._circuit_breaker = SinkCircuitBreaker(
                self.name,
                SinkCircuitBreakerConfig(
                    enabled=True,
                    failure_threshold=self._config.circuit_breaker_threshold,
                ),
            )

        await self._start_batching()

    async def stop(self) -> None:
        await self._stop_batching()
        self._client = None

    async def write(self, entry: dict[str, Any]) -> None:
        await self._enqueue_for_batch(entry)

    async def write_serialized(self, view: SerializedView) -> None:
        """Fast path for pre-serialized payloads."""
        try:
            # CloudWatch expects JSON string in message field
            # SerializedView is already JSON, use directly
            event = {
                "timestamp": int(time.time() * 1000),
                "message": bytes(view.data).decode("utf-8"),
            }
            await self._enqueue_for_batch(event)
        except Exception:
            pass  # Contain errors

    async def _send_batch(self, batch: list[dict[str, Any]]) -> None:
        if self._circuit_breaker and not self._circuit_breaker.should_allow():
            diagnostics.warn(
                "cloudwatch-sink",
                "circuit breaker open, dropping batch",
                batch_size=len(batch),
            )
            return

        # Transform entries to CloudWatch format
        log_events = []
        for entry in batch:
            if "timestamp" in entry and "message" in entry:
                # Already formatted (from write_serialized)
                log_events.append(entry)
            else:
                log_events.append({
                    "timestamp": int(time.time() * 1000),
                    "message": json.dumps(entry, default=str),
                })

        # Sort by timestamp (CloudWatch requirement)
        log_events.sort(key=lambda x: x["timestamp"])

        await self._put_log_events_with_retry(log_events)

    async def _put_log_events_with_retry(
        self, log_events: list[dict[str, Any]]
    ) -> None:
        """Send log events with retry and sequence token handling."""
        for attempt in range(self._config.max_retries):
            try:
                kwargs: dict[str, Any] = {
                    "logGroupName": self._config.log_group_name,
                    "logStreamName": self._config.log_stream_name,
                    "logEvents": log_events,
                }
                if self._sequence_token:
                    kwargs["sequenceToken"] = self._sequence_token

                response = await asyncio.to_thread(
                    self._client.put_log_events, **kwargs
                )
                self._sequence_token = response.get("nextSequenceToken")

                if self._circuit_breaker:
                    self._circuit_breaker.record_success()
                return

            except ClientError as e:
                error_code = e.response.get("Error", {}).get("Code", "")

                if error_code in ("InvalidSequenceTokenException", "DataAlreadyAcceptedException"):
                    # Extract correct token and retry immediately
                    self._sequence_token = e.response.get("Error", {}).get(
                        "expectedSequenceToken"
                    )
                    continue

                if error_code == "ThrottlingException":
                    delay = self._config.retry_base_delay * (2 ** attempt)
                    await asyncio.sleep(delay)
                    continue

                # Other errors
                if self._circuit_breaker:
                    self._circuit_breaker.record_failure()

                diagnostics.warn(
                    "cloudwatch-sink",
                    "failed to send logs",
                    error_code=error_code,
                    attempt=attempt + 1,
                    batch_size=len(log_events),
                )

                if attempt == self._config.max_retries - 1:
                    return  # Give up, contain error

                delay = self._config.retry_base_delay * (2 ** attempt)
                await asyncio.sleep(delay)

    async def _ensure_log_group(self) -> None:
        try:
            await asyncio.to_thread(
                self._client.create_log_group,
                logGroupName=self._config.log_group_name,
            )
        except ClientError as e:
            if e.response.get("Error", {}).get("Code") != "ResourceAlreadyExistsException":
                raise

    async def _ensure_log_stream(self) -> None:
        if not self._config.log_stream_name:
            # Auto-generate stream name
            import socket
            self._config.log_stream_name = f"{socket.gethostname()}-{int(time.time())}"

        try:
            await asyncio.to_thread(
                self._client.create_log_stream,
                logGroupName=self._config.log_group_name,
                logStreamName=self._config.log_stream_name,
            )
        except ClientError as e:
            if e.response.get("Error", {}).get("Code") != "ResourceAlreadyExistsException":
                raise

    async def health_check(self) -> bool:
        if not self._client:
            return False

        if self._circuit_breaker and self._circuit_breaker.is_open:
            return False

        try:
            await asyncio.to_thread(
                self._client.describe_log_streams,
                logGroupName=self._config.log_group_name,
                limit=1,
            )
            return True
        except Exception:
            return False


PLUGIN_METADATA = {
    "name": "cloudwatch",
    "version": "1.0.0",
    "plugin_type": "sink",
    "entry_point": "fapilog.plugins.sinks.contrib.cloudwatch:CloudWatchSink",
    "description": "AWS CloudWatch Logs sink with batching and retry.",
    "author": "Fapilog Core",
    "compatibility": {"min_fapilog_version": "0.4.0"},
    "api_version": "1.0",
    "dependencies": ["boto3>=1.26.0"],
}
```

### 3. Settings Integration

```python
# In src/fapilog/core/settings.py

class CloudWatchSinkSettings(BaseModel):
    log_group_name: str = Field(default="/fapilog/default")
    log_stream_name: str | None = Field(default=None)
    region: str | None = Field(default=None)
    create_log_group: bool = Field(default=True)
    create_log_stream: bool = Field(default=True)
    batch_size: int = Field(default=100, ge=1, le=10000)
    batch_timeout_seconds: float = Field(default=5.0, gt=0)
    endpoint_url: str | None = Field(default=None)  # For LocalStack

class SinkConfigSettings(BaseModel):
    # ... existing ...
    cloudwatch: CloudWatchSinkSettings = Field(default_factory=CloudWatchSinkSettings)
```

### 4. LocalStack CI Testing

```yaml
# .github/workflows/test-cloudwatch-sink.yml
name: CloudWatch Sink Tests

on:
  push:
    paths:
      - "src/fapilog/plugins/sinks/contrib/cloudwatch.py"
      - "tests/integration/test_cloudwatch_*.py"
  pull_request:
    paths:
      - "src/fapilog/plugins/sinks/contrib/cloudwatch.py"
      - "tests/integration/test_cloudwatch_*.py"

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      localstack:
        image: localstack/localstack:latest
        ports:
          - 4566:4566
        env:
          SERVICES: logs
          DEFAULT_REGION: us-east-1

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          pip install -e ".[aws,test]"

      - name: Wait for LocalStack
        run: |
          timeout 30 bash -c 'until curl -s http://localhost:4566/_localstack/health | grep -q "running"; do sleep 1; done'

      - name: Run CloudWatch tests
        run: |
          pytest tests/integration/test_cloudwatch_sink.py -v
        env:
          AWS_ACCESS_KEY_ID: test
          AWS_SECRET_ACCESS_KEY: test
          AWS_DEFAULT_REGION: us-east-1
          LOCALSTACK_ENDPOINT: http://localhost:4566
```

### 5. Test Implementation

```python
# tests/integration/test_cloudwatch_sink.py

import os
import pytest
import boto3

# Skip if LocalStack not available
LOCALSTACK_ENDPOINT = os.getenv("LOCALSTACK_ENDPOINT", "http://localhost:4566")

@pytest.fixture
def cloudwatch_client():
    """Create a CloudWatch Logs client pointing to LocalStack."""
    return boto3.client(
        "logs",
        endpoint_url=LOCALSTACK_ENDPOINT,
        region_name="us-east-1",
        aws_access_key_id="test",
        aws_secret_access_key="test",
    )

@pytest.fixture
def log_group_name():
    return f"/fapilog/test-{int(time.time())}"

class TestCloudWatchSink:
    async def test_creates_log_group_and_stream(self, cloudwatch_client, log_group_name):
        from fapilog.plugins.sinks.contrib.cloudwatch import (
            CloudWatchSink,
            CloudWatchSinkConfig,
        )

        config = CloudWatchSinkConfig(
            log_group_name=log_group_name,
            log_stream_name="test-stream",
            endpoint_url=LOCALSTACK_ENDPOINT,
        )
        sink = CloudWatchSink(config)
        await sink.start()

        # Verify resources created
        groups = cloudwatch_client.describe_log_groups(
            logGroupNamePrefix=log_group_name
        )
        assert len(groups["logGroups"]) == 1

        await sink.stop()

    async def test_sends_logs_successfully(self, cloudwatch_client, log_group_name):
        from fapilog.plugins.sinks.contrib.cloudwatch import (
            CloudWatchSink,
            CloudWatchSinkConfig,
        )

        config = CloudWatchSinkConfig(
            log_group_name=log_group_name,
            log_stream_name="test-stream",
            endpoint_url=LOCALSTACK_ENDPOINT,
            batch_size=1,  # Flush immediately for testing
        )
        sink = CloudWatchSink(config)
        await sink.start()

        await sink.write({"level": "INFO", "message": "test log"})
        await sink.stop()

        # Verify log was written
        events = cloudwatch_client.get_log_events(
            logGroupName=log_group_name,
            logStreamName="test-stream",
        )
        assert len(events["events"]) == 1
        assert "test log" in events["events"][0]["message"]

    async def test_handles_batch_correctly(self, cloudwatch_client, log_group_name):
        from fapilog.plugins.sinks.contrib.cloudwatch import (
            CloudWatchSink,
            CloudWatchSinkConfig,
        )

        config = CloudWatchSinkConfig(
            log_group_name=log_group_name,
            log_stream_name="test-stream",
            endpoint_url=LOCALSTACK_ENDPOINT,
            batch_size=10,
        )
        sink = CloudWatchSink(config)
        await sink.start()

        # Send 25 logs (should create 3 batches: 10, 10, 5)
        for i in range(25):
            await sink.write({"level": "INFO", "message": f"log {i}"})

        await sink.stop()

        events = cloudwatch_client.get_log_events(
            logGroupName=log_group_name,
            logStreamName="test-stream",
        )
        assert len(events["events"]) == 25

    async def test_health_check_returns_true_when_healthy(self, log_group_name):
        from fapilog.plugins.sinks.contrib.cloudwatch import (
            CloudWatchSink,
            CloudWatchSinkConfig,
        )

        config = CloudWatchSinkConfig(
            log_group_name=log_group_name,
            log_stream_name="test-stream",
            endpoint_url=LOCALSTACK_ENDPOINT,
        )
        sink = CloudWatchSink(config)
        await sink.start()

        assert await sink.health_check() is True

        await sink.stop()
```

---

## Documentation

### docs/plugins/sinks/cloudwatch.md

````markdown
# AWS CloudWatch Logs Sink

Send structured logs to AWS CloudWatch Logs with automatic batching,
retry, and resource provisioning.

## Installation

```bash
pip install fapilog[aws]
```
````

## Quick Start

```python
import fapilog

# Configure via environment
# AWS_REGION=us-east-1
# FAPILOG_CLOUDWATCH__LOG_GROUP_NAME=/myapp/logs

with fapilog.runtime() as logger:
    logger.info("Hello CloudWatch!")
```

## Configuration

### Environment Variables

| Variable                               | Description             | Default          |
| -------------------------------------- | ----------------------- | ---------------- |
| `FAPILOG_CLOUDWATCH__LOG_GROUP_NAME`   | CloudWatch log group    | Required         |
| `FAPILOG_CLOUDWATCH__LOG_STREAM_NAME`  | Log stream name         | Auto-generated   |
| `FAPILOG_CLOUDWATCH__REGION`           | AWS region              | `AWS_REGION` env |
| `FAPILOG_CLOUDWATCH__BATCH_SIZE`       | Events per batch        | 100              |
| `FAPILOG_CLOUDWATCH__CREATE_LOG_GROUP` | Create group if missing | true             |

### Programmatic Configuration

```python
from fapilog import Settings
from fapilog.plugins.sinks.contrib.cloudwatch import CloudWatchSink, CloudWatchSinkConfig

config = CloudWatchSinkConfig(
    log_group_name="/myapp/production",
    log_stream_name="web-server-1",
    region="us-west-2",
    batch_size=200,
)

sink = CloudWatchSink(config)
```

## IAM Permissions

Minimum required permissions:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/myapp/*"
    }
  ]
}
```

## CloudWatch Limits

| Limit           | Value         | Handling               |
| --------------- | ------------- | ---------------------- |
| Max event size  | 256 KB        | Use SizeGuardProcessor |
| Max batch size  | 10,000 events | Auto-chunked           |
| Max batch bytes | 1 MB          | Auto-chunked           |

## Troubleshooting

### InvalidSequenceTokenException

This is handled automatically. The sink retries with the correct token.

### ThrottlingException

The sink uses exponential backoff. If persistent, reduce `batch_size`
or increase `batch_timeout_seconds`.

### ResourceNotFoundException

Ensure `create_log_group` and `create_log_stream` are `true`, or
create resources manually before starting the sink.

## Building Similar Sinks

This sink demonstrates the **SDK-based integration pattern**:

1. Lazy-load the SDK to avoid hard dependencies
2. Wrap blocking SDK calls in `asyncio.to_thread()`
3. Handle provider-specific authentication via environment
4. Manage cloud resources (create if not exists)
5. Handle provider-specific quirks (sequence tokens)

See [Building SDK-Based Sinks](../patterns/sdk-sinks.md) for a detailed guide.

```

---

## Example Application

### examples/cloudwatch_logging/

```

examples/cloudwatch_logging/
├── README.md
├── docker-compose.yml # LocalStack for local testing
├── main.py # FastAPI app example
└── requirements.txt

````

**docker-compose.yml:**
```yaml
version: '3.8'
services:
  localstack:
    image: localstack/localstack:latest
    ports:
      - "4566:4566"
    environment:
      - SERVICES=logs
      - DEFAULT_REGION=us-east-1
````

**main.py:**

```python
"""Example FastAPI app logging to CloudWatch."""
import os
from fastapi import FastAPI
import fapilog

app = FastAPI()

@app.on_event("startup")
async def setup_logging():
    # For local testing with LocalStack
    os.environ.setdefault("FAPILOG_CLOUDWATCH__ENDPOINT_URL", "http://localhost:4566")
    os.environ.setdefault("FAPILOG_CLOUDWATCH__LOG_GROUP_NAME", "/example/fastapi")

@app.get("/")
async def root():
    logger = fapilog.get_logger()
    logger.info("Request received", path="/")
    return {"status": "ok"}
```

---

## Migration Notes

- Remove `examples/sinks/cloudwatch_sink.py`
- Update any imports from `examples.sinks.cloudwatch_sink` to `fapilog.plugins.sinks.contrib.cloudwatch`

---

## Success Metrics

- [ ] All LocalStack integration tests pass
- [ ] Documentation complete with IAM permissions
- [ ] Example application works with LocalStack
- [ ] No regression in existing functionality
- [ ] Health check correctly reports status

---

## Related Stories

- Story 5.5: Cloud Sink Examples (parent epic)
- Story 5.13: SizeGuardProcessor (recommended for CloudWatch's 256KB limit)
- Story 5.15: Loki Sink (HTTP-based pattern counterpart)
