# Story 5.5: Cloud Sink Examples and Documentation (Epic)

## Status: In Progress (Examples Complete, Official Sinks Pending)

## Priority: High

## Type: Epic

## Dependencies: None

## Implementing Stories

| Story    | Title                                              | Status     |
| -------- | -------------------------------------------------- | ---------- |
| **5.14** | [AWS CloudWatch Sink](./5.14.aws-cloudwatch-sink.md) | Ready      |
| **5.15** | [Grafana Loki Sink](./5.15.loki-sink.md)            | Ready      |

---

## Overview

This epic covers cloud sink integrations for fapilog. The approach is:

1. **Official Sinks** (Stories 5.14, 5.15): Two fully-tested, production-ready sinks that serve as reference implementations:
   - **AWS CloudWatch** (5.14) — SDK-based pattern (testable via LocalStack)
   - **Grafana Loki** (5.15) — HTTP-based pattern (testable via Docker)

2. **Example Code** (this story): Additional examples in `examples/sinks/` for users who need patterns for other providers.

The two official sinks are selected because they can be **fully integration tested in CI** without requiring cloud accounts, making them maintainable and trustworthy.

---

## Context

Enterprise users typically need to send logs to cloud observability platforms:

- AWS CloudWatch Logs
- Google Cloud Logging (formerly Stackdriver)
- Azure Monitor Logs
- Datadog
- Splunk
- Elastic/OpenSearch

fapilog currently lacks examples and guidance for these integrations. While the HTTP sink can be used, users need:

1. Authentication patterns
2. Batch formatting for each platform
3. Rate limiting considerations
4. Retry strategies for cloud APIs

---

## Problem Statement

Users evaluating fapilog for enterprise use cannot easily:

1. See how to integrate with their cloud provider
2. Understand authentication requirements
3. Implement proper batching for cost optimization
4. Handle cloud-specific error responses

---

## Quick Start: End-to-End Example

Before diving into implementation details, here's a complete working example:

```python
# app.py - Complete AWS CloudWatch integration
from fapilog import get_logger, Settings

# Import the cloud sink (after installing from examples/)
from examples.sinks.cloudwatch_sink import CloudWatchSink, CloudWatchSinkConfig

# 1. Configure the sink
sink_config = CloudWatchSinkConfig(
    log_group_name="/myapp/production",
    log_stream_name="api-server",
    region="us-west-2",
)
cloudwatch_sink = CloudWatchSink(config=sink_config)

# 2. Get a logger with the custom sink
settings = Settings()
logger = get_logger(settings=settings, sinks=[cloudwatch_sink])

# 3. Start logging!
logger.info("Application started", version="1.2.3")
logger.error("Database connection failed", host="db.example.com")

# 4. Ensure graceful shutdown
import asyncio
asyncio.run(logger.stop_and_drain())
```

### Alternative: Entry Point Registration

For production deployments, register sinks via `pyproject.toml`:

```toml
# pyproject.toml
[project.entry-points."fapilog.sinks"]
cloudwatch = "myapp.sinks.cloudwatch_sink:CloudWatchSink"
datadog = "myapp.sinks.datadog_sink:DatadogSink"
```

Then use by name in settings:

```yaml
# config.yaml or environment variables
core:
  sinks: ["cloudwatch"]

sink_config:
  cloudwatch:
    log_group_name: "/myapp/production"
    log_stream_name: "api-server"
    region: "us-west-2"
```

```python
# app.py - Using entry point registration
from fapilog import get_logger

# Sink loaded automatically from entry points
logger = get_logger()
logger.info("Using CloudWatch via entry point")
```

---

## Acceptance Criteria

### AC1: AWS CloudWatch Example

- [ ] Complete example sink for CloudWatch Logs
- [ ] IAM authentication (boto3 credentials)
- [ ] Batch PutLogEvents implementation
- [ ] Sequence token handling
- [ ] Retry with exponential backoff

### AC2: GCP Cloud Logging Example

- [ ] Complete example using google-cloud-logging
- [ ] Service account authentication
- [ ] Structured logging format
- [ ] Resource labeling

### AC3: Datadog Example

- [ ] HTTP-based sink using Datadog API
- [ ] API key authentication
- [ ] Proper JSON format for Datadog
- [ ] Tag propagation

### AC4: Generic Cloud Sink Pattern

- [ ] Abstract base class for cloud sinks
- [ ] Common patterns documented
- [ ] Authentication abstraction
- [ ] Batch buffer with flush

### AC5: Documentation

- [ ] Step-by-step integration guide for each cloud
- [ ] Configuration examples (env vars, settings)
- [ ] Troubleshooting common issues
- [ ] Cost optimization tips

### AC6: Usage Examples

- [ ] End-to-end quick start example at top of each guide
- [ ] Direct instantiation pattern with `get_logger(sinks=[...])`
- [ ] Entry point registration via `pyproject.toml`
- [ ] Environment variable configuration for secrets
- [ ] Settings-based configuration via YAML/env

---

## Technical Design

### 1. AWS CloudWatch Sink Example

```python
# examples/sinks/cloudwatch_sink.py
"""
AWS CloudWatch Logs sink for fapilog.

Requires: pip install boto3

Configuration via environment:
- AWS_REGION or AWS_DEFAULT_REGION
- AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
- FAPILOG_CLOUDWATCH_LOG_GROUP
- FAPILOG_CLOUDWATCH_LOG_STREAM
"""

from __future__ import annotations

import asyncio
import json
import os
import time
from dataclasses import dataclass, field
from typing import Any

import boto3
from botocore.exceptions import ClientError


@dataclass
class CloudWatchSinkConfig:
    """Configuration for CloudWatch sink."""

    log_group_name: str = field(
        default_factory=lambda: os.getenv("FAPILOG_CLOUDWATCH_LOG_GROUP", "/fapilog/default")
    )
    log_stream_name: str = field(
        default_factory=lambda: os.getenv("FAPILOG_CLOUDWATCH_LOG_STREAM", "default")
    )
    region: str = field(
        default_factory=lambda: os.getenv("AWS_REGION", os.getenv("AWS_DEFAULT_REGION", "us-east-1"))
    )
    batch_size: int = 100
    batch_timeout_seconds: float = 5.0
    create_log_group: bool = True
    create_log_stream: bool = True


class CloudWatchSink:
    """Async sink that batches and sends logs to AWS CloudWatch Logs."""

    name = "cloudwatch"

    def __init__(self, config: CloudWatchSinkConfig | None = None) -> None:
        self._config = config or CloudWatchSinkConfig()
        self._client: Any = None
        self._sequence_token: str | None = None
        self._batch: list[dict[str, Any]] = []
        self._batch_lock = asyncio.Lock()
        self._last_flush = time.monotonic()
        self._flush_task: asyncio.Task[None] | None = None

    async def start(self) -> None:
        """Initialize CloudWatch client and ensure log group/stream exist."""
        # Create boto3 client in thread to avoid blocking
        self._client = await asyncio.to_thread(
            boto3.client,
            "logs",
            region_name=self._config.region,
        )

        if self._config.create_log_group:
            await self._ensure_log_group()

        if self._config.create_log_stream:
            await self._ensure_log_stream()

        # Start background flush task
        self._flush_task = asyncio.create_task(self._flush_loop())

    async def stop(self) -> None:
        """Flush remaining logs and cleanup."""
        if self._flush_task:
            self._flush_task.cancel()
            try:
                await self._flush_task
            except asyncio.CancelledError:
                pass

        # Final flush
        await self._flush_batch()

    async def write(self, entry: dict) -> None:
        """Add entry to batch for sending."""
        try:
            async with self._batch_lock:
                self._batch.append({
                    "timestamp": int(time.time() * 1000),
                    "message": json.dumps(entry, default=str),
                })

                if len(self._batch) >= self._config.batch_size:
                    await self._flush_batch()
        except Exception:
            # Contain errors
            pass

    async def health_check(self) -> bool:
        """Check CloudWatch connectivity."""
        try:
            await asyncio.to_thread(
                self._client.describe_log_streams,
                logGroupName=self._config.log_group_name,
                limit=1,
            )
            return True
        except Exception:
            return False

    async def _ensure_log_group(self) -> None:
        """Create log group if it doesn't exist."""
        try:
            await asyncio.to_thread(
                self._client.create_log_group,
                logGroupName=self._config.log_group_name,
            )
        except ClientError as e:
            if e.response["Error"]["Code"] != "ResourceAlreadyExistsException":
                raise

    async def _ensure_log_stream(self) -> None:
        """Create log stream if it doesn't exist."""
        try:
            await asyncio.to_thread(
                self._client.create_log_stream,
                logGroupName=self._config.log_group_name,
                logStreamName=self._config.log_stream_name,
            )
        except ClientError as e:
            if e.response["Error"]["Code"] != "ResourceAlreadyExistsException":
                raise

    async def _flush_loop(self) -> None:
        """Background loop for time-based flushing."""
        while True:
            await asyncio.sleep(self._config.batch_timeout_seconds)
            if self._batch:
                await self._flush_batch()

    async def _flush_batch(self) -> None:
        """Send batch to CloudWatch."""
        async with self._batch_lock:
            if not self._batch:
                return

            batch = self._batch[:]
            self._batch = []

        try:
            kwargs: dict[str, Any] = {
                "logGroupName": self._config.log_group_name,
                "logStreamName": self._config.log_stream_name,
                "logEvents": sorted(batch, key=lambda x: x["timestamp"]),
            }

            if self._sequence_token:
                kwargs["sequenceToken"] = self._sequence_token

            response = await asyncio.to_thread(
                self._client.put_log_events,
                **kwargs,
            )

            self._sequence_token = response.get("nextSequenceToken")

        except ClientError as e:
            error_code = e.response["Error"]["Code"]

            if error_code in ("InvalidSequenceTokenException", "DataAlreadyAcceptedException"):
                # Get correct sequence token and retry
                self._sequence_token = e.response["Error"].get("expectedSequenceToken")
                # Re-add batch for retry
                async with self._batch_lock:
                    self._batch = batch + self._batch
            else:
                # Log and drop
                pass


# Plugin metadata
PLUGIN_METADATA = {
    "name": "cloudwatch",
    "version": "1.0.0",
    "plugin_type": "sink",
    "entry_point": "examples.sinks.cloudwatch_sink:CloudWatchSink",
    "description": "AWS CloudWatch Logs sink with batching.",
    "author": "Fapilog Examples",
    "compatibility": {"min_fapilog_version": "0.4.0"},
    "api_version": "1.0",
    "dependencies": ["boto3>=1.26.0"],
}
```

#### Usage: AWS CloudWatch

**Direct instantiation:**

```python
from fapilog import get_logger, Settings
from examples.sinks.cloudwatch_sink import CloudWatchSink, CloudWatchSinkConfig

# Configure via code
sink = CloudWatchSink(config=CloudWatchSinkConfig(
    log_group_name="/myapp/logs",
    log_stream_name="web-server",
    region="eu-west-1",
    batch_size=50,
))

logger = get_logger(settings=Settings(), sinks=[sink])
logger.info("CloudWatch logging initialized")
```

**Via environment variables:**

```bash
export AWS_REGION=us-east-1
export FAPILOG_CLOUDWATCH_LOG_GROUP=/production/api
export FAPILOG_CLOUDWATCH_LOG_STREAM=instance-001
```

```python
from fapilog import get_logger
from examples.sinks.cloudwatch_sink import CloudWatchSink

# Config auto-populated from environment
logger = get_logger(sinks=[CloudWatchSink()])
```

**Entry point registration (`pyproject.toml`):**

```toml
[project.entry-points."fapilog.sinks"]
cloudwatch = "myapp.sinks.cloudwatch_sink:CloudWatchSink"
```

```yaml
# settings.yaml
core:
  sinks: ["cloudwatch"]
sink_config:
  cloudwatch:
    log_group_name: "/myapp/logs"
    batch_size: 100
```

### 2. Datadog Sink Example

```python
# examples/sinks/datadog_sink.py
"""
Datadog Logs sink for fapilog.

Configuration via environment:
- DD_API_KEY: Datadog API key (required)
- DD_SITE: Datadog site (default: datadoghq.com)
- DD_SERVICE: Service name for logs
- DD_ENV: Environment tag
"""

from __future__ import annotations

import asyncio
import json
import os
from dataclasses import dataclass, field

import httpx


@dataclass
class DatadogSinkConfig:
    """Configuration for Datadog sink."""

    api_key: str = field(
        default_factory=lambda: os.getenv("DD_API_KEY", "")
    )
    site: str = field(
        default_factory=lambda: os.getenv("DD_SITE", "datadoghq.com")
    )
    service: str = field(
        default_factory=lambda: os.getenv("DD_SERVICE", "fapilog")
    )
    env: str = field(
        default_factory=lambda: os.getenv("DD_ENV", "dev")
    )
    source: str = "python"
    batch_size: int = 100
    timeout_seconds: float = 10.0


class DatadogSink:
    """Async sink that sends logs to Datadog."""

    name = "datadog"

    def __init__(self, config: DatadogSinkConfig | None = None) -> None:
        self._config = config or DatadogSinkConfig()
        self._client: httpx.AsyncClient | None = None
        self._url = f"https://http-intake.logs.{self._config.site}/api/v2/logs"
        self._batch: list[dict] = []
        self._batch_lock = asyncio.Lock()

    async def start(self) -> None:
        self._client = httpx.AsyncClient(
            timeout=self._config.timeout_seconds,
            headers={
                "DD-API-KEY": self._config.api_key,
                "Content-Type": "application/json",
            },
        )

    async def stop(self) -> None:
        await self._flush_batch()
        if self._client:
            await self._client.aclose()

    async def write(self, entry: dict) -> None:
        try:
            dd_entry = {
                "message": entry.get("message", ""),
                "ddsource": self._config.source,
                "ddtags": f"env:{self._config.env}",
                "service": self._config.service,
                "status": self._level_to_status(entry.get("level", "INFO")),
                **{k: v for k, v in entry.items() if k not in ("message", "level")},
            }

            async with self._batch_lock:
                self._batch.append(dd_entry)

                if len(self._batch) >= self._config.batch_size:
                    await self._flush_batch()
        except Exception:
            pass

    async def _flush_batch(self) -> None:
        async with self._batch_lock:
            if not self._batch or not self._client:
                return

            batch = self._batch[:]
            self._batch = []

        try:
            await self._client.post(
                self._url,
                content=json.dumps(batch),
            )
        except Exception:
            pass

    def _level_to_status(self, level: str) -> str:
        mapping = {
            "DEBUG": "debug",
            "INFO": "info",
            "WARNING": "warn",
            "WARN": "warn",
            "ERROR": "error",
            "CRITICAL": "critical",
        }
        return mapping.get(level.upper(), "info")

    async def health_check(self) -> bool:
        return self._client is not None and not self._client.is_closed


PLUGIN_METADATA = {
    "name": "datadog",
    "version": "1.0.0",
    "plugin_type": "sink",
    "entry_point": "examples.sinks.datadog_sink:DatadogSink",
    "description": "Datadog Logs HTTP sink.",
    "author": "Fapilog Examples",
    "compatibility": {"min_fapilog_version": "0.4.0"},
    "api_version": "1.0",
    "dependencies": ["httpx>=0.24.0"],
}
```

#### Usage: Datadog

**Direct instantiation:**

```python
from fapilog import get_logger, Settings
from examples.sinks.datadog_sink import DatadogSink, DatadogSinkConfig

sink = DatadogSink(config=DatadogSinkConfig(
    api_key="your-api-key-here",
    site="datadoghq.eu",  # For EU region
    service="my-api",
    env="production",
))

logger = get_logger(settings=Settings(), sinks=[sink])
logger.info("Hello Datadog!", user_id=123)
```

**Via environment variables (recommended for secrets):**

```bash
export DD_API_KEY=your-api-key-here
export DD_SITE=datadoghq.com
export DD_SERVICE=my-api
export DD_ENV=production
```

```python
from fapilog import get_logger
from examples.sinks.datadog_sink import DatadogSink

# Config auto-populated from DD_* environment variables
logger = get_logger(sinks=[DatadogSink()])
```

**Entry point registration:**

```toml
[project.entry-points."fapilog.sinks"]
datadog = "myapp.sinks.datadog_sink:DatadogSink"
```

```yaml
# settings.yaml (DO NOT put API key here - use env var)
core:
  sinks: ["datadog"]
sink_config:
  datadog:
    service: "my-api"
    env: "production"
    batch_size: 50
```

### 3. Abstract Cloud Sink Base

```python
# examples/sinks/cloud_sink_base.py
"""
Abstract base class for cloud logging sinks.

Provides common patterns for:
- Batching with size and time limits
- Retry with exponential backoff
- Error containment
- Health checks
"""

from __future__ import annotations

import abc
import asyncio
import time
from dataclasses import dataclass
from typing import Any, Generic, TypeVar

T = TypeVar("T")


@dataclass
class CloudSinkConfig:
    """Base configuration for cloud sinks."""

    batch_size: int = 100
    batch_timeout_seconds: float = 5.0
    max_retries: int = 3
    retry_base_delay: float = 1.0
    retry_max_delay: float = 30.0


class CloudSinkBase(abc.ABC, Generic[T]):
    """Abstract base for cloud logging sinks."""

    name: str = "cloud_base"

    def __init__(self, config: CloudSinkConfig) -> None:
        self._config = config
        self._batch: list[T] = []
        self._batch_lock = asyncio.Lock()
        self._last_flush = time.monotonic()
        self._flush_task: asyncio.Task[None] | None = None

    async def start(self) -> None:
        await self._initialize_client()
        self._flush_task = asyncio.create_task(self._flush_loop())

    async def stop(self) -> None:
        if self._flush_task:
            self._flush_task.cancel()
        await self._flush_batch()
        await self._cleanup_client()

    async def write(self, entry: dict) -> None:
        try:
            transformed = self._transform_entry(entry)

            async with self._batch_lock:
                self._batch.append(transformed)

                if len(self._batch) >= self._config.batch_size:
                    await self._flush_batch()
        except Exception:
            pass

    async def _flush_loop(self) -> None:
        while True:
            await asyncio.sleep(self._config.batch_timeout_seconds)
            if self._batch:
                await self._flush_batch()

    async def _flush_batch(self) -> None:
        async with self._batch_lock:
            if not self._batch:
                return
            batch = self._batch[:]
            self._batch = []

        await self._send_with_retry(batch)

    async def _send_with_retry(self, batch: list[T]) -> None:
        delay = self._config.retry_base_delay

        for attempt in range(self._config.max_retries):
            try:
                await self._send_batch(batch)
                return
            except Exception:
                if attempt == self._config.max_retries - 1:
                    break
                await asyncio.sleep(delay)
                delay = min(delay * 2, self._config.retry_max_delay)

    @abc.abstractmethod
    async def _initialize_client(self) -> None:
        """Initialize cloud client."""

    @abc.abstractmethod
    async def _cleanup_client(self) -> None:
        """Cleanup cloud client."""

    @abc.abstractmethod
    def _transform_entry(self, entry: dict) -> T:
        """Transform entry to cloud-specific format."""

    @abc.abstractmethod
    async def _send_batch(self, batch: list[T]) -> None:
        """Send batch to cloud."""

    @abc.abstractmethod
    async def health_check(self) -> bool:
        """Check cloud connectivity."""
```

#### Usage: Extending CloudSinkBase

Create custom cloud sinks by extending the base:

```python
# myapp/sinks/custom_cloud_sink.py
from examples.sinks.cloud_sink_base import CloudSinkBase, CloudSinkConfig

class MyCloudSink(CloudSinkBase[dict]):
    """Custom sink for MyCloud platform."""

    name = "mycloud"

    async def _initialize_client(self) -> None:
        self._client = MyCloudClient(api_key="...")

    async def _cleanup_client(self) -> None:
        await self._client.close()

    def _transform_entry(self, entry: dict) -> dict:
        return {"timestamp": entry["timestamp"], "data": entry}

    async def _send_batch(self, batch: list[dict]) -> None:
        await self._client.send_logs(batch)

    async def health_check(self) -> bool:
        return await self._client.ping()
```

```python
# Usage
from fapilog import get_logger
from myapp.sinks.custom_cloud_sink import MyCloudSink

logger = get_logger(sinks=[MyCloudSink(CloudSinkConfig(batch_size=50))])
```

---

## Documentation Structure

### docs/examples/cloud-sinks/

```
cloud-sinks/
├── index.md           # Overview of cloud integrations
├── aws-cloudwatch.md  # AWS CloudWatch step-by-step
├── gcp-logging.md     # Google Cloud Logging guide
├── datadog.md         # Datadog integration
├── azure-monitor.md   # Azure Monitor (placeholder)
├── elastic.md         # Elasticsearch/OpenSearch
└── common-patterns.md # Shared patterns (batching, auth)
```

---

## Test Plan

1. Unit tests for each sink with mocked clients
2. Integration tests against LocalStack (AWS)
3. Integration tests against emulators where available
4. Manual testing with real cloud accounts (documented)

---

## Related Stories

- **Story 5.14**: [AWS CloudWatch Sink](./5.14.aws-cloudwatch-sink.md) — Official SDK-based sink
- **Story 5.15**: [Grafana Loki Sink](./5.15.loki-sink.md) — Official HTTP-based sink
- Story 5.6: HTTP sink batching (uses similar patterns)
- Story 5.4: Plugin testing guide (examples use testing patterns)
- Story 5.13: SizeGuardProcessor (recommended for CloudWatch's 256KB limit)
