# Issue #23: ストリーミング処理

**Phase:** 2a
**優先度:** 中
**依存関係:** #22 (async_pipeline完全サポート)
**見積もり:** 4日

---

## 概要

大量データを効率的に処理するため、ストリーミング処理をサポートします。
イテレータやジェネレータを使用して、メモリ効率の良いデータ処理を実現します。

---

## TDD実装手順

### Step 1: ストリーミングパイプラインのテスト (Red)

```python
# tests/unit/core/test_streaming_pipeline.py
"""Tests for streaming pipeline."""
import pytest
from typing import Iterator, AsyncIterator


class TestStreamingPipeline:
    """Test streaming pipeline execution."""

    def test_streaming_with_generator(self):
        """Should process data in chunks using generator."""
        from railway.core.decorators import node
        from railway.core.pipeline import streaming_pipeline

        @node
        def generate_data() -> Iterator[int]:
            """Generate data in chunks."""
            for i in range(1000):
                yield i

        @node
        def double(x: int) -> int:
            return x * 2

        @node
        def sum_all(items: Iterator[int]) -> int:
            return sum(items)

        result = streaming_pipeline(
            generate_data(),
            double,
            sum_all
        )

        assert result == sum(i * 2 for i in range(1000))

    def test_streaming_memory_efficient(self):
        """Should not load all data into memory at once."""
        from railway.core.decorators import node
        from railway.core.pipeline import streaming_pipeline
        import sys

        max_memory = 0

        @node
        def generate_large_data() -> Iterator[list]:
            """Generate large chunks."""
            for _ in range(100):
                yield [0] * 10000

        @node
        def count_chunk(chunk: list) -> int:
            nonlocal max_memory
            current = sys.getsizeof(chunk)
            max_memory = max(max_memory, current)
            return len(chunk)

        result = list(streaming_pipeline(
            generate_large_data(),
            count_chunk
        ))

        assert len(result) == 100
        # Memory should not accumulate (all chunks combined)
        assert max_memory < 1000000  # Less than 1MB per chunk


class TestAsyncStreamingPipeline:
    """Test async streaming pipeline."""

    @pytest.mark.asyncio
    async def test_async_streaming_pipeline(self):
        """Should process async stream."""
        from railway.core.decorators import node
        from railway.core.pipeline import async_streaming_pipeline
        import asyncio

        @node
        async def async_generate() -> AsyncIterator[int]:
            """Generate data asynchronously."""
            for i in range(10):
                await asyncio.sleep(0.01)
                yield i

        @node
        async def async_process(x: int) -> int:
            await asyncio.sleep(0.01)
            return x * 2

        results = []
        async for item in async_streaming_pipeline(
            async_generate(),
            async_process
        ):
            results.append(item)

        assert results == [i * 2 for i in range(10)]
```

---

### Step 2: ストリーミングパイプラインの実装 (Green)

```python
# railway/core/pipeline.py に追加

from typing import Iterator, AsyncIterator, TypeVar
from collections.abc import Iterable

T = TypeVar('T')


def streaming_pipeline(
    source: Iterator[T] | Iterable[T],
    *steps: Callable,
    buffer_size: int = 100
) -> Iterator[Any]:
    """
    Execute a pipeline on a stream of data.

    Processes data in chunks for memory efficiency.

    Args:
        source: Data source (iterator or iterable)
        *steps: Processing functions
        buffer_size: Number of items to buffer

    Yields:
        Processed items one at a time

    Example:
        results = streaming_pipeline(
            generate_large_dataset(),
            filter_valid,
            transform,
            validate
        )
        for result in results:
            save_to_db(result)
    """
    from loguru import logger

    logger.debug(f"Streaming pipeline starting with {len(steps)} steps")

    # Ensure source is an iterator
    if not isinstance(source, Iterator):
        source = iter(source)

    # Process items one by one through the pipeline
    for item in source:
        current_value = item

        for step_idx, step in enumerate(steps, 1):
            try:
                current_value = step(current_value)
            except Exception as e:
                logger.error(f"Streaming step {step_idx} failed: {e}")
                raise

        yield current_value


async def async_streaming_pipeline(
    source: AsyncIterator[T] | Iterator[T],
    *steps: Callable,
    buffer_size: int = 100
) -> AsyncIterator[Any]:
    """
    Execute an async pipeline on a stream of data.

    Args:
        source: Async data source
        *steps: Processing functions (sync or async)
        buffer_size: Number of items to buffer

    Yields:
        Processed items one at a time

    Example:
        async for result in async_streaming_pipeline(
            async_fetch_stream(),
            process,
            validate
        ):
            await save_to_db(result)
    """
    from loguru import logger
    import asyncio

    logger.debug(f"Async streaming pipeline starting with {len(steps)} steps")

    # Process items asynchronously
    if hasattr(source, '__aiter__'):
        # Async iterator
        async for item in source:
            current_value = item

            for step_idx, step in enumerate(steps, 1):
                try:
                    if asyncio.iscoroutinefunction(step):
                        current_value = await step(current_value)
                    else:
                        loop = asyncio.get_event_loop()
                        current_value = await loop.run_in_executor(
                            None, step, current_value
                        )
                except Exception as e:
                    logger.error(f"Async streaming step {step_idx} failed: {e}")
                    raise

            yield current_value
    else:
        # Regular iterator in async context
        for item in source:
            current_value = item

            for step_idx, step in enumerate(steps, 1):
                try:
                    if asyncio.iscoroutinefunction(step):
                        current_value = await step(current_value)
                    else:
                        current_value = step(current_value)
                except Exception as e:
                    logger.error(f"Async streaming step {step_idx} failed: {e}")
                    raise

            yield current_value
```

---

## 完了条件

- [x] `streaming_pipeline()` 関数の実装
- [x] `async_streaming_pipeline()` 関数の実装
- [x] イテレータ/ジェネレータのサポート
- [x] メモリ効率の確認
- [x] バッファリング機能
- [x] エラーハンドリング
- [x] テスト (10テスト以上)
- [x] テストカバレッジ 90%以上
- [x] ドキュメント更新
- [x] 大規模データ処理の実例

---

## ユースケース

- 大量CSVファイルの処理
- APIからのページネーション取得
- ログファイルのリアルタイム処理
- データベースクエリ結果のストリーミング

---

## 関連Issue

- Issue #22: async_pipeline完全サポート
- Issue #39: 外部ストレージ連携 (S3/GCSストリーミング)
