---
tags: [pipeline, orchestration, write, read, async, staged]
modules: [src/arandu/write/, src/arandu/read/]
applies_to: [pipelines, orchestrators]
confidence: inferred
---
# Pattern: Pipeline Orchestration

<!-- vibeflow:auto:start -->
## What
Multi-stage async pipelines that compose domain operations sequentially. Each pipeline has a single `run_*_pipeline()` entry point that orchestrates stages, handles timing, and returns a result dict/dataclass.

## Where
- `src/arandu/write/pipeline.py` — Write pipeline (4 stages)
- `src/arandu/read/pipeline.py` — Read pipeline (5+ stages)

## The Pattern
1. A top-level `run_*_pipeline()` function receives all dependencies as parameters (session, user_id, llm, embeddings, config).
2. Each stage is a separate module with its own entry point function.
3. Timing is measured with `time.perf_counter()`.
4. Results are accumulated in a dict (write) or dataclass (read).
5. If `trace` (PipelineTrace) is provided, each stage records intermediate data via `trace.add_step()`.
6. Stages use savepoints (`session.begin_nested()`) for atomicity.
7. The caller (MemoryClient) manages the session transaction and commit.

## Rules
- Pipeline functions do NOT commit — the caller (MemoryClient) calls `session.commit()`.
- Each stage imports its dependencies at the module level (not lazy).
- Stage functions receive explicit parameters (no shared state object).
- Tracing is opt-in: `if trace:` guard before recording.
- Duration is calculated as `round((time.perf_counter() - t_start) * 1000, 1)` (ms, 1 decimal).

## Examples from this codebase
File: src/arandu/write/pipeline.py
```python
async def run_write_pipeline(
    session: AsyncSession,
    user_id: str,
    message: str,
    llm: LLMProvider,
    embeddings: EmbeddingProvider,
    config: MemoryConfig,
    source: str = "api",
    recent_messages: list[str] | None = None,
    trace: PipelineTrace | None = None,
) -> dict:
    t_start = time.perf_counter()
    # 1. Create event
    event = await _create_event(session, user_id, message, source, embeddings)
    # 2. Extract
    extraction = await extract_from_message(message, llm=llm, ...)
    # 3. Resolve entities (inside savepoint)
    async with session.begin_nested():
        entity_map = await resolve_entities(session, user_id, ...)
        # 4. Reconcile
        decisions = await reconcile_facts(session, user_id, ...)
        # 5. Upsert
        upsert_result = await execute_upsert(session, user_id, ...)
    duration_ms = round((time.perf_counter() - t_start) * 1000, 1)
    return {...}
```

File: src/arandu/read/pipeline.py
```python
async def run_read_pipeline(
    session: AsyncSession, user_id: str, query: str,
    llm: LLMProvider, embeddings: EmbeddingProvider,
    config: MemoryConfig, trace: PipelineTrace | None = None,
) -> ReadResult:
    # 0. Plan retrieval strategy
    plan = await plan_retrieval(session, user_id, query, llm)
    # 1. Multi-signal retrieval (parallel via asyncio.gather)
    candidates = await retrieve_candidates(...)
    # 2. Rerank via LLM
    candidates = await rerank_with_llm(query, candidates, llm, config)
    # 3. Top-K + format
    return ReadResult(facts=scored_facts, context=context, ...)
```
<!-- vibeflow:auto:end -->

## Anti-patterns (if found)
- Write pipeline returns a raw `dict` instead of a typed dataclass. The read pipeline uses `ReadResult` dataclass — the write pipeline should follow the same pattern.
