# VoiceEval Monitoring Pipeline — Implementation Spec

## Overview

This spec covers the end-to-end pipeline for ingesting traces and audio from customer voice agents, mapping them via call_id, and triggering automated evaluations. It supports two customer integration modes:

- **SDK customers**: Install `voiceeval-sdk` in their LiveKit agent. Traces flow automatically via OTel. Audio is pushed separately.
- **Non-SDK customers**: Push both traces (as structured JSON) and audio recording links to a single REST endpoint.

---

## 1. Data Architecture

### 1.1 Use existing `v2_calls` collection for both simulation and monitoring calls

Add these fields to `V2CallDocument`:

```python
class V2CallDocument(Document):
    call_id: str                          # Primary key (indexed, unique)
    sdk_call_id: Optional[str] = None     # SDK's voiceeval.call_id (indexed, unique, sparse)
    source: str = "simulation"            # "simulation" | "monitoring"
    tenant_id: str

    # Status lifecycle (same for both sources)
    status: CallStatus                    # ongoing → call_done → transcribing → evaluating → evals_done

    # Simulation-specific (null for monitoring)
    test_case_id: Optional[str] = None
    folder_id: Optional[str] = None

    # Monitoring-specific (null for simulation)
    trace_id: Optional[str] = None        # Langfuse trace ID or internal trace reference
    agent_name: Optional[str] = None      # Extracted from trace metadata

    # Readiness flags for eval trigger
    has_trace: bool = False
    has_audio: bool = False

    created_at: datetime
    updated_at: datetime
```

**Call ID strategy:**
- **Monitoring calls (SDK customer):** `call_id = sdk_call_id = voiceeval.call_id` from the SDK. The SDK-generated UUID is the primary key.
- **Monitoring calls (non-SDK customer):** `call_id` is generated by the customer or by our backend on ingestion. `sdk_call_id` is null.
- **Simulation calls (no SDK):** `call_id` = backend-generated via `generate_uuid("call")`. `sdk_call_id` = null.
- **Simulation calls (SDK installed):** `call_id` = backend-generated. `sdk_call_id` = extracted from trace when it arrives. Linked post-hoc.

Index on both `call_id` (unique) and `sdk_call_id` (unique, sparse). When looking up a call, query: `{$or: [{call_id: id}, {sdk_call_id: id}]}`.

### 1.2 Use existing `v2_recordings` collection — no changes needed

```python
class V2RecordingDocument(Document):
    call_id: str              # FK to v2_calls.call_id
    recording_url: str        # GCS path or external URL
    duration: Optional[float]
    created_at: datetime
```

### 1.3 New `monitoring_traces` collection — stores raw trace data for monitoring calls

```python
class MonitoringTraceDocument(Document):
    call_id: str              # FK to v2_calls.call_id (indexed)
    tenant_id: str            # Resolved from API key
    source: str               # "sdk" | "push" (how the trace arrived)
    spans: List[Dict]         # Raw span data (from OTLP protobuf or JSON push)
    metadata: Dict            # Agent name, model, provider, token counts, cost, etc.
    total_tokens: int
    total_cost_usd: float
    llm_calls: int
    error_count: int
    duration_ms: float
    created_at: datetime
```

This replaces `observability_traces` for monitoring calls. Simulation calls that also produce traces (when SDK is installed) get linked here too via `sdk_call_id`.

### 1.4 Differentiation via `source` field — not separate collections

| Use case | `v2_calls.source` | `monitoring_traces.source` | `has_trace` | `has_audio` |
|----------|-------------------|---------------------------|-------------|-------------|
| Simulation (no SDK) | `"simulation"` | N/A | false | true |
| Simulation (SDK installed) | `"simulation"` | `"sdk"` | true | true |
| Monitoring (SDK customer) | `"monitoring"` | `"sdk"` | true | true (after push) |
| Monitoring (non-SDK customer) | `"monitoring"` | `"push"` | true | true (pushed together) |

---

## 2. Ingestion Endpoints

### 2.1 Dual-write at the existing SDK ingestion proxy (`server/main.py`)

Currently the proxy at `POST /v1/traces` only forwards to Langfuse. Add a direct MongoDB write:

```
SDK → POST /v1/traces (OTLP protobuf) → Proxy:
  1. Validate API key (existing)
  2. Parse OTLP protobuf (existing)
  3. Inject tenant email into spans (existing)
  4. Forward to Langfuse (existing — keep for dashboard)
  5. [NEW] Extract voiceeval.call_id from span attributes
  6. [NEW] Write trace to monitoring_traces collection
  7. [NEW] Upsert v2_calls record: {call_id, source: "monitoring", has_trace: true}
  8. [NEW] Check if has_audio is also true → if yes, trigger eval pipeline
```

Extract call_id from the parsed protobuf:

```python
# After parsing trace_request in server/main.py
call_ids = set()
for resource_span in trace_request.resource_spans:
    for scope_span in resource_span.scope_spans:
        for span in scope_span.spans:
            for attr in span.attributes:
                if attr.key == "voiceeval.call_id":
                    call_ids.add(attr.value.string_value)
```

### 2.2 New unified push endpoint — `POST /api/v1/monitoring/ingest`

This is the endpoint for ALL customers to push data. It handles three scenarios:
- Non-SDK customer pushing traces + audio link together
- SDK customer pushing audio link only (traces already arrived via SDK)
- Any customer pushing audio file directly

**Location:** Add to `voiceeval-backend/backend/api/v1/` (not the SDK proxy server — this is on the main backend).

**Auth:** Same VoiceEval API key, Bearer token. Resolve tenant from API key.

```python
class MonitoringIngestRequest(BaseModel):
    call_id: str                                  # Required — customer's call identifier

    # Audio (at least one required)
    audio_url: Optional[str] = None               # URL to recording (GCS, S3, HTTP)
    audio_file: Optional[UploadFile] = None       # Direct file upload (multipart)

    # Traces (optional — not needed if SDK is sending traces)
    traces: Optional[List[TraceSpan]] = None      # Structured trace data

    # Metadata (optional)
    agent_name: Optional[str] = None
    metadata: Optional[Dict[str, Any]] = None

class TraceSpan(BaseModel):
    """Structured span for non-SDK customers pushing traces as JSON."""
    span_id: Optional[str] = None                 # Auto-generated if not provided
    parent_span_id: Optional[str] = None
    name: str                                     # e.g., "llm_call", "tool_use", "agent_response"
    start_time: datetime
    end_time: datetime
    attributes: Optional[Dict[str, Any]] = None   # model, provider, tokens, cost, etc.
    input: Optional[Any] = None                   # LLM input / function args
    output: Optional[Any] = None                  # LLM output / function return
    status: str = "OK"                            # "OK" | "ERROR"
```

**Endpoint logic:**

```python
@router.post("/api/v1/monitoring/ingest")
async def ingest_monitoring_data(
    request: MonitoringIngestRequest,  # JSON body
    # OR for file upload: call_id: str = Form(), audio_file: UploadFile = File(), ...
    tenant: TenantContext = Depends(get_tenant_from_api_key)
):
    call_id = request.call_id

    # 1. Upsert v2_calls record
    call = await call_repo.find_by_call_id_or_sdk_call_id(call_id)
    if not call:
        call = await call_repo.create({
            "call_id": call_id,
            "source": "monitoring",
            "tenant_id": tenant.tenant_id,
            "status": "call_done",
            "has_trace": False,
            "has_audio": False,
            "agent_name": request.agent_name,
            "created_at": now(),
        })

    # 2. Handle audio
    if request.audio_url:
        # Store external URL directly
        await recording_repo.create({
            "call_id": call_id,
            "recording_url": request.audio_url,
        })
        await call_repo.update(call_id, {"has_audio": True})

    elif request.audio_file:
        # Upload to GCS, store URL
        gcs_url = await gcs_service.upload(request.audio_file, f"monitoring/{tenant.tenant_id}/{call_id}")
        await recording_repo.create({
            "call_id": call_id,
            "recording_url": gcs_url,
        })
        await call_repo.update(call_id, {"has_audio": True})

    # 3. Handle traces (non-SDK customers)
    if request.traces:
        trace_doc = build_monitoring_trace(call_id, tenant.tenant_id, request.traces, source="push")
        await trace_repo.save(trace_doc)
        await call_repo.update(call_id, {"has_trace": True})

    # 4. Check if both ready → trigger eval
    updated_call = await call_repo.find_by_call_id(call_id)
    if updated_call.has_trace and updated_call.has_audio:
        asyncio.create_task(trigger_evaluation(call_id))

    return {"status": "accepted", "call_id": call_id}
```

**Multipart variant** (for direct file upload):

```
POST /api/v1/monitoring/ingest
Content-Type: multipart/form-data

call_id=abc-123
audio_file=@recording.mp3
traces=[{"name": "llm_call", ...}]  (optional, as JSON string in form field)
```

### 2.3 Audio-only push endpoint (convenience alias)

For SDK customers who only need to push audio (traces arrive via SDK):

```
POST /api/v1/monitoring/audio/{call_id}
Authorization: Bearer <api_key>
Content-Type: multipart/form-data OR application/json

# Multipart: audio_file=@recording.mp3
# JSON: {"audio_url": "gs://bucket/recording.mp3"}
```

This is a thin wrapper that calls the same logic as the unified endpoint with `traces=None`.

---

## 3. Evaluation Trigger Pipeline

### 3.1 Readiness check — both `has_trace` and `has_audio` must be true

The evaluation is triggered when both flags are set on a `v2_calls` record. This can happen in two ways:

**For SDK customers:**
1. Traces arrive at proxy → `has_trace = true` → check: audio missing, wait
2. Customer pushes audio to `/monitoring/ingest` or `/monitoring/audio/{call_id}` → `has_audio = true` → check: trace present, TRIGGER EVAL

**For non-SDK customers:**
1. Customer pushes both traces + audio_url in a single `POST /monitoring/ingest` → both flags set → TRIGGER EVAL immediately

**For simulation calls (with SDK):**
1. Backend starts call → creates `v2_calls` with `has_audio = false, has_trace = false`
2. Call completes → audio recorded → `has_audio = true`
3. Traces arrive via SDK → matched via `sdk_call_id` → `has_trace = true` → TRIGGER EVAL

### 3.2 Evaluation pipeline (reuse existing infrastructure)

```python
async def trigger_evaluation(call_id: str):
    """
    Triggered when both trace and audio are ready for a call.
    Reuses existing evaluation infrastructure.
    """
    call = await call_repo.find_by_call_id(call_id)
    recording = await recording_repo.find_by_call(call_id)
    trace = await trace_repo.find_by_call_id(call_id)

    # 1. Update status
    await call_repo.update_status(call_id, "transcribing")

    # 2. Download audio + transcribe (existing: SarvamDiarizationService)
    audio_path = await gcs_service.download(recording.recording_url)
    diarization = await sarvam_service.diarize_audio(audio_path)
    transcript = convert_to_transcript_result(diarization)
    await transcript_repo.save(call_id, transcript)

    await call_repo.update_status(call_id, "evaluating")

    # 3. Build evaluation input
    eval_input = EvaluationInput(
        transcript=transcript,
        trace=trace,          # NEW: pass trace data for trace-based metrics
        agent_name=call.agent_name,
        source=call.source,
    )

    # 4. Run evaluation engine (existing: EvaluationEngine)
    #    Audio-based metrics: pronunciation, sentiment, audio quality, silence, interruptions
    #    Trace-based metrics: LLM latency, token usage, cost, error rate, tool call accuracy
    #    Combined metrics: correlate LLM response time with audio silence gaps
    result = await evaluation_engine.evaluate(eval_input)

    # 5. Store results (existing: v2_evals collection)
    await eval_repo.save({
        "call_id": call_id,
        "tenant_id": call.tenant_id,
        "source": call.source,
        "overall_score": result.overall_score,
        "passed": result.passed,
        "metric_results": result.metric_results,
        "kpi_results": result.kpi_results,
    })

    await call_repo.update_status(call_id, "evals_done")
```

### 3.3 What metrics run depends on what data is available

```python
has_audio = recording exists for call_id
has_trace = trace exists for call_id

if has_audio:
    # Existing audio-based metrics (already implemented)
    run: pronunciation, sentiment, audio quality, silence duration,
         interruption rate, endpointing, speech quality

if has_trace:
    # Trace-based metrics (new, from span attributes)
    run: LLM latency (time-to-first-token), token usage per turn,
         cost per call, error rate, tool call success rate, model used

if has_audio and has_trace:
    # Combined metrics (new, correlate both data sources)
    run: silence-to-LLM-latency correlation,
         transcript-to-LLM-output alignment (hallucination detection)
```

---

## 4. Handling the Non-SDK Customer (Traces + Audio in One Push)

For customers who won't install the SDK but will push their own trace data along with audio links:

### What they send:

```json
POST /api/v1/monitoring/ingest
Authorization: Bearer <their_api_key>
Content-Type: application/json

{
    "call_id": "customer-generated-uuid",
    "audio_url": "https://their-bucket.s3.amazonaws.com/recordings/call-123.mp3",
    "agent_name": "booking-agent",
    "traces": [
        {
            "name": "agent_entrypoint",
            "start_time": "2026-03-25T10:00:00Z",
            "end_time": "2026-03-25T10:02:30Z",
            "attributes": {"agent.type": "booking"},
            "status": "OK"
        },
        {
            "name": "llm_call",
            "parent_span_id": null,
            "start_time": "2026-03-25T10:00:01Z",
            "end_time": "2026-03-25T10:00:02Z",
            "input": {"messages": [{"role": "user", "content": "Book a table for 2"}]},
            "output": {"content": "I'd be happy to help..."},
            "attributes": {
                "model": "gpt-4o",
                "provider": "openai",
                "input_tokens": 150,
                "output_tokens": 80,
                "cost_usd": 0.003
            },
            "status": "OK"
        },
        {
            "name": "tool_call",
            "start_time": "2026-03-25T10:00:02Z",
            "end_time": "2026-03-25T10:00:03Z",
            "input": {"function": "check_availability", "args": {"date": "2026-03-26", "party_size": 2}},
            "output": {"available": true},
            "status": "OK"
        }
    ],
    "metadata": {
        "customer_id": "cust_456",
        "call_direction": "inbound"
    }
}
```

### How the backend handles it:

Since both traces and audio arrive together, both `has_trace` and `has_audio` are set to true in the same request, and evaluation is triggered immediately. No waiting or coordination needed.

The `TraceSpan` schema is intentionally simple — non-SDK customers don't need to understand OTel. They just send name, timestamps, input/output, and optional attributes. Our backend normalizes this into the same `MonitoringTraceDocument` format.

---

## 5. Linking Simulation Calls to SDK Traces

When a customer has the SDK installed and we run a simulation:

1. Backend creates call: `call_id = "call_abc123"`, `sdk_call_id = null`, `source = "simulation"`
2. Call runs through LiveKit → SDK's `@observe` generates `voiceeval.call_id = "uuid-xyz-789"`
3. Traces arrive at proxy → we extract `voiceeval.call_id = "uuid-xyz-789"`
4. No matching `call_id` in v2_calls, but we can match by:
   - **Timing**: trace arrived within expected window of the simulation call
   - **Tenant**: same tenant_id
   - **Better approach**: Pass our `call_id` into the simulation's agent context before the call starts, so the SDK uses our ID

**Recommended linking approach for simulation + SDK:**

Before starting a simulation call, configure the SDK context so our call_id is used:

```python
# In simulation orchestration code
from voiceeval.context import set_call_metadata, CallMetadata

# Set our call_id before the agent entrypoint runs
set_call_metadata(CallMetadata(call_id=our_generated_call_id))
```

If this isn't possible (customer's agent runs on their infra), fall back to:
- Store `sdk_call_id` when trace arrives
- Match unlinked traces to recent simulation calls by tenant + time window
- Link post-hoc: `UPDATE v2_calls SET sdk_call_id = ? WHERE call_id = ? AND sdk_call_id IS NULL`

---

## 6. Summary of Endpoints

| Endpoint | Method | Auth | Purpose | Who calls it |
|----------|--------|------|---------|-------------|
| `/v1/traces` (proxy) | POST | Bearer API key | OTLP protobuf trace ingestion | SDK (automatic) |
| `/v1/auth/validate` (proxy) | GET | Bearer API key | API key validation | SDK on init |
| `/v1/auth/keys` (proxy) | POST | Admin | Create API key for customer | VoiceEval admin |
| `/api/v1/monitoring/ingest` (backend) | POST | Bearer API key | Unified push: traces + audio (JSON or multipart) | Non-SDK customers, or SDK customers pushing audio |
| `/api/v1/monitoring/audio/{call_id}` (backend) | POST | Bearer API key | Audio-only push (convenience) | SDK customers pushing just audio |

---

## 7. Files to Create/Modify

### New files (backend):
- `backend/api/v1/monitoring.py` — New router with `/monitoring/ingest` and `/monitoring/audio/{call_id}`
- `backend/models/odm/v2/monitoring_trace.py` — `MonitoringTraceDocument` ODM model
- `backend/database/repositories/v2/monitoring_trace_repository.py` — CRUD for monitoring traces
- `backend/services/monitoring_pipeline.py` — Orchestrates the readiness check + eval trigger

### Modified files (backend):
- `backend/models/odm/v2/call.py` — Add `source`, `sdk_call_id`, `trace_id`, `agent_name`, `has_trace`, `has_audio` fields
- `backend/database/repositories/v2/call_repository.py` — Add `find_by_call_id_or_sdk_call_id()` method
- `backend/api/v1/__init__.py` — Register new monitoring router
- `backend/services/v2_post_call_evaluator.py` — Accept optional trace data for trace-based metrics
- `backend/engines/evaluation/engine.py` — Add trace-based metric execution path (when trace data is available)

### Modified files (SDK proxy):
- `server/main.py` — Add MongoDB write in `/v1/traces` handler (dual-write): extract `voiceeval.call_id`, upsert `v2_calls`, save trace, check readiness
- `server/database.py` — Add connection to the same MongoDB used by backend (or a shared connection config)

### No changes needed:
- `v2_recordings` collection/model — works as-is
- `v2_evals` collection/model — works as-is
- `v2_transcripts` collection/model — works as-is
- SDK package (`src/voiceeval/`) — no changes needed
