# Upstream Sync Guide

This document tracks all custom changes made to this pipecat fork and provides instructions for syncing with the upstream repository while preserving these changes.

**Base Version**: This fork is based on [pipecat-ai/pipecat](https://github.com/pipecat-ai/pipecat) version **0.0.91**

---

## Table of Contents

1. [Upstream Sync Process](#upstream-sync-process)
2. [Custom Services (New)](#custom-services-new)
3. [Custom Serializers (New)](#custom-serializers-new)
4. [Modified Core Files](#modified-core-files)
5. [Current Feature Branch Changes](#current-feature-branch-changes)
6. [Merge Conflict Resolution Guide](#merge-conflict-resolution-guide)

---

## Upstream Sync Process

### Step 1: Set Up Upstream Remote (One-time)

```bash
git remote add upstream https://github.com/pipecat-ai/pipecat.git
```

Verify with:
```bash
git remote -v
# Should show:
# upstream        https://github.com/pipecat-ai/pipecat.git (fetch)
# upstream        https://github.com/pipecat-ai/pipecat.git (push)
```

### Step 2: Checkout Your Target Branch

```bash
git checkout dv-stage
```

### Step 3: Fetch Upstream with Tags

```bash
git fetch upstream --tags
```

### Step 4: View Available Version Tags

You will see versions like: `v0.0.87`, `v0.0.88`, `v0.0.89`, `v0.0.90`, etc.

```bash
git tag -l "v0.0.*" | sort -V | tail -20
```

### Step 5: Create New Branch from Target Version

For example, to bump to version 0.0.90:

```bash
git checkout -b dv0.0.90 tags/v0.0.90
```

### Step 6: Merge Your Branch into the New Version Branch

```bash
git merge dv-stage
```

### Step 7: Fix Merge Conflicts

Resolve conflicts following the [Merge Conflict Resolution Guide](#merge-conflict-resolution-guide) below.

---

## Custom Services (New)

These are entirely custom services **not present in upstream**. They should be preserved during any merge and will not have conflicts.

| Service | Path | Purpose |
|---------|------|---------|
| **Ringg STT** | `src/pipecat/services/ringg/` | Custom STT service for Ringg integration |
| **Vistaar LLM** | `src/pipecat/services/vistaar/` | Custom LLM service with authentication |
| **Salesforce LLM** | `src/pipecat/services/salesforce/` | Salesforce Agent integration |
| **Soniox STT** | `src/pipecat/services/soniox/` | Soniox STT service |

---

## Custom Serializers (New)

These are entirely custom serializers **not present in upstream**.

| Serializer | Path | Purpose |
|------------|------|---------|
| **VI** | `src/pipecat/serializers/vi.py` | VI platform serializer |
| **Convox** | `src/pipecat/serializers/convox.py` | Convox serializer |
| **Asterisk Chan WS** | `src/pipecat/serializers/asterisk_chan_websocket_original.py` | Asterisk channel websocket |

---

## Modified Core Files

These files have custom modifications that must be preserved during upstream sync.

### HIGH Risk Files (Core Logic Changes)

#### A. STT Service
**File**: `src/pipecat/services/stt_service.py`

**Changes**:
- Voicemail detection support with `_voicemail_detect` and `_first_speech_handled` flags
- Modified audio processing logic to allow audio when voicemail detect is enabled
- `BotStoppedSpeakingFrame` handling for first speech detection
- Metadata parsing for `voicemail_detect` in start method

**Key Code Additions**:
```python
# In __init__:
self._first_speech_handled: bool = False
self._voicemail_detect: bool = False

# In process_frame:
# Only process audio if not muted, unless voicemail_detect is enabled and first speech not handled
if self._muted and ((not self._voicemail_detect) or self._first_speech_handled):
    return

# BotStoppedSpeakingFrame handling:
elif isinstance(frame, BotStoppedSpeakingFrame):
    if not self._first_speech_handled:
        self._first_speech_handled = True
    await self.push_frame(frame, direction)

# In start method:
if hasattr(frame, "metadata") and "voicemail_detect" in frame.metadata:
    self._voicemail_detect = frame.metadata["voicemail_detect"]
```

---

#### B. STT Mute Filter
**File**: `src/pipecat/processors/filters/stt_mute_filter.py`

**Changes**:
- `_voicemail_detection_enabled` flag (default False)
- Conditional suppression types based on voicemail detection
- DTMF capture handling with `StartDTMFCaptureFrame` / `EndDTMFCaptureFrame`
- `STTMuteStrategy.DTMF_CAPTURE` mute strategy

**Key Code Additions**:
```python
# In __init__:
self._voicemail_detection_enabled = False
self._dtmf_capture_active = False

# Suppression types logic:
suppression_types = (
    InterruptionFrame,
    StartInterruptionFrame,
    VADUserStartedSpeakingFrame,
    VADUserStoppedSpeakingFrame,
    UserStartedSpeakingFrame,
    UserStoppedSpeakingFrame,
    InterimTranscriptionFrame,
    TranscriptionFrame,
)
if not self._voicemail_detection_enabled:
    suppression_types = suppression_types + (InputAudioRawFrame,)

# In process_frame - DTMF handling:
elif isinstance(frame, StartDTMFCaptureFrame):
    self._dtmf_capture_active = True
    should_mute = await self._should_mute()
elif isinstance(frame, EndDTMFCaptureFrame):
    self._dtmf_capture_active = False
    should_mute = await self._should_mute()

# In mute strategies:
case STTMuteStrategy.DTMF_CAPTURE:
    if self._dtmf_capture_active:
        return True
```

---

#### C. Base Output Transport
**File**: `src/pipecat/transports/base_output.py`

**Changes**:
- `BOT_VAD_STOP_SECS = 0.30` timing constant
- `FIRST_BOT_VAD_STOP_SECS = 0.12` for faster initial response
- `on_output_terminated` event handler registration
- Event handler calls in `EndFrame` and `CancelFrame` processing

**Key Code Additions**:
```python
# Constants:
BOT_VAD_STOP_SECS = 0.30
FIRST_BOT_VAD_STOP_SECS = 0.12

# Event handler registration:
self._register_event_handler("on_output_terminated")

# In EndFrame handling:
elif isinstance(frame, EndFrame):
    await self._call_event_handler("on_output_terminated", frame)

# In CancelFrame handling:
elif isinstance(frame, CancelFrame):
    await self._call_event_handler("on_output_terminated", frame)
```

---

#### D. LLM Response Aggregators
**Files**:
- `src/pipecat/processors/aggregators/llm_response.py`
- `src/pipecat/processors/aggregators/llm_response_universal.py`

**Changes**:
- Stricter VAD handling before LLM trigger
- `StartInterruptionFrame` handling changes
- Dropped messages handling
- Filter `TTSTextFrame` by `add_to_llm_context` flag (staged change)

---

#### E. Frames
**File**: `src/pipecat/frames/frames.py`

**Changes**:
- `FunctionCallTrackingFrame` added
- `InputTransportMessageUrgentFrame` added
- `add_to_llm_context` field in `TTSTextFrame` and `TTSSpeakFrame` (staged change)
- Various frame enhancements

---

### MEDIUM Risk Files

#### F. Azure STT
**File**: `src/pipecat/services/azure/stt.py`

**Changes**:
- `vocab` support via `PhraseListGrammar` for keyword boosting
- `additional_languages` parameter for auto language detection
- `endpoint_id` parameter for custom Azure models
- `set_language()` method for runtime language switching
- `_apply_phrase_list()` helper method
- Optimized settings: `Speech_SegmentationSilenceTimeoutMs = 400`

**Key Code Additions**:
```python
# In __init__:
self._vocab: Optional[List[str]] = kwargs.pop("vocab", None)
self._additional_languages = additional_languages
if endpoint_id:
    self._speech_config.endpoint_id = endpoint_id

# Optimization:
self._speech_config.set_property(PropertyId.Speech_SegmentationSilenceTimeoutMs, "400")

# New methods:
def _apply_phrase_list(self, recognizer: SpeechRecognizer):
    """Applies the configured vocabulary as a phrase list to the recognizer."""
    if self._vocab and recognizer:
        phrase_list_grammar = PhraseListGrammar.from_recognizer(recognizer)
        for phrase in self._vocab:
            if isinstance(phrase, str) and phrase.strip():
                phrase_list_grammar.addPhrase(phrase.strip())

async def set_language(self, language: Language):
    """Updates the primary recognition language for Azure STT service."""
    # Full implementation for runtime language switching
```

---

#### G. FastAPI WebSocket Transport
**File**: `src/pipecat/transports/websocket/fastapi.py`

**Changes**:
- `_conversation_id` tracking for call-based logging
- Enhanced exception handling with `call_id` in logs
- `WebSocketDisconnect` specific handling

**Key Code Additions**:
```python
# In class:
self._conversation_id = None

# In setup:
if _.metadata and "call_id" in _.metadata:
    self._conversation_id = _.metadata["call_id"]

# Exception handling:
except Exception as e:
    if isinstance(e, WebSocketDisconnect):
        logger.warning(
            f"{self} WebSocket disconnected during send: {e}, application_state: {self._websocket.application_state}",
            call_id=self._conversation_id,
        )
    else:
        logger.error(
            f"{self} exception sending data: {e.__class__.__name__} ({e}), application_state: {self._websocket.application_state}",
            call_id=self._conversation_id,
        )
```

---

#### H. Frame Processor
**File**: `src/pipecat/processors/frame_processor.py`

**Changes**:
- `self.logger` attribute for custom logger binding
- Logger binding with `call_id` from frame metadata in `__start`

**Key Code Additions**:
```python
# In class:
self.logger = logger  # Will later be replaced with a bound logger

# In __start:
if frame.metadata and "call_id" in frame.metadata:
    self.logger = logger.bind(call_id=frame.metadata["call_id"])
    self._metrics.set_logger(self.logger)
```

---

#### I. Transcript Processor
**File**: `src/pipecat/processors/transcript_processor.py`

**Changes**:
- `FunctionCallTrackingFrame` support
- ID bug fix for dropped messages in `UserTranscriptProcessor`
- Drop words from user transcript that don't go to LLM

---

#### J. Smart Turn Analyzer
**Directory**: `src/pipecat/audio/turn/smart_turn/`

**Changes**:
- `local_smart_turn_v3.py` with CPU count optimization
- Local smart turn v2 with MPS backend auto-detection
- Bundled smart-turn-v3 without torch dependency

---

### LOW Risk Files

#### K. OpenAI LLM Context
**File**: `src/pipecat/processors/aggregators/openai_llm_context.py`

**Changes**:
- `get_messages_for_logging()` function removes system message from logs

**Key Code**:
```python
def get_messages_for_logging(self):
    msgs = copy.deepcopy(self._messages)
    # Remove the first message if it's a system message
    if len(msgs) > 0 and msgs[0].get("role") == "system":
        msgs.pop(0)
    return json.dumps(msgs, ensure_ascii=False)
```

---

#### L. Deepgram STT
**File**: `src/pipecat/services/deepgram/stt.py`

**Changes**:
- Retry faster logic for reconnection
- Custom logger usage

---

#### M. ElevenLabs
**Directory**: `src/pipecat/services/elevenlabs/`

**Changes**:
- STT: Scribe V2 Realtime WebSocket support (`stt.py`)
- TTS: Additional loggers (`tts.py`)

---

#### N. Cartesia TTS
**File**: `src/pipecat/services/cartesia/tts.py`

**Changes**:
- Self logger usage
- Sonic-3 complete support
- Additional languages support

---

#### O. Connection Metrics (New File)
**File**: `src/pipecat/metrics/connection_metrics.py`

**Changes**:
- New file added for tracking connection metrics
- Service connection monitoring

---

#### P. Base Input Transport
**File**: `src/pipecat/transports/base_input.py`

**Changes**:
- Ordering of `VADParamsUpdateFrame` processing
- Base logger for VAD update

---

#### Q. User Idle Processor
**File**: `src/pipecat/processors/user_idle_processor.py`

**Changes**:
- Pushing LLM frames for idle processor upstream
- Don't trigger user idle callback if function call is in progress
- `ResetUserIdleProcessorFrame` support
- Custom loggers

---

#### R. OpenAI Base LLM
**File**: `src/pipecat/services/openai/base_llm.py`

**Changes**:
- Use `max_completion_tokens` for GPT-5 series models
- Stricter VAD before LLM trigger integration

---

#### S. Sarvam TTS
**File**: `src/pipecat/services/sarvam/tts.py`

**Changes**:
- Removed `StartInterruptionFrame` (deprecated)
- TTS switcher integration

---

#### T. OpenAI LLM (Legacy)
**File**: `src/pipecat/services/openai.py`

**Changes**:
- Various optimizations and logger additions

---

## Current Feature Branch Changes

### Branch: `feat/remove_prefix_message_from_llm_context`

These are currently staged changes that control whether TTS output is added to LLM context.

| File | Change | Purpose |
|------|--------|---------|
| `src/pipecat/frames/frames.py` | Added `add_to_llm_context: bool = True` to `TTSTextFrame` and `TTSSpeakFrame` | Control whether TTS output is added to LLM context |
| `src/pipecat/processors/aggregators/llm_response.py` | Filter `TTSTextFrame` by `add_to_llm_context` flag | Skip processing if flag is False |
| `src/pipecat/processors/aggregators/llm_response_universal.py` | Same filter logic as llm_response.py | Consistency across aggregators |
| `src/pipecat/services/tts_service.py` | Propagate `add_to_llm_context` through TTS pipeline; `WordTTSService` deque tracking | Full flag propagation for word-level TTS |

---

## Merge Conflict Resolution Guide

### Priority Order

When resolving conflicts, prioritize in this order:

1. **Preserve our custom logic** - Our changes are for production use cases
2. **Adopt upstream API changes** - If method signatures or imports change upstream
3. **Keep both if compatible** - If upstream adds new features that don't conflict

### Common Conflict Patterns

#### Pattern 1: Import Changes
Upstream may reorganize imports. Keep both our custom imports and upstream's new organization.

#### Pattern 2: Method Signature Changes
If upstream changes method signatures, update our custom code to use new signatures while preserving our logic.

#### Pattern 3: New Fields in Classes
Upstream may add new fields. Keep their fields and ensure our custom fields remain.

### Testing Checklist After Merge

- [ ] Run `pytest` to ensure tests pass
- [ ] Test voicemail detection flow
- [ ] Test DTMF capture functionality
- [ ] Test language switching (Azure STT)
- [ ] Test call_id logging propagation
- [ ] Test custom services (Ringg, Vistaar, Salesforce, Soniox)
- [ ] Test TTS `add_to_llm_context` flag (if merged)
- [ ] Run a sample bot to verify end-to-end flow

### Files to Watch Carefully

These files change frequently in upstream and have our custom changes:

1. `src/pipecat/frames/frames.py` - Frame definitions
2. `src/pipecat/services/stt_service.py` - Core STT logic
3. `src/pipecat/processors/aggregators/llm_response.py` - LLM context aggregation
4. `src/pipecat/transports/base_output.py` - Output timing

---

## Quick Reference: Files Changed Since October 25, 2024

To see all commits affecting `src/pipecat/` since October 25th:

```bash
# Count total commits
git log --since="2024-10-25" --oneline -- src/pipecat/ | wc -l

# View commits with file names
git log --since="2024-10-25" --oneline --name-only -- src/pipecat/ | head -200

# Filter for custom commits (not upstream merges)
git log --since="2024-10-25" --oneline -- src/pipecat/ | grep -E "^[a-f0-9]+ (feat|fix|Added|Adds|Removed|Self|FIRST|Drop|Ordering|Support|Id bug|code cleanup|ringg|VI|Deepgram|Merged|Enables|Send Bot)"

# See most frequently modified files
git log --since="2024-10-25" --oneline --name-only -- src/pipecat/ | grep -E "^src/pipecat/" | sort | uniq -c | sort -rn | head -30
```

---

## Updating This Document

When making new custom changes to the fork:

1. Add the file to the appropriate section above
2. Document the specific changes made
3. Assign a conflict risk level (HIGH/MEDIUM/LOW)
4. Add relevant code snippets for reference

When syncing with upstream:

1. Update the **Base Version** at the top of this document
2. Note any new conflicts encountered and how they were resolved
3. Update the testing checklist if new features are added
