<!-- cd04e3ec-916f-4418-922a-390f05ece83b -->
---
todos:
  - id: "1-init-expo"
    content: "Create Expo app: npx create-expo-app@latest mobile --template blank-typescript"
    status: pending
  - id: "2-install-deps"
    content: "Install deps: expo-document-picker, expo-file-system, expo-media-library, ffmpeg-kit-react-native, base-64"
    status: pending
  - id: "3-configure"
    content: "Configure app.json with plugins for native modules"
    status: pending
  - id: "4-sync-module"
    content: "Create src/sync.ts - FFmpeg PCM extraction + waveform cross-correlation"
    status: pending
  - id: "5-ffmpeg-module"
    content: "Create src/ffmpeg.ts - merge command with hardware accel + CFR conversion"
    status: pending
  - id: "6-app-ui"
    content: "Build App.tsx - file pickers, progress indicator, sync button, save to gallery"
    status: pending
  - id: "7-eas-build"
    content: "Build with EAS for Android"
    status: pending
  - id: "8-test"
    content: "Test on Android device"
    status: pending
isProject: false
---
# React Native Audio-Video Sync App

## Architecture

```mermaid
flowchart TD
    subgraph ui [User Interface]
        SelectVideo[Select Video Button]
        SelectAudio[Select Audio Button]
        SyncButton[Sync Button]
        Progress[Progress Indicator]
    end
    
    subgraph processing [Processing]
        ExtractVideo[Extract PCM from Video]
        ExtractAudio[Extract PCM from Audio]
        Correlate[Cross-correlate Waveforms]
        CalcOffset[Calculate Offset]
        Merge[FFmpeg Merge]
    end
    
    subgraph output [Output]
        SaveGallery[Save to Gallery]
        ShowSuccess[Show Success]
    end
    
    SelectVideo --> SyncButton
    SelectAudio --> SyncButton
    SyncButton --> ExtractVideo
    SyncButton --> ExtractAudio
    ExtractVideo --> Correlate
    ExtractAudio --> Correlate
    Correlate --> CalcOffset
    CalcOffset --> Merge
    Merge --> SaveGallery
    SaveGallery --> ShowSuccess
```

## Files to Create

- `mobile/App.tsx` - Main UI with file pickers, progress, and sync button
- `mobile/src/sync.ts` - PCM extraction + waveform cross-correlation
- `mobile/src/ffmpeg.ts` - FFmpeg commands (merge, probe duration/fps)
- `mobile/src/utils.ts` - Base64 to Float32Array conversion

## Algorithm (based on [src/audio_video_sync/sync.py](src/audio_video_sync/sync.py))

**Audio Extraction** (same FFmpeg approach as Python):
```
ffmpeg -i <input> -t 30 -ac 1 -ar 8000 -f f32le <output.pcm>
```

**Cross-Correlation** (waveform only, simplified from Python's dual-method):
- Correlate raw samples using `scipy.signal.correlate` equivalent
- Find peak index, calculate offset: `(peak - zero_lag) / sample_rate`

**Merge** (matches [src/audio_video_sync/ffmpeg.py](src/audio_video_sync/ffmpeg.py)):
- Positive offset: `-ss <offset>` before video input
- Negative offset: `-ss <trim>` before audio input
- CFR conversion: `-fps_mode cfr -r <fps>`
- Hardware encoding on iOS: `h264_videotoolbox`
- Software encoding on Android: `libx264 -preset fast -crf 18`

## Dependencies

- `expo-document-picker` - File selection
- `expo-file-system` - Read/write temp files
- `expo-media-library` - Save to gallery
- `ffmpeg-kit-react-native` - FFmpeg execution (RebootMotion fork)
- `base-64` - Decode PCM data

## Build Steps

1. Create Expo app with TypeScript template
2. Install Expo modules via `npx expo install`
3. Install ffmpeg-kit-react-native via npm
4. Configure `app.json` with native module plugins
5. Implement sync algorithm and UI
6. Build with EAS: `npx eas build --platform android`
7. Test on Android device

## Simplifications from Python Version

- Waveform correlation only (skip chromagram - requires complex DSP)
- 8kHz sample rate (vs 22050 Hz - faster on mobile)
- 30s analysis duration (vs 40s - sufficient accuracy)
- Software encoding on Android (libx264 - more reliable than MediaCodec)
