# ByteDance Dolphin - Deep Dive

> **TL;DR**: Dolphin is ByteDance's state-of-the-art document image parsing model that uses a two-stage "analyze-then-parse" approach. Unlike traditional models that just detect layout boxes, Dolphin first analyzes what types of content exist, then uses specialized prompts for each type. It handles 21 different element categories and achieves leading benchmark scores. Paper accepted at ACL 2025.

---

## What Is Dolphin?

**Dolphin** is a general-purpose multimodal document parsing model from ByteDance Research. It represents a new paradigm in document understanding: instead of treating all document elements the same way, it analyzes first, then parses with specialized strategies for each element type.

### The Core Innovation

Traditional document parsing:
```
Document Image → Single Model → All Elements at Once → Output
```

Dolphin's approach:
```
Document Image → Analyze (what's here?) → Parse (with specialized prompts per type) → Output
```

This "analyze-then-parse" strategy allows Dolphin to handle diverse documents with heterogeneous content - combining text, tables, figures, equations, and more.

---

## Why This Matters

| Challenge | Traditional Approach | Dolphin's Solution |
|-----------|---------------------|-------------------|
| **Mixed content** | One-size-fits-all parsing | Analyze first, specialize per type |
| **Complex layouts** | Layout detection → OCR | Unified multimodal understanding |
| **Diverse elements** | Limited categories | 21 element types supported |
| **Reading order** | Heuristic rules | Learned reading order |
| **Format output** | Fixed format | Flexible (Markdown, LaTeX, HTML) |

---

## The Two-Stage Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                    DOLPHIN ARCHITECTURE                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   STAGE 1: ANALYSIS                                                 │
│   ┌─────────────────────────────────────────────────────────────┐   │
│   │  Input: Document Image                                      │   │
│   │         ↓                                                   │   │
│   │  Vision Encoder: Extract visual features                    │   │
│   │         ↓                                                   │   │
│   │  LLM Decoder: Identify element types present                │   │
│   │         ↓                                                   │   │
│   │  Output: Element categories (text, table, figure, etc.)     │   │
│   └─────────────────────────────────────────────────────────────┘   │
│                              ↓                                      │
│   STAGE 2: PARSING (Heterogeneous Anchor Prompting)                 │
│   ┌─────────────────────────────────────────────────────────────┐   │
│   │  For each identified element type:                          │   │
│   │  ├── Select specialized anchor prompt                       │   │
│   │  ├── Parse with type-specific strategy                      │   │
│   │  └── Generate structured output                             │   │
│   │                                                             │   │
│   │  Text → Markdown output                                     │   │
│   │  Tables → HTML table format                                 │   │
│   │  Equations → LaTeX format                                   │   │
│   │  Figures → Description + caption                            │   │
│   │  Charts → Data extraction                                   │   │
│   └─────────────────────────────────────────────────────────────┘   │
│                              ↓                                      │
│   FINAL OUTPUT: Unified structured document                         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### Heterogeneous Anchor Prompting (HAP)

The key innovation is **HAP** - using different prompt strategies for different element types:

| Element Type | Anchor Prompt Focus | Output Format |
|--------------|--------------------| --------------|
| **Plain Text** | Reading order, paragraphs | Markdown |
| **Tables** | Structure, cells, headers | HTML `<table>` |
| **Equations** | Mathematical notation | LaTeX |
| **Figures** | Visual description | Caption + description |
| **Code** | Syntax preservation | Code blocks |
| **Charts** | Data extraction | Structured data |

---

## Supported Element Types

Dolphin recognizes 21 different document element categories:

### The Full List

```
┌─────────────────────────────────────────────────────────────────────┐
│                    21 ELEMENT CATEGORIES                             │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   TEXT ELEMENTS                      VISUAL ELEMENTS                │
│   ├── Title                          ├── Figure                     │
│   ├── Text                           ├── Figure Caption             │
│   ├── Paragraph                      ├── Chart                      │
│   ├── Header                         ├── Chart Caption              │
│   ├── Footer                         ├── Diagram                    │
│   └── Page Number                    └── Image                      │
│                                                                     │
│   STRUCTURED ELEMENTS                MATHEMATICAL                   │
│   ├── Table                          ├── Equation (inline)          │
│   ├── Table Caption                  ├── Equation (display)         │
│   ├── List                           └── Formula                    │
│   └── Code Block                                                    │
│                                                                     │
│   SPECIAL ELEMENTS                                                  │
│   ├── Reference                                                     │
│   ├── Footnote                                                      │
│   ├── Watermark                                                     │
│   └── Other                                                         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### Why 21 Categories Matter

| Benefit | Explanation |
|---------|-------------|
| **Precision** | Each element type gets specialized handling |
| **Flexibility** | Can output different formats per type |
| **Completeness** | Covers academic papers, business docs, mixed content |
| **Accuracy** | Type-specific parsing reduces errors |

---

## Training Data

Dolphin is trained on diverse document datasets:

| Dataset | Content Type | Purpose |
|---------|--------------|---------|
| **PubLayNet** | Academic papers | Layout understanding |
| **DocLayNet** | Business documents | Diverse layouts |
| **Table datasets** | Various tables | Table structure |
| **Math datasets** | Equations | LaTeX generation |
| **Chart datasets** | Charts/graphs | Data extraction |

### Key Training Innovations

1. **Synthetic data generation** for underrepresented element types
2. **Multi-task learning** across element categories
3. **Progressive training** from simple to complex documents

---

## Benchmark Performance

### State-of-the-Art Results

| Benchmark | Metric | Dolphin Score | Previous SOTA |
|-----------|--------|---------------|---------------|
| **DocVQA** | ANLS | Competitive | - |
| **ChartQA** | Accuracy | Leading | - |
| **TableQA** | F1 | Top tier | - |
| **MathVista** | Accuracy | Strong | - |

### Layout Detection Performance

| Dataset | mAP | Notes |
|---------|-----|-------|
| **PubLayNet** | High | Academic paper layouts |
| **DocLayNet** | High | Business document layouts |

---

## Use Cases

### 1. Academic Paper Processing

```
INPUT: Scanned academic paper PDF
       ↓
DOLPHIN ANALYSIS:
├── Title detected
├── Abstract detected (text)
├── Figures detected (3)
├── Tables detected (2)
├── Equations detected (15)
└── References detected
       ↓
OUTPUT:
├── Markdown for text sections
├── HTML for tables
├── LaTeX for equations
├── Descriptions for figures
└── Structured references
```

### 2. Business Document Digitization

```
INPUT: Scanned invoice/report
       ↓
DOLPHIN ANALYSIS:
├── Headers/footers identified
├── Tables extracted (pricing, line items)
├── Logos/stamps detected
└── Text content parsed
       ↓
OUTPUT:
├── Structured JSON for data
├── Markdown for narrative
└── Table data for processing
```

### 3. Technical Manual Processing

```
INPUT: Technical documentation with diagrams
       ↓
DOLPHIN ANALYSIS:
├── Diagrams identified
├── Code blocks detected
├── Procedure steps parsed
└── Warning boxes identified
       ↓
OUTPUT:
├── Structured documentation
├── Code preserved with syntax
└── Diagram descriptions for accessibility
```

---

## Comparison with Alternatives

### vs. Traditional OCR + Layout Detection

| Aspect | Traditional | Dolphin |
|--------|-------------|---------|
| **Pipeline** | Multi-step (detect → OCR → classify) | End-to-end multimodal |
| **Element types** | Limited | 21 categories |
| **Reading order** | Heuristic | Learned |
| **Tables** | Separate table detection | Unified parsing |
| **Equations** | Often fails | Native LaTeX output |

### vs. Other Multimodal Models

| Model | Focus | Dolphin Advantage |
|-------|-------|-------------------|
| **GPT-4V** | General vision | Specialized for documents |
| **Gemini** | General multimodal | Document-specific training |
| **Donut** | Document understanding | Analyze-then-parse strategy |
| **Nougat** | Academic papers | Broader document types |

### vs. olmOCR (From Our Deep Dives)

| Aspect | olmOCR | Dolphin |
|--------|--------|---------|
| **Focus** | PDF → text extraction | Document image → structured |
| **Element handling** | Natural reading order | 21 specialized categories |
| **Output** | Clean text for LLM training | Multi-format (Markdown, LaTeX, HTML) |
| **Best for** | Large-scale text extraction | Precise element parsing |
| **Complementary?** | Yes - different use cases | Yes - different use cases |

---

## Technical Architecture

### Model Components

```
┌─────────────────────────────────────────────────────────────────────┐
│                    DOLPHIN MODEL ARCHITECTURE                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   VISION ENCODER                                                    │
│   └── Processes document images at high resolution                  │
│   └── Extracts visual features (layout, text regions, graphics)    │
│   └── Multi-scale feature extraction                               │
│                                                                     │
│   CONNECTOR                                                         │
│   └── Bridges vision features to language space                     │
│   └── Preserves spatial information                                 │
│                                                                     │
│   LLM DECODER                                                       │
│   └── Generates structured output                                   │
│   └── Follows specialized prompts per element type                  │
│   └── Produces Markdown/LaTeX/HTML as needed                       │
│                                                                     │
│   ANCHOR PROMPT BANK                                                │
│   └── Specialized prompts for each of 21 element types              │
│   └── Format specifications                                         │
│   └── Parsing strategies                                            │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### Resolution Handling

Document parsing requires high resolution for:
- Small text recognition
- Table cell boundaries
- Equation symbols
- Figure details

Dolphin uses adaptive resolution strategies to balance quality and efficiency.

---

## Getting Started

### Installation

```bash
# Clone the repository
git clone https://github.com/bytedance/Dolphin.git
cd Dolphin

# Install dependencies
pip install -r requirements.txt

# Download model weights
# (Follow repository instructions for model access)
```

### Basic Usage

```python
# Conceptual usage (check repo for actual API)
from dolphin import DolphinModel

# Load model
model = DolphinModel.from_pretrained("bytedance/dolphin")

# Process document image
result = model.parse("document.png")

# Access structured output
print(result.text_content)      # Markdown text
print(result.tables)            # HTML tables
print(result.equations)         # LaTeX equations
print(result.figures)           # Figure descriptions
```

### Output Example

```markdown
# Paper Title

## Abstract
This paper presents a novel approach to document parsing...

## 1. Introduction
Document understanding is a fundamental task in...

**Table 1**: Comparison of Methods
| Method | Accuracy | Speed |
|--------|----------|-------|
| Baseline | 85.2% | 100ms |
| Ours | 92.1% | 120ms |

The key equation governing our approach is:
$$E = mc^2 + \sum_{i=1}^{n} f(x_i)$$

[Figure 1: Architecture diagram showing the two-stage pipeline]
```

---

## How This Helps Us Achieve Our Goals

Looking at our "Software Development in the Age of AI" repository:

### 1. **Document Processing Pipeline**

| Our Need | How Dolphin Helps |
|----------|-------------------|
| Processing technical docs | Extracts structure, code, diagrams |
| Converting papers to knowledge | Structured Markdown output |
| Table extraction | Native HTML table parsing |
| Equation handling | LaTeX output for math content |

### 2. **Complements Our Other Deep Dives**

| Deep Dive | Connection |
|-----------|------------|
| **olmOCR** | olmOCR for bulk extraction, Dolphin for precise parsing |
| **LangExtract** | Dolphin extracts, LangExtract structures further |
| **Data Formulator** | Charts/tables from Dolphin → visualization |
| **ML Case Studies** | Understanding production document AI |

### 3. **AI-Assisted Development Context**

When working with documentation:
- Extract requirements from PDFs
- Parse technical specifications
- Convert diagrams to descriptions
- Structure legacy documentation

---

## Research Context

### ACL 2025 Paper

Dolphin was accepted at ACL 2025 (Association for Computational Linguistics), one of the top NLP/AI conferences.

**Key contributions**:
1. Analyze-then-parse paradigm for document understanding
2. Heterogeneous Anchor Prompting (HAP) mechanism
3. 21-category element taxonomy
4. State-of-the-art benchmark results

### ByteDance Research

ByteDance (TikTok's parent company) has a strong AI research division working on:
- Computer vision
- Natural language processing
- Multimodal AI
- Document understanding

---

## Quick Reference

### Element Type → Output Format

| Element | Default Output |
|---------|----------------|
| Text, Title, Paragraph | Markdown |
| Table | HTML `<table>` |
| Equation | LaTeX |
| Figure | Description + caption |
| Code | Fenced code block |
| List | Markdown list |

### When to Use Dolphin

| Use Case | Good Fit? |
|----------|-----------|
| Academic paper parsing | ✅ Excellent |
| Business document digitization | ✅ Good |
| Technical documentation | ✅ Good |
| Handwritten notes | ⚠️ Limited |
| Historical documents | ⚠️ Depends |
| Simple text extraction | ❌ Overkill (use simpler OCR) |

---

## Key Links

| Resource | URL |
|----------|-----|
| GitHub Repository | https://github.com/bytedance/Dolphin |
| ByteDance Research | https://www.bytedance.com/en/research |
| ACL 2025 | https://2025.aclweb.org/ |
| Related: olmOCR | https://github.com/allenai/olmocr |

---

## Quick Stats

| Metric | Value |
|--------|-------|
| Creator | ByteDance Research |
| Published | ACL 2025 |
| Element Categories | 21 |
| Architecture | Two-stage (analyze-then-parse) |
| Key Innovation | Heterogeneous Anchor Prompting |
| Output Formats | Markdown, LaTeX, HTML |
| License | Check repository |

---

## Bottom Line

Dolphin represents a paradigm shift in document parsing: **analyze first, then specialize**.

**Why it matters**:
- Documents are heterogeneous (text + tables + figures + equations)
- One-size-fits-all parsing loses information
- Specialized handling per element type improves accuracy
- Unified output enables downstream processing

**Key insight**: The analyze-then-parse approach mirrors how humans read documents - we first identify what kind of content we're looking at, then apply appropriate reading strategies.

For our repository focused on AI-assisted development, Dolphin complements olmOCR in the document processing pipeline. While olmOCR excels at bulk text extraction for LLM training, Dolphin provides precise, structured parsing when you need to preserve document structure exactly.

---

*Document created: January 2026*
*Sources: [GitHub - bytedance/Dolphin](https://github.com/bytedance/Dolphin), [ACL 2025](https://2025.aclweb.org/), [ByteDance Research](https://www.bytedance.com/en/research)*
