# Research

## Research Results

<details>
<summary>According to official Gemini documentation and model cards, what inputs are supported for images and PDFs (inline bytes, Base64, public URLs, Google Cloud Storage URIs), what are the size/format limits and tiling rules, and what are recommended practices for enterprise use?</summary>

### Source [6]: https://ai.google.dev/gemini-api/docs/document-processing

Query: According to official Gemini documentation and model cards, what inputs are supported for images and PDFs (inline bytes, Base64, public URLs, Google Cloud Storage URIs), what are the size/format limits and tiling rules, and what are recommended practices for enterprise use?

Answer: **Supported Inputs for PDFs:**  
The Gemini API accepts PDF documents as input using two primary methods:
- **Inline bytes:** You can pass PDF data as raw bytes directly in the API call.
- **Base64 encoding:** You can encode the PDF as a Base64 string and submit it as part of the request.

**Size and Format Limits:**  
- The maximum allowed PDF payload for inline uploads is **20MB**.
- The API supports standard PDF MIME type ('application/pdf').
- There are code examples for both Python and JavaScript showing how to convert a PDF (fetched from a public URL) into bytes or Base64, but actual remote URLs or Google Cloud Storage URIs are not directly passed to the API—they are fetched and then sent as data.

**Tiling Rules:**  
- No explicit mention of document tiling rules is provided in this source.

**Recommended Practices for Enterprise Use:**  
- No detailed enterprise recommendations are given, but the documentation suggests fetching PDFs, handling them securely, and encoding them properly before submission.

-----

-----

-----

### Source [9]: https://ai.google.dev/gemini-api/docs/image-understanding

Query: According to official Gemini documentation and model cards, what inputs are supported for images and PDFs (inline bytes, Base64, public URLs, Google Cloud Storage URIs), what are the size/format limits and tiling rules, and what are recommended practices for enterprise use?

Answer: **Supported Inputs for Images:**
- **Inline image data:** Images can be provided as inline bytes (raw binary data) or as Base64-encoded strings in the API call.
- **File API:** For larger images or when reusing images in multiple requests, uploading via the File API is recommended.

**Size and Format Limits:**
- The total request size, including images and prompts, must be **less than 20MB** when passing inline data.
- Supported image formats include JPEG and PNG, as indicated by code samples and standard usage.

**Tiling Rules:**
- No explicit tiling or chunking rules are provided for images in this documentation.

**Recommended Practices for Enterprise Use:**
- Use the File API for larger files or frequent reuse to optimize resource management and performance.
- For small images and ad hoc requests, inline data is preferred.

-----

-----

</details>

<details>
<summary>Which production APIs in 2024–2025 provide multimodal (text–image) embeddings in a shared vector space (e.g., Voyage AI, Cohere, Google Vertex AI, OpenAI/CLIP), and what are their documented capabilities, limitations, and pricing?</summary>

### Source [11]: https://blog.voyageai.com/2024/11/12/voyage-multimodal-3/

Query: Which production APIs in 2024–2025 provide multimodal (text–image) embeddings in a shared vector space (e.g., Voyage AI, Cohere, Google Vertex AI, OpenAI/CLIP), and what are their documented capabilities, limitations, and pricing?

Answer: "Voyage Multimodal 3" is Voyage AI’s dedicated **multimodal embedding model** for production, allowing **interleaved text and image inputs** in a shared vector space. The model’s architecture is a modern vision-language transformer, distinct from earlier models like OpenAI CLIP and Cohere Multimodal v3. Voyage Multimodal 3 outperforms OpenAI CLIP large and Cohere Multimodal v3 by 2.1–2.2x on table/figure retrieval, 25–26% on document screenshot retrieval, and about 6% on text-to-photo retrieval. It also surpasses OpenAI v3 large and Cohere v3 on text-only datasets.

**Capabilities:**
- Handles interleaved text–image input.
- Superior performance across multiple retrieval tasks and datasets.
- Demonstrated with example notebooks for developers.

**Limitations:** Not specifically listed, but the model is presented as a significant upgrade over previous generation models in both performance and flexibility.

Pricing is not explicitly stated in this source, but Voyage's general pricing for embeddings is $0.06 per 1M tokens for Voyage-3[2].

-----

-----

</details>

<details>
<summary>What authoritative sources explain the Base64 storage overhead (~33%) and best practices for storing large media in object storage (S3/GCS) with URL‑based access instead of raw bytes/Base64 in relational databases, including I/O and scalability considerations for AI pipelines?</summary>

### Source [20]: https://lemire.me/blog/2019/01/30/what-is-the-space-overhead-of-base64-encoding/

Query: What authoritative sources explain the Base64 storage overhead (~33%) and best practices for storing large media in object storage (S3/GCS) with URL‑based access instead of raw bytes/Base64 in relational databases, including I/O and scalability considerations for AI pipelines?

Answer: The **Base64 encoding process increases file size by at least 33%** compared to the original binary data, as Base64 encodes every 3 bytes of data into 4 bytes of text, resulting in a 4/3 size ratio. For large files, this overhead is approximately 33%, but for very small files, the overhead can be even higher due to padding. The blog clarifies that the Base64 method "wastes 1/4 of the bits," meaning the efficiency of Base64 is 75%, and the storage overhead is 33% or greater depending on data size and alignment. The post highlights that while Base64 may be useful for embedding small resources, it is inefficient and not recommended for storing large binary data like images, especially where storage and I/O efficiency are concerns.

-----

-----

-----

### Source [23]: https://en.wikipedia.org/wiki/Base64

Query: What authoritative sources explain the Base64 storage overhead (~33%) and best practices for storing large media in object storage (S3/GCS) with URL‑based access instead of raw bytes/Base64 in relational databases, including I/O and scalability considerations for AI pipelines?

Answer: Base64 encoding introduces an **overhead of 33–37%** relative to the original binary data. The encoding itself adds a 33% increase, with an additional few percent possible due to line breaks or padding, depending on the implementation and use case. This overhead makes Base64 unsuitable for large binaries or media files if storage or network efficiency is a priority. Wikipedia also notes that Base64 is primarily intended for text-safe transmission of binary data where direct binary transport is problematic, not for efficient storage or large-scale transfer. For scalable, high-performance systems (such as AI pipelines), storing raw binary data in object stores and referencing it by URL is more efficient and a common best practice.

-----

-----

</details>

<details>
<summary>What comparative studies or official papers detail trade‑offs among multimodal LLM architectures—unified token concatenation vs cross‑attention vs hybrid designs—especially regarding high‑resolution efficiency and OCR‑style tasks (e.g., NVLM, Llama 3.2/3.x Vision, Molmo, Pixtral)?</summary>

### Source [34]: https://magazine.sebastianraschka.com/p/understanding-multimodal-llms

Query: What comparative studies or official papers detail trade‑offs among multimodal LLM architectures—unified token concatenation vs cross‑attention vs hybrid designs—especially regarding high‑resolution efficiency and OCR‑style tasks (e.g., NVLM, Llama 3.2/3.x Vision, Molmo, Pixtral)?

Answer: This overview distinguishes two primary multimodal LLM architectures: **Unified Embedding Decoder Architecture** (token concatenation) and **Cross-Modality Attention Architecture**. In the unified approach, images are converted into tokens of the same embedding size as text, allowing the LLM to jointly process text and image tokens through simple concatenation. This method leverages an unmodified decoder (as in Llama) and is conceptually straightforward. In contrast, the cross-attention approach fuses image and text representations directly within the attention mechanism, enabling more explicit interactions between modalities. The article notes that the unified approach may face challenges with efficiency for high-resolution images due to the sheer number of image tokens required, while cross-attention offers more flexible integration but adds architectural complexity. The article references recent research (e.g., LLaVA, Flamingo) and mentions ongoing ablation studies that analyze trade-offs, including the impact of data mixtures and coordinate tokens, but does not provide detailed benchmarking data or direct OCR task comparisons.

-----

-----

-----

### Source [35]: https://arxiv.org/html/2409.11402v1

Query: What comparative studies or official papers detail trade‑offs among multimodal LLM architectures—unified token concatenation vs cross‑attention vs hybrid designs—especially regarding high‑resolution efficiency and OCR‑style tasks (e.g., NVLM, Llama 3.2/3.x Vision, Molmo, Pixtral)?

Answer: The NVLM paper provides a comprehensive comparative analysis of **decoder-only (token concatenation)** and **cross-attention** multimodal LLMs, specifically evaluating trade-offs for **high-resolution efficiency** and **OCR-style tasks**:
- **Architecture comparison:** Using identical LLM backbones and vision encoders, their findings show that the **cross-attention-based NVLM-X** is superior for high-resolution images due to better computational efficiency, while the **decoder-only NVLM-D** offers stronger unified multimodal reasoning and achieves higher accuracy on OCR-related tasks.
- **Hybrid design:** The proposed **NVLM-H hybrid** combines strengths of both, excelling in multimodal reasoning while improving efficiency for high-res images.
- **Tile-tagging for high-res:** For high-resolution and OCR, a dynamic tiling mechanism ("tile-tagging") further boosts accuracy by introducing text-based 1-D tile tags before corresponding image tokens in the decoder, as demonstrated through extensive ablation.
- **Training data:** Diverse, high-quality multimodal pretraining data is critical for both architectures; cross-attention models particularly benefit from this, but decoder-only models also see significant gains from curated data.
The paper includes detailed ablation studies and direct benchmarking across OCR (OCRBench) and multimodal reasoning (MMMU) tasks, offering one of the most rigorous official comparisons currently available for these architectures.

-----

-----

</details>

<details>
<summary>What does the ColPali/ViDoRe literature report about its architecture (multi‑vector embeddings, MaxSim late interaction), indexing/retrieval latency, storage footprint, and benchmark performance (e.g., nDCG@5 averages), including guidance on reranking use?</summary>

### Source [40]: https://github.com/illuin-tech/colpali

Query: What does the ColPali/ViDoRe literature report about its architecture (multi‑vector embeddings, MaxSim late interaction), indexing/retrieval latency, storage footprint, and benchmark performance (e.g., nDCG@5 averages), including guidance on reranking use?

Answer: The ColPali codebase and associated paper detail a **multi-vector embedding architecture** based on ColBERT and PaliGemma, leveraging VLMs for document image retrieval. Visual patches from PaliGemma-3B’s ViT are projected to form document embeddings; queries are encoded similarly. The **MaxSim late interaction** mechanism compares each query token with each document patch, maximizing similarity scores for efficient, fine-grained retrieval. ColPali is benchmarked on the **ViDoRe leaderboard** using the `vidore-benchmark` package.

For interpretability, ColPali provides similarity maps that visualize salient image patches per query term, helping users understand which document regions are most relevant. To use these maps, the `colpali-engine[interpretability]` package is required, allowing overlays of similarity scores on the original images. Guidance on reranking is not explicit here, but the architecture is designed for efficient first-pass retrieval, with the option to visualize and further process results.

-----

-----

</details>

<details>
<summary>What do peer‑reviewed evaluations report about OCR accuracy and failure modes on complex, visually rich documents (multi‑column PDFs, tables, charts/diagrams, engineering drawings/blueprints, medical scans, and handwriting), and which benchmark datasets or competitions (e.g., ICDAR Robust Reading, DocVQA, PubLayNet, Chart/Tab tasks) provide CER/WER and structure‑extraction metrics?</summary>

### Source [51]: https://rrc.cvc.uab.es/?ch=13&com=evaluation&task=3

Query: What do peer‑reviewed evaluations report about OCR accuracy and failure modes on complex, visually rich documents (multi‑column PDFs, tables, charts/diagrams, engineering drawings/blueprints, medical scans, and handwriting), and which benchmark datasets or competitions (e.g., ICDAR Robust Reading, DocVQA, PubLayNet, Chart/Tab tasks) provide CER/WER and structure‑extraction metrics?

Answer: The ICDAR 2019 Robust Reading Challenge provides **leaderboards and detailed evaluation results** for document parsing and structured text understanding.

- Methods like **StrucTexT** (Baidu-OCR) and **GraphDoc** (USTC) are evaluated on datasets including **PDFs, invoices, and receipts**; these datasets commonly feature complex layouts such as multi-column formats and tables.
- Evaluation protocols **exclude OCR mismatch errors** when appropriate, and focus on **structure extraction accuracy** in addition to text recognition.
- Leaderboards show high performance (over 90% on certain metrics) for advanced models, but detailed per-metric breakdowns (e.g., CER/WER, structure extraction) are available for individual submissions.
- Benchmarks used for these evaluations (e.g., SROIE, RVL-CDIP) are widely referenced in peer-reviewed literature for assessing document understanding.

These competitions and their metrics provide ground truth for peer-reviewed evaluations, with structure extraction and recognition performance tracked separately.

-----

-----

</details>

<details>
<summary>According to Google’s official Gemini and Vertex AI documentation, which input methods are supported for images and PDFs (inline bytes, Base64, File API, public URLs, Google Cloud Storage URIs, and any web/URL tools), what are the size/format limits and tiling/token accounting rules, and what are the recommended practices for secure enterprise workflows?</summary>

### Source [54]: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/document-understanding

Query: According to Google’s official Gemini and Vertex AI documentation, which input methods are supported for images and PDFs (inline bytes, Base64, File API, public URLs, Google Cloud Storage URIs, and any web/URL tools), what are the size/format limits and tiling/token accounting rules, and what are the recommended practices for secure enterprise workflows?

Answer: **Supported input methods for PDFs and images**:  
Vertex AI Gemini supports adding PDFs and text files to requests via the Google Cloud Console and Vertex AI API. The documentation provides sample code for including a PDF in a prompt. The main supported input methods mentioned are PDF uploads via the API and through the console interface.

**Supported models**:  
A range of Gemini models support document (PDF) understanding, including Gemini 2.5 Pro, 2.5 Flash, and their variants.

**Size/format limits, tiling, and token accounting**:  
The documentation references a quota metric: `generate_content_document_input_per_base_model_id_and_resolution`, which relates to the maximum tokens per minute from document inputs across all requests in a project. For detailed token limits or tiling, you are referred to model-specific documentation.

**Recommended practices for secure enterprise workflows**:  
The page suggests reviewing responsible AI best practices and Vertex AI's safety filters, directing users to additional documentation for secure and compliant deployment.

-----

-----

</details>

<details>
<summary>Which production APIs currently offer shared text–image embedding spaces (e.g., Google Vertex AI Multimodal Embeddings, Voyage Multimodal 3, Cohere, OpenAI/CLIP), what do their model cards and docs say about capabilities, limits, pricing, and embedding dimensions, and what vendor guides show vector database integration patterns for multimodal RAG?</summary>

### Source [66]: https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings

Query: Which production APIs currently offer shared text–image embedding spaces (e.g., Google Vertex AI Multimodal Embeddings, Voyage Multimodal 3, Cohere, OpenAI/CLIP), what do their model cards and docs say about capabilities, limits, pricing, and embedding dimensions, and what vendor guides show vector database integration patterns for multimodal RAG?

Answer: The **multimodal embeddings model** from Vertex AI produces **1408-dimensional vectors** for images, text, or video. These vectors are in a **shared semantic space**, making cross-modal retrieval straightforward (e.g., searching images using text queries). For text-only scenarios, the documentation recommends the text-embeddings API for better results. The documentation provides best practices such as distinguishing between text within images (OCR-like) and image content, and notes that similarity metrics (like dot product) are not calibrated probabilities—ranking-based approaches are recommended for retrieval tasks. The only officially supported model for multimodal embeddings is `multimodalembedding`.

-----

-----

</details>

<details>
<summary>What do the ColPali paper and official repositories report about its multi‑vector architecture (PaliGemma+SigLIP), MaxSim late interaction, ViDoRe benchmark results (e.g., nDCG@5), indexing/query latencies and storage footprint, and recommended practices such as reranking and token pooling?</summary>

### Source [69]: https://arxiv.org/html/2407.01449v6

Query: What do the ColPali paper and official repositories report about its multi‑vector architecture (PaliGemma+SigLIP), MaxSim late interaction, ViDoRe benchmark results (e.g., nDCG@5), indexing/query latencies and storage footprint, and recommended practices such as reranking and token pooling?

Answer: ColPali extends PaliGemma-3B to generate ColBERT-style **multi-vector representations** for both text and images, allowing for detailed, token-level alignment between queries and documents. The architecture feeds SigLIP-generated image patch embeddings into a language model, mapping both text and image tokens to a shared latent space. **MaxSim late interaction** is employed: at inference, each query token is compared to all document patch vectors, with the maximum similarity per query token summed to yield the final relevance score. This architecture demonstrates a significant performance improvement over prior approaches, especially on visually complex benchmarks such as InfographicVQA, ArxivQA, and TabFQuAD, as well as on text-centric benchmarks across domains and languages. Negative results showed that simply applying late interaction to BiSigLIP or mixing SigLIP and PaliGemma embeddings without alignment led to poor performance, highlighting the importance of coordinated multi-modal training. The paper details these results and architectural choices, attributing superior results to the synergy between multi-vector late interaction and well-aligned vision-language embeddings.

-----

-----

</details>

<details>
<summary>What do the ColPali paper and the ViDoRe leaderboard report about retrieval quality (e.g., nDCG@5 across tasks), indexing/query latencies, storage footprint, and best‑practice techniques like late‑interaction reranking and token pooling?</summary>

### Source [93]: https://pypi.org/project/vidore-benchmark/

Query: What do the ColPali paper and the ViDoRe leaderboard report about retrieval quality (e.g., nDCG@5 across tasks), indexing/query latencies, storage footprint, and best‑practice techniques like late‑interaction reranking and token pooling?

Answer: The `vidore-benchmark` package allows users to **evaluate retrievers like ColPali on the ViDoRe leaderboard**. It provides **command-line tools** for running retrieval evaluations and reproducing paper results. The documentation does not specify **retrieval quality metrics (e.g., nDCG@5), latency, or storage footprint** directly, but it is implied that using this tool will yield the same metrics reported in the ColPali paper and on the ViDoRe leaderboard. Users are warned about a known initialization issue that may cause minor metric differences compared to published results.
-----

-----

</details>

<details>
<summary>What do Google’s official Gemini and Vertex AI docs specify about supplying images and PDFs (inline bytes/Base64, Files API uploads, public URLs, GCS URIs), including exact size limits, page-based token accounting for PDFs, and image tiling rules—and is “URL context” officially supported for fetching public PDFs?</summary>

### Source [102]: https://ai.google.dev/api/files

Query: What do Google’s official Gemini and Vertex AI docs specify about supplying images and PDFs (inline bytes/Base64, Files API uploads, public URLs, GCS URIs), including exact size limits, page-based token accounting for PDFs, and image tiling rules—and is “URL context” officially supported for fetching public PDFs?

Answer: The Gemini API documentation on using files provides:
- **Files API:** You can **upload media files (images, PDFs, etc.) separately** from the prompt input, allowing reuse across requests and prompts.
- **Example:** Files are uploaded via `client.files.upload()` (Python) or equivalent in Node.js, and then referenced in prompts using their returned file references or URIs.
- **Supported formats:** Example shows images; the text and broader docs confirm PDFs are also supported.
- **No direct support for public URLs:** Files must be uploaded before referencing them in prompts. There is **no official support for fetching public PDFs by URL context**; the file must be uploaded to Gemini's managed storage via the Files API.
- **No mention of inline base64/bytes** for Gemini API (unlike Document AI).
- **No explicit documentation** here about size limits, page-based token accounting for PDFs, or image tiling.

-----

-----

</details>

<details>
<summary>According to the ColPali paper, code, and ViDoRe leaderboard, what are the reported nDCG@5 results, indexing/query latencies, storage footprint, and best‑practice techniques (MaxSim late interaction, token pooling/reranking) for document image retrieval?</summary>

### Source [116]: https://huggingface.co/blog/manu/colpali

Query: According to the ColPali paper, code, and ViDoRe leaderboard, what are the reported nDCG@5 results, indexing/query latencies, storage footprint, and best‑practice techniques (MaxSim late interaction, token pooling/reranking) for document image retrieval?

Answer: This source provides an overview of ColPali's technical foundation and practical workflow:

- **Pipeline comparison**: Traditional document retrieval relies on OCR, layout detection, captioning, and chunking, then maps text chunks using neural embedding models (e.g., BGE M3)[2]. These steps are resource-intensive and can propagate errors.
- **ColPali's approach**: ColPali directly embeds the page image using **Vision Language Models** (notably PaliGemma) and leverages **multi-vector retrieval with late interaction** (as in ColBERT)[2].
- **Best-practice techniques**:
  - **Multi-vector retrieval**: Query tokens and image patches are embedded in the same vector space. Relevance is computed using MaxSim late interaction, enabling robust matching between queries and visually-rich document content[2].
  - **End-to-end training**: ColPali is trained end-to-end to optimize for retrieval tasks, simplifying the pipeline and improving speed and accuracy[2].

This source does not report exact benchmark metrics (nDCG@5, latency, storage footprint), but it confirms the use of MaxSim late interaction and direct image embedding as best-practice techniques.

-----

-----

</details>

<details>
<summary>What peer‑reviewed benchmarks and challenge reports quantify OCR performance and failure modes on complex documents (multi‑column PDFs, tables, charts/diagrams, handwriting, low‑quality scans), including CER/WER and structure‑extraction metrics (e.g., DocVQA, ICDAR RRC, PubLayNet, Chart/Tab tasks)?</summary>

### Source [131]: https://arxiv.org/html/2502.06445v1

Query: What peer‑reviewed benchmarks and challenge reports quantify OCR performance and failure modes on complex documents (multi‑column PDFs, tables, charts/diagrams, handwriting, low‑quality scans), including CER/WER and structure‑extraction metrics (e.g., DocVQA, ICDAR RRC, PubLayNet, Chart/Tab tasks)?

Answer: This open-source benchmark by VideoDB systematically evaluated **Vision-Language Models (VLMs) and traditional OCR systems** using **character error rate (CER), word error rate (WER), and average accuracy**. Tested models included RapidOCR, EasyOCR, Claude-3 Sonnet, Gemini-1.5 Pro, and GPT-4o. Results show **GPT-4o and Gemini-1.5 Pro achieve the lowest error rates and highest accuracy (76.22% and 76.13% respectively)**, while traditional tools underperform, with RapidOCR and EasyOCR showing higher error rates and lower accuracy (CER: 0.43–0.51, WER: 0.76–0.82). The benchmark also analyzed **domain-specific performance** (e.g., legal/educational, finance/business/news, handwritten text), revealing that modern VLMs maintain robust accuracy on challenging domains like handwriting and multi-structure layouts, whereas traditional OCR models struggle with these complexities. Processing speed per image is also reported, with Gemini-1.5 Pro being the fastest.

-----

-----

-----

### Source [132]: https://www.primaresearch.org/www/assets/papers/HIP21_CNeudecker_OcrEvalSurvey.pdf

Query: What peer‑reviewed benchmarks and challenge reports quantify OCR performance and failure modes on complex documents (multi‑column PDFs, tables, charts/diagrams, handwriting, low‑quality scans), including CER/WER and structure‑extraction metrics (e.g., DocVQA, ICDAR RRC, PubLayNet, Chart/Tab tasks)?

Answer: This survey from PRImA provides a comprehensive overview of **OCR evaluation tools and metrics**. It discusses the most prominent metrics, including **CER, WER, and layout evaluation metrics**, and emphasizes the importance of **structure extraction** (reading order, over-/under-segmentation of text lines) for documents with complex layouts (multi-column, tables, diagrams). Figures in the paper illustrate how metrics are applied to datasets such as IMPACT and ENP, comparing performance across models and layouts. The survey notes that **layout analysis is crucial for digital scholarship and linguistic analysis**, but is insufficiently addressed by most standard metrics. It advocates for **standard evaluation profiles tailored to specific use cases** and highlights the need for metrics that robustly capture layout and structure extraction failures, not just text recognition errors.

-----

-----

-----

### Source [133]: https://arxiv.org/html/2412.02210v2

Query: What peer‑reviewed benchmarks and challenge reports quantify OCR performance and failure modes on complex documents (multi‑column PDFs, tables, charts/diagrams, handwriting, low‑quality scans), including CER/WER and structure‑extraction metrics (e.g., DocVQA, ICDAR RRC, PubLayNet, Chart/Tab tasks)?

Answer: The CC-OCR benchmark is designed to evaluate **Large Multimodal Models (LMMs)** on four key OCR-centric tasks: **multilingual text reading, text spotting, layout-aware document parsing, and key information extraction**. CC-OCR features a diverse set of document scenarios, including **orientation-sensitivity, grounding, natural noise, and artistic text**, to assess performance on fine-grained visual and structural challenges. The benchmark covers **structured inputs and outputs**, making it suitable for quantifying both recognition accuracy and structure extraction. Five generalist models (including GPT-4o and Gemini-1.5) are assessed, with results highlighting strengths and weaknesses in **real-world applications** such as document digitization and urban monitoring. CC-OCR is positioned as a comprehensive standard for assessing OCR robustness and failure modes on complex, multi-layout, and noisy documents.

-----

</details>

<details>
<summary>What peer‑reviewed evaluations or leaderboard studies quantify how OCR pipelines compare to vision‑language models on complex documents (multi‑column PDFs, tables, charts/diagrams, handwritten or low‑quality scans), including CER/WER, structure‑extraction scores, and end‑to‑end QA/retrieval on DocVQA/ChartQA/PubLayNet/ViDoRe?</summary>

### Source [149]: https://arxiv.org/html/2505.05666v1

Query: What peer‑reviewed evaluations or leaderboard studies quantify how OCR pipelines compare to vision‑language models on complex documents (multi‑column PDFs, tables, charts/diagrams, handwritten or low‑quality scans), including CER/WER, structure‑extraction scores, and end‑to‑end QA/retrieval on DocVQA/ChartQA/PubLayNet/ViDoRe?

Answer: This peer-reviewed study provides a **systematic comparison of a vision-based RAG (Retrieval-Augmented Generation) system and traditional OCR-based pipelines** using large language models (Llama 3.2, 90B, and Llama 3.3) on document QA tasks. The study evaluates both pipelines on metrics including **computational efficiency** and **retrieval accuracy**. The findings indicate:

- **VLM-based pipelines** are more efficient in embedding generation and retrieval latency: embedding time was 0.4739s per document for VLMs vs. 0.5503s for OCR; retrieval latency was 0.0010s per query for VLMs vs. 0.0311s for OCR.

- **Memory usage** was substantially lower for VLM-based pipelines (1.38 GB per 1,000 documents) than for OCR-based systems (9.5 GB per 1,000 documents), attributed to the absence of extensive OCR-derived textual embeddings.

- **OCR pipelines** require significant time for text extraction (6s per document), which is not needed for VLMs.

While this study does not report CER/WER or structure-extraction scores directly, it provides **quantitative leaderboard-style comparisons for computational metrics and end-to-end QA/retrieval performance** on complex documents. The results are based on large-scale experiments conducted with state-of-the-art hardware, ensuring reliability for real-world deployments.

-----

</details>


## Sources Scraped From Research Results

<details>
<summary>Title: ColPali: Efficient Document Retrieval with Vision Language Models</summary>

# Title: ColPali: Efficient Document Retrieval with Vision Language Models

Authors: [Manuel Faysse](https://arxiv.org/search/cs?searchtype=author&query=Faysse,+M), [Hugues Sibille](https://arxiv.org/search/cs?searchtype=author&query=Sibille,+H), [Tony Wu](https://arxiv.org/search/cs?searchtype=author&query=Wu,+T), [Bilel Omrani](https://arxiv.org/search/cs?searchtype=author&query=Omrani,+B), [Gautier Viaud](https://arxiv.org/search/cs?searchtype=author&query=Viaud,+G), [Céline Hudelot](https://arxiv.org/search/cs?searchtype=author&query=Hudelot,+C), [Pierre Colombo](https://arxiv.org/search/cs?searchtype=author&query=Colombo,+P)

[View PDF](https://arxiv.org/pdf/2407.01449) [HTML (experimental)](https://arxiv.org/html/2407.01449v6)

> Abstract: Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at [this https URL](https://hf.co/vidore).

|     |     |
| --- | --- |
| Comments: | Published as a conference paper at ICLR 2025 |
| Subjects: | Information Retrieval (cs.IR); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | [arXiv:2407.01449](https://arxiv.org/abs/2407.01449) \[cs.IR\] |
|  | (or [arXiv:2407.01449v6](https://arxiv.org/abs/2407.01449v6) \[cs.IR\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2407.01449](https://doi.org/10.48550/arXiv.2407.01449) |

## Submission history

From: Tony Wu \[ [view email](https://arxiv.org/show-email/4cb4f549/2407.01449)\]

**[\[v1\]](https://arxiv.org/abs/2407.01449v1)**  
Thu, 27 Jun 2024 15:45:29 UTC (12,102 KB)

**[\[v2\]](https://arxiv.org/abs/2407.01449v2)**  
Tue, 2 Jul 2024 13:02:58 UTC (12,102 KB)

**[\[v3\]](https://arxiv.org/abs/2407.01449v3)**  
Mon, 7 Oct 2024 07:46:00 UTC (12,117 KB)

**[\[v4\]](https://arxiv.org/abs/2407.01449v4)**  
Wed, 5 Feb 2025 08:42:57 UTC (15,111 KB)

**[\[v5\]](https://arxiv.org/abs/2407.01449v5)**  
Thu, 6 Feb 2025 09:57:56 UTC (15,111 KB)

**\[v6\]**  
Fri, 28 Feb 2025 08:51:57 UTC (15,110 KB)

Full-text links:

- [View PDF](https://arxiv.org/pdf/2407.01449)  
- [HTML (experimental)](https://arxiv.org/html/2407.01449v6)  
- [TeX Source](https://arxiv.org/src/2407.01449)  
- [Other Formats](https://arxiv.org/format/2407.01449)

[View license](http://creativecommons.org/publicdomain/zero/1.0/)

### References & Citations

- [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:2407.01449)  
- [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=2407.01449)  
- [Semantic Scholar](https://api.semanticscholar.org/arXiv:2407.01449)

</details>

<details>
<summary>Computer Science > Computation and Language</summary>

# Computer Science > Computation and Language

**arXiv:2409.11402** (cs)

\[Submitted on 17 Sep 2024 (v1), last revised 22 Oct 2024 (this version, v2)\]

# Title: NVLM: Open Frontier-Class Multimodal LLMs

Authors: [Wenliang Dai](https://arxiv.org/search/cs?searchtype=author&query=Dai,+W), [Nayeon Lee](https://arxiv.org/search/cs?searchtype=author&query=Lee,+N), [Boxin Wang](https://arxiv.org/search/cs?searchtype=author&query=Wang,+B), [Zhuolin Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+Z), [Zihan Liu](https://arxiv.org/search/cs?searchtype=author&query=Liu,+Z), [Jon Barker](https://arxiv.org/search/cs?searchtype=author&query=Barker,+J), [Tuomas Rintamaki](https://arxiv.org/search/cs?searchtype=author&query=Rintamaki,+T), [Mohammad Shoeybi](https://arxiv.org/search/cs?searchtype=author&query=Shoeybi,+M), [Bryan Catanzaro](https://arxiv.org/search/cs?searchtype=author&query=Catanzaro,+B), [Wei Ping](https://arxiv.org/search/cs?searchtype=author&query=Ping,+W)

[View PDF](https://arxiv.org/pdf/2409.11402) [HTML (experimental)](https://arxiv.org/html/2409.11402v2)

> Abstract:We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. In terms of model design, we perform a comprehensive comparison between decoder-only multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g., Flamingo). Based on the strengths and weaknesses of both approaches, we propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for tile-based dynamic high-resolution images, which significantly boosts performance on multimodal reasoning and OCR-related tasks. Regarding training data, we meticulously curate and provide detailed information on our multimodal pretraining and supervised fine-tuning datasets. Our findings indicate that dataset quality and task diversity are more important than scale, even during the pretraining phase, across all architectures. Notably, we develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks while maintaining and even improving text-only performance compared to their LLM backbones. To achieve this, we craft and integrate a high-quality text-only dataset into multimodal training, alongside a substantial amount of multimodal math and reasoning data, leading to enhanced math and coding capabilities across modalities. To advance research in the field, we release the model weights at [this https URL](https://huggingface.co/nvidia/NVLM-D-72B) and will open-source the training code for the community soon.

|     |     |
| --- | --- |
| Comments: | Fixed the typos. For more information, please visit our project page at: [this https URL](https://research.nvidia.com/labs/adlr/NVLM-1) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM) |
| Cite as: | [arXiv:2409.11402](https://arxiv.org/abs/2409.11402) \[cs.CL\] (or [arXiv:2409.11402v2](https://arxiv.org/abs/2409.11402v2) \[cs.CL\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2409.11402](https://doi.org/10.48550/arXiv.2409.11402) |

## Submission history

From: Wei Ping \[ [view email](https://arxiv.org/show-email/6d6c7b58/2409.11402)\]

**[v1]**  
Tue, 17 Sep 2024 17:59:06 UTC (20,301 KB)

**[v2]**  
Tue, 22 Oct 2024 23:13:34 UTC (12,259 KB)

Full-text links:

- [View PDF](https://arxiv.org/pdf/2409.11402)
- [HTML (experimental)](https://arxiv.org/html/2409.11402v2)
- [TeX Source](https://arxiv.org/src/2409.11402)
- [Other Formats](https://arxiv.org/format/2409.11402)

View license: [Creative Commons Attribution 4.0 International (CC BY 4.0)](http://creativecommons.org/licenses/by/4.0/)

### References & Citations

- [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:2409.11402)
- [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=2409.11402)
- [Semantic Scholar](https://api.semanticscholar.org/arXiv:2409.11402)

</details>

<details>
<summary>Computer Science > Computer Vision and Pattern Recognition</summary>

# Computer Science > Computer Vision and Pattern Recognition

**arXiv:2505.05666** (cs)

[Submitted on 8 May 2025]

# Title: Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval

Authors: [Alexander Most](https://arxiv.org/search/cs?searchtype=author&query=Most,+A), [Joseph Winjum](https://arxiv.org/search/cs?searchtype=author&query=Winjum,+J), [Ayan Biswas](https://arxiv.org/search/cs?searchtype=author&query=Biswas,+A), [Shawn Jones](https://arxiv.org/search/cs?searchtype=author&query=Jones,+S), [Nishath Rajiv Ranasinghe](https://arxiv.org/search/cs?searchtype=author&query=Ranasinghe,+N+R), [Dan O'Malley](https://arxiv.org/search/cs?searchtype=author&query=O%27Malley,+D), [Manish Bhattarai](https://arxiv.org/search/cs?searchtype=author&query=Bhattarai,+M)

[View PDF](https://arxiv.org/pdf/2505.05666) [HTML (experimental)](https://arxiv.org/html/2505.05666v1)

> Abstract: Retrieval-Augmented Generation (RAG) has become a popular technique for enhancing the reliability and utility of Large Language Models (LLMs) by grounding responses in external documents. Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text. However, even state-of-the-art OCRs can introduce errors, especially in degraded or complex documents. Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR. This study presents a systematic comparison between a vision-based RAG system (ColPali) and more traditional OCR-based pipelines utilizing Llama 3.2 (90B) and Nougat OCR across varying document qualities. Beyond conventional retrieval accuracy metrics, we introduce a semantic answer evaluation benchmark to assess end-to-end question-answering performance. Our findings indicate that while vision-based RAG performs well on documents it has been fine-tuned on, OCR-based RAG is better able to generalize to unseen documents of varying quality. We highlight the key trade-offs between computational efficiency and semantic accuracy, offering practical guidance for RAG practitioners in selecting between OCR-dependent and vision-based document retrieval systems in production environments.

|     |     |
| --- | --- |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | [arXiv:2505.05666](https://arxiv.org/abs/2505.05666) \[cs.CV\] |
|  | (or [arXiv:2505.05666v1](https://arxiv.org/abs/2505.05666v1) \[cs.CV\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2505.05666](https://doi.org/10.48550/arXiv.2505.05666) |

## Submission history

From: Manish Bhattarai \[ [view email](https://arxiv.org/show-email/6e8adf70/2505.05666)\]

**\[v1\]**
Thu, 8 May 2025 21:54:02 UTC (2,160 KB)

Full-text links:

- [View PDF](https://arxiv.org/pdf/2505.05666)
- [HTML (experimental)](https://arxiv.org/html/2505.05666v1)
- [TeX Source](https://arxiv.org/src/2505.05666)
- [Other Formats](https://arxiv.org/format/2505.05666)

[view license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/ "Rights to this article")

### References & Citations

- [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:2505.05666)
- [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=2505.05666)
- [Semantic Scholar](https://api.semanticscholar.org/arXiv:2505.05666)

</details>

<details>
<summary>Best practices design patterns: optimizing Amazon S3 performance</summary>

# Best practices design patterns: optimizing Amazon S3 performance

[PDF](https://docs.aws.amazon.com/pdfs/AmazonS3/latest/userguide/s3-userguide.pdf#optimizing-performance)  
[RSS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-userguide-rss-updates.rss)

Your applications can easily achieve thousands of transactions per second in request performance when uploading and retrieving storage from Amazon S3. Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by using parallelization. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second. Similarly, you can scale write operations by writing to multiple prefixes. The scaling, in the case of both read and write operations, happens gradually and is not instantaneous, and actual performance will vary based on your specific workload characteristics, usage patterns, and system configuration.

While Amazon S3 is scaling to your new higher request rate, you may see some 503 (Slow Down) errors. These errors will dissipate when the scaling is complete. For more information about creating and using prefixes, see [Organizing objects using prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html).

Some data lake applications on Amazon S3 scan millions or billions of objects for queries that run over petabytes of data. These data lake applications achieve single-instance transfer rates that maximize the network interface use for their [Amazon EC2](https://docs.aws.amazon.com/ec2/index.html) instance, which can be up to 100 Gb/s on a single instance. These applications then aggregate throughput across multiple instances to get multiple terabits per second.

Other applications are sensitive to latency, such as social media messaging applications. These applications can achieve consistent small object latencies (and first-byte-out latencies for larger objects) of roughly 100–200 milliseconds.

Other AWS services can also help accelerate performance for different application architectures. For example, if you want higher transfer rates over a single HTTP connection or single-digit millisecond latencies, use [Amazon CloudFront](https://docs.aws.amazon.com/cloudfront/index.html) or [Amazon ElastiCache](https://docs.aws.amazon.com/elasticache/index.html) for caching with Amazon S3.

Additionally, if you want fast data transport over long distances between a client and an S3 bucket, use [Configuring fast, secure file transfers using Amazon S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html). Transfer Acceleration uses the globally distributed edge locations in CloudFront to accelerate data transport over geographical distances. If your Amazon S3 workload uses server-side encryption with AWS KMS, see [AWS KMS Limits](https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) in the AWS Key Management Service Developer Guide for information about the request rates supported for your use case.

The following topics describe best practice guidelines and design patterns for optimizing performance for applications that use Amazon S3. Refer to the [Performance guidelines for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html) and [Performance design patterns for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html) for the most current information about performance optimization for Amazon S3.

Note: For more information about using the Amazon S3 Express One Zone storage class with directory buckets, see [S3 Express One Zone](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-bucket-high-performance.html#s3-express-one-zone) and [Working with directory buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html).

Topics
- [Performance guidelines for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html)
- [Performance design patterns for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html)

Related documentation
- [Document Conventions](https://docs.aws.amazon.com/general/latest/gr/docconventions.html)
- [Working with the Object ACL field](https://docs.aws.amazon.com/AmazonS3/latest/userguide/objectacl.html)
- [Performance guidelines for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html)

Next topic: [Performance guidelines for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html)  
Previous topic: [Working with the Object ACL field](https://docs.aws.amazon.com/AmazonS3/latest/userguide/objectacl.html)

</details>

<details>
<summary>HTML conversions [sometimes display errors](https://info.dev.arxiv.org/about/accessibility_html_error_messages.html) due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.</summary>

HTML conversions [sometimes display errors](https://info.dev.arxiv.org/about/accessibility_html_error_messages.html) due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.

- failed: inconsolata
- failed: fvextra

Authors: achieve the best HTML results from your LaTeX submissions by following these [best practices](https://info.arxiv.org/help/submit_latex_best_practices.html).

[License: CC Zero](https://info.arxiv.org/help/license/index.html#licenses-available)

arXiv:2407.01449v6 \[cs.IR\] 28 Feb 2025

# ColPali: Efficient Document Retrieval with    Vision Language Models

Report issue for preceding element

Manuel Faysse 1,3 Hugues Sibille∗1,4  Tony Wu∗1  Bilel Omrani1

Gautier Viaud1Céline Hudelot3Pierre Colombo2,3

1Illuin Technology  2Equall.ai  3CentraleSupélec, Paris-Saclay  4ETH Zürich

[manuel.faysse@centralesupelec.fr](https://arxiv.org/html/2407.01449v6/manuel.faysse@centralesupelec.fr "")Equal Contribution

Report issue for preceding element

###### Abstract

Report issue for preceding element

Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG).
To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings.
The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable.
We release models, data, code and benchmarks under open licenses at [https://hf.co/vidore](https://hf.co/vidore "").

Report issue for preceding element

## 1 Introduction

Report issue for preceding element

Document Retrieval consists of matching a user query to relevant documents in a given corpus. It is central to many widespread industrial applications, either as a standalone ranking system (search engines) or as part of more complex information extraction or Retrieval Augmented Generation (RAG) pipelines.

Report issue for preceding element

Over recent years, pretrained language models have enabled large improvements in text embedding models. In practical industrial settings, however, the primary performance bottleneck for efficient document retrieval stems not from embedding model performance but from the prior data ingestion pipeline. Indexing a standard PDF document involves several steps. First, PDF parsers or Optical Character Recognition (OCR) systems are used to extract words from the pages. Document layout detection models can then be run to segment paragraphs, titles, and other page objects such as tables, figures, and headers. A chunking strategy is then defined to group text passages with some semantical coherence, and modern retrieval setups may even integrate a captioning step to describe visually rich elements in a natural language form, more suitable for embedding models.
In our experiments ( [Table 2](https://arxiv.org/html/2407.01449v6#S5.T2 "Table 2 ‣ 5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models")), we typically find that optimizing the ingestion pipeline yields much better performance on visually rich document retrieval than optimizing the text embedding model.

Report issue for preceding element

Contribution 1: ViDoRe.
In this work, we argue that document retrieval systems should not be evaluated solely on the capabilities of text embedding models (Bajaj et al., [2016](https://arxiv.org/html/2407.01449v6#bib.bib6 ""); Thakur et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib50 ""); Muennighoff et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib40 "")), but should also consider the context and visual elements of the documents to be retrieved. To this end, we create and openly release ViDoRe, a comprehensive benchmark to evaluate systems on page-level document retrieval with a wide coverage of domains, visual elements, and languages. ViDoRe addresses practical document retrieval scenarios, where queries often necessitate both textual and visual understanding for accurate document matching. We highlight the shortcomings of current text-centric systems in these settings.111The ViDoRe benchmark leaderboard is hosted publicly at [https://huggingface.co/spaces/vidore/vidore-leaderboard](https://huggingface.co/spaces/vidore/vidore-leaderboard "") to encourage further developments.

Report issue for preceding element

Contribution 2: ColPali.
We propose a novel concept and model architecture based on Vision Language Models (VLMs) to efficiently index documents purely from their visual features, allowing for subsequent fast query matching with late interaction mechanisms (Khattab & Zaharia, [2020](https://arxiv.org/html/2407.01449v6#bib.bib26 "")). Our method, ColPali, significantly outperforms all other retrieval systems on ViDoRe while being fast and end-to-end trainable.
These results demonstrate the potential and the many benefits of this novel Retrieval in Vision Space concept, which could significantly alter the way document retrieval is approached in the industry moving forward.
We release all resources at [https://hf.co/vidore](https://hf.co/vidore "").

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/final_architecture.png)Figure 1: ColPali simplifies document retrieval w.r.t. standard retrieval methods while achieving stronger performances with better latencies. Latencies and results are detailed in [section 5](https://arxiv.org/html/2407.01449v6#S5 "5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models") and [subsection B.4](https://arxiv.org/html/2407.01449v6#A2.SS4 "B.4 Latency computations ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models").Report issue for preceding element

## 2 Problem Formulation & Related Work

Report issue for preceding element

Problem Setting.
In our setting, a retrieval system scores how relevant a document d𝑑ditalic\_d from corpus 𝒟𝒟\\mathcal{D}caligraphic\_D is with respect to a query q𝑞qitalic\_q. Computing the similarity score s⁢(q,d)∈ℝ𝑠𝑞𝑑ℝs(q,d)\\in\\mathbb{R}italic\_s ( italic\_q , italic\_d ) ∈ blackboard\_R for each of the \|𝒟\|𝒟\|\\mathcal{D}\|\| caligraphic\_D \| documents in the corpus creates a ranking we can use to extract the most relevant documents. In this work, we focus on page-level retrieval: given a query, is the correct document page retrieved by the system? For coherence with existing literature, we further use the term document to refer to individual pages, i.e. the atomic retrieved elements in our setting. As we focus on practical industrial retrieval applications (RAG, search engines) with potentially large corpora sizes, latency constraints are imposed on scoring systems. Most current retrieval systems can be decomposed into (1) an offline indexation phase in which a document index is built and (2) an online querying phase in which a query is matched to documents from the index and where low latency is vital to the user experience.

Report issue for preceding element

Under these industrial constraints, we identify three main properties an efficient document retrieval systems should exhibit: (R1) strong retrieval performance, as measured by standard retrieval metrics; (R2) fast online querying, measured through average latencies; (R3) high throughput corpus indexation, ie. the number of pages that can be embedded in a given timeframe.

Report issue for preceding element

### 2.1 Textual Retrieval Methods

Report issue for preceding element

Document Retrieval in Text Space.

Report issue for preceding element

Statistical methods based on word frequency like TF-IDF (Sparck Jones, [1972](https://arxiv.org/html/2407.01449v6#bib.bib48 "")) and BM25 (Robertson et al., [1994](https://arxiv.org/html/2407.01449v6#bib.bib45 "")) are still widely used due to their simplicity and efficiency. More recently, neural embedding models based on fine-tuned large language models display state-of-the-art performance on a variety of text embedding tasks and top the retrieval leaderboards (Muennighoff et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib40 "")).

Report issue for preceding element

Neural Retrievers.
In bi-encoder models (Reimers & Gurevych, [2019](https://arxiv.org/html/2407.01449v6#bib.bib44 ""); Karpukhin et al., [2020](https://arxiv.org/html/2407.01449v6#bib.bib25 ""); Wang et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib52 "")), documents are independently mapped offline to a dense vector space. Queries are embedded online and matched to documents through a fast cosine distance computation.
A slower, but slightly more performant alternative, cross-encoder systems (Wang et al., [2020](https://arxiv.org/html/2407.01449v6#bib.bib55 ""); Cohere, [2024](https://arxiv.org/html/2407.01449v6#bib.bib13 "")) concatenate query and document as a single input sequence and iteratively attribute matching scores to each possible combination. This enables full attention computation between query and document terms but comes at the cost of computational efficiency, as \|𝒟\|𝒟\|\\mathcal{D}\|\| caligraphic\_D \| encoding passes must be done online.

Report issue for preceding element

Multi-Vector retrieval via late interaction.
In the late interaction paradigm introduced by ColBERT (Khattab & Zaharia, [2020](https://arxiv.org/html/2407.01449v6#bib.bib26 "")), an embedding is pre-computed and indexed per document token. At runtime, similarity can be computed with individual query token embeddings. The idea is to benefit from the rich interaction between individual query and document terms while taking advantage of the offline computation and fast query matching enabled by bi-encoders. See [Appendix E](https://arxiv.org/html/2407.01449v6#A5.SSx3 "ColBERT ‣ Appendix E Model glossary ‣ ColPali: Efficient Document Retrieval with Vision Language Models") for more details.

Report issue for preceding element

Retrieval Evaluation.
Although benchmarks and leaderboards have been developed to evaluate text embedding models (Thakur et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib50 ""); Muennighoff et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib40 "")), much of the performance improvements in industrial use cases of embedding models stem from the prior data ingestion pipeline. While documents often rely on visual elements to more efficiently convey information to human readers, text-only systems barely tap into these visual cues. Other work has also independently studied table or chart retrieval systems through repurposed Question Answering datasets (Zhang et al., [2019](https://arxiv.org/html/2407.01449v6#bib.bib59 ""); Nowak et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib42 "")) but only assessing specialized methods for each task.

Report issue for preceding element

To our knowledge, no benchmark evaluates document retrieval systems in practical settings; in an end-to-end manner, across several document types and topics, and by evaluating the use of both textual and visual document features.

Report issue for preceding element

### 2.2 Integrating Visual features

Report issue for preceding element

Contrastive Vision Language Models.
Mapping latent representations of textual content to corresponding representations of visual content has been done by aligning disjoint visual and text encoders through contrastive losses (Radford et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib43 ""); Zhai et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib58 "")). While some OCR capabilities exist in these models, the visual component is often not optimized for text understanding.

Report issue for preceding element

The Fine-grained Interactive Language-Image Pre-training (Yao et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib56 "")) framework extends the late interaction mechanism to cross-modal Vision Language Models, relying on max similarity operations between text tokens and image patches.

Report issue for preceding element

Visually Rich Document Understanding.
To go beyond text, some document-focused models jointly encode text tokens alongside visual or document layout features (Appalaraju et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib4 ""); Kim et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib27 ""); Huang et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib22 ""); Tang et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib49 "")).
Large Language transformer Models (LLMs) with strong reasoning capabilities have recently been combined with Vision Transformers (ViTs) (Dosovitskiy et al., [2020](https://arxiv.org/html/2407.01449v6#bib.bib17 "")) to create VLMs (Alayrac et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib2 ""); Liu et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib35 ""); Bai et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib5 ""); Laurençon et al., [2024b](https://arxiv.org/html/2407.01449v6#bib.bib30 "")) where image patch vectors from contrastively trained ViT models (Zhai et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib58 "")) are fed as input embeddings to the LLM and concatenated with the text-token embeddings.

Report issue for preceding element

PaliGemma.
The PaliGemma-3B model (Beyer et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib7 "")) extends concepts from Pali3 (Chen et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib11 "")), and projects SigLIP-So400m/14(Alabdulmohsin et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib1 "")) patch embeddings into Gemma-2B’s text vector space (Gemma Team et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib19 "")). Along with its reasonable size w.r.t. other performant VLMs, an interesting property of PaliGemma’s text model is that it is fine-tuned with full-block attention on the prefix (instruction text and image tokens). See [Appendix E](https://arxiv.org/html/2407.01449v6#A5 "Appendix E Model glossary ‣ ColPali: Efficient Document Retrieval with Vision Language Models") for more details.

Report issue for preceding element

VLMs display enhanced capabilities in Visual Question Answering, captioning, and document understanding (Yue et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib57 "")), but are not optimized for retrieval tasks.

Report issue for preceding element

## 3 The ViDoRe Benchmark

Report issue for preceding element

Existing benchmarks for contrastive vision-language models primarily evaluate retrieval for natural images (Lin et al., [2014](https://arxiv.org/html/2407.01449v6#bib.bib34 ""); Borchmann et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib9 ""); Thapliyal et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib51 "")). On the other hand, textual retrieval benchmarks (Muennighoff et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib40 "")) are evaluated at at textual passage level and are not tailored for document retrieval tasks. We fill the gap with ViDoRe, a comprehensive benchmark for document retrieval using visual features.

Report issue for preceding element

### 3.1 Benchmark Design

Report issue for preceding element

ViDoRe is designed to comprehensively evaluate retrieval systems on their capacity to match queries to relevant documents at the page level. This benchmark encompasses multiple orthogonal subtasks, with focuses on various modalities - text, figures, infographics, tables; thematic domains - medical, business, scientific, administrative; or languages - English, French. Tasks also span varying levels of complexity, in order to capture signals from both weaker and stronger systems.
As many systems require large amounts of time to index pages (captioning-based approaches can take dozens of seconds per page for instance), we limit the number of candidate documents for each retrieval task in order to evaluate even complex systems in a reasonable timeframe without sacrificing quality. For trainable retrieval systems, we provide a reference training set that can be used to facilitate comparisons.

Report issue for preceding element

| Dataset | Language | \# Queries | \# Documents | Description |
| --- | --- | --- | --- | --- |
| Academic Tasks |  |  |  |  |
| DocVQA | English | 500 | 500 | Scanned documents from UCSF Industry |
| InfoVQA | English | 500 | 500 | Infographics scrapped from the web |
| TAT-DQA | English | 1600 | 1600 | High-quality financial reports |
| arXiVQA | English | 500 | 500 | Scientific Figures from arXiv |
| TabFQuAD | French | 210 | 210 | Tables scrapped from the web |
| Practical Tasks |  |  |  |  |
| Energy | English | 100 | 1000 | Documents about energy |
| Government | English | 100 | 1000 | Administrative documents |
| Healthcare | English | 100 | 1000 | Medical documents |
| AI | English | 100 | 1000 | Scientific documents related to AI |
| Shift Project | French | 100 | 1000 | Environmental reports |

Table 1: ViDoRe comprehensively evaluates multimodal retrieval methods.Report issue for preceding element

Academic Tasks.
We repurpose widely used visual question-answering benchmarks for retrieval tasks: for each page-question-answer triplet, we use the question as the query, and the associated page as the gold document ( [Table 1](https://arxiv.org/html/2407.01449v6#S3.T1 "Table 1 ‣ 3.1 Benchmark Design ‣ 3 The ViDoRe Benchmark ‣ ColPali: Efficient Document Retrieval with Vision Language Models")). These academic datasets either focus on single specific modalities (Mathew et al., [2020](https://arxiv.org/html/2407.01449v6#bib.bib38 ""); [2021](https://arxiv.org/html/2407.01449v6#bib.bib39 ""); Li et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib33 "")) or target more varied visually rich documents (Zhu et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib61 "")). Moreover, we consider TabFQuAD, a human-labeled dataset on tables extracted from French industrial PDF documents released with this work. Details can be found in [subsection A.1](https://arxiv.org/html/2407.01449v6#A1.SS1 "A.1 Academic Datasets ‣ Appendix A Benchmark Datasets ‣ ColPali: Efficient Document Retrieval with Vision Language Models").

Report issue for preceding element

Practical tasks.
We construct topic-specific retrieval benchmarks spanning multiple domains to go beyond repurposed QA datasets and evaluate retrieval in more realistic industrial situations (e.g. RAG). To achieve this, we collect publicly accessible PDF documents and generate queries pertaining to document pages using Claude-3 Sonnet, a high-quality proprietary vision-language model (Anthropic, [2024](https://arxiv.org/html/2407.01449v6#bib.bib3 "")). In total, we collect 1,000 document pages per topic, which we associate with 100 queries extensively filtered for quality and relevance by human annotators. The corpus topics are intentionally specific to maximize syntactic proximity between documents, creating more challenging retrieval tasks and covering an array of orthogonal domains ( [Table 1](https://arxiv.org/html/2407.01449v6#S3.T1 "Table 1 ‣ 3.1 Benchmark Design ‣ 3 The ViDoRe Benchmark ‣ ColPali: Efficient Document Retrieval with Vision Language Models")).
222Answers are generated alongside queries to (1) ground queries and improve their quality and (2) provide resources to foster future work.

Report issue for preceding element

Evaluation Metrics.
We evaluate performance on our benchmark (Requirement R1) using standard metrics from the retrieval literature (nDCG, Recall@K, MRR). We report nDCG@5 values as the main performance metric in this work and release the complete sets of results along with the models333 [https://huggingface.co/vidore](https://huggingface.co/vidore ""). To validate compliance with practical industrial requirements ( [section 2](https://arxiv.org/html/2407.01449v6#S2 "2 Problem Formulation & Related Work ‣ ColPali: Efficient Document Retrieval with Vision Language Models")), we also consider query latencies (R2) and indexing throughputs (R3).

Report issue for preceding element

### 3.2 Assessing Current Systems

Report issue for preceding element

Unstructured. We evaluate retrieval systems representative of those found in standard industrial RAG pipelines. As is common practice, we rely on the Unstructured444 [www.unstructured.io](https://arxiv.org/html/2407.01449v6/www.unstructured.io "") off-the-shelf tool in the highest resolution settings to construct high-quality text chunks from PDF documents. Unstructured orchestrates the document parsing pipeline, relying on deep learning vision models to detect titles and document layouts (Ge et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib18 "")), OCR engines (Smith, [2007](https://arxiv.org/html/2407.01449v6#bib.bib47 "")) to extract text in non-native PDFs, specialized methods or models to detect and reconstruct tables, and implements a chunking strategy (by-title) that leverages the detected document structure to preserve section boundaries when concatenating texts. As is common practice, in our simplest Unstructured configuration (text-only), only textual elements are kept and figures, images, and tables are considered noisy information and are filtered out.

Report issue for preceding element

Unstructured + X. While Unstructured is a strong baseline by itself, we further augment Unstructured’s output by integrating the visual elements. In (\+ OCR), tables, charts, and images are run through an OCR engine, processed by Unstructured, and chunked independently. In (\+ Captioning), we set up a fully-fledged captioning strategy (Zhao et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib60 "")), in which we feed visual elements to a strong proprietary Vision Language Model (Claude-3 Sonnet (Anthropic, [2024](https://arxiv.org/html/2407.01449v6#bib.bib3 ""))) to obtain highly detailed textual descriptions of the elements.
Both strategies aim to integrate visual elements in the retrieval pipeline but incur significant latency and resource costs ( [subsection 5.2](https://arxiv.org/html/2407.01449v6#S5.SS2 "5.2 Latencies & Memory Footprint ‣ 5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models")).

Report issue for preceding element

Embedding Model. To embed textual chunks, we evaluate Okapi BM25, the de facto standard sparse statistical retrieval method, and the dense encoder of BGE-M3 (Chen et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib10 "")), a multilingual neural method with SOTA performance in its size category. Chunks are embedded and scored independently, and page-level scores are obtained by max-pooling over the page’s chunk scores.555We empirically validated the max-pooling strategy over sub-page chunks to be more effective than concatenating all page chunks before embedding pagewise.

Report issue for preceding element

Contrastive VLMs. We also evaluate the strongest available vision-language embedding models; Jina CLIP (Koukounas et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib28 "")), Nomic Embed Vision (Nomic, [2024](https://arxiv.org/html/2407.01449v6#bib.bib41 "")), and SigLIP-So400m/14(Alabdulmohsin et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib1 "")).

Report issue for preceding element

Results. From a performance perspective, best results are obtained by combining the Unstructured parser with visual information, either from captioning strategies or by running OCR on the visual elements ( [Table 2](https://arxiv.org/html/2407.01449v6#S5.T2 "Table 2 ‣ 5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models")). Little difference is seen between BM25 and BGE-M3 embeddings highlighting the visual information bottleneck. Contrastive VLMs lag behind. Beyond retrieval performance (R1), the indexing latencies (R2) reported in [Figure 2](https://arxiv.org/html/2407.01449v6#S3.F2 "Figure 2 ‣ 3.2 Assessing Current Systems ‣ 3 The ViDoRe Benchmark ‣ ColPali: Efficient Document Retrieval with Vision Language Models") illustrate that PDF parsing pipelines can be very lengthy, especially when incorporating OCR or captioning strategies. Querying latencies at runtime (R3) are very good for all evaluated systems (≤22absent22\\leq 22≤ 22 ms on a NVIDIA L4) due to fast query encoding and cosine similarity matching.

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2407.01449v6/x1.png)

Figure 2: Offline document indexing with ColPali is much simpler and faster compared to standard retrieval methods. The PDF Parser results are obtained following the Unstructured settings with BGE-M3 detailed in [subsection 3.2](https://arxiv.org/html/2407.01449v6#S3.SS2 "3.2 Assessing Current Systems ‣ 3 The ViDoRe Benchmark ‣ ColPali: Efficient Document Retrieval with Vision Language Models"). All indexing speeds are averaged per-page latencies. More details in [subsection B.4](https://arxiv.org/html/2407.01449v6#A2.SS4 "B.4 Latency computations ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models")

.

Report issue for preceding element

Report issue for preceding element

## 4 Late interaction based Vision Retrieval

Report issue for preceding element

### 4.1 Architecture

Report issue for preceding element

Vision-Language Models.
Encouraged by their strong document understanding capabilities, we propose adapting recent VLMs for retrieval. The key concept is to leverage the alignment between output embeddings of text and image tokens acquired during multi-modal fine-tuning.
To this extent, we introduce ColPali, a Paligemma-3B extension that is capable of generating ColBERT-style multi-vector representations of text and images ( [Figure 1](https://arxiv.org/html/2407.01449v6#S1.F1 "Figure 1 ‣ 1 Introduction ‣ ColPali: Efficient Document Retrieval with Vision Language Models")).
PaliGemma-3B is a strong candidate due to its small size, the many released checkpoints fine-tuned for different image resolutions and tasks, and the promising performances on various document understanding benchmarks.
We add a projection layer to map each of the language model’s output token embeddings (whether from text or image tokens) to a vector space of reduced dimension D=128𝐷128D=128italic\_D = 128 as used in the ColBERT paper (Khattab & Zaharia, [2020](https://arxiv.org/html/2407.01449v6#bib.bib26 "")) to keep lightweight bag-of-embedding representations.

Report issue for preceding element

Late Interaction.
Given query q𝑞qitalic\_q and document d𝑑ditalic\_d, we denote as 𝐄𝐪∈ℝNq×Dsubscript𝐄𝐪superscriptℝsubscript𝑁𝑞𝐷\\mathbf{E\_{q}}\\in\\mathbb{R}^{{N\_{q}}\\times D}bold\_E start\_POSTSUBSCRIPT bold\_q end\_POSTSUBSCRIPT ∈ blackboard\_R start\_POSTSUPERSCRIPT italic\_N start\_POSTSUBSCRIPT italic\_q end\_POSTSUBSCRIPT × italic\_D end\_POSTSUPERSCRIPT and 𝐄𝐝∈ℝNd×Dsubscript𝐄𝐝superscriptℝsubscript𝑁𝑑𝐷\\mathbf{E\_{d}}\\in\\mathbb{R}^{N\_{d}\\times D}bold\_E start\_POSTSUBSCRIPT bold\_d end\_POSTSUBSCRIPT ∈ blackboard\_R start\_POSTSUPERSCRIPT italic\_N start\_POSTSUBSCRIPT italic\_d end\_POSTSUBSCRIPT × italic\_D end\_POSTSUPERSCRIPT their respective multi-vector representation in the common embedding space ℝDsuperscriptℝ𝐷\\mathbb{R}^{D}blackboard\_R start\_POSTSUPERSCRIPT italic\_D end\_POSTSUPERSCRIPT, where Nqsubscript𝑁𝑞N\_{q}italic\_N start\_POSTSUBSCRIPT italic\_q end\_POSTSUBSCRIPT and Ndsubscript𝑁𝑑N\_{d}italic\_N start\_POSTSUBSCRIPT italic\_d end\_POSTSUBSCRIPT are respectively the number of vectors in the query and in the document page embeddings. The late interaction operator, LI⁢(q,d)LI𝑞𝑑\\text{LI}\\left(q,d\\right)LI ( italic\_q , italic\_d ), is the sum over all query vectors 𝐄𝐪(j)superscriptsubscript𝐄𝐪𝑗\\mathbf{E\_{q}}^{(j)}bold\_E start\_POSTSUBSCRIPT bold\_q end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT ( italic\_j ) end\_POSTSUPERSCRIPT, of its maximum dot product ⟨⋅\|⋅⟩\\langle\\cdot\|\\cdot\\rangle⟨ ⋅ \| ⋅ ⟩ with each of the Ndsubscript𝑁𝑑N\_{d}italic\_N start\_POSTSUBSCRIPT italic\_d end\_POSTSUBSCRIPT document embedding vectors 𝐄𝐝(1:Nd)subscriptsubscript𝐄𝐝:1subscript𝑁𝑑\\mathbf{E\_{d}}\_{(1:N\_{d})}bold\_E start\_POSTSUBSCRIPT bold\_d end\_POSTSUBSCRIPT start\_POSTSUBSCRIPT ( 1 : italic\_N start\_POSTSUBSCRIPT italic\_d end\_POSTSUBSCRIPT ) end\_POSTSUBSCRIPT.

Report issue for preceding element

|     |     |     |     |
| --- | --- | --- | --- |
|  | LI⁢(q,d)=∑i∈\[\|1,Nq\|\]maxj∈\[\|1,Nd\|\]⁢⟨𝐄𝐪(i)\|𝐄𝐝(j)⟩\\text{LI}\\left(q,d\\right)=\\sum\_{i\\in\[\|1,N\_{q}\|\]}\\max\_{j\\in\[\|1,N\_{d}\|\]}\\langle%<br>\\mathbf{E\_{q}}^{(i)}\|\\mathbf{E\_{d}}^{(j)}\\rangleLI ( italic\_q , italic\_d ) = ∑ start\_POSTSUBSCRIPT italic\_i ∈ \[ \| 1 , italic\_N start\_POSTSUBSCRIPT italic\_q end\_POSTSUBSCRIPT \| \] end\_POSTSUBSCRIPT roman\_max start\_POSTSUBSCRIPT italic\_j ∈ \[ \| 1 , italic\_N start\_POSTSUBSCRIPT italic\_d end\_POSTSUBSCRIPT \| \] end\_POSTSUBSCRIPT ⟨ bold\_E start\_POSTSUBSCRIPT bold\_q end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT ( italic\_i ) end\_POSTSUPERSCRIPT \| bold\_E start\_POSTSUBSCRIPT bold\_d end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT ( italic\_j ) end\_POSTSUPERSCRIPT ⟩ |  | (1) |

Contrastive Loss.
The Late Interaction operation is fully differentiable, enabling backpropagation.
Let a batch {qk,dk}k∈\[\|1,b\|\]\\left\\{q\_{k},d\_{k}\\right\\}\_{k\\in\[\|1,b\|\]}{ italic\_q start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT , italic\_d start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT } start\_POSTSUBSCRIPT italic\_k ∈ \[ \| 1 , italic\_b \| \] end\_POSTSUBSCRIPT composed of b𝑏bitalic\_b query-page pairs, where for all k∈\[\|1,b\|\]k\\in\[\|1,b\|\]italic\_k ∈ \[ \| 1 , italic\_b \| \], the document page dksubscript𝑑𝑘d\_{k}italic\_d start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT is the document corresponding to query qksubscript𝑞𝑘q\_{k}italic\_q start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT.
Following Khattab & Zaharia ( [2020](https://arxiv.org/html/2407.01449v6#bib.bib26 "")), we define our in-batch contrastive loss ℒℒ\\mathcal{L}caligraphic\_L as the softmaxed cross-entropy of the positive scores sk+=LI⁢(qk,dk)superscriptsubscript𝑠𝑘LIsubscript𝑞𝑘subscript𝑑𝑘s\_{k}^{+}=\\text{LI}\\left(q\_{k},d\_{k}\\right)italic\_s start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT + end\_POSTSUPERSCRIPT = LI ( italic\_q start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT , italic\_d start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT ) w.r.t. to the maximal in-batch negative scores sk−=maxl,l≠k⁡LI⁢(qk,dl)superscriptsubscript𝑠𝑘subscript𝑙𝑙𝑘LIsubscript𝑞𝑘subscript𝑑𝑙s\_{k}^{-}=\\max\\limits\_{l,l\\neq k}\\hskip 8.53581pt\\text{LI}\\left(q\_{k},d\_{l}\\right)italic\_s start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT - end\_POSTSUPERSCRIPT = roman\_max start\_POSTSUBSCRIPT italic\_l , italic\_l ≠ italic\_k end\_POSTSUBSCRIPT LI ( italic\_q start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT , italic\_d start\_POSTSUBSCRIPT italic\_l end\_POSTSUBSCRIPT )666We reformulate the loss to leverage the numerically stable softplus function where softplus(x)=log⁡(1+exp⁡(x))softplus(x)1𝑥\\texttt{softplus(x)}=\\log\\left(1+\\exp\\left(x\\right)\\right)softplus(x) = roman\_log ( 1 + roman\_exp ( italic\_x ) ):

Report issue for preceding element

|     |     |     |     |
| --- | --- | --- | --- |
|  | ℒ=−1b⁢∑k=1blog⁡\[exp⁡(sk+)exp⁡(sk+)+exp⁡(sk−)\]=1b⁢∑k=1blog⁡(1+exp⁡(sk−−sk+))ℒ1𝑏superscriptsubscript𝑘1𝑏superscriptsubscript𝑠𝑘superscriptsubscript𝑠𝑘superscriptsubscript𝑠𝑘1𝑏superscriptsubscript𝑘1𝑏1superscriptsubscript𝑠𝑘superscriptsubscript𝑠𝑘\\mathcal{L}=-\\frac{1}{b}\\sum\_{k=1}^{b}\\log\\left\[\\frac{\\exp\\left(s\_{k}^{+}%<br>\\right)}{\\exp\\left(s\_{k}^{+}\\right)+\\exp\\left(s\_{k}^{-}\\right)}\\right\]=\\frac{1%<br>}{b}\\sum\_{k=1}^{b}\\log\\left(1+\\exp\\left(s\_{k}^{-}-s\_{k}^{+}\\right)\\right)caligraphic\_L = - divide start\_ARG 1 end\_ARG start\_ARG italic\_b end\_ARG ∑ start\_POSTSUBSCRIPT italic\_k = 1 end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT italic\_b end\_POSTSUPERSCRIPT roman\_log \[ divide start\_ARG roman\_exp ( italic\_s start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT + end\_POSTSUPERSCRIPT ) end\_ARG start\_ARG roman\_exp ( italic\_s start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT + end\_POSTSUPERSCRIPT ) + roman\_exp ( italic\_s start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT - end\_POSTSUPERSCRIPT ) end\_ARG \] = divide start\_ARG 1 end\_ARG start\_ARG italic\_b end\_ARG ∑ start\_POSTSUBSCRIPT italic\_k = 1 end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT italic\_b end\_POSTSUPERSCRIPT roman\_log ( 1 + roman\_exp ( italic\_s start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT - end\_POSTSUPERSCRIPT - italic\_s start\_POSTSUBSCRIPT italic\_k end\_POSTSUBSCRIPT start\_POSTSUPERSCRIPT + end\_POSTSUPERSCRIPT ) ) |  | (2) |

### 4.2 Model training

Report issue for preceding element

Dataset. Our training dataset of 118,695 query-page pairs is comprised of train sets of openly available academic datasets (63%percent6363\\%63 %) and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions (37%percent3737\\%37 %). Dataset split details are given in [subsection A.3](https://arxiv.org/html/2407.01449v6#A1.SS3 "A.3 Training Dataset ‣ Appendix A Benchmark Datasets ‣ ColPali: Efficient Document Retrieval with Vision Language Models").
Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages777Multilingual data is present in the pretraining corpus of the language model (Gemma-2B) and potentially occurs during PaliGemma-3B’s multimodal training.. We explicitly verify no multi-page PDF document is used both ViDoRe and in the train set to prevent evaluation contamination. A validation set is created with 2%percent22\\%2 % of the samples to tune hyperparameters. We openly release the training dataset888 [https://huggingface.co/datasets/vidore/colpali\_train\_set](https://huggingface.co/datasets/vidore/colpali_train_set "") for reproducibility and to encourage further research.

Report issue for preceding element

Parameters. All models are trained for 1 epoch on the train set. Unless specified otherwise, we train models in bfloat16 format, use low-rank adapters (LoRA, Hu et al. ( [2021](https://arxiv.org/html/2407.01449v6#bib.bib21 ""))) with α=32𝛼32\\alpha=32italic\_α = 32 and r=32𝑟32r=32italic\_r = 32 on the transformer layers from the language model, as well as the final randomly initialized projection layer, and use a paged\_adamw\_8bit optimizer. We train on an 8 GPU setup with data parallelism, a learning rate of 5⁢e−55𝑒55e-55 italic\_e - 5 with linear decay with 2.5% warmup steps, and a batch size of 32.

Report issue for preceding element

Query Augmentation. As in Khattab & Zaharia ( [2020](https://arxiv.org/html/2407.01449v6#bib.bib26 "")), we append 5 <unused0> tokens to the query tokens to serve as a soft, differentiable query expansion or re-weighting mechanism.

Report issue for preceding element

## 5 Results

Report issue for preceding element

|     |     |     |     |     |     |     |     |     |     |     |     |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ArxivQ | DocQ | InfoQ | TabF | TATQ | Shift | AI | Energy | Gov. | Health. | Avg. |
| Unstructured text-only |  |  |  |  |  |  |  |  |  |  |  |
| \- BM25 | - | 34.1 | - | - | 44.0 | 59.6 | 90.4 | 78.3 | 78.8 | 82.6 | - |
| \- BGE-M3 | - | 28.4↓↓\\downarrow↓5.7 | - | - | 36.1↓↓\\downarrow↓7.9 | 68.5↑↑\\uparrow↑8.9 | 88.4↓↓\\downarrow↓2.0 | 76.8↓↓\\downarrow↓1.5 | 77.7↓↓\\downarrow↓1.1 | 84.6↑↑\\uparrow↑2.0 | - |
| Unstructured \+ OCR |  |  |  |  |  |  |  |  |  |  |  |
| \- BM25 | 31.6 | 36.8 | 62.9 | 46.5 | 62.7 | 64.3 | 92.8 | 85.9 | 83.9 | 87.2 | 65.5 |
| \- BGE-M3 | 31.4↓↓\\downarrow↓0.2 | 25.7↓↓\\downarrow↓11.1 | 60.1↓↓\\downarrow↓2.8 | 70.8↑↑\\uparrow↑24.3 | 50.5↓↓\\downarrow↓12.2 | 73.2↑↑\\uparrow↑8.9 | 90.2↓↓\\downarrow↓2.6 | 83.6↓↓\\downarrow↓2.3 | 84.9↑↑\\uparrow↑1.0 | 91.1↑↑\\uparrow↑3.9 | 66.1↑↑\\uparrow↑0.6 |
| Unstructured \+ Captioning |  |  |  |  |  |  |  |  |  |  |  |
| \- BM25 | 40.1 | 38.4 | 70.0 | 35.4 | 61.5 | 60.9 | 88.0 | 84.7 | 82.7 | 89.2 | 65.1 |
| \- BGE-M3 | 35.7↓↓\\downarrow↓4.4 | 32.9↓↓\\downarrow↓5.4 | 71.9↑↑\\uparrow↑1.9 | 69.1↑↑\\uparrow↑33.7 | 43.8↓↓\\downarrow↓17.7 | 73.1↑↑\\uparrow↑12.2 | 88.8↑↑\\uparrow↑0.8 | 83.3↓↓\\downarrow↓1.4 | 80.4↓↓\\downarrow↓2.3 | 91.3↑↑\\uparrow↑2.1 | 67.0↑↑\\uparrow↑1.9 |
| Contrastive VLMs |  |  |  |  |  |  |  |  |  |  |  |
| Jina-CLIP | 25.4 | 11.9 | 35.5 | 20.2 | 3.3 | 3.8 | 15.2 | 19.7 | 21.4 | 20.8 | 17.7 |
| Nomic-vision | 17.1 | 10.7 | 30.1 | 16.3 | 2.7 | 1.1 | 12.9 | 10.9 | 11.4 | 15.7 | 12.9 |
| SigLIP (Vanilla) | 43.2 | 30.3 | 64.1 | 58.1 | 26.2 | 18.7 | 62.5 | 65.7 | 66.1 | 79.1 | 51.4 |
| Ours |  |  |  |  |  |  |  |  |  |  |  |
| SigLIP (Vanilla) | 43.2 | 30.3 | 64.1 | 58.1 | 26.2 | 18.7 | 62.5 | 65.7 | 66.1 | 79.1 | 51.4 |
| BiSigLIP (+fine-tuning) | 58.5↑↑\\uparrow↑15.3 | 32.9↑↑\\uparrow↑2.6 | 70.5↑↑\\uparrow↑6.4 | 62.7↑↑\\uparrow↑4.6 | 30.5↑↑\\uparrow↑4.3 | 26.5↑↑\\uparrow↑7.8 | 74.3↑↑\\uparrow↑11.8 | 73.7↑↑\\uparrow↑8.0 | 74.2↑↑\\uparrow↑8.1 | 82.3↑↑\\uparrow↑3.2 | 58.6↑↑\\uparrow↑7.2 |
| BiPali (+LLM) | 56.5↓↓\\downarrow↓-2.0 | 30.0↓↓\\downarrow↓-2.9 | 67.4↓↓\\downarrow↓-3.1 | 76.9↑↑\\uparrow↑14.2 | 33.4↑↑\\uparrow↑2.9 | 43.7↑↑\\uparrow↑17.2 | 71.2↓↓\\downarrow↓-3.1 | 61.9↓↓\\downarrow↓-11.7 | 73.8↓↓\\downarrow↓-0.4 | 73.6↓↓\\downarrow↓-8.8 | 58.8↑↑\\uparrow↑0.2 |
| ColPali (+Late Inter.) | 79.1↑↑\\uparrow↑22.6 | 54.4↑↑\\uparrow↑24.5 | 81.8↑↑\\uparrow↑14.4 | 83.9↑↑\\uparrow↑7.0 | 65.8↑↑\\uparrow↑32.4 | 73.2↑↑\\uparrow↑29.5 | 96.2↑↑\\uparrow↑25.0 | 91.0↑↑\\uparrow↑29.1 | 92.7↑↑\\uparrow↑18.9 | 94.4↑↑\\uparrow↑20.8 | 81.3↑↑\\uparrow↑22.5 |

Table 2: Comprehensive evaluation of baseline models and our proposed method on ViDoRe. Results are presented using nDCG@5 metrics, and illustrate the impact of different components. Text-only metrics are not computed for benchmarks with only visual elements.Report issue for preceding element

### 5.1 Performance (R1)

Report issue for preceding element

We show performance is achieved iteratively through the combination of three factors; (1) a carefully crafted task-specific dataset, (2) pairing a pretrained LLM to a vision model to better leverage text semantics from the image, and (3) using multi-vector embeddings rather than a single vector representation to better capture the vast amount of visual information present in a document.

Report issue for preceding element

Fine-tuning a Vision Model on a document retrieval oriented dataset: BiSigLIP. SigLIP999 [https://huggingface.co/google/siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384 "") is a strong vision-language bi-encoder producing single vector embeddings, and pretrained on billions of image-text pairs from the English split of WebLI (Chen et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib11 "")). Further fine-tuning the textual component of this model on our document-oriented dataset (BiSigLIP) yields clear improvements across the board, particularly on figure retrieval (ArxivQA) and table retrieval tasks (TabFQuAD).

Report issue for preceding element

Feeding image patches to a LLM: BiPali. In the PaliGemma model architecture, SigLIP-generated patch embeddings are fed to a text language model and we can obtain LLM contextualized output patch embeddings.101010Note that the SigLIP model used in PaliGemma slightly differs in terms of number patches - 1024 patches for PaliGemma’s vision encoder, and 729 for the standalone SigLIP model. This technique aligns the image token representations with the text token embeddings in the LLM’s embeddings space, and augments the vision model embeddings with the language model’s text understanding capabilities. We average pool these representations to obtain a single dense vector, effectively creating a PaliGemma bi-encoder model (BiPali). After fine-tuning on the training dataset, we obtain a model that performs slightly worse in English than the tuned BiSigLIP variant.111111This can be explained by the fact that contrary to SigLIP, the original PaliGemma is not trained on contrastive matching tasks, but rather on next token prediction. Our contrastive fine-tuning phase on 119K images to transform PaliGemma into a bi-encoder is 5 orders of magnitude smaller than SigLIP’s original contrastive training. However, we see notable improvements in French tasks, indicating that BiPali’s LLM (Gemma 2B) helps multilingual text understanding. This is particularly notable as our training dataset does not contain non-English samples.

Report issue for preceding element

Leveraging Multi-Vector Embeddings through Late Interaction: ColPali.
One benefit of inputting image patch embeddings through a language model is that they are natively mapped to a latent space similar to the textual input (query). This enables leveraging the ColBERT strategy to construct one embedding per image patch token, and at inference compute all interactions between text tokens and image patches, resulting in a step-change improvement in performance compared to BiPali.
Results in [Table 2](https://arxiv.org/html/2407.01449v6#S5.T2 "Table 2 ‣ 5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models") show that our ColPali model also largely outperforms the strong baselines based on Unstructured and captioning, as well as all evaluated text-image embedding models. The difference is particularly stark on the more visually complex benchmark tasks, such as InfographicVQA, ArxivQA, and TabFQuAD, respectively representing infographics, figures, and tables. However, text-centric documents are also better retrieved by the ColPali models across all evaluated domains and languages, making our approach the overall best-performing document-retrieval model.

Report issue for preceding element

Negative Results. For extensiveness, we also train ColSigLIP, a late interaction variant of the BiSigLIP model but obtain abysmal performances. We attribute this to the large gaps w.r.t. SigLIP’s pre-training, in which only a pooled latent representation is used in the contrastive loss, which does not optimize the representations of individual patch and token embeddings. Similarly, we train a BiSigLIPPaliGemma variant, in which we retrieve the image representations from the SigLIP model that has been further updated by PaliGemma fine-tuning, and use the text representations from PaliGemma’s text model. After fine-tuning on our dataset, performance is severely inferior to SigLIPVanilla which simply encodes with SigLIP’s original text and vision components. This indicates a logical misalignment between SigLIP embeddings, and Gemma embeddings after PaliGemma training. We detail these results in [subsection C.1](https://arxiv.org/html/2407.01449v6#A3.SS1 "C.1 Other Metrics ‣ Appendix C Additional results ‣ ColPali: Efficient Document Retrieval with Vision Language Models").

Report issue for preceding element

### 5.2 Latencies & Memory Footprint

Report issue for preceding element

Online Querying. (R2) Logically, querying latencies differ between ColPali and a BGE-M3 embedding model. For BGE, encoding takes about 22222222 ms for 15 tokens, while encoding a query with ColPali’s language model takes about 30303030 ms121212Computed for a batch size of 1111 (online), and averaged over 1000 queries. See [subsection B.4](https://arxiv.org/html/2407.01449v6#A2.SS4 "B.4 Latency computations ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models"). For smaller corpus sizes, computing the late interaction operation induces marginally small overheads (≈1absent1\\approx 1≈ 1 ms per 1000 pages in the corpus), and the cosine similarity computation between bi-encoder vectors is even faster. Optimized late interaction engines (Santhanam et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib46 ""); Lee et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib32 "")) enable to easily scale corpus sizes to millions of documents with reduced latency degradations.

Report issue for preceding element

Offline Indexing. (R3)
Standard retrieval methods using bi-encoders represent each chunk as a single vector embedding, which is easy to store and fast to compute. However, processing a PDF to get the different chunks is the most time-consuming part (layout detection, OCR, chunking), and using captioning to handle multimodal data will only exacerbate this already lengthy process. On the other hand, ColPali directly encodes pages from their image representation. Although the model is larger than standard retrieval encoders, skipping the preprocessing allows large speedups at indexing131313Measures a NVIDIA L4 GPU, averaged on 100 pages, with a batch size of 4 pages for ColPali and 8 text chunks for Bi-Encoders. On average, a page is divided into 2.1 chunks. See [subsection B.4](https://arxiv.org/html/2407.01449v6#A2.SS4 "B.4 Latency computations ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models"). ( [Figure 2](https://arxiv.org/html/2407.01449v6#S3.F2 "Figure 2 ‣ 3.2 Assessing Current Systems ‣ 3 The ViDoRe Benchmark ‣ ColPali: Efficient Document Retrieval with Vision Language Models")). As pages are embedded end-to-end in single forward pass, the VRAM usage depends exclusively on the sequence length (number of patches per image) which is fixed as well, enabling efficient batching strategies to fully leverage hardware acceleration. ColPali also benefits from most LLM efficiency improvements introduced in the ecosystem such as Flash Attention (Dao, [2023](https://arxiv.org/html/2407.01449v6#bib.bib14 "")).

Report issue for preceding element

Storage Footprint. Our method requires storing a vector per image patch, along with 6 extra text tokens “Describe the image” concatenated to image patches. We project each PaliGemma vector to a lower dimensional space (D=128𝐷128D=128italic\_D = 128) to maximize efficiency, leading to a memory footprint of 257.5257.5257.5257.5 KB per page ( [subsection B.3](https://arxiv.org/html/2407.01449v6#A2.SS3 "B.3 Embedding size ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models")). Importantly, the memory footprint of the naive ColBERT indexing strategy can be drastically improved through compression and clustering mechanisms (Santhanam et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib46 ""); Clavié et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib12 "")).

Report issue for preceding element

Token pooling.
Token pooling (Clavié et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib12 "")) is a CRUDE-compliant method (document addition/deletion-friendly) that aims amountto reduce the amount of multi-vector embeddings. For ColPali, many image patches share redundant information, e.g. white background patches. By pooling these patches together, we can reduce the amount of embeddings while retaining most information. Retrieval performance with hierarchical mean token pooling on image embeddings is shown in [Figure 3](https://arxiv.org/html/2407.01449v6#S5.F3 "Figure 3 ‣ 5.3 Interpretability ‣ 5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models") (left).
With a pool factor of 3, the total number of vectors is reduced by 66.7%percent66.766.7\\%66.7 % while 97.8%percent97.897.8\\%97.8 % of the original performance is maintained. We note that the Shift dataset—composed of the most text-dense documents—is a clear outlier, showcasing more information dense documents contain less redundant patches and may be prone to worse performance degradation with such pooling techniques.

Report issue for preceding element

### 5.3 Interpretability

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2407.01449v6/x2.png)

![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/similarity_map_energy.png)

Figure 3: (Left: Token Pooling) Relative performance degradation when reducing the number of stored embeddings per document. (Right: Interpretability) For each term in a user query, ColPali identifies the most relevant document image patches (highlighted zones) and computes a query-to-page matching score.Report issue for preceding element

By superimposing the late interaction heatmap on top of the original image, we can visualize the most salient image patches with respect to each term of the query, yielding interpretable insights into model focus zones. As epitomized in [Figure 3](https://arxiv.org/html/2407.01449v6#S5.F3 "Figure 3 ‣ 5.3 Interpretability ‣ 5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models") (right), we observe ColPali exhibits strong OCR capabilities as both the words “hourly” and “hours” present a high similarity score with the query token <\_hour>. We also note particular focus on other non-trivial image features such as the x-axis representing hours being salient. Other visualization examples are shown in [Appendix D](https://arxiv.org/html/2407.01449v6#A4 "Appendix D More similarity maps ‣ ColPali: Efficient Document Retrieval with Vision Language Models").

Report issue for preceding element

## 6 Ablation study

Report issue for preceding element

We run various ablations to better understand the mechanisms at play. By default, result deltas reported below refer to nDCG@5 values averaged over all ViDoRe tasks. Detailed results in [C.2](https://arxiv.org/html/2407.01449v6#A3.SS2 "C.2 Model Variants ‣ Appendix C Additional results ‣ ColPali: Efficient Document Retrieval with Vision Language Models").

Report issue for preceding element

Tradeoffs between model size and the number of image patches. We train a variant of PaliGemma with half the number of image patches (512). While we observe a clear performance degradation with respects to to the 1024-patch ColPali model (−24.824.8-24.8\- 24.8 nDCG@5), memory usage is much lower.
As an alternative to PaliGemma, we train Idefics2-8B (Laurençon et al., [2024b](https://arxiv.org/html/2407.01449v6#bib.bib30 "")), a VLM with a similar architecture and based on a Mistral-7B (Jiang et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib23 "")) language backbone and a SigLIP vision encoder paired with a perceiver resampler. The most notable differences with PaliGemma lie in the size of the language model (2B and 7B resp.) and the number of image patches (between 512 and 2048 for PaliGemma, and 64 post-resampling for Idefics2141414With the option of adding 4 sub-image crops of 64 tokens each to the sequence, for a total of 320 tokens). Our results suggest better language models enable more efficient representations of image embeddings - ColIdefics2 with 64 patches largely outperforms out ColPali with 512 patches (+20.1 nDCG@5). However ColIdefics2 (64) remains less accurate than ColPali (1024) (−4.74.7-4.7\- 4.7 nDCG@5) while being about twice as slow in terms of training and inference latency.
These results suggest there are tradeoffs between performance (R1), latencies during online querying (R2) and offline indexation phases (R3), and index memory size.

Report issue for preceding element

Unfreezing the vision component. We train a ColPali variant by also backpropagating through and updating the vision encoder and the projection layer. This leads to a slight performance degradation (−0.70.7-0.7\- 0.7 nDCG@5). These conclusions may change with larger scales of training data.

Report issue for preceding element

Impact of “query augmentation” tokens. In ColBERT, special tokens are concatenated to the input query to serve as soft query augmentation buffers. Training without these tokens, we observe no significant performance difference in the English benchmarks. However, performance on the French tasks seems to improve (+9.89.8+9.8\+ 9.8 nDCG@G on Shift, +6.36.3+6.3\+ 6.3 nDCG@5 on TabFQuAD, [Table 7](https://arxiv.org/html/2407.01449v6#A3.T7 "Table 7 ‣ C.2 Model Variants ‣ Appendix C Additional results ‣ ColPali: Efficient Document Retrieval with Vision Language Models")).

Report issue for preceding element

Impact of the Pairwise CE loss. Training with an in-batch negative contrastive loss, instead of the pairwise CE loss that only considers the hardest negative sample, leads to a slight performance degradation (−1.61.6-1.6\- 1.6 nDCG@5) on the aggregated benchmark.

Report issue for preceding element

Adapting models to new tasks. Contrary to more complex multi-step retrieval pipelines, ColPali can be trained end-to-end, directly optimizing the downstream retrieval task which greatly facilitates fine-tuning to boost performance on specialized domains, multilingual retrieval, or specific visual elements the model struggles with. To demonstrate, we add 1552 samples representing French tables and associated queries to the training set. This represents the only French data in the training set, with all other examples being kept unchanged. We see clear nDCG@5 improvements (+2.62.6+2.6\+ 2.6) and even starker Recall@1 gains (+55+5\+ 5) on the TabFQuAD benchmark, with no performance degradation on the rest of the benchmark tasks (+0.40.4+0.4\+ 0.4 nDCG@5 overall).

Report issue for preceding element

Better VLMs lead to better visual retrievers. As improved VLMs are released, it is interesting to observe if improved performances on generative tasks translate once these models are adapted for image retrieval tasks through ColPali training strategies. We train the recently released Qwen2-VL 2B (Wang et al., [2024b](https://arxiv.org/html/2407.01449v6#bib.bib54 "")), a SOTA 2 billion parameter generative VLM, with the same data and training strategy, obtaining ColQwen2-VL. To approximately match ColPali’s memory requirements, we limit the number of image patches to 768, slightly less than ColPali’s 1024 patches. We observe clear performance improvements of +5.35.3+5.3\+ 5.3 nDCG@5 values over ColPali showcasing clear performance correlations between generative benchmarks performance and retrieving metrics.

Report issue for preceding element

Out-of-domain generalization. Some of the datasets in the ViDoRe benchmark have train sets, which we have integrated within the ColPali train set (eg. academic tasks). This is standard in embedding models (Wang et al., [2024a](https://arxiv.org/html/2407.01449v6#bib.bib53 ""); Lee et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib31 "")), and while ColPali also exhibits strong performance on tasks in which this is not the case (French data is never seen by the model during training for instance), it remains interesting to evaluate model performance when training is done on a fully disjoint data distribution. We train a ColPali variant solely using the recent DocMatix dataset (Laurençon et al., [2024a](https://arxiv.org/html/2407.01449v6#bib.bib29 "")), a large scale, synthetically annotated visual document question answering dataset, which we subsample to obtain a comparably-sized train set. Results on ViDoRe show the performance drop is minor (−2.22.2-2.2\- 2.2 nDCG@5), still outperforming the closest baseline method by over 12 points. These results showcase
ColPali generalizes well outside of its training distribution, and demonstrate that our results are not unreasonably boosted with respect to baselines (BGE-M3) that cannot be fine-tuned on the same data151515To train with data resembling the one BGE-M3 models would see at inference time would require running complex extraction pipelines for the more than 100K documents in the training set, notably relying on external proprietary captioning models which is both too costly and lengthy. This is not needed to train vision-based models.

Report issue for preceding element

## 7 Conclusions

Report issue for preceding element

In this work, we introduced the Visual Document Retrieval Benchmark (ViDoRe), which evaluates document retrieval systems in realistic settings involving visually complex documents. We demonstrated that current retrieval pipelines and contrastive vision-language models struggle to efficiently exploit visual information embedded in documents, leading to suboptimal performance. To address this, we presented ColPali, a novel retrieval method that leverages Vision-Language Models to create high-quality, multi-vector embeddings purely from visual document features. ColPali largely outperforms the best existing document retrieval methods while enabling faster corpus indexing times and maintaining low querying latencies, thus circumventing many pain points of modern document retrieval applications. We hope to drive industrial adoption, and to encourage future work by publicly releasing the ViDoRe benchmark, the data, the codebase, and all models and baselines from our work.

Report issue for preceding element

Future Work.
Beyond performance improvements that could be obtained through better data, backbone models or training strategies, our vision at term is to combine visual retrieval systems and visually grounded query answering to create end-to-end RAG systems that purely function from image features. This idea is supported by concurrent work (Ma et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib37 "")) showcasing the strong promises of VLMs for visual QA, and may eventually become a new industrial standard for document processing. In this line of work, reliability is key, and confidence estimation techniques for Information Retrieval methods could become central to implement abstention mechanisms (Gisserot-Boukhlef et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib20 "")), and are particularly interesting given the information rich multi-vector scoring mechanisms of late interaction systems.
Expanding benchmarking efforts to cover more languages, modalities, and tasks is also a crucial future research direction (Jiang et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib24 "")).

Report issue for preceding element

## Reproducibility Statement

Report issue for preceding element

For transparency, reproducibility and to foster future work, we release our training data, model checkpoints (adapters), entire codebase, and complete evaluation benchmark under MIT licenses as detailed in the main paper. We also host a public ViDoRe leaderboard161616 [https://huggingface.co/spaces/vidore/vidore-leaderboard](https://huggingface.co/spaces/vidore/vidore-leaderboard "") to foster concurrent work in the space. The supplementary material further details training configurations for our models (also specified in HuggingFace model repositories), and dives into the process we used to generate synthetic data, how latency computations are performed, as well as provides further detailed evaluation results.

Report issue for preceding element

## Acknowledgements

Report issue for preceding element

This work is partially supported by Illuin Technology, and by a grant from ANRT France. This work was performed using HPC resources from the CINES ADASTRA through Grant 2024-AD011015443 and from IDRIS with grant 2024-AD011015724R1.
We extend our warm thanks to Jonathan Dong, Caio Corro, Victor Pellegrain and Ender Konukoglu for their valuable feedback on the paper.

Report issue for preceding element

## References

Report issue for preceding element

- Alabdulmohsin et al. (2023)↑
Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer.

Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design.

2023.

doi: 10.48550/ARXIV.2305.13035.

URL [https://arxiv.org/abs/2305.13035](https://arxiv.org/abs/2305.13035 "").

Publisher: arXiv Version Number: 5.

- Alayrac et al. (2022)↑
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.

Flamingo: a Visual Language Model for Few-Shot Learning.

2022.

doi: 10.48550/ARXIV.2204.14198.

URL [https://arxiv.org/abs/2204.14198](https://arxiv.org/abs/2204.14198 "").

Publisher: arXiv Version Number: 2.

- Anthropic (2024)↑
Anthropic.

The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024.

URL [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model\_Card\_Claude\_3.pdf](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf "").

- Appalaraju et al. (2021)↑
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha.

DocFormer: End-to-End Transformer for Document Understanding, 2021.

URL [https://arxiv.org/abs/2106.11539](https://arxiv.org/abs/2106.11539 "").

Version Number: 2.

- Bai et al. (2023)↑
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.

Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.

2023.

doi: 10.48550/ARXIV.2308.12966.

URL [https://arxiv.org/abs/2308.12966](https://arxiv.org/abs/2308.12966 "").

Publisher: arXiv Version Number: 3.

- Bajaj et al. (2016)↑
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang.

MS MARCO: A Human Generated MAchine Reading COmprehension Dataset, 2016.

URL [https://arxiv.org/abs/1611.09268](https://arxiv.org/abs/1611.09268 "").

Version Number: 3.

- Beyer et al. (2024)↑
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai.

Paligemma: A versatile 3b vlm for transfer, 2024.

URL [https://arxiv.org/abs/2407.07726](https://arxiv.org/abs/2407.07726 "").

- Bloom (1970)↑
Burton H. Bloom.

Space/time trade-offs in hash coding with allowable errors.

_Commun. ACM_, 13(7):422–426, July 1970.

ISSN 0001-0782.

doi: 10.1145/362686.362692.

URL [https://doi.org/10.1145/362686.362692](https://doi.org/10.1145/362686.362692 "").

Place: New York, NY, USA Publisher: Association for Computing Machinery.

- Borchmann et al. (2021)↑
Łukasz Borchmann, Michał Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michał Turski, Karolina Szyndler, and Filip Graliński.

DUE: End-to-End Document Understanding Benchmark.

In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_, 2021.

URL [https://openreview.net/forum?id=rNs2FvJGDK](https://openreview.net/forum?id=rNs2FvJGDK "").

- Chen et al. (2024)↑
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.

BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation, 2024.

URL [https://arxiv.org/abs/2402.03216](https://arxiv.org/abs/2402.03216 "").

Version Number: 3.

- Chen et al. (2023)↑
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut.

PaLI-3 Vision Language Models: Smaller, Faster, Stronger, 2023.

URL [https://arxiv.org/abs/2310.09199](https://arxiv.org/abs/2310.09199 "").

Version Number: 2.

- Clavié et al. (2024)↑
Benjamin Clavié, Antoine Chaffin, and Griffin Adams.

Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling, 2024.

URL [https://arxiv.org/abs/2409.14683](https://arxiv.org/abs/2409.14683 "").

Version Number: 1.

- Cohere (2024)↑
Cohere.

Introducing Rerank 3: A New Foundation Model for Efficient Enterprise Search & Retrieval, April 2024.

URL [https://cohere.com/blog/rerank-3](https://cohere.com/blog/rerank-3 "").

- Dao (2023)↑
Tri Dao.

Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.

URL [https://arxiv.org/abs/2307.08691](https://arxiv.org/abs/2307.08691 "").

- Darcet et al. (2023)↑
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski.

Vision Transformers Need Registers.

2023.

doi: 10.48550/ARXIV.2309.16588.

URL [https://arxiv.org/abs/2309.16588](https://arxiv.org/abs/2309.16588 "").

Publisher: \[object Object\] Version Number: 2.

- Devlin et al. (2018)↑
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018.

URL [https://arxiv.org/abs/1810.04805](https://arxiv.org/abs/1810.04805 "").

Version Number: 2.

- Dosovitskiy et al. (2020)↑
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.

2020.

doi: 10.48550/ARXIV.2010.11929.

URL [https://arxiv.org/abs/2010.11929](https://arxiv.org/abs/2010.11929 "").

Publisher: arXiv Version Number: 2.

- Ge et al. (2021)↑
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun.

YOLOX: Exceeding YOLO Series in 2021, 2021.

URL [https://arxiv.org/abs/2107.08430](https://arxiv.org/abs/2107.08430 "").

Version Number: 2.

- Gemma Team et al. (2024)↑
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem,
Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.

Gemma: Open Models Based on Gemini Research and Technology, 2024.

URL [https://arxiv.org/abs/2403.08295](https://arxiv.org/abs/2403.08295 "").

Version Number: 4.

- Gisserot-Boukhlef et al. (2024)↑
Hippolyte Gisserot-Boukhlef, Manuel Faysse, Emmanuel Malherbe, Céline Hudelot, and Pierre Colombo.

Towards trustworthy reranking: A simple yet effective abstention mechanism, 2024.

URL [https://arxiv.org/abs/2402.12997](https://arxiv.org/abs/2402.12997 "").

- Hu et al. (2021)↑
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.

LoRA: Low-Rank Adaptation of Large Language Models.

2021.

doi: 10.48550/ARXIV.2106.09685.

URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685 "").

Publisher: arXiv Version Number: 2.

- Huang et al. (2022)↑
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking.

2022.

doi: 10.48550/ARXIV.2204.08387.

URL [https://arxiv.org/abs/2204.08387](https://arxiv.org/abs/2204.08387 "").

Publisher: arXiv Version Number: 3.

- Jiang et al. (2023)↑
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.

Mistral 7B.

2023.

doi: 10.48550/ARXIV.2310.06825.

URL [https://arxiv.org/abs/2310.06825](https://arxiv.org/abs/2310.06825 "").

Publisher: arXiv Version Number: 1.

- Jiang et al. (2024)↑
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen.

Vlm2vec: Training vision-language models for massive multimodal embedding tasks, 2024.

URL [https://arxiv.org/abs/2410.05160](https://arxiv.org/abs/2410.05160 "").

- Karpukhin et al. (2020)↑
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.

Dense Passage Retrieval for Open-Domain Question Answering, 2020.

URL [https://arxiv.org/abs/2004.04906](https://arxiv.org/abs/2004.04906 "").

Version Number: 3.

- Khattab & Zaharia (2020)↑
Omar Khattab and Matei Zaharia.

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.

2020.

doi: 10.48550/ARXIV.2004.12832.

URL [https://arxiv.org/abs/2004.12832](https://arxiv.org/abs/2004.12832 "").

- Kim et al. (2021)↑
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.

OCR-free Document Understanding Transformer, 2021.

URL [https://arxiv.org/abs/2111.15664](https://arxiv.org/abs/2111.15664 "").

Version Number: 5.

- Koukounas et al. (2024)↑
Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan Wang, and Han Xiao.

Jina CLIP: Your CLIP Model Is Also Your Text Retriever, 2024.

URL [https://arxiv.org/abs/2405.20204](https://arxiv.org/abs/2405.20204 "").

Version Number: 1.

- Laurençon et al. (2024a)↑
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon.

Building and better understanding vision-language models: insights and future directions., 2024a.

- Laurençon et al. (2024b)↑
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh.

What matters when building vision-language models?, May 2024b.

URL [http://arxiv.org/abs/2405.02246](http://arxiv.org/abs/2405.02246 "").

arXiv:2405.02246 \[cs\].

- Lee et al. (2024)↑
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.

Nv-embed: Improved techniques for training llms as generalist embedding models, 2024.

URL [https://arxiv.org/abs/2405.17428](https://arxiv.org/abs/2405.17428 "").

- Lee et al. (2023)↑
Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, and Vincent Y. Zhao.

Rethinking the Role of Token Retrieval in Multi-Vector Retrieval, 2023.

URL [https://arxiv.org/abs/2304.01982](https://arxiv.org/abs/2304.01982 "").

Version Number: 3.

- Li et al. (2024)↑
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu.

Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024.

- Lin et al. (2014)↑
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.

Microsoft COCO: Common Objects in Context, 2014.

URL [https://arxiv.org/abs/1405.0312](https://arxiv.org/abs/1405.0312 "").

Version Number: 3.

- Liu et al. (2023)↑
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.

Visual Instruction Tuning.

2023.

doi: 10.48550/ARXIV.2304.08485.

URL [https://arxiv.org/abs/2304.08485](https://arxiv.org/abs/2304.08485 "").

Publisher: arXiv Version Number: 1.

- Lucas Beyer\* et al. (2024)↑
Lucas Beyer\*, Andreas Steiner\*, André Susano Pinto\*, Alexander Kolesnikov\*, Xiao Wang\*, Xiaohua Zhai\*, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Jeremiah Harmsen, Daniel Keysers, Neil Houlsby, Xi Chen, Emanuele Bugliarello, Thomas Unterthiner, Keran Rong, Matthias Minderer, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Julian Eisenschlos, Manoj Kumar, Matko Bošnjak, Matthias Bauer, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Paul Voigtlaender, Pinelopi Papalampidi, Olivier Henaff, Skanda Koppula, Xi Xiong, Radu Soricut, Model release contributors and general support, Tris Warkentin, Kat Black, Luiz Gustavo Martins, Glenn Cameron, Raj Gundluru, Manvinder Singh, Meg Risdal, Nilay Chauhan, Nate Keating, Nesh Devanathan, Elisa Bandy, Joe Fernandez, Antonia Paterson, Jenny Brennan, Tom Eccles, Pankil Botadra, Ben Bariach, Lav Rai, Minwoo Park, Dustin Luong, Daniel Vlasic,
Bo Wu, Wenming Ye, Divyashree Sreepathihalli, Kiranbir Sodhia, Alek Andreev, Armand Joulin, Surya Bhupatiraju, Minh Giang, Joelle Barral, and Zoubin Ghahramani.

PaliGemma, 2024.

URL [https://www.kaggle.com/m/23393](https://www.kaggle.com/m/23393 "").

- Ma et al. (2024)↑
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun.

Mmlongbench-doc: Benchmarking long-context document understanding with visualizations, 2024.

URL [https://arxiv.org/abs/2407.01523](https://arxiv.org/abs/2407.01523 "").

- Mathew et al. (2020)↑
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar.

DocVQA: A Dataset for VQA on Document Images.

2020.

doi: 10.48550/ARXIV.2007.00398.

URL [https://arxiv.org/abs/2007.00398](https://arxiv.org/abs/2007.00398 "").

- Mathew et al. (2021)↑
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar.

InfographicVQA, 2021.

URL [https://arxiv.org/abs/2104.12756](https://arxiv.org/abs/2104.12756 "").

Version Number: 2.

- Muennighoff et al. (2022)↑
Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers.

MTEB: Massive Text Embedding Benchmark, 2022.

URL [https://arxiv.org/abs/2210.07316](https://arxiv.org/abs/2210.07316 "").

Version Number: 3.

- Nomic (2024)↑
Nomic.

Nomic Embed Vision: Expanding The Nomic Latent Space, June 2024.

URL [https://blog.nomic.ai/posts/nomic-embed-vision](https://blog.nomic.ai/posts/nomic-embed-vision "").

- Nowak et al. (2024)↑
Averi Nowak, Francesco Piccinno, and Yasemin Altun.

Multimodal chart retrieval: A comparison of text, table and image based approaches.

In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pp.  5488–5505, Mexico City, Mexico, June 2024. Association for Computational Linguistics.

doi: 10.18653/v1/2024.naacl-long.307.

URL [https://aclanthology.org/2024.naacl-long.307](https://aclanthology.org/2024.naacl-long.307 "").

- Radford et al. (2021)↑
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.

Learning Transferable Visual Models From Natural Language Supervision.

2021.

doi: 10.48550/ARXIV.2103.00020.

URL [https://arxiv.org/abs/2103.00020](https://arxiv.org/abs/2103.00020 "").

Publisher: arXiv Version Number: 1.

- Reimers & Gurevych (2019)↑
Nils Reimers and Iryna Gurevych.

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, 2019.

URL [https://arxiv.org/abs/1908.10084](https://arxiv.org/abs/1908.10084 "").

Version Number: 1.

- Robertson et al. (1994)↑
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford.

Okapi at TREC-3.

In Donna K. Harman (ed.), _Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994_, volume 500-225 of _NIST Special Publication_, pp.  109–126. National Institute of Standards and Technology (NIST), 1994.

URL [http://trec.nist.gov/pubs/trec3/papers/city.ps.gz](http://trec.nist.gov/pubs/trec3/papers/city.ps.gz "").

- Santhanam et al. (2022)↑
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia.

PLAID: An Efficient Engine for Late Interaction Retrieval, 2022.

URL [https://arxiv.org/abs/2205.09707](https://arxiv.org/abs/2205.09707 "").

Version Number: 1.

- Smith (2007)↑
R. Smith.

An Overview of the Tesseract OCR Engine.

In _Ninth International Conference on Document Analysis and Recognition (ICDAR 2007) Vol 2_, pp.  629–633, Curitiba, Parana, Brazil, September 2007. IEEE.

ISBN 978-0-7695-2822-9.

doi: 10.1109/ICDAR.2007.4376991.

URL [http://ieeexplore.ieee.org/document/4376991/](http://ieeexplore.ieee.org/document/4376991/ "").

ISSN: 1520-5363.

- Sparck Jones (1972)↑
Karen Sparck Jones.

A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL.

_Journal of Documentation_, 28(1):11–21, January 1972.

ISSN 0022-0418.

doi: 10.1108/eb026526.

URL [https://www.emerald.com/insight/content/doi/10.1108/eb026526/full/html](https://www.emerald.com/insight/content/doi/10.1108/eb026526/full/html "").

- Tang et al. (2022)↑
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal.

Unifying Vision, Text, and Layout for Universal Document Processing, 2022.

URL [https://arxiv.org/abs/2212.02623](https://arxiv.org/abs/2212.02623 "").

Version Number: 3.

- Thakur et al. (2021)↑
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych.

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, 2021.

URL [https://arxiv.org/abs/2104.08663](https://arxiv.org/abs/2104.08663 "").

Version Number: 4.

- Thapliyal et al. (2022)↑
Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut.

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset, 2022.

URL [https://arxiv.org/abs/2205.12522](https://arxiv.org/abs/2205.12522 "").

Version Number: 2.

- Wang et al. (2022)↑
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei.

Text Embeddings by Weakly-Supervised Contrastive Pre-training, 2022.

URL [https://arxiv.org/abs/2212.03533](https://arxiv.org/abs/2212.03533 "").

Version Number: 2.

- Wang et al. (2024a)↑
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei.

Improving text embeddings with large language models, 2024a.

URL [https://arxiv.org/abs/2401.00368](https://arxiv.org/abs/2401.00368 "").

- Wang et al. (2024b)↑
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin.

Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024b.

URL [https://arxiv.org/abs/2409.12191](https://arxiv.org/abs/2409.12191 "").

- Wang et al. (2020)↑
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers, April 2020.

URL [http://arxiv.org/abs/2002.10957](http://arxiv.org/abs/2002.10957 "").

arXiv:2002.10957 \[cs\].

- Yao et al. (2021)↑
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.

FILIP: Fine-grained Interactive Language-Image Pre-Training, 2021.

URL [https://arxiv.org/abs/2111.07783](https://arxiv.org/abs/2111.07783 "").

Version Number: 1.

- Yue et al. (2023)↑
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, 2023.

URL [https://arxiv.org/abs/2311.16502](https://arxiv.org/abs/2311.16502 "").

Version Number: 3.

- Zhai et al. (2023)↑
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.

Sigmoid Loss for Language Image Pre-Training.

2023.

doi: 10.48550/ARXIV.2303.15343.

URL [https://arxiv.org/abs/2303.15343](https://arxiv.org/abs/2303.15343 "").

Publisher: \[object Object\] Version Number: 4.

- Zhang et al. (2019)↑
Li Zhang, Shuo Zhang, and Krisztian Balog.

Table2vec: Neural word and entity embeddings for table population and retrieval.

In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR ’19. ACM, July 2019.

doi: 10.1145/3331184.3331333.

URL [http://dx.doi.org/10.1145/3331184.3331333](http://dx.doi.org/10.1145/3331184.3331333 "").

- Zhao et al. (2023)↑
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty.

Retrieving Multimodal Information for Augmented Generation: A Survey, 2023.

URL [https://arxiv.org/abs/2303.10868](https://arxiv.org/abs/2303.10868 "").

Version Number: 3.

- Zhu et al. (2022)↑
Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua.

Towards Complex Document Understanding By Discrete Reasoning.

2022.

doi: 10.48550/ARXIV.2207.11871.

URL [https://arxiv.org/abs/2207.11871](https://arxiv.org/abs/2207.11871 "").

Publisher: arXiv Version Number: 3.


## Appendix A Benchmark Datasets

Report issue for preceding element

### A.1 Academic Datasets

Report issue for preceding element

DocVQA(Mathew et al., [2020](https://arxiv.org/html/2407.01449v6#bib.bib38 "")) includes collected images from the UCSF Industry Documents Library. Questions and answers were manually annotated.

Report issue for preceding element

InfoVQA(Mathew et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib39 "")) includes infographics collected from the Internet using the search query “infographics”. Questions and answers were manually annotated.

Report issue for preceding element

TAT-DQA(Zhu et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib61 "")) is a large-scale Document VQA dataset that was constructed from publicly available real-world financial reports. It focuses on rich tabular and textual content requiring numerical reasoning. Questions and answers were manually annotated by human experts in finance.

Report issue for preceding element

arXivQA(Li et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib33 "")) is a VQA dataset based on figures extracted from arXiv publications. The questions were generated synthetically using GPT-4 Vision.

Report issue for preceding element

TabFQuAD (Table French Question Answering Dataset) is designed to evaluate TableQA models in realistic industry settings. We create additional queries to augment the existing human-annotated ones using the same method described in [subsection A.2](https://arxiv.org/html/2407.01449v6#A1.SS2 "A.2 Practical Datasets ‣ Appendix A Benchmark Datasets ‣ ColPali: Efficient Document Retrieval with Vision Language Models").

Report issue for preceding element

### A.2 Practical Datasets

Report issue for preceding element

Methodology. Creating a relevant retrieval dataset close to real use cases is a major challenge as the dataset needs to be both sufficiently large for effective fine-tuning and sufficiently diverse to cover a broad range of modalities (full text, tables, charts, …), domains (industry, healthcare, …), and query-document interactions (extractive questions, open-ended questions, …). Our approach to building this dataset involves several steps: (1) we use a web crawler to collect publicly available documents on various themes and sources, (2) we convert these PDFs into a series of images, one per page, and (3) we generate queries related to each image using a VLM.

Report issue for preceding element

Web-Crawler. We implemented a web crawler to efficiently collect large volumes of documents related to a given topic. The crawler is seeded with a user-defined query (e.g. “artificial intelligence”) and then uses GPT-3.5 Turbo to brainstorm related topics and subtopics. This query augmentation strategy aims at both broadening and deepening the search. GPT-3.5 Turbo is further used to generate diverse search queries from each subtopic. This query set is then consumed by a pool of parallel workers whose job is to fetch the associated most relevant documents. We use SerpAPI171717 [https://serpapi.com/](https://serpapi.com/ "") along with a filetype filter (PDF documents only) to programmatically scrape Google Search rankings. Each file is hashed and stored in a Bloom filter (Bloom, [1970](https://arxiv.org/html/2407.01449v6#bib.bib8 "")) shared among workers to avoid duplicate documents in the final corpus. Unique scraped files are downloaded, and inserted into a SQLite database along with additional metadata.

Report issue for preceding element

Datamix. Using the web crawler, we collected approximately 100 documents for each of the following four seeds: “energy”, “government reports”, “healthcare industry”, and “artificial intelligence”. These seeds were meticulously hand-picked to align with real-use cases for retrieval models and visually rich pages. We also removed all documents containing any private information.

Report issue for preceding element

Query Generation. To increase the efficiency of our query generation scheme and to limit API calls, we generate at most 3 questions per image. From all the documents collected, we randomly sample 10,000 images per theme and call Claude-3 Sonnet with the following prompt:

Report issue for preceding element

You are an assistant specialized in Multimodal RAG tasks.The task is the following: given an image from a pdf page, you will have togenerate questions that can be asked by a user to retrieve information froma large documentary corpus.The question should be relevant to the page, and should not be too specificor too general. The question should be about the subject of the page, andthe answer needs to be found in the page.Remember that the question is asked by a user to get some information from alarge documentary corpus that contains multimodal data. Generate a questionthat could be asked by a user without knowing the existence and the contentof the corpus.Generate as well the answer to the question, which should be found in thepage. And the format of the answer should be a list of words answering thequestion.Generate at most THREE pairs of questions and answers per page in a dictionarywith the following format, answer ONLY this dictionary NOTHING ELSE:{    "questions": \[        {            "question": "XXXXXX",            "answer": \["YYYYYY"\]        },        {            "question": "XXXXXX",            "answer": \["YYYYYY"\]        },        {            "question": "XXXXXX",            "answer": \["YYYYYY"\]        },    \]}where XXXXXX is the question and \[’YYYYYY’\] is the corresponding list of answersthat could be as long as needed.Note: If there are no questions to ask about the page, return an empty list.Focus on making relevant questions concerning the page.Here is the page:Report issue for preceding element

Human Validation. We manually validate every single synthetically created query in ViDoRe to ensure quality, query relevance, and consistency with the benchmark objective of evaluating retrieval in practical industrial settings. During this step, we randomly assign document-pair queries to 4 volunteer annotators and instruct them to filter out queries that do not fit the above-listed criteria. We also instruct annotators to flag any documents they deem to contain PII information or content not suited for an academic benchmark. No flag was raised during the entirety of the process, validating our prior PDF collection strategy. 100 queries per topic are collected in this manner. Annotators are colleagues and collaborators of the authors who volunteered to help. Each annotator spent approximately 3 hours filtering the larger query set down to 100 high-quality queries per topic.

Report issue for preceding element

### A.3 Training Dataset

Report issue for preceding element

The statistics of the train set are given in the following table. The creation of the train set follows the same methodology as in [subsection A.2](https://arxiv.org/html/2407.01449v6#A1.SS2 "A.2 Practical Datasets ‣ Appendix A Benchmark Datasets ‣ ColPali: Efficient Document Retrieval with Vision Language Models"). We made sure that a PDF document cannot have pages in both the training set and the test set to prevent data leakage and that there are no duplicate documents in each split.

Report issue for preceding element

| Dataset Split | Split Size | Language | Domain |
| --- | --- | --- | --- |
| DocVQA | 39,463 | English | Scanned documents from UCSF Industry |
| InfoVQA | 10,074 | English | Infographics scrapped from the web |
| TATDQA | 13,251 | English | High-quality financial reports |
| arXivQA | 10,000 | English | Scientific Scientific Figures from arXiv |
| Scrapped PDFs | 45,940 | English | Varied PDFs from 3885 distinct URL domains |
| TOTAL | 118,695 | English-only | Mixed |

Table 3: Details on the different splits in the dataset used to train ColPali.Report issue for preceding element

## Appendix B Implementation details

Report issue for preceding element

### B.1 Codebase

Report issue for preceding element

The codebase is written in PyTorch181818 [https://pytorch.org/](https://pytorch.org/ "") and leverages HuggingFace tooling for model implementations and trainers191919 [https://huggingface.co](https://huggingface.co/ "").

Report issue for preceding element

### B.2 Hyperparameters

Report issue for preceding element

Hyperparameters are tuned on a validation split composed of 2%percent22\\%2 % of the training dataset. We find bi-encoder methods to be more sensible to learning rate variations than late interaction-based models and achieve the best performance for all models with a learning rate of 5⁢e−55𝑒55e-55 italic\_e - 5. We experiment with LoRA rank and α𝛼\\alphaitalic\_α values and do not notice particular improvements past r=α=32𝑟𝛼32r=\\alpha=32italic\_r = italic\_α = 32. Per-device batch sizes are kept small due to long sequence lengths that complicate scaling past b=4𝑏4b=4italic\_b = 4. We simulate larger batch sizes with multi-GPU training and train with a total batch size b=32𝑏32b=32italic\_b = 32 with no accumulation, for 1 epoch on our training set.

Report issue for preceding element

### B.3 Embedding size

Report issue for preceding element

Minimizing storage footprint can be essential to industrial retrieval systems if databases contain millions of documents. With this criterion in view, we have compared the embedding sizes of the models in our study. As shown in [Table 4](https://arxiv.org/html/2407.01449v6#A2.T4 "Table 4 ‣ B.3 Embedding size ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models"), ColPali’s embedding size is an order of magnitude larger than BM25 and two orders of magnitude larger than BGE-M3. However, in practical scenarios, pooling multi-vector embeddings by centroid cluster, or quantizing embeddings to binary representations 202020 [https://blog.vespa.ai/scaling-colpali-to-billions/](https://blog.vespa.ai/scaling-colpali-to-billions/ "") can reduce storage costs by two orders of magnitude (Santhanam et al., [2022](https://arxiv.org/html/2407.01449v6#bib.bib46 "")) with minimal performance hits, and make storage costs competitive with other systems.

Report issue for preceding element

| Model | Embedding size (KB) |
| BGE-M3 | 8.60 |
| BM25 (dense emb.) | 3.00 |
| BM25 (sparse emb.) | 1.56 ± 0.51 |
| ColPali (float16) | 257.5 |

Table 4: Comparison of the embedding sizes for the DocVQA test set from ViDoRe w.r.t. different retrieval models. The mean ± std size is given for the sparse embeddings. In general multiple vectors (2-5) per page are used for BGE-M3 and BM25.Report issue for preceding element

### B.4 Latency computations

Report issue for preceding element

To ensure comparison fairness, the latencies of the different retrieval systems shown in [Figure 2](https://arxiv.org/html/2407.01449v6#S3.F2 "Figure 2 ‣ 3.2 Assessing Current Systems ‣ 3 The ViDoRe Benchmark ‣ ColPali: Efficient Document Retrieval with Vision Language Models") are measured on the same g2-standard-8 GCP VM with a NVIDIA L4 GPU. Document pages are embedded using the highest settings of Unstructured with captioning (see [subsection 3.2](https://arxiv.org/html/2407.01449v6#S3.SS2 "3.2 Assessing Current Systems ‣ 3 The ViDoRe Benchmark ‣ ColPali: Efficient Document Retrieval with Vision Language Models")). SigLIP and ColPali are both loaded with bfloat16 parameter dtypes. The reported times in [Table 5](https://arxiv.org/html/2407.01449v6#A2.T5 "Table 5 ‣ B.4 Latency computations ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models") are the average per-page latencies for each indexing operation on 1000 randomly chosen documents across all splits of the ViDoRe benchmark test set. A batch size of 8888 was used for the BGE-M3 model used with Unstructured, and a batch size of 4444 was used for SigLIP and ColPali.

Report issue for preceding element

| Indexing operation | Latency (s) |
| --- | --- |
| Unstructured | SigLIP | ColPali |
| --- | --- | --- |
| Layout detection | 0.81 | NA | NA |
| OCR | 2.67 | NA | NA |
| Captioning | 3.71 | NA | NA |
| Page encoding | 0.03 | 0.12 | 0.39 |
| Total | 7.22 | 0.12 | 0.39 |

Table 5: Page-level latencies for document indexing using various retrieval systems. SigLIP and ColPali are much faster than Unstructured because they don’t require the layout detection, OCR, and captioning operations.Report issue for preceding element

### B.5 Captioning

Report issue for preceding element

Examples of captions generated for visually rich document chunks with Claude-3 Sonnet are shown in [Figure 5](https://arxiv.org/html/2407.01449v6#A2.F5 "Figure 5 ‣ B.5 Captioning ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models") and [Figure 4](https://arxiv.org/html/2407.01449v6#A2.F4 "Figure 4 ‣ B.5 Captioning ‣ Appendix B Implementation details ‣ ColPali: Efficient Document Retrieval with Vision Language Models"). The prompt used for generating the description is the following:

Report issue for preceding element

You are an assistant specialized in document analysis. Given a table or a figure, you have to provide a detailed summary of the content in maximum 3000 characters. Your summary should be qualitative and not quantitative. Here is the table/figure to analyze: {image}. Answer ONLY with the caption of the table/figure.Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2407.01449v6/x3.png)Figure 4: Example from the “Energy” test set.

Caption:The image depicts the hourly energy generation profile, illustrating the contributions of various energy sources over 24 hours. The data is presented as a stacked bar chart, with the x-axis representing the hours of the day from 1 to 2, and the y-axis showing the average hourly generation in MW. The bars are segmented into different colors, each representing a distinct energy source: nuclear, bio, geothermal, solar, wind, hydro, natural gas, and other imports. The chart provides insights into the temporal variations in energy generation across different sources, highlighting the interplay between baseload and intermittent sources throughout the day.Report issue for preceding element![Refer to caption](https://arxiv.org/html/2407.01449v6/x4.png)Figure 5: Example from the “Government Reports” test set.

Caption:The image shows a table titled “System of Record” which outlines the different types of documents or records maintained across various systems or departments within an organization related to project management and construction. The rows list documents like project plans, budgets, schedules, contracts, purchase orders, invoices, change requests, bid submissions, drawings, manuals, meeting minutes, and reports. The columns indicate the system or department responsible for maintaining each record, such as County Servers, Project View, OnBase, CGI Advantage Financial System, and Purchasing Department. The table uses ”W” and ”T” markers to denote which system or department serves as the primary source (writer) or storage location (trailer) for each type of document.Report issue for preceding element

## Appendix C Additional results

Report issue for preceding element

### C.1 Other Metrics

Report issue for preceding element

|     |     |     |     |     |     |     |     |     |     |     |     |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ArxivQ | DocQ | InfoQ | TabF | TATQ | Shift | AI | Energy | Gov. | Health. | Avg. |
| Unstructured text-only |  |  |  |  |  |  |  |  |  |  |  |
| BM25 | - | 26.6 | - | - | 34.6 | 45.0 | 86.0 | 70.0 | 68.0 | 74.0 | - |
| BGE-M3 | - | 22.8↓↓\\downarrow↓3.8 | - | - | 26.1↓↓\\downarrow↓8.5 | 51.0↑↑\\uparrow↑6.0 | 81.0↓↓\\downarrow↓5.0 | 72.0↑↑\\uparrow↑2.0 | 67.0↓↓\\downarrow↓1.0 | 77.0↑↑\\uparrow↑3.0 | - |
| Unstructured + OCR |  |  |  |  |  |  |  |  |  |  |  |
| BM25 | 26.7 | 28.9 | 54.0 | 30.4 | 50.0 | 52.0 | 86.0 | 77.0 | 74.0 | 80.0 | 55.9 |
| BGE-M3 | 28.1↑↑\\uparrow↑1.4 | 22.9↓↓\\downarrow↓6.0 | 53.8↓↓\\downarrow↓0.2 | 55.7↑↑\\uparrow↑25.3 | 38.6↓↓\\downarrow↓11.4 | 56.0↑↑\\uparrow↑4.0 | 82.0↓↓\\downarrow↓4.0 | 79.0↑↑\\uparrow↑2.0 | 76.0↑↑\\uparrow↑2.0 | 83.0↑↑\\uparrow↑3.0 | 57.5↑↑\\uparrow↑1.6 |
| Unstructured \+ Captioning |  |  |  |  |  |  |  |  |  |  |  |
| BM25 | 35.5 | 30.2 | 61.5 | 24.3 | 49.0 | 47.0 | 79.0 | 76.0 | 75.0 | 81.0 | 55.9 |
| BGE-M3 | 29.3↓↓\\downarrow↓6.2 | 26.0↓↓\\downarrow↓4.2 | 62.1 ↑↑\\uparrow↑0.6 | 58.6↑↑\\uparrow↑34.3 | 30.6↓↓\\downarrow↓18.4 | 55.0↑↑\\uparrow↑8.0 | 80.0↑↑\\uparrow↑1.0 | 78.0↑↑\\uparrow↑2.0 | 69.0↓↓\\downarrow↓6.0 | 83.0↑↑\\uparrow↑2.0 | 57.2↑↑\\uparrow↑1.3 |
| Contrastive VLMs |  |  |  |  |  |  |  |  |  |  |  |
| Jina-CLIP | 19.4 | 7.3 | 26.7 | 12.5 | 1.6 | 2.0 | 11.0 | 13.0 | 15.0 | 17.0 | 12.6 |
| Nomic-vision | 10.4 | 6.7 | 22.1 | 9.6 | 1.6 | 0.0 | 9.0 | 9.0 | 7.0 | 13.0 | 8.8 |
| SigLIP (Vanilla) | 34.2 | 21.3 | 51.8 | 46.1 | 17.9 | 13.0 | 50.0 | 51.0 | 47.0 | 65.0 | 39.7 |
| Ours |  |  |  |  |  |  |  |  |  |  |  |
| SigLIP (Vanilla) | 34.2 | 21.3 | 51.8 | 46.1 | 17.9 | 13.0 | 50.0 | 51.0 | 47.0 | 65.0 | 39.7 |
| BiSigLIP (+fine-tuning) | 49.2↑↑\\uparrow↑15.0 | 23.8↑↑\\uparrow↑2.5 | 59.0↑↑\\uparrow↑7.2 | 52.1↑↑\\uparrow↑6.0 | 20.7↑↑\\uparrow↑2.8 | 16.0↑↑\\uparrow↑3.0 | 62.0↑↑\\uparrow↑12.0 | 61.0↑↑\\uparrow↑10.0 | 55.0↑↑\\uparrow↑8.0 | 72.0↑↑\\uparrow↑7.0 | 47.1↑↑\\uparrow↑7.4 |
| BiPali (+LLM) | 46.4↓↓\\downarrow↓-2.8 | 20.0↓↓\\downarrow↓-3.8 | 54.6↓↓\\downarrow↓-4.4 | 63.2↑↑\\uparrow↑11.1 | 20.4↓↓\\downarrow↓-0.4 | 34.0↑↑\\uparrow↑18.0 | 59.0↓↓\\downarrow↓-3.0 | 45.0↓↓\\downarrow↓-16.0 | 57.0↑↑\\uparrow↑2.0 | 56.0↓↓\\downarrow↓-16.0 | 45.6↓↓\\downarrow↓-1.5 |
| ColPali (+Late Inter.) | 72.4↑↑\\uparrow↑26.0 | 45.6↑↑\\uparrow↑25.6 | 74.6↑↑\\uparrow↑20.0 | 75.4↑↑\\uparrow↑12.1 | 53.1↑↑\\uparrow↑32.7 | 55.0↑↑\\uparrow↑21.0 | 93.0↑↑\\uparrow↑34.0 | 85.0↑↑\\uparrow↑40.0 | 85.0↑↑\\uparrow↑28.0 | 88.0↑↑\\uparrow↑32.0 | 72.7↑↑\\uparrow↑27.1 |

Table 6: Comprehensive evaluation of baseline models and our proposed method on ViDoRe. Results are presented using Recall@1 metrics. Text-only metrics are not computed for benchmarks with only visual elements.Report issue for preceding element

### C.2 Model Variants

Report issue for preceding element

|     |     |     |     |     |     |     |     |     |     |     |     |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ArxivQ | DocQ | InfoQ | TabF | TATQ | Shift | AI | Energy | Gov. | Health. | Avg. |
| ColSigLIP (PaliGemma) | 3.1 | 3.0 | 5.1 | 6.2 | 2.5 | 1.0 | 3.4 | 3.4 | 2.3 | 2.2 | 3.2 |
| BiSigLIP (PaliGemma) | 18.5 | 14.6 | 33.4 | 39.5 | 16.1 | 5.2 | 27.6 | 32.6 | 36.6 | 35.7 | 26.0 |
| ColSigLIP (Original) | 2.6 | 2.2 | 2.3 | 5.7 | 1.8 | 1.0 | 2.6 | 4.1 | 1.4 | 1.5 | 2.5 |
| ColPali (No Q.A. Tokens) | 80.4 | 53.2 | 82.4 | 77.4 | 65.7 | 63.4 | 97.0 | 89.9 | 93.6 | 92.4 | 79.6 |
| ColPali (Docmatix) | 71.3 | 48.0 | 80.0 | 83.9 | 59.1 | 73.8 | 95.7 | 93.8 | 92.5 | 93.1 | 79.1 |
| ColPali (224) | 71.0 | 37.4 | 62.3 | 65.7 | 28.6 | 20.4 | 65.7 | 66.8 | 73.9 | 73.0 | 56.5 |
| ColPali (Vision Trained) | 78.8 | 53.9 | 81.3 | 81.7 | 64.4 | 70.6 | 95.3 | 91.7 | 93.5 | 94.7 | 80.6 |
| ColPali (No Pairwise) | 79.0 | 53.0 | 82.1 | 85.3 | 63.2 | 66.2 | 94.9 | 88.9 | 92.7 | 92.1 | 79.7 |
| ColPali (+TabFQuAD training) | 77.6 | 54.7 | 82.6 | 86.5 | 65.4 | 73.9 | 94.8 | 92.4 | 94.2 | 94.8 | 81.7 |
| ColIdefics2 (64) | 73.6 | 48.0 | 82.4 | 81.6 | 63.0 | 57.2 | 95.5 | 86.9 | 86.6 | 91.2 | 76.6 |
| ColQwen2 (768) | 86.4 | 56.2 | 89.8 | 88.7 | 75.2 | 85.7 | 98.8 | 94.8 | 93.6 | 97.3 | 86.6 |
| ColPali (Reference: 448) | 79.1 | 54.4 | 81.8 | 83.9 | 65.8 | 73.2 | 96.2 | 91.0 | 92.7 | 94.4 | 81.3 |

Table 7: Benchmark scores for the “negative results” and various ablations on ViDoRe; ColPali for reference. Results are presented using nDCG@5 metrics. Text-only metrics are not computed for benchmarks with only visual elements.Report issue for preceding element

## Appendix D More similarity maps

Report issue for preceding element

In [Figure 6](https://arxiv.org/html/2407.01449v6#A4.F6 "Figure 6 ‣ Appendix D More similarity maps ‣ ColPali: Efficient Document Retrieval with Vision Language Models"), ColPali assigns a high similarity to all patches with the word “Kazakhstan” when given the token <\_Kazakhstan>. Moreover, our model seems to exhibit world knowledge capabilities as the patch around the word ”Kashagan”—an offshore oil field in Kazakhstan—also shows a high similarity score.

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/similarity_maps/similarity_map_kazakhstan.png)Figure 6: Similarity of the image patches w.r.t. the underlined token in the user query. This example is from the Shift test set.Report issue for preceding element

It is also interesting to highlight that both this similarity map and the one displayed in [Figure 3](https://arxiv.org/html/2407.01449v6#S5.F3 "Figure 3 ‣ 5.3 Interpretability ‣ 5 Results ‣ ColPali: Efficient Document Retrieval with Vision Language Models") (right) showcase a few white patches with high similarity scores. This behavior might first seem surprising as the white patches should not carry a meaningful signal from the original images. We believe the vectors associated with these patches share a similar role with the ViT registers (Darcet et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib15 "")), i.e. these patches were repurposed for internal computations and stored the global information from the whole image.

Report issue for preceding element

## Appendix E Model glossary

Report issue for preceding element

### SigLIP

Report issue for preceding element

SigLIP (Sigmoid Loss for Language Image Pre-Training) builds upon CLIP (Contrastive Language-Image Pretraining)—a foundational model that aligns images and text by maximizing the similarity between correct image-text pairs while minimizing it for incorrect ones, leveraging a contrastive loss (Zhai et al., [2023](https://arxiv.org/html/2407.01449v6#bib.bib58 "")). Unlike CLIP (Radford et al., [2021](https://arxiv.org/html/2407.01449v6#bib.bib43 "")), which applies the softmax function to the logits, SigLIP uses the sigmoid activation function. This innovation eliminates the need for a global view of all pairwise similarities between images and texts within a batch, enabling more flexible batch size scaling (up to 1M items per batch, with an effective optimal batch size of 32k). This approach allows SigLIP to achieve state-of-the-art performance in zero-shot image classification tasks.

Report issue for preceding element

### PaliGemma

Report issue for preceding element

PaliGemma is a 3B-parameter vision-language model. It integrates the SigLIP vision encoder with a Gemma-2B language decoder, connected via a multimodal linear projection layer (Lucas Beyer\* et al., [2024](https://arxiv.org/html/2407.01449v6#bib.bib36 "")). The model processes images by segmenting them into a fixed number of Vision Transformer (Dosovitskiy et al., [2020](https://arxiv.org/html/2407.01449v6#bib.bib17 "")) tokens, which are prepended to an optional text prompt.

Report issue for preceding element

A distinguishing feature of PaliGemma is its operation as a Prefix-Language Model (Prefix-LM). This design ensures full attention between image tokens and the user-provided input (prefix) while generating outputs auto-regressively (suffix). This architecture allows image tokens to access the task-specific query during processing, facilitating more effective task-dependent reasoning.

Report issue for preceding element

PaliGemma was trained in four stages: unimodal pretraining with existing components, extended multimodal pretraining, short high-resolution pretraining, and task-specific fine-tuning.

Report issue for preceding element

### ColBERT

Report issue for preceding element

ColBERT (Contextualized Late Interaction over BERT) is a retrieval model designed to balance speed and effectiveness in information retrieval tasks (Khattab & Zaharia, [2020](https://arxiv.org/html/2407.01449v6#bib.bib26 "")). Traditional retrieval models are typically categorized based on their type of interaction: either processing queries and documents independently for efficiency (bi-encoders) or jointly to capture rich contextual relationships (cross-encoders). ColBERT combines the advantages of both approaches through a novel late interaction mechanism.

Report issue for preceding element

Queries and documents are encoded separately using BERT, enabling offline pre-computation of document representations for scalability. Instead of pooling embeddings into a single vector, ColBERT retains token-level embeddings and employs a MaxSim operator to compute fine-grained similarity scores. For each query token, the model determines the maximum similarity with document tokens, summing these scores to compute relevance.

Report issue for preceding element

This architecture preserves the contextual richness of deep language models while significantly improving computational efficiency. By delaying the interaction step, ColBERT supports vector similarity indexing, facilitating end-to-end retrieval from large collections without prohibitive costs. Empirical evaluations on passage search datasets demonstrate that ColBERT achieves competitive effectiveness compared to existing BERT-based models (Devlin et al., [2018](https://arxiv.org/html/2407.01449v6#bib.bib16 "")), while executing queries orders of magnitude faster and with drastically reduced computational requirements.

Report issue for preceding element

## Appendix F Examples from the ViDoRe benchmark

Report issue for preceding element

Energy

Report issue for preceding element

Query: What types of accounts or products allow investors to defer paying taxes?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/energy_1.jpeg)Report issue for preceding element

Query: What is the estimated total savings for a PV system in Durham under the net metering (flat rate) billing option over the system’s useful life of 25 years?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/energy_2.jpeg)Report issue for preceding element

Query: What is the projected peak electricity demand in California for the year 2030?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/energy_3.jpeg)Report issue for preceding element

Report issue for preceding element

Artificial Intelligence

Report issue for preceding element

Query: What are some common outcome areas targeted by TAII for different age groups?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/ai_1.jpeg)Report issue for preceding element

Query: What did the robot monitor to determine when to activate or deactivate the blower motor and blinker?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/ai_2.jpeg)Report issue for preceding element

Query: What is the key approach used in the PDP architecture?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/ai_3.jpeg)Report issue for preceding element

Report issue for preceding element

Healthcare Industry

Report issue for preceding element

Query: What is the chemical formula for the ferroelectric material Lead Zirconium Titanate (PZT)?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/healthcare_1.jpeg)Report issue for preceding element

Query: What government entities are involved in public financing for healthcare in the US?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/healthcare_2.jpeg)Report issue for preceding element

Query: What does the AVPU scale stand for in assessing the level of consciousness of a seriously ill child?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/healthcare_3.jpeg)Report issue for preceding element

Report issue for preceding element

Government Reports

Report issue for preceding element

Query: What are some mandates for the EPA under the Pollution Prevention Act?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/gov_1.jpeg)Report issue for preceding element

Query: What is the strategy of KPMG Hazem Hassan?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/gov_2.jpeg)Report issue for preceding element

Query: What is the trust signal score for the consumer industry best-in-class archetype?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/gov_3.jpeg)Report issue for preceding element

Report issue for preceding element

Shift

Report issue for preceding element

Query: Selon le graphique, quelle est la capacité d’import et la consommation réelle de carburants SAF (biocarburants durables pour l’aviation) prévues en 2050 ?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/shift_1.jpeg)Report issue for preceding element

Query: Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/shift_2.jpeg)Report issue for preceding element

Query: Quels sont les pays ayant la plus grande part des découvertes cumulées de pétrole brut en 2020 (en milliers de barils, hors découvertes cumulées) ?![Refer to caption](https://arxiv.org/html/2407.01449v6/extracted/6240861/images/dataset_samples/shift_3.jpeg)Report issue for preceding element

Report issue for preceding element

Report IssueReport Issue for Selection

Generated by
[L\\
A\\
T\\
Exml![[LOGO]](<Base64-Image-Removed>)](https://math.nist.gov/~BMiller/LaTeXML/)

</details>

<details>
<summary>Document understanding</summary>

# Document understanding

Gemini models can process documents in PDF format, using native
vision to understand entire document contexts. This goes beyond
simple text extraction, allowing Gemini to:

- Analyze and interpret content, including text, images, diagrams,
  charts, and tables, even in long documents up to 1000 pages.
- Extract information into [structured output](https://ai.google.dev/gemini-api/docs/structured-output) formats.
- Summarize and answer questions based on both the visual and textual elements
  in a document.
- Transcribe document content (e.g. to HTML), preserving layouts and
  formatting, for use in downstream applications.

## Passing inline PDF data

You can pass inline PDF data in the request to `generateContent`.
For PDF payloads under 20MB, you can choose between uploading base64
encoded documents or directly uploading locally stored files.

The following example shows you how to fetch a PDF from a URL and convert it to
bytes for processing:

```python
from google import genai
from google.genai import types
import httpx

client = genai.Client()

doc_url = "https://discovery.ucl.ac.uk/id/eprint/10089234/1/343019_3_art_0_py4t4l_convrt.pdf"

# Retrieve and encode the PDF byte
doc_data = httpx.get(doc_url).content

prompt = "Summarize this document"
response = client.models.generate_content(
  model="gemini-2.5-flash",
  contents=[\
      types.Part.from_bytes(\
        data=doc_data,\
        mime_type='application/pdf',\
      ),\
      prompt])
print(response.text)
```

```javascript
import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "GEMINI_API_KEY" });

async function main() {
    const pdfResp = await fetch('https://discovery.ucl.ac.uk/id/eprint/10089234/1/343019_3_art_0_py4t4l_convrt.pdf')
        .then((response) => response.arrayBuffer());

    const contents = [\
        { text: "Summarize this document" },\
        {\
            inlineData: {\
                mimeType: 'application/pdf',\
                data: Buffer.from(pdfResp).toString("base64")\
            }\
        }\
    ];

    const response = await ai.models.generateContent({
        model: "gemini-2.5-flash",
        contents: contents
    });
    console.log(response.text);
}

main();
```

```go
package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "google.golang.org/genai"
)

func main() {

    ctx := context.Background()
    client, _ := genai.NewClient(ctx, &genai.ClientConfig{
        APIKey:  os.Getenv("GEMINI_API_KEY"),
        Backend: genai.BackendGeminiAPI,
    })

    pdfResp, _ := http.Get("https://discovery.ucl.ac.uk/id/eprint/10089234/1/343019_3_art_0_py4t4l_convrt.pdf")
    var pdfBytes []byte
    if pdfResp != nil && pdfResp.Body != nil {
        pdfBytes, _ = io.ReadAll(pdfResp.Body)
        pdfResp.Body.Close()
    }

    parts := []*genai.Part{
        &genai.Part{
            InlineData: &genai.Blob{
                MIMEType: "application/pdf",
                Data:     pdfBytes,
            },
        },
        genai.NewPartFromText("Summarize this document"),
    }

    contents := []*genai.Content{
        genai.NewContentFromParts(parts, genai.RoleUser),
    }

    result, _ := client.Models.GenerateContent(
        ctx,
        "gemini-2.5-flash",
        contents,
        nil,
    )

    fmt.Println(result.Text())
}
```

```bash
DOC_URL="https://discovery.ucl.ac.uk/id/eprint/10089234/1/343019_3_art_0_py4t4l_convrt.pdf"
PROMPT="Summarize this document"
DISPLAY_NAME="base64_pdf"

# Download the PDF
wget -O "${DISPLAY_NAME}.pdf" "${DOC_URL}"

# Check for FreeBSD base64 and set flags accordingly
if [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then
  B64FLAGS="--input"
else
  B64FLAGS="-w0"
fi

# Base64 encode the PDF
ENCODED_PDF=$(base64 $B64FLAGS "${DISPLAY_NAME}.pdf")

# Generate content using the base64 encoded PDF
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"inline_data": {"mime_type": "application/pdf", "data": "'"$ENCODED_PDF"'"}},\
          {"text": "'$PROMPT'"}\
        ]\
      }]
    }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json

# Clean up the downloaded PDF
rm "${DISPLAY_NAME}.pdf"
```

You can also read a PDF from a local file for processing:

```python
from google import genai
from google.genai import types
import pathlib

client = genai.Client()

# Retrieve and encode the PDF byte
filepath = pathlib.Path('file.pdf')

prompt = "Summarize this document"
response = client.models.generate_content(
  model="gemini-2.5-flash",
  contents=[\
      types.Part.from_bytes(\
        data=filepath.read_bytes(),\
        mime_type='application/pdf',\
      ),\
      prompt])
print(response.text)
```

```javascript
import { GoogleGenAI } from "@google/genai";
import * as fs from 'fs';

const ai = new GoogleGenAI({ apiKey: "GEMINI_API_KEY" });

async function main() {
    const contents = [\
        { text: "Summarize this document" },\
        {\
            inlineData: {\
                mimeType: 'application/pdf',\
                data: Buffer.from(fs.readFileSync("content/343019_3_art_0_py4t4l_convrt.pdf")).toString("base64")\
            }\
        }\
    ];

    const response = await ai.models.generateContent({
        model: "gemini-2.5-flash",
        contents: contents
    });
    console.log(response.text);
}

main();
```

```go
package main

import (
    "context"
    "fmt"
    "os"
    "google.golang.org/genai"
)

func main() {

    ctx := context.Background()
    client, _ := genai.NewClient(ctx, &genai.ClientConfig{
        APIKey:  os.Getenv("GEMINI_API_KEY"),
        Backend: genai.BackendGeminiAPI,
    })

    pdfBytes, _ := os.ReadFile("path/to/your/file.pdf")

    parts := []*genai.Part{
        &genai.Part{
            InlineData: &genai.Blob{
                MIMEType: "application/pdf",
                Data:     pdfBytes,
            },
        },
        genai.NewPartFromText("Summarize this document"),
    }
    contents := []*genai.Content{
        genai.NewContentFromParts(parts, genai.RoleUser),
    }

    result, _ := client.Models.GenerateContent(
        ctx,
        "gemini-2.5-flash",
        contents,
        nil,
    )

    fmt.Println(result.Text())
}
```

## Uploading PDFs using the File API

You can use the [File API](https://ai.google.dev/gemini-api/docs/files) to upload larger documents. Always use the File
API when the total request size (including the files, text prompt, system
instructions, etc.) is larger than 20MB.

Call [`media.upload`](https://ai.google.dev/api/rest/v1beta/media/upload) to upload a file using the
File API. The following code uploads a document file and then uses the file in a
call to
[`models.generateContent`](https://ai.google.dev/api/generate-content#method:-models.generatecontent).

### Large PDFs from URLs

Use the File API to simplify uploading and processing large PDF files from URLs:

```python
from google import genai
from google.genai import types
import io
import httpx

client = genai.Client()

long_context_pdf_path = "https://www.nasa.gov/wp-content/uploads/static/history/alsj/a17/A17_FlightPlan.pdf"

# Retrieve and upload the PDF using the File API
doc_io = io.BytesIO(httpx.get(long_context_pdf_path).content)

sample_doc = client.files.upload(
  # You can pass a path or a file-like object here
  file=doc_io,
  config=dict(
    mime_type='application/pdf')
)

prompt = "Summarize this document"

response = client.models.generate_content(
  model="gemini-2.5-flash",
  contents=[sample_doc, prompt])
print(response.text)
```

```javascript
import { createPartFromUri, GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "GEMINI_API_KEY" });

async function main() {

    const pdfBuffer = await fetch("https://www.nasa.gov/wp-content/uploads/static/history/alsj/a17/A17_FlightPlan.pdf")
        .then((response) => response.arrayBuffer());

    const fileBlob = new Blob([pdfBuffer], { type: 'application/pdf' });

    const file = await ai.files.upload({
        file: fileBlob,
        config: {
            displayName: 'A17_FlightPlan.pdf',
        },
    });

    // Wait for the file to be processed.
    let getFile = await ai.files.get({ name: file.name });
    while (getFile.state === 'PROCESSING') {
        getFile = await ai.files.get({ name: file.name });
        console.log(`current file status: ${getFile.state}`);
        console.log('File is still processing, retrying in 5 seconds');

        await new Promise((resolve) => {
            setTimeout(resolve, 5000);
        });
    }
    if (file.state === 'FAILED') {
        throw new Error('File processing failed.');
    }

    // Add the file to the contents.
    const content = [\
        'Summarize this document',\
    ];

    if (file.uri && file.mimeType) {
        const fileContent = createPartFromUri(file.uri, file.mimeType);
        content.push(fileContent);
    }

    const response = await ai.models.generateContent({
        model: 'gemini-2.5-flash',
        contents: content,
    });

    console.log(response.text);

}

main();
```

```go
package main

import (
  "context"
  "fmt"
  "io"
  "net/http"
  "os"
  "google.golang.org/genai"
)

func main() {

  ctx := context.Background()
  client, _ := genai.NewClient(ctx, &genai.ClientConfig{
    APIKey:  os.Getenv("GEMINI_API_KEY"),
    Backend: genai.BackendGeminiAPI,
  })

  pdfURL := "https://www.nasa.gov/wp-content/uploads/static/history/alsj/a17/A17_FlightPlan.pdf"
  localPdfPath := "A17_FlightPlan_downloaded.pdf"

  respHttp, _ := http.Get(pdfURL)
  defer respHttp.Body.Close()

  outFile, _ := os.Create(localPdfPath)
  defer outFile.Close()

  _, _ = io.Copy(outFile, respHttp.Body)

  uploadConfig := &genai.UploadFileConfig{MIMEType: "application/pdf"}
  uploadedFile, _ := client.Files.UploadFromPath(ctx, localPdfPath, uploadConfig)

  promptParts := []*genai.Part{
    genai.NewPartFromURI(uploadedFile.URI, uploadedFile.MIMEType),
    genai.NewPartFromText("Summarize this document"),
  }
  contents := []*genai.Content{
    genai.NewContentFromParts(promptParts, genai.RoleUser), // Specify role
  }

    result, _ := client.Models.GenerateContent(
        ctx,
        "gemini-2.5-flash",
        contents,
        nil,
    )

  fmt.Println(result.Text())
}
```

```bash
PDF_PATH="https://www.nasa.gov/wp-content/uploads/static/history/alsj/a17/A17_FlightPlan.pdf"
DISPLAY_NAME="A17_FlightPlan"
PROMPT="Summarize this document"

# Download the PDF from the provided URL
wget -O "${DISPLAY_NAME}.pdf" "${PDF_PATH}"

MIME_TYPE=$(file -b --mime-type "${DISPLAY_NAME}.pdf")
NUM_BYTES=$(wc -c < "${DISPLAY_NAME}.pdf")

echo "MIME_TYPE: ${MIME_TYPE}"
echo "NUM_BYTES: ${NUM_BYTES}"

tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GOOGLE_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${DISPLAY_NAME}.pdf" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo "file_uri: ${file_uri}"

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"text": "'$PROMPT'"},\
          {"file_data":{"mime_type": "application/pdf", "file_uri": '$file_uri'}}]\
        }]
      }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json

# Clean up the downloaded PDF
rm "${DISPLAY_NAME}.pdf"
```

### Large PDFs stored locally

```python
from google import genai
from google.genai import types
import pathlib
import httpx

client = genai.Client()

# Retrieve and encode the PDF byte
file_path = pathlib.Path('large_file.pdf')

# Upload the PDF using the File API
sample_file = client.files.upload(
  file=file_path,
)

prompt="Summarize this document"

response = client.models.generate_content(
  model="gemini-2.5-flash",
  contents=[sample_file, "Summarize this document"])
print(response.text)
```

```javascript
import { createPartFromUri, GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "GEMINI_API_KEY" });

async function main() {
    const file = await ai.files.upload({
        file: 'path-to-localfile.pdf'
        config: {
            displayName: 'A17_FlightPlan.pdf',
        },
    });

    // Wait for the file to be processed.
    let getFile = await ai.files.get({ name: file.name });
    while (getFile.state === 'PROCESSING') {
        getFile = await ai.files.get({ name: file.name });
        console.log(`current file status: ${getFile.state}`);
        console.log('File is still processing, retrying in 5 seconds');

        await new Promise((resolve) => {
            setTimeout(resolve, 5000);
        });
    }
    if (file.state === 'FAILED') {
        throw new Error('File processing failed.');
    }

    // Add the file to the contents.
    const content = [\
        'Summarize this document',\
    ];

    if (file.uri && file.mimeType) {
        const fileContent = createPartFromUri(file.uri, file.mimeType);
        content.push(fileContent);
    }

    const response = await ai.models.generateContent({
        model: 'gemini-2.5-flash',
        contents: content,
    });

    console.log(response.text);

}

main();
```

```go
package main

import (
    "context"
    "fmt"
    "os"
    "google.golang.org/genai"
)

func main() {

    ctx := context.Background()
    client, _ := genai.NewClient(ctx, &genai.ClientConfig{
        APIKey:  os.Getenv("GEMINI_API_KEY"),
        Backend: genai.BackendGeminiAPI,
    })
    localPdfPath := "/path/to/file.pdf"

    uploadConfig := &genai.UploadFileConfig{MIMEType: "application/pdf"}
    uploadedFile, _ := client.Files.UploadFromPath(ctx, localPdfPath, uploadConfig)

    promptParts := []*genai.Part{
        genai.NewPartFromURI(uploadedFile.URI, uploadedFile.MIMEType),
        genai.NewPartFromText("Give me a summary of this pdf file."),
    }
    contents := []*genai.Content{
        genai.NewContentFromParts(promptParts, genai.RoleUser),
    }

    result, _ := client.Models.GenerateContent(
        ctx,
        "gemini-2.5-flash",
        contents,
        nil,
    )

    fmt.Println(result.Text())
}
```

```bash
NUM_BYTES=$(wc -c < "${PDF_PATH}")
DISPLAY_NAME=TEXT
tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \
  -D "${tmp_header_file}" \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: application/pdf" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the PDF
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${PDF_PATH}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"text": "Can you add a few more lines to this poem?"},\
          {"file_data":{"mime_type": "application/pdf", "file_uri": '$file_uri'}}]\
        }]
      }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json
```

You can verify the API successfully stored the uploaded file and get its
metadata by calling [`files.get`](https://ai.google.dev/api/rest/v1beta/files/get). Only the `name`
(and by extension, the `uri`) are unique.

```python
from google import genai
import pathlib

client = genai.Client()

fpath = pathlib.Path('example.txt')
fpath.write_text('hello')

file = client.files.upload(file='example.txt')

file_info = client.files.get(name=file.name)
print(file_info.model_dump_json(indent=4))
```

```bash
name=$(jq ".file.name" file_info.json)
# Get the file of interest to check state
curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json
# Print some information about the file you got
name=$(jq ".file.name" file_info.json)
echo name=$name
file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri
```

## Passing multiple PDFs

The Gemini API is capable of processing multiple PDF documents (up to 1000 pages)
in a single request, as long as the combined size of the documents and the text
prompt stays within the model's context window.

```python
from google import genai
import io
import httpx

client = genai.Client()

doc_url_1 = "https://arxiv.org/pdf/2312.11805"
doc_url_2 = "https://arxiv.org/pdf/2403.05530"

# Retrieve and upload both PDFs using the File API
doc_data_1 = io.BytesIO(httpx.get(doc_url_1).content)
doc_data_2 = io.BytesIO(httpx.get(doc_url_2).content)

sample_pdf_1 = client.files.upload(
  file=doc_data_1,
  config=dict(mime_type='application/pdf')
)
sample_pdf_2 = client.files.upload(
  file=doc_data_2,
  config=dict(mime_type='application/pdf')
)

prompt = "What is the difference between each of the main benchmarks between these two papers? Output these in a table."

response = client.models.generate_content(
  model="gemini-2.5-flash",
  contents=[sample_pdf_1, sample_pdf_2, prompt])
print(response.text)
```

```javascript
import { createPartFromUri, GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "GEMINI_API_KEY" });

async function uploadRemotePDF(url, displayName) {
    const pdfBuffer = await fetch(url)
        .then((response) => response.arrayBuffer());

    const fileBlob = new Blob([pdfBuffer], { type: 'application/pdf' });

    const file = await ai.files.upload({
        file: fileBlob,
        config: {
            displayName: displayName,
        },
    });

    // Wait for the file to be processed.
    let getFile = await ai.files.get({ name: file.name });
    while (getFile.state === 'PROCESSING') {
        getFile = await ai.files.get({ name: file.name });
        console.log(`current file status: ${getFile.state}`);
        console.log('File is still processing, retrying in 5 seconds');

        await new Promise((resolve) => {
            setTimeout(resolve, 5000);
        });
    }
    if (file.state === 'FAILED') {
        throw new Error('File processing failed.');
    }

    return file;
}

async function main() {
    const content = [\
        'What is the difference between each of the main benchmarks between these two papers? Output these in a table.',\
    ];

    let file1 = await uploadRemotePDF("https://arxiv.org/pdf/2312.11805", "PDF 1")
    if (file1.uri && file1.mimeType) {
        const fileContent = createPartFromUri(file1.uri, file1.mimeType);
        content.push(fileContent);
    }
    let file2 = await uploadRemotePDF("https://arxiv.org/pdf/2403.05530", "PDF 2")
    if (file2.uri && file2.mimeType) {
        const fileContent = createPartFromUri(file2.uri, file2.mimeType);
        content.push(fileContent);
    }

    const response = await ai.models.generateContent({
        model: 'gemini-2.5-flash',
        contents: content,
    });

    console.log(response.text);
}

main();
```

```go
package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "google.golang.org/genai"
)

func main() {

    ctx := context.Background()
    client, _ := genai.NewClient(ctx, &genai.ClientConfig{
        APIKey:  os.Getenv("GEMINI_API_KEY"),
        Backend: genai.BackendGeminiAPI,
    })

    docUrl1 := "https://arxiv.org/pdf/2312.11805"
    docUrl2 := "https://arxiv.org/pdf/2403.05530"
    localPath1 := "doc1_downloaded.pdf"
    localPath2 := "doc2_downloaded.pdf"

    respHttp1, _ := http.Get(docUrl1)
    defer respHttp1.Body.Close()

    outFile1, _ := os.Create(localPath1)
    _, _ = io.Copy(outFile1, respHttp1.Body)
    outFile1.Close()

    respHttp2, _ := http.Get(docUrl2)
    defer respHttp2.Body.Close()

    outFile2, _ := os.Create(localPath2)
    _, _ = io.Copy(outFile2, respHttp2.Body)
    outFile2.Close()

    uploadConfig1 := &genai.UploadFileConfig{MIMEType: "application/pdf"}
    uploadedFile1, _ := client.Files.UploadFromPath(ctx, localPath1, uploadConfig1)

    uploadConfig2 := &genai.UploadFileConfig{MIMEType: "application/pdf"}
    uploadedFile2, _ := client.Files.UploadFromPath(ctx, localPath2, uploadConfig2)

    promptParts := []*genai.Part{
        genai.NewPartFromURI(uploadedFile1.URI, uploadedFile1.MIMEType),
        genai.NewPartFromURI(uploadedFile2.URI, uploadedFile2.MIMEType),
        genai.NewPartFromText("What is the difference between each of the " +
                              "main benchmarks between these two papers? " +
                              "Output these in a table."),
    }
    contents := []*genai.Content{
        genai.NewContentFromParts(promptParts, genai.RoleUser),
    }

    modelName := "gemini-2.5-flash"
    result, _ := client.Models.GenerateContent(
        ctx,
        modelName,
        contents,
        nil,
    )

    fmt.Println(result.Text())
}
```

```bash
DOC_URL_1="https://arxiv.org/pdf/2312.11805"
DOC_URL_2="https://arxiv.org/pdf/2403.05530"
DISPLAY_NAME_1="Gemini_paper"
DISPLAY_NAME_2="Gemini_1.5_paper"
PROMPT="What is the difference between each of the main benchmarks between these two papers? Output these in a table."

# Function to download and upload a PDF
upload_pdf() {
  local doc_url="$1"
  local display_name="$2"

  # Download the PDF
  wget -O "${display_name}.pdf" "${doc_url}"

  local MIME_TYPE=$(file -b --mime-type "${display_name}.pdf")
  local NUM_BYTES=$(wc -c < "${display_name}.pdf")

  echo "MIME_TYPE: ${MIME_TYPE}"
  echo "NUM_BYTES: ${NUM_BYTES}"

  local tmp_header_file=upload-header.tmp

  # Initial resumable request
  curl "${BASE_URL}/upload/v1beta/files?key=${GOOGLE_API_KEY}" \
    -D "${tmp_header_file}" \
    -H "X-Goog-Upload-Protocol: resumable" \
    -H "X-Goog-Upload-Command: start" \
    -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
    -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
    -H "Content-Type: application/json" \
    -d "{'file': {'display_name': '${display_name}'}}" 2> /dev/null

  local upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
  rm "${tmp_header_file}"

  # Upload the PDF
  curl "${upload_url}" \
    -H "Content-Length: ${NUM_BYTES}" \
    -H "X-Goog-Upload-Offset: 0" \
    -H "X-Goog-Upload-Command: upload, finalize" \
    --data-binary "@${display_name}.pdf" 2> /dev/null > "file_info_${display_name}.json"

  local file_uri=$(jq ".file.uri" "file_info_${display_name}.json")
  echo "file_uri for ${display_name}: ${file_uri}"

  # Clean up the downloaded PDF
  rm "${display_name}.pdf"

  echo "${file_uri}"
}

# Upload the first PDF
file_uri_1=$(upload_pdf "${DOC_URL_1}" "${DISPLAY_NAME_1}")

# Upload the second PDF
file_uri_2=$(upload_pdf "${DOC_URL_2}" "${DISPLAY_NAME_2}")

# Now generate content using both files
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"file_data": {"mime_type": "application/pdf", "file_uri": '$file_uri_1'}},\
          {"file_data": {"mime_type": "application/pdf", "file_uri": '$file_uri_2'}},\
          {"text": "'$PROMPT'"}\
        ]\
      }]
    }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json
```

## Technical details

Gemini supports a maximum of 1,000 document pages.
Each document page is equivalent to 258 tokens.

While there are no specific limits to the number of pixels in a document besides
the model's [context window](https://ai.google.dev/gemini-api/docs/long-context), larger pages are
scaled down to a maximum resolution of 3072x3072 while preserving their original
aspect ratio, while smaller pages are scaled up to 768x768 pixels. There is no
cost reduction for pages at lower sizes, other than bandwidth, or performance
improvement for pages at higher resolution.

### Document types

Technically, you can pass other MIME types for document understanding, like
TXT, Markdown, HTML, XML, etc. However, document vision **_only meaningfully_**
**_understands PDFs_**. Other types will be extracted as pure text, and the model
won't be able to interpret what we see in the rendering of those files. Any
file-type specifics like charts, diagrams, HTML tags, Markdown formatting, etc.,
will be lost.

### Best practices

For best results:

- Rotate pages to the correct orientation before uploading.
- Avoid blurry pages.
- If using a single page, place the text prompt after the page.

## What's next

To learn more, see the following resources:

- [File prompting strategies](https://ai.google.dev/gemini-api/docs/files#prompt-guide): The
  Gemini API supports prompting with text, image, audio, and video data, also
  known as multimodal prompting.
- [System instructions](https://ai.google.dev/gemini-api/docs/text-generation#system-instructions):
  System instructions let you steer the behavior of the model based on your
  specific needs and use cases.

</details>

<details>
<summary>Get multimodal embeddings</summary>

# Get multimodal embeddings

The multimodal embeddings model generates 1408-dimension vectors based on the input you provide, which can include a combination of image, text, and video data. The embedding vectors can then be used for subsequent tasks like image classification or video content moderation.

The image embedding vector and text embedding vector are in the same semantic space with the same dimensionality. Consequently, these vectors can be used interchangeably for use cases like searching image by text, or searching video by image.

For text-only embedding use cases, we recommend using the Vertex AI text-embeddings API instead. For example, the text-embeddings API might be better for text-based semantic search, clustering, long-form document analysis, and other text retrieval or question-answering use cases. For more information, see Get text embeddings.

## Supported models

You can get multimodal embeddings by using the following model:

- `multimodalembedding`

## Best practices

Consider the following input aspects when using the multimodal embeddings model:

- **Text in images** - The model can distinguish text in images, similar to optical character recognition (OCR). If you need to distinguish between a description of the image content and the text within an image, consider using prompt engineering to specify your target content. For example: instead of just "cat", specify "picture of a cat" or "the text 'cat'", depending on your use case.

|     |     |
| --- | --- |
| **the text** 'cat' | https://cloud.google.com/static/vertex-ai/generative-ai/docs/image/images/txt-embedding-cat.png |
| **picture of** a cat | https://cloud.google.com/static/vertex-ai/generative-ai/docs/image/images/img-embedding-cat.jpg_Image credit: [Manja Vitolic](https://unsplash.com/photos/gKXKBY-C-Dk)_<br>_on [Unsplash](https://unsplash.com/)._ |

- **Embedding similarities** - The dot product of embeddings isn't a calibrated probability. The dot product is a similarity metric and might have different score distributions for different use cases. Consequently, avoid using a fixed value threshold to measure quality. Instead, use ranking approaches for retrieval, or use sigmoid for classification.

## API usage

### API limits

The following limits apply when you use the `multimodalembedding` model for text and image embeddings:

| Limit | Value and description |
| --- | --- |
| Maximum number of API requests per minute per project | 120-600 depending on region |
| Maximum text length | 32 tokens (~32 words). The maximum text length is 32 tokens (approximately 32 words). If the input exceeds 32 tokens, the model internally shortens the input to this length. |
| Language | English |
| Image formats | BMP, GIF, JPG, PNG |
| Image size | Base64-encoded images: 20 MB (when transcoded to PNG). Cloud Storage images: 20MB (original file format). The maximum image size accepted is 20 MB. To avoid increased network latency, use smaller images. Additionally, the model resizes images to 512 x 512 pixel resolution. Consequently, you don't need to provide higher resolution images. |
| Audio supported (video) | N/A - The model doesn't consider audio content when generating video embeddings |
| Video formats | AVI, FLV, MKV, MOV, MP4, MPEG, MPG, WEBM, and WMV |
| Maximum video length (Cloud Storage) | No limit. However, only 2 minutes of content can be analyzed at a time. |

### Before you begin

- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
- In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
- Verify that billing is enabled for your Google Cloud project.
- Enable the Vertex AI API.
- Set up authentication for your environment (Application Default Credentials) and install the Google Cloud CLI if needed.

To use the language-specific samples on this page, choose the appropriate tab: Java, Node.js, Python, REST, Go. Each sample includes notes about setting up authentication and any additional client libraries.

### Locations

A location is a region you can specify in a request to control where data is stored at rest. For a list of available regions, see Generative AI on Vertex AI locations.

### Error messages

#### Quota exceeded error

```
google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for
aiplatform.googleapis.com/online_prediction_requests_per_base_model with base
model: multimodalembedding. Please submit a quota increase request.
```

If this is the first time you receive this error, use the Google Cloud console to request a quota adjustment for your project. Use the following filters before requesting your adjustment:

- Service ID: aiplatform.googleapis.com
- metric: aiplatform.googleapis.com/online_prediction_requests_per_base_model
- base_model:multimodalembedding

### Specify lower-dimension embeddings

By default an embedding request returns a 1408 float vector for a data type. You can also specify lower-dimension embeddings (128, 256, or 512 float vectors) for text and image data. This option lets you optimize for latency and storage or quality based on how you plan to use the embeddings. Lower-dimension embeddings provide decreased storage needs and lower latency for subsequent embedding tasks (like search or recommendation), while higher-dimension embeddings offer greater accuracy for the same tasks.

Low-dimension can be accessed by adding the `parameters.dimension` field. The parameter accepts one of the following values: `128`, `256`, `512` or `1408`. The response includes the embedding of that dimension.

Before using any of the request data, make the following replacements:

- LOCATION: Your project's region. For example, `us-central1`, `europe-west2`, or `asia-northeast3`.
- PROJECT_ID: Your Google Cloud project ID.
- IMAGE_URI: The Cloud Storage URI of the target image to get embeddings for, for example `gs://my-bucket/embeddings/supermarket-img.png`.

You can also provide the image as a base64-encoded byte string.

HTTP method and URL:

```
POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/multimodalembedding@001:predict
```

Request JSON body:

```
{
  "instances": [
    {
      "image": {
        "gcsUri": "IMAGE_URI"
      },
      "text": "TEXT"
    }
  ],
  "parameters": {
    "dimension": EMBEDDING_DIMENSION
  }
}
```

To send your request using curl:

```
curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/multimodalembedding@001:predict"
```

Sample JSON response (shortened):

**128 dimensions:**

```
{
  "predictions": [
    {
      "imageEmbedding": [ 0.0279239565, [...128 dimension vector...], 0.00403284049 ],
      "textEmbedding": [ 0.202921599, [...128 dimension vector...], -0.0365431122 ]
    }
  ],
  "deployedModelId": "DEPLOYED_MODEL_ID"
}
```

**256 dimensions** and **512 dimensions** sample responses follow the same schema with longer vectors.

## Send an embedding request (image and text)

Use the following code samples to send an embedding request with image and text data. The samples show how to send a request with both data types, but you can also use the service with an individual data type.

### Python example (lower-dimension)

```python
import vertexai

from vertexai.vision_models import Image, MultiModalEmbeddingModel

# TODO(developer): Update & uncomment line below
# PROJECT_ID = "your-project-id"
vertexai.init(project=PROJECT_ID, location="us-central1")

# TODO(developer): Try different dimenions: 128, 256, 512, 1408
embedding_dimension = 128

model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
image = Image.load_from_file(
    "gs://cloud-samples-data/vertex-ai/llm/prompts/landmark1.png"
)

embeddings = model.get_embeddings(
    image=image,
    contextual_text="Colosseum",
    dimension=embedding_dimension,
)

print(f"Image Embedding: {embeddings.image_embedding}")
print(f"Text Embedding: {embeddings.text_embedding}")

# Example response:
# Image Embedding: [0.0622573346, -0.0406507477, 0.0260440577, ...]
# Text Embedding: [0.27469793, -0.146258667, 0.0222803634, ...]
```

### Go example (lower-dimension)

```go
import (
	"context"
	"encoding/json"
	"fmt"
	"io"

	aiplatform "cloud.google.com/go/aiplatform/apiv1beta1"
	aiplatformpb "cloud.google.com/go/aiplatform/apiv1beta1/aiplatformpb"
	"google.golang.org/api/option"
	"google.golang.org/protobuf/encoding/protojson"
	"google.golang.org/protobuf/types/known/structpb"
)

// generateWithLowerDimension shows how to generate lower-dimensional embeddings for text and image inputs.
func generateWithLowerDimension(w io.Writer, project, location string) error {
	ctx := context.Background()
	apiEndpoint := fmt.Sprintf("%s-aiplatform.googleapis.com:443", location)
	client, err := aiplatform.NewPredictionClient(ctx, option.WithEndpoint(apiEndpoint))
	if err != nil {
		return fmt.Errorf("failed to construct API client: %w", err)
	}
	defer client.Close()

	model := "multimodalembedding@001"
	endpoint := fmt.Sprintf("projects/%s/locations/%s/publishers/google/models/%s", project, location, model)

	instance, err := structpb.NewValue(map[string]any{
		"image": map[string]any{
			"gcsUri": "gs://cloud-samples-data/vertex-ai/llm/prompts/landmark1.png",
		},
		"text": "Colosseum",
	})
	if err != nil {
		return fmt.Errorf("failed to construct request payload: %w", err)
	}

	// TODO(developer): Try different dimenions: 128, 256, 512, 1408
	outputDimensionality := 128
	params, err := structpb.NewValue(map[string]any{
		"dimension": outputDimensionality,
	})
	if err != nil {
		return fmt.Errorf("failed to construct request params: %w", err)
	}

	req := &aiplatformpb.PredictRequest{
		Endpoint: endpoint,
		// The model supports only 1 instance per request.
		Instances:  []*structpb.Value{instance},
		Parameters: params,
	}

	resp, err := client.Predict(ctx, req)
	if err != nil {
		return fmt.Errorf("failed to generate embeddings: %w", err)
	}

	instanceEmbeddingsJson, err := protojson.Marshal(resp.GetPredictions()[0])
	if err != nil {
		return fmt.Errorf("failed to convert protobuf value to JSON: %w", err)
	}
	var instanceEmbeddings struct {
		ImageEmbeddings []float32 `json:"imageEmbedding"`
		TextEmbeddings  []float32 `json:"textEmbedding"`
	}
	if err := json.Unmarshal(instanceEmbeddingsJson, &instanceEmbeddings); err != nil {
		return fmt.Errorf("failed to unmarshal JSON: %w", err)
	}

	imageEmbedding := instanceEmbeddings.ImageEmbeddings
	textEmbedding := instanceEmbeddings.TextEmbeddings

	fmt.Fprintf(w, "Text embedding (length=%d): %v\n", len(textEmbedding), textEmbedding)
	fmt.Fprintf(w, "Image embedding (length=%d): %v\n", len(imageEmbedding), imageEmbedding)
	// Example response:
	// Text Embedding (length=128): [0.27469793 -0.14625867 0.022280363 ... ]
	// Image Embedding (length=128): [0.06225733 -0.040650766 0.02604402 ... ]

	return nil
}
```

## Send an embedding request (video, image, or text)

When sending an embedding request you can specify an input video alone, or you can specify a combination of video, image, and text data.

### Video embedding modes

There are three modes you can use with video embeddings: Essential, Standard, or Plus. The mode corresponds to the density of the embeddings generated, which can be specified by the `interval_sec` config in the request. For each video interval with `interval_sec` length, an embedding is generated. The minimal video interval length is 4 seconds. Interval lengths greater than 120 seconds might negatively affect the quality of the generated embeddings.

Pricing for video embedding depends on the mode you use.

| Mode | Maximum number of embeddings per minute | Video embedding interval (minimum value) |
| --- | --- | --- |
| Essential | 4 | 15 (intervalSec >= 15) |
| Standard | 8 | 8 (8 <= intervalSec < 15) |
| Plus | 15 | 4 (4 <= intervalSec < 8) |

### Video embeddings best practices

- To generate a single embedding for the first two minutes of an input video of any length, use the following `videoSegmentConfig` setting:

```
"videoSegmentConfig": {
    "intervalSec": 120
}
```

- To generate embedding for a video with a length greater than two minutes, you can send multiple requests that specify the start and end times in the `videoSegmentConfig`:

Example for multiple segments:

```
"videoSegmentConfig": {
    "startOffsetSec": 0,
    "endOffsetSec": 120
}
```

and

```
"videoSegmentConfig": {
    "startOffsetSec": 120,
    "endOffsetSec": 240
}
```

If you omit `videoSegmentConfig`, the service uses the following default values:

```
"videoSegmentConfig": { "startOffsetSec": 0, "endOffsetSec": 120, "intervalSec": 16 }
```

This default interval (16 seconds) falls in the Essential video embedding mode.

### Get video embeddings (HTTP)

HTTP method and URL:

```
POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/multimodalembedding@001:predict
```

Request JSON body for video:

```
{
  "instances": [
    {
      "video": {
        "gcsUri": "VIDEO_URI",
        "videoSegmentConfig": {
          "startOffsetSec": START_SECOND,
          "endOffsetSec": END_SECOND,
          "intervalSec": INTERVAL_SECONDS
        }
      }
    }
  ]
}
```

Sample response (shortened) with videoEmbeddings array and per-segment embeddings.

## Examples: Get image, text, and video embeddings (Python)

```python
import vertexai

from vertexai.vision_models import Image, MultiModalEmbeddingModel, Video
from vertexai.vision_models import VideoSegmentConfig

# TODO(developer): Update & uncomment line below
# PROJECT_ID = "your-project-id"
vertexai.init(project=PROJECT_ID, location="us-central1")

model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")

image = Image.load_from_file(
    "gs://cloud-samples-data/vertex-ai/llm/prompts/landmark1.png"
)
video = Video.load_from_file(
    "gs://cloud-samples-data/vertex-ai-vision/highway_vehicles.mp4"
)

embeddings = model.get_embeddings(
    image=image,
    video=video,
    video_segment_config=VideoSegmentConfig(end_offset_sec=1),
    contextual_text="Cars on Highway",
)

print(f"Image Embedding: {embeddings.image_embedding}")

# Video Embeddings are segmented based on the video_segment_config.
print("Video Embeddings:")
for video_embedding in embeddings.video_embeddings:
    print(
        f"Video Segment: {video_embedding.start_offset_sec} - {video_embedding.end_offset_sec}"
    )
    print(f"Embedding: {video_embedding.embedding}")

print(f"Text Embedding: {embeddings.text_embedding}")
# Example response:
# Image Embedding: [-0.0123144267, 0.0727186054, ...]
# Video Embeddings:
# Video Segment: 0.0 - 1.0
# Embedding: [-0.0206376351, 0.0345234685, ...]
# Text Embedding: [-0.0207006838, -0.00251058186, ...]
```

## Send an embedding request (image and text) - HTTP request example

Request JSON body:

```
{
  "instances": [
    {
      "text": "TEXT",
      "image": {
        "bytesBase64Encoded": "B64_ENCODED_IMG"
      }
    }
  ]
}
```

To send your request, use curl or PowerShell as shown earlier.

Sample response (shortened):

```
{
  "predictions": [
    {
      "textEmbedding": [...],
      "imageEmbedding": [...],
    }
  ],
  "deployedModelId": "DEPLOYED_MODEL_ID"
}
```

## Get text and image embeddings (additional code examples)

The page includes language-specific client library examples (Node.js, Java, Go) showing how to construct the request, base64-encode images, and call the Prediction API. The same request and response schema apply across languages: include `text`, `image` (either `gcsUri` or `bytesBase64Encoded`), and optional `parameters.dimension`.

## What's next

- Read the blog "What is Multimodal Search: 'LLMs with vision' change businesses".
- For information about text-only use cases (text-based semantic search, clustering, long-form document analysis, and other text retrieval or question-answering use cases), read Get text embeddings.
- View all Vertex AI image generative AI offerings in the Imagen on Vertex AI overview.
- Explore more pretrained models in Model Garden.
- Learn about responsible AI best practices and safety filters in Vertex AI.

</details>

<details>
<summary>HTML conversions [sometimes display errors](https://info.dev.arxiv.org/about/accessibility_html_error_messages.html) due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.</summary>

HTML conversions [sometimes display errors](https://info.dev.arxiv.org/about/accessibility_html_error_messages.html) due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.

- failed: arydshln
- failed: arydshln

Authors: achieve the best HTML results from your LaTeX submissions by following these [best practices](https://info.arxiv.org/help/submit_latex_best_practices.html).

[License: CC BY 4.0](https://info.arxiv.org/help/license/index.html#licenses-available)

arXiv:2409.11402v1 \[cs.CL\] 17 Sep 2024

††∗ Equal contributions, listed alphabetically.††† Leads the effort.††1NVIDIA Vision Language Model.

# NVLM: Open Frontier-Class Multimodal LLMs

Report issue for preceding element

Wenliang Dai∗  Nayeon Lee∗  Boxin Wang∗  Zhuoling Yang∗Zihan Liu   Jon Barker   Tuomas Rintamaki   Mohammad Shoeybi   Bryan Catanzaro
Wei Ping∗,†

NVIDIA

∗{wdai, nayeonl, boxinw, zhuoliny, wping}@nvidia.com

Report issue for preceding element

###### Abstract

Report issue for preceding element

We introduce NVLM 1.0, 1 a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training.

Report issue for preceding element

In terms of model design, we perform a comprehensive comparison between decoder-only multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g., Flamingo). Based on the strengths and weaknesses of both approaches, we propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities.
Furthermore, we introduce a 1-D tile-tagging design for tile-based dynamic high-resolution images, which significantly boosts performance on multimodal reasoning and OCR-related tasks.
Regarding training data, we meticulously curate and provide detailed information on our multimodal pretraining and supervised fine-tuning datasets. Our findings indicate that dataset quality and task diversity are more important than scale, even during the pretraining phase, across all architectures.
Notably, we develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks while maintaining and even improving text-only performance compared to their LLM backbones.
To achieve this, we craft and integrate a high-quality text-only dataset into multimodal training, alongside a substantial amount of multimodal math and reasoning data, leading to enhanced math and coding capabilities across modalities.

Report issue for preceding element

To advance research in the field, we are releasing the model weights and will open-source the code for the community: [https://nvlm-project.github.io/](https://nvlm-project.github.io/ "").

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x1.png)

Report issue for preceding element

Figure 1: Qualitative examples generated by our NVLM-D1.072B model. We demonstrate diverse capabilities of our model, including chart and table understanding, OCR, localization, knowledge-grounded image description, humorous meme understanding, scene understanding, math reasoning and coding capabilities. For more examples, refer to Appendix [A](https://arxiv.org/html/2409.11402v1#A1 "Appendix A Qualitative Examples from the NVLM-1.0-D 72B Model ‣ NVLM: Open Frontier-Class Multimodal LLMs").
Report issue for preceding element

## 1 Introduction

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x2.png)

Report issue for preceding elementFigure 2: NVLM 1.0 _versus_ leading proprietary and open-access multimodal LLMs. Note that the model weights for ∗Llama 3-V have not been released as of the time of this report.
The results demonstrate that NVLM 1.0 achieves performance on par with leading models across both vision-language and text-only tasks.
Additionally, we compare multimodal LLM to its backbone LLM on text-only tasks. Llama 3-V 70B and 405B show no degradation in text-only tasks, as their LLM backbones are frozen during multimodal training. In contrast, our NVLM-D1.0 72B model demonstrates significant improvements over its text backbone on text-only math and coding benchmarks, with average accuracy increasing by 4.3 points after multimodal training.
See Table [7](https://arxiv.org/html/2409.11402v1#S6.T7 "Table 7 ‣ 6.2 Baseline Models ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs") and Table [8](https://arxiv.org/html/2409.11402v1#S6.T8 "Table 8 ‣ 6.3 Main Results ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs") for full details.
Report issue for preceding element

Large language models (LLMs) \[ [11](https://arxiv.org/html/2409.11402v1#bib.bib11 "")\] have laid the foundation for the rapid progress in AI recently.
Since the introduction of ChatGPT \[ [104](https://arxiv.org/html/2409.11402v1#bib.bib104 "")\], LLMs have revolutionized the text domain and are becoming universal task solvers for natural language processing, math and coding problems.
Simultaneously, multimodal LLMs (MLLMs) \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 ""); [107](https://arxiv.org/html/2409.11402v1#bib.bib107 "")\], which bridge the physical world with language models, have gained significant traction.
The release of GPT-4V \[ [107](https://arxiv.org/html/2409.11402v1#bib.bib107 "")\] has sparked a competitive race in the development of proprietary multimodal LLMs for vision-language intelligence \[ [35](https://arxiv.org/html/2409.11402v1#bib.bib35 ""); [108](https://arxiv.org/html/2409.11402v1#bib.bib108 ""); [5](https://arxiv.org/html/2409.11402v1#bib.bib5 ""); [6](https://arxiv.org/html/2409.11402v1#bib.bib6 ""); [153](https://arxiv.org/html/2409.11402v1#bib.bib153 ""); [154](https://arxiv.org/html/2409.11402v1#bib.bib154 ""); [122](https://arxiv.org/html/2409.11402v1#bib.bib122 "")\].
However, the model architectures, training data, and methods used to build these proprietary models remain undisclosed, preventing the research community from building upon them.

Report issue for preceding element

A notable feature of leading proprietary multimodal LLMs is their exceptional performance on both multimodal and text-only tasks, a quality we refer to as _production-grade multimodality_\[ [108](https://arxiv.org/html/2409.11402v1#bib.bib108 ""); [35](https://arxiv.org/html/2409.11402v1#bib.bib35 ""); [36](https://arxiv.org/html/2409.11402v1#bib.bib36 "")\].
For example, GPT-4o is a single neural network trained end-to-end on text and images, achieving state-of-the-art results in both text-only and vision-language tasks \[ [110](https://arxiv.org/html/2409.11402v1#bib.bib110 "")\].
This unified approach simplifies deployment by eliminating the need to route different input modalities to separate LLMs, offering users a seamless experience for switching between modalities without losing text or multimodal context.

Report issue for preceding element

The community has made significant progress in advancing the capabilities of open-access multimodal LLMs \[ [26](https://arxiv.org/html/2409.11402v1#bib.bib26 ""); [79](https://arxiv.org/html/2409.11402v1#bib.bib79 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [71](https://arxiv.org/html/2409.11402v1#bib.bib71 ""); [139](https://arxiv.org/html/2409.11402v1#bib.bib139 "")\]. Notable families of open models include BLIP \[ [66](https://arxiv.org/html/2409.11402v1#bib.bib66 ""); [67](https://arxiv.org/html/2409.11402v1#bib.bib67 ""); [26](https://arxiv.org/html/2409.11402v1#bib.bib26 "")\], LLaVA \[ [79](https://arxiv.org/html/2409.11402v1#bib.bib79 ""); [78](https://arxiv.org/html/2409.11402v1#bib.bib78 ""); [80](https://arxiv.org/html/2409.11402v1#bib.bib80 ""); [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\], InternVL \[ [19](https://arxiv.org/html/2409.11402v1#bib.bib19 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\], and Llama 3-V \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\].
The most common architectures used to build these multimodal LLMs are the decoder-only architecture (e.g., LLaVA \[ [79](https://arxiv.org/html/2409.11402v1#bib.bib79 "")\] and InternVL \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\]), which processes image tokens within the LLM self-attention layers, and the cross-attention-based architecture (e.g., Flamingo \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 "")\] and Llama 3-V \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\]), which handles image tokens through LLM cross-attention layers.

Report issue for preceding element

However, the previous studies of multimodal LLMs have several limitations:

Report issue for preceding element

1. ∙∙\\bullet∙


In contrast to the convergence of model architectures to build LLM in the text domain \[ [11](https://arxiv.org/html/2409.11402v1#bib.bib11 ""); [133](https://arxiv.org/html/2409.11402v1#bib.bib133 ""); [105](https://arxiv.org/html/2409.11402v1#bib.bib105 ""); [5](https://arxiv.org/html/2409.11402v1#bib.bib5 ""); [35](https://arxiv.org/html/2409.11402v1#bib.bib35 "")\], i.e., the decoder-only transformer \[ [143](https://arxiv.org/html/2409.11402v1#bib.bib143 "")\], existing multimodal LLM architectures (e.g., decoder-only vs. cross-attention models) have not been studied and compared in an apples-to-apples manner. There is no information regarding the architectures of proprietary models. Furthermore, studies on open-access models differ in their choice of LLM backbones, vision encoders, and, most importantly, training data, making direct comparisons challenging. For these reasons, IDEFICS-80B, an open-access reproduction of Flamingo \[ [62](https://arxiv.org/html/2409.11402v1#bib.bib62 "")\] based on LLaMA-65B \[ [140](https://arxiv.org/html/2409.11402v1#bib.bib140 "")\], is perceived as significantly lagging behind LLaVA-1.5-13B \[ [78](https://arxiv.org/html/2409.11402v1#bib.bib78 "")\], which is based on Vicuna-13B \[ [21](https://arxiv.org/html/2409.11402v1#bib.bib21 "")\], in VQA tasks.


Report issue for preceding element

2. ∙∙\\bullet∙


Model designs that handle high-resolution image input (e.g, dynamic high-resolution \[ [80](https://arxiv.org/html/2409.11402v1#bib.bib80 ""); [30](https://arxiv.org/html/2409.11402v1#bib.bib30 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\]) significantly boost performance on OCR-related tasks (e.g., OCRBench \[ [81](https://arxiv.org/html/2409.11402v1#bib.bib81 "")\]), but sometimes show reduced accuracy on reasoning-related tasks (e.g., MMMU \[ [166](https://arxiv.org/html/2409.11402v1#bib.bib166 "")\]) compared to their low-resolution counterparts.

Report issue for preceding element

3. ∙∙\\bullet∙


Although open-access multimodal LLMs achieve impressive benchmark results on vision-language tasks, we observe a significant degradation in text-only performance (see Table [8](https://arxiv.org/html/2409.11402v1#S6.T8 "Table 8 ‣ 6.3 Main Results ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs")), unlike leading proprietary models (e.g., GPT-4o).
The only work that provides substantial technical details addressing this issue is Llama 3-V \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\], which freezes the LLM parameters and trains only the cross-attention layers. However, these models have not yet been made publicly available.

Report issue for preceding element


To address these limitations, we introduce NVLM-1.0, a family of frontier multimodal LLMs (see Figure [2](https://arxiv.org/html/2409.11402v1#S1.F2 "Figure 2 ‣ 1 Introduction ‣ NVLM: Open Frontier-Class Multimodal LLMs") for a comparison with leading models) featuring three distinct architectures: _i)_ NVLM-D, a Decoder-only architecture, _ii)_ NVLM-X, a cross (X)-attention-based architecture, and _iii)_ NVLM-H, a novel Hybrid architecture.
Trained on the same curated data blend, all three architectures achieve state-of-the-art performance, rivaling leading proprietary and open-access models, while offering practitioners flexible and feature-rich model options. Specifically, we make the following contributions:

Report issue for preceding element

1. 1.


Model architecture:
We compare the pros and cons of the decoder-only and the cross-attention-based models using the same LLM backbones, vision encoder, and well-curated training data. Our findings show that the cross-attention-based NVLM-X offers superior computational efficiency when handling high-resolution images, whereas the decoder-only NVLM-D provides unified multimodal reasoning and achieves higher accuracy in OCR-related tasks. Building on these insights, we propose NVLM-H, a novel hybrid architecture that excels in multimodal reasoning while also delivering improved computational efficiency for high-resolution images.

Report issue for preceding element

2. 2.


High-resolution:
To achieve strong accuracy on both OCR-related tasks (e.g., OCRBench \[ [81](https://arxiv.org/html/2409.11402v1#bib.bib81 "")\]) and multimodal reasoning tasks (e.g., MMMU \[ [166](https://arxiv.org/html/2409.11402v1#bib.bib166 "")\]), we propose a tile-tagging design for the dynamic tiling of high-resolution image inputs. Through comprehensive ablation studies, we find that adding a text-based 1-D tile tag before the image tokens of the corresponding tile in the decoder achieves the best accuracy.

Report issue for preceding element

3. 3.


Training data:
We meticulously collect and provide detailed information on our multimodal pretraining and supervised fine-tuning (SFT) datasets, which will support and benefit future research.
In the dataset selection and filtering process, we find that _the data quality and task diversity are more important than the scale, even during the pretraining stage_. Furthermore, previous studies have shown that abundant and diverse multimodal pretraining data is crucial for the success of cross-attention-based models, such as Flamingo \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 "")\]. In this work, we found that such pretraining data can also significantly improve the performance of decoder-only models, like LLaVA \[ [78](https://arxiv.org/html/2409.11402v1#bib.bib78 "")\], even with a simplified design that involves training only an MLP projection layer during pretraining.
For the curation of SFT data, we collected a much larger set of task-oriented datasets compared to previous studies \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\].

Report issue for preceding element

4. 4.


Production-grade multimodality:
We develop _production-grade multimodality_ for NVLM models, enabling them to excel in both vision-language tasks (e.g., multimodal reasoning, OCR, natural image understanding) and text-only tasks (e.g., multidisciplinary knowledge reasoning, coding, and math).
To maintain text-only performance during multimodal training, we investigate two approaches:
_i)_ For the cross-attention-based NVLM-X, we find that freezing the LLM’s parameters and training only the cross-attention layers \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 "")\] during both the pretraining and SFT stages works reasonably well, with a moderate performance trade-off on vision-language tasks.
_ii)_ We curate a high-quality text-only dataset and integrate it into the multimodal SFT stage, effectively preserving text-only performance with no degradation, and even achieving noticeable improvements on text-only math and coding benchmarks after multimodal training across all NVLM models.
We attribute this to the superb quality of text-only data and the significant amount of multimodal math data (e.g., geometry) incorporated into multimdoal SFT blend, which improves NVLM’s reasoning capabilities, regardless of modality.

Report issue for preceding element


We organize the rest of this paper as follows.
In § [2](https://arxiv.org/html/2409.11402v1#S2 "2 Qualitative Study ‣ NVLM: Open Frontier-Class Multimodal LLMs"), we present a qualitative study of our model’s capabilities, showcasing generated samples.
In § [3](https://arxiv.org/html/2409.11402v1#S3 "3 Preliminaries ‣ NVLM: Open Frontier-Class Multimodal LLMs"), we introduce the preliminaries of multimodal LLMs and discuss related work.
In § [4](https://arxiv.org/html/2409.11402v1#S4 "4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs"), we present the NVLM-1.0 model family, followed by details on the training data in § [5](https://arxiv.org/html/2409.11402v1#S5 "5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs").
We introduce the evaluation benchmarks and report results in § [6](https://arxiv.org/html/2409.11402v1#S6 "6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs").
We conclude the paper in § [7](https://arxiv.org/html/2409.11402v1#S7 "7 Conclusion ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

## 2 Qualitative Study

Report issue for preceding element

We conduct a qualitative analysis of NVLM-1.0 with diverse images and instructions.
As illustrated in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs"),
NVLM-1.0 can handle diverse types of images including memes in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (a), object-centric images in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (b), real-world scene images in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (c), hand-written pseudo code in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (d), table in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (e), and charts in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (f).

Report issue for preceding element

Our NVLM-D1.072B demonstrates versatile capabilities in various multimodal tasks by jointly utilizing OCR, reasoning, localization, common sense, world knowledge, and coding ability.
For instance, our model can understand the humor behind the “abstract vs. paper” meme in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (a) by performing OCR to recognize the text labels for each image and using reasoning to grasp why
juxtaposing “the abstract” — labeled with a fierce-looking lynx — and “the paper” — labeled with a domestic cat — is humorous.
NVLM accurately performs localization to effectively answer location-sensitive questions, such as “What is the difference between the left, middle, and right objects in the image?” in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (b).
NVLM is capable of performing mathematical reasoning and coding based on visual information, such as tables and handwritten pseudocode, as illustrated in Figure [1](https://arxiv.org/html/2409.11402v1#S0.F1 "Figure 1 ‣ NVLM: Open Frontier-Class Multimodal LLMs") (d) and (e).
For more examples, refer to Appendix [A](https://arxiv.org/html/2409.11402v1#A1 "Appendix A Qualitative Examples from the NVLM-1.0-D 72B Model ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

## 3 Preliminaries

Report issue for preceding element

Vision language models \[ [120](https://arxiv.org/html/2409.11402v1#bib.bib120 ""); [9](https://arxiv.org/html/2409.11402v1#bib.bib9 ""); [149](https://arxiv.org/html/2409.11402v1#bib.bib149 ""); [4](https://arxiv.org/html/2409.11402v1#bib.bib4 ""); [146](https://arxiv.org/html/2409.11402v1#bib.bib146 ""); [174](https://arxiv.org/html/2409.11402v1#bib.bib174 ""); [17](https://arxiv.org/html/2409.11402v1#bib.bib17 ""); [145](https://arxiv.org/html/2409.11402v1#bib.bib145 ""); [169](https://arxiv.org/html/2409.11402v1#bib.bib169 "")\] build the connection between the visual world and open text domain.
Among these works, the multimodal LLMs augmented from pretrained large language models (LLMs) \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 ""); [7](https://arxiv.org/html/2409.11402v1#bib.bib7 ""); [158](https://arxiv.org/html/2409.11402v1#bib.bib158 ""); [66](https://arxiv.org/html/2409.11402v1#bib.bib66 ""); [67](https://arxiv.org/html/2409.11402v1#bib.bib67 ""); [26](https://arxiv.org/html/2409.11402v1#bib.bib26 ""); [79](https://arxiv.org/html/2409.11402v1#bib.bib79 ""); [78](https://arxiv.org/html/2409.11402v1#bib.bib78 ""); [80](https://arxiv.org/html/2409.11402v1#bib.bib80 ""); [65](https://arxiv.org/html/2409.11402v1#bib.bib65 ""); [19](https://arxiv.org/html/2409.11402v1#bib.bib19 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [111](https://arxiv.org/html/2409.11402v1#bib.bib111 ""); [71](https://arxiv.org/html/2409.11402v1#bib.bib71 ""); [147](https://arxiv.org/html/2409.11402v1#bib.bib147 ""); [8](https://arxiv.org/html/2409.11402v1#bib.bib8 ""); [15](https://arxiv.org/html/2409.11402v1#bib.bib15 ""); [139](https://arxiv.org/html/2409.11402v1#bib.bib139 ""); [159](https://arxiv.org/html/2409.11402v1#bib.bib159 "")\] have become visual assistants and universal task solvers for various vision-language tasks, including image / video captioning \[ [72](https://arxiv.org/html/2409.11402v1#bib.bib72 ""); [157](https://arxiv.org/html/2409.11402v1#bib.bib157 "")\], visual understanding and reasoning \[ [166](https://arxiv.org/html/2409.11402v1#bib.bib166 "")\], chart and diagram-related QA \[ [93](https://arxiv.org/html/2409.11402v1#bib.bib93 "")\], math reasoning in visual context \[ [87](https://arxiv.org/html/2409.11402v1#bib.bib87 "")\], and optical character recognition (OCR) \[ [81](https://arxiv.org/html/2409.11402v1#bib.bib81 "")\].

Report issue for preceding element

### 3.1 Essential Building Blocks

Report issue for preceding element

Multimodal LLM typically consists of two indispensable components: large language model (LLM) and vision encoder.

Report issue for preceding element

#### Large Languge Model

Report issue for preceding element

A multimodal LLM typically builds upon a text-only LLM for initialization. While there are exceptions where multimodal LLMs are pretrained from scratch using multimodal data \[ [35](https://arxiv.org/html/2409.11402v1#bib.bib35 ""); [3](https://arxiv.org/html/2409.11402v1#bib.bib3 "")\], these approaches, though conceptually compelling, lack clear evidence of superior performance in vision-language tasks compared to multimodal LLMs built on a text-only LLM.

Report issue for preceding element

Instruction-tuned LLMs \[ [150](https://arxiv.org/html/2409.11402v1#bib.bib150 ""); [23](https://arxiv.org/html/2409.11402v1#bib.bib23 ""); [115](https://arxiv.org/html/2409.11402v1#bib.bib115 "")\] serve as universal task solvers in the text domain, as they can follow user-provided instructions to address a variety of tasks.
As a result, it is common to build multimodal LLMs on instruction-tuned LLMs rather than base LLMs in previous studies, \[ [65](https://arxiv.org/html/2409.11402v1#bib.bib65 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [71](https://arxiv.org/html/2409.11402v1#bib.bib71 ""); [139](https://arxiv.org/html/2409.11402v1#bib.bib139 ""); [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\], as the instruction-following capability is essential for solving a wide range of vision-language tasks.
Various instruction-tuned LLMs have been used to build multimodal LLMs in different study, including Vicuna-1.5 \[ [21](https://arxiv.org/html/2409.11402v1#bib.bib21 "")\], LLaMA-2-Chat \[ [141](https://arxiv.org/html/2409.11402v1#bib.bib141 "")\], Mistral 7B \[ [50](https://arxiv.org/html/2409.11402v1#bib.bib50 "")\], Yi-34B \[ [161](https://arxiv.org/html/2409.11402v1#bib.bib161 "")\], Llama3-Instruct \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\], and Qwen2-Instruct \[ [119](https://arxiv.org/html/2409.11402v1#bib.bib119 "")\].
In this work, we use Qwen2-72B-Instruct \[ [119](https://arxiv.org/html/2409.11402v1#bib.bib119 "")\] as the default text-only LLM backbone.
We also employ Nous-Hermes-2-Yi-34B \[ [102](https://arxiv.org/html/2409.11402v1#bib.bib102 "")\] for ablation study and faster experimentation.

Report issue for preceding element

#### Vision Encoder.

Report issue for preceding element

Multimodal LLMs \[e.g., [4](https://arxiv.org/html/2409.11402v1#bib.bib4 ""); [65](https://arxiv.org/html/2409.11402v1#bib.bib65 ""); [67](https://arxiv.org/html/2409.11402v1#bib.bib67 ""); [19](https://arxiv.org/html/2409.11402v1#bib.bib19 "")\] typically leverage pretrained vision encoders (e.g., CLIP \[ [120](https://arxiv.org/html/2409.11402v1#bib.bib120 "")\]) to extract visual features from input images or video frames, with only a very few exceptions \[ [3](https://arxiv.org/html/2409.11402v1#bib.bib3 "")\].
These vision encoders \[ [120](https://arxiv.org/html/2409.11402v1#bib.bib120 ""); [47](https://arxiv.org/html/2409.11402v1#bib.bib47 ""); [19](https://arxiv.org/html/2409.11402v1#bib.bib19 ""); [167](https://arxiv.org/html/2409.11402v1#bib.bib167 ""); [27](https://arxiv.org/html/2409.11402v1#bib.bib27 "")\] are often trained on large-scale, diverse, and noisy text-image pairs sourced from the web \[ [124](https://arxiv.org/html/2409.11402v1#bib.bib124 ""); [12](https://arxiv.org/html/2409.11402v1#bib.bib12 ""); [33](https://arxiv.org/html/2409.11402v1#bib.bib33 "")\].
This allows for large-scale training and enhances the generalization needed to effectively process visual input in unseen domains.
The other types of datasets, such as those used for optical character recognition (OCR) \[ [19](https://arxiv.org/html/2409.11402v1#bib.bib19 "")\] and image segmentation \[ [58](https://arxiv.org/html/2409.11402v1#bib.bib58 "")\], are also incorporated to enhance the specific capabilities of vision encoders.
In this study, we use InternViT-6B \[ [19](https://arxiv.org/html/2409.11402v1#bib.bib19 "")\] as the default vision encoder due to its strong performance.
We keep this vision encoder frozen at all stages of training, as this simplifies the training process while still delivering strong results.

Report issue for preceding element

### 3.2 Architectural Designs

Report issue for preceding element

There are various architectural designs for constructing multimodal LLMs (MLLMs) using existing LLMs and vision encoders \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 ""); [66](https://arxiv.org/html/2409.11402v1#bib.bib66 ""); [79](https://arxiv.org/html/2409.11402v1#bib.bib79 ""); [147](https://arxiv.org/html/2409.11402v1#bib.bib147 ""); [8](https://arxiv.org/html/2409.11402v1#bib.bib8 "")\]. We discuss the two most common architectures.

Report issue for preceding element

#### Decoder-only MLLMs.

Report issue for preceding element

Decoder-only architectures are popular mainly for their simplicity and unified handling of all modalities by aligning other modality tokens into the text token embedding space.
It also facilitates the extension to generating other modalities \[ [35](https://arxiv.org/html/2409.11402v1#bib.bib35 ""); [108](https://arxiv.org/html/2409.11402v1#bib.bib108 "")\].
The notable examples of decoder-only multimodal LLMs include LLaVA \[ [79](https://arxiv.org/html/2409.11402v1#bib.bib79 ""); [78](https://arxiv.org/html/2409.11402v1#bib.bib78 ""); [80](https://arxiv.org/html/2409.11402v1#bib.bib80 ""); [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\], InternVL \[ [19](https://arxiv.org/html/2409.11402v1#bib.bib19 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\], and Cambrian-1 \[ [139](https://arxiv.org/html/2409.11402v1#bib.bib139 "")\].
In these models, image tokens from the vision encoder are projected into the text-embedding space via a projector module, e.g., position-wise multi-layer perceptron (MLP), and then directly fed into the decoder-only LLM, just like the text tokens.
Some variants, such as Qwen-VL \[ [8](https://arxiv.org/html/2409.11402v1#bib.bib8 "")\], utilize more advanced modules, e.g., _Perceiver_\[ [48](https://arxiv.org/html/2409.11402v1#bib.bib48 "")\], to down-sample the image tokens before they are fed into the LLM.

Report issue for preceding element

Training decoder-only multimodal LLMs typically involves two stages: _pretraining_ and _supervised fine-tuning_ (SFT). At the start of pretraining, the randomly initialized MLP or projector module needs to be trained while keeping the LLM frozen to avoid disrupting the LLM’s weights \[ [79](https://arxiv.org/html/2409.11402v1#bib.bib79 ""); [80](https://arxiv.org/html/2409.11402v1#bib.bib80 "")\].
Related work has also shown cases where both the projector and vision encoder are jointly trained during the pretraining stage \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [8](https://arxiv.org/html/2409.11402v1#bib.bib8 "")\].
Due to the limited capacity of the MLP or projector module, the LLM need to be unfrozen during multimodal supervised fine-tuning (SFT) to achieve good performance on vision-language tasks \[ [71](https://arxiv.org/html/2409.11402v1#bib.bib71 "")\].
The vision encoder is typically kept frozen during the SFT stage.
There are some exceptions, though, where the entire multimodal LLM is trained end-to-end \[ [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\], usually with smaller vision encoder \[ [167](https://arxiv.org/html/2409.11402v1#bib.bib167 "")\].

Report issue for preceding element

#### Cross-attention-based MLLMs.

Report issue for preceding element

Cross-attention-based architectures are similar to encoder-decoder transformer models for machine translation \[ [143](https://arxiv.org/html/2409.11402v1#bib.bib143 "")\], where the text decoder processes flattened image tokens via cross-attention layers, treating them as if they were a foreign language.
One of the early successful cross(X𝑋Xitalic\_X)-attention architectures is Flamingo \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 "")\], which is built on frozen pretrained LLMs \[ [42](https://arxiv.org/html/2409.11402v1#bib.bib42 "")\] and often serves as the starting point for many studies on this type of model.
The Flamingo model has two sets of trainable modules: _i)_ a _perceiver resampler_\[ [48](https://arxiv.org/html/2409.11402v1#bib.bib48 "")\] positioned after the frozen vision encoder \[ [120](https://arxiv.org/html/2409.11402v1#bib.bib120 "")\], which is designed to down-sample the vision encoder output to a specified size of representations, and _ii)_ the _gated x-attention layers_ interleaved with frozen LLM layers, which read output representations from the perceiver resampler.
In contrast, our NVLM-1.0-X and the concurrent Llama 3-V \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\] models utilize only gated cross-attention layers to process image tokens and do not include the Perceiver module.

Report issue for preceding element

The Flamingo model was trained in two stages: 1) pretraining with a large (and possibly noisy) set of image-text pairs or interleaved image-text data, and 2) supervised fine-tuning (SFT) with high-quality data. It always freezes self-attention layers in LLM decoder and only trains cross-attention layers and perceiver during both pretraining and supervised fine-tuning (SFT) to maintain text-only performance.
At inference time, the gate of the X𝑋Xitalic\_X-attention layers can be turned ON for multimodal tasks and OFF for text-only tasks.
Thanks to the frozen LLM and gated X𝑋Xitalic\_X-attention designs, the text-only performance is guaranteed not to degrade after multimodal training.
The follow-up work includes IDEFICS \[ [62](https://arxiv.org/html/2409.11402v1#bib.bib62 "")\] and OpenFlamingo \[ [7](https://arxiv.org/html/2409.11402v1#bib.bib7 "")\], which are open-source reproductions of Flamingo.

Report issue for preceding element

In contrast to decoder-only models, cross-attention-based MLLMs are generally considered more complex to implement. This complexity arises from the introduction of additional modules, the need for proper cross-attention masking in interleaved image-text settings, and the significantly heavier pretraining data requirements \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 ""); [62](https://arxiv.org/html/2409.11402v1#bib.bib62 ""); [176](https://arxiv.org/html/2409.11402v1#bib.bib176 "")\].
However, a notable advantage of the X𝑋Xitalic\_X-attention-based architecture is its computational efficiency, as it does not require unrolling all image tokens in the LLM decoder, which typically results in long sequences during both training and inference, especially for high-resolution images.
See § [4.3](https://arxiv.org/html/2409.11402v1#S4.SS3 "4.3 NVLM-X: X-attention Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs") for further study.

Report issue for preceding element

### 3.3 High-Resolution Inputs

Report issue for preceding element

Properly handling high-resolution images is crucial for achieving strong performance in many OCR-related tasks.
However, vision encoders are typically trained with static resolution of 2242superscript2242224^{2}224 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT or 3362superscript3362336^{2}336 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT pixels for efficiency \[ [120](https://arxiv.org/html/2409.11402v1#bib.bib120 ""); [100](https://arxiv.org/html/2409.11402v1#bib.bib100 "")\], when the image patch size per token is usually 142superscript14214^{2}14 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT or 162superscript16216^{2}16 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT. For example, feeding a 2242superscript2242224^{2}224 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT image to ViT-L/14 (patch size 142superscript14214^{2}14 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT) results in (22414)2=256superscript224142256(\\frac{224}{14})^{2}=256( divide start\_ARG 224 end\_ARG start\_ARG 14 end\_ARG ) start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT = 256 tokens.
There are specialized vision encoders that can directly handle static high-resolution images. For instance, the SAM encoder \[ [58](https://arxiv.org/html/2409.11402v1#bib.bib58 "")\], designed for image segmentation, can process images of 10242superscript102421024^{2}1024 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT pixels with a ViT-L/16 backbone (162superscript16216^{2}16 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT pixels per patch), producing a 4096-token output. This can be costly, especially when training datasets and downstream tasks contain a mix of low-resolution and high-resolution images.

Report issue for preceding element

The dynamic high-resolution mechanism \[ [160](https://arxiv.org/html/2409.11402v1#bib.bib160 ""); [80](https://arxiv.org/html/2409.11402v1#bib.bib80 ""); [30](https://arxiv.org/html/2409.11402v1#bib.bib30 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\] has been proposed to address the waste of compute in such scenarios.
For example, given a ViT-L/14 vision encoder trained on low-resolution images (e.g., 2242superscript2242224^{2}224 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT), a high-resolution image (e.g., 896×672896672896\\times 672896 × 672) is divided into tiles based on the aspect ratio and resolution of the input image (896224×672224=1289622467222412\\frac{896}{224}\\times\\frac{672}{224}=12divide start\_ARG 896 end\_ARG start\_ARG 224 end\_ARG × divide start\_ARG 672 end\_ARG start\_ARG 224 end\_ARG = 12 tiles in this case). Each tile is independently fed into the ViT-L/14, producing 256 tokens per tile and 3072 tokens in total.
Meanwhile, it only produces 512 tokens for an input image with 448×224448224448\\times 224448 × 224 resolution.
This dynamic approach is particularly well-suited for multimodal LLMs, which need to handle different types of tasks with varying image resolutions.

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x3.png)

Report issue for preceding elementFigure 3: NVLM-1.0 offers three architectural options: the cross-attention-based NVLM-X (top), the hybrid NVLM-H (middle), and the decoder-only NVLM-D (bottom).
The dynamic high-resolution vision pathway is shared by all three models.
However, different architectures process the image features from thumbnails and regular local tiles in distinct ways.
Report issue for preceding element

## 4 NVLM: Models and Training Methods

Report issue for preceding element

In this section, we introduce NVLM-1.0, a family of frontier-class multimodal LLMs featuring three architectures:
_i)_ Decoder-only NVLM-D,
_ii)_ Cross (X)-attention based NVLM-X,
and _iii)_ NVLM-H with Hybrid architecture.
Figure [3](https://arxiv.org/html/2409.11402v1#S3.F3 "Figure 3 ‣ 3.3 High-Resolution Inputs ‣ 3 Preliminaries ‣ NVLM: Open Frontier-Class Multimodal LLMs") illustrates these architectures.
We will begin by detailing the vision pathway shared by all NVLM models.

Report issue for preceding element

### 4.1 Shared Vision Pathway

Report issue for preceding element

Several studies have compared various vision encoders in multimodal LLMs, suggesting that unfreezing and combining multiple smaller vision encoders offer advantages \[ [139](https://arxiv.org/html/2409.11402v1#bib.bib139 "")\]. In this work, we employ a single, large, and powerful vision encoder, InternViT-6B-448px-V1-5 \[ [113](https://arxiv.org/html/2409.11402v1#bib.bib113 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\], as the default for all three architectures, keeping it frozen throughout all training stages. It processes images at a fixed resolution of 4482superscript4482448^{2}448 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT, generating 1,024 output tokens.

Report issue for preceding element

We use the similar dynamic high-resolution (DHR) approach outlined in Chen et al. \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\]. See the left part of Figure [3](https://arxiv.org/html/2409.11402v1#S3.F3 "Figure 3 ‣ 3.3 High-Resolution Inputs ‣ 3 Preliminaries ‣ NVLM: Open Frontier-Class Multimodal LLMs") for an illustration.
We allow a maximum of 6 tiles at training. Thus, the predefined aspect ratios are: {1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 2:1, 2:2, 2:3, 3:1, 3:2, 4:1, 5:1, 6:1}, encompassing all possible combinations of aspect ratios formed by 1 to 6 tiles.
For each input image, we dynamically match it to a predefined aspect ratio and divide it into 1 to 6 tiles, each corresponding to 448×448 pixels, based on the image’s resolution.
We include a thumbnail tile, which is a scaled-down version of the entire image to capture the global context.
Each tile is then fed into InternViT-6B-448px-V1-5 \[ [113](https://arxiv.org/html/2409.11402v1#bib.bib113 "")\], generating 1,024 tokens.
We apply a _downsampling_ operation to reduce the 1,024 image tokens to 256, reducing the processing overhead for the LLM. This operation groups four neighboring image tokens into one by concatenating them along the channel dimension, a.k.a. pixel shuffle \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\].
See Figure [4](https://arxiv.org/html/2409.11402v1#S4.F4 "Figure 4 ‣ 4.2 NVLM-D: Decoder-only Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs") for a detailed illustration of this process.

Report issue for preceding element

This dynamic high-resolution (DHR) design significantly improves performance on OCR-related tasks \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [30](https://arxiv.org/html/2409.11402v1#bib.bib30 "")\], but sometimes results in degraded performance on reasoning-related tasks \[ [166](https://arxiv.org/html/2409.11402v1#bib.bib166 "")\] when all image tokens from the tiles are simply concatenated and fed directly into the LLM.
We will address this issue across the three architectures, respectively.

Report issue for preceding element

### 4.2 NVLM-D: Decoder-only Model

Report issue for preceding element

Similar to previous decoder-only multimodal LLMs \[ [79](https://arxiv.org/html/2409.11402v1#bib.bib79 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\], NVLM-D model connects the pretrained vision encoder to the LLM using a 2-layer MLP as the projector or modality-alignment module.

Report issue for preceding element

Training NVLM-D involves two stages: pretraining and supervised fine-tuning (SFT). The MLP is randomly initialized and needs to undergo pretraining first, with both the vision encoder and LLM backbone kept frozen.
In our early exploration, we found that joint pretraining of the MLP projector and vision encoder is beneficial when the vision encoder is relatively weak (e.g., ViT-L/14 \[ [100](https://arxiv.org/html/2409.11402v1#bib.bib100 "")\]) and the pretraining datasets are sufficiently diverse. However, after upgrading to the more powerful InternViT-6B-448px-V1-5 \[ [113](https://arxiv.org/html/2409.11402v1#bib.bib113 "")\], the performance gains became marginal. Consequently, we opt to keep the vision encoder frozen during pretraining for the sake of simplicity.
During the SFT stage, both the MLP projector and LLM are trained to learn new vision-language tasks with novel instructions, while the vision encoder remains frozen.
However, a less frequently discussed point in decoder-only MLLM literature is that leaving the LLM unfrozen during multimodal SFT training often results in significant degradation in text-only performance.
Our NVLM-D model effectively maintains text-only performance by incorporating a high-quality text-only SFT dataset.
The model configuration and training details for NVLM-D models are in § [4.5](https://arxiv.org/html/2409.11402v1#S4.SS5 "4.5 Model Configurations and Training Method ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/extracted/5857645/figures/tile-details-bold.png)

Report issue for preceding elementFigure 4: Dynamic tiling of high-resolution input images.
Each tile is encoded separately by InternViT-6B, producing 1,024 tokens, which are downsampled to 256 tokens using a pixel shuffle operation.
Report issue for preceding element

#### Tile Tag for Dynamic High-Resolution.

Report issue for preceding element

As illustrated in Figure [3](https://arxiv.org/html/2409.11402v1#S3.F3 "Figure 3 ‣ 3.3 High-Resolution Inputs ‣ 3 Preliminaries ‣ NVLM: Open Frontier-Class Multimodal LLMs"), the LLM backbone needs to process the flattened image tokens from all dynamic high-resolution tiles, including an additional thumbnail tile. Directly concatenating flattened tokens without delimiters could confuse the LLM, as LLM lacks prior knowledge of the dynamic tiling process.
To address this, we insert a text-based tile tag in the input sequence to signal the start of a tile and the position of this tile within the whole tiling structure.
After the tile tag, we append the flattened 256 image tokens of the tile.
Note that our design differs from previous work \[ [30](https://arxiv.org/html/2409.11402v1#bib.bib30 "")\], which globally flattens the image tokens from different tiles and inserts a newline symbol at the end of each row of tokens.
We observe improved results with our approach, particularly as we scale up the model size and training data.

Report issue for preceding element

Table 1: Ablation study of tile tag formats for dynamic high-resolution (DHR) using the decoder-only NVLM-D with Yi-34B as the backbone LLM.
All models are trained for 20K iterations with batch size 128 without checkpoint selection to ensure a straightforward comparison.

Tile tag formatMMMU (val)MathVistaAI2D (test)ChartQADocVQATextVQAOCRBenchLow-resolution (4482superscript4482448^{2}448 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT)50.946.167.064.852.978.2622DHR + No tag50.051.779.976.180.278.4728DHR + 2-D grid tag51.152.881.781.186.779.4787DHR + 2-D bbox tag50.350.681.280.886.779.7791DHR + 1-D tag52.053.882.181.187.479.9806
Report issue for preceding element

Report issue for preceding element

We introduce three different tile tags, and perform an ablation study on NVLM-D with Yi-34B \[ [102](https://arxiv.org/html/2409.11402v1#bib.bib102 "")\] as the LLM backbone using the following variants of tile tags:

Report issue for preceding element

1. a)


No tag: Simple concatenation without tile tag, which is the design of InternVL-1.5 \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\].

Report issue for preceding element

2. b)


1-D flattened tile tag: <𝚝𝚒𝚕𝚎⁢\_⁢𝟷𝚝𝚒𝚕𝚎\_1\\mathtt{tile\\\_1}typewriter\_tile \_ typewriter\_1>,  <𝚝𝚒𝚕𝚎⁢\_⁢𝟸𝚝𝚒𝚕𝚎\_2\\mathtt{tile\\\_2}typewriter\_tile \_ typewriter\_2>,  ⋯⋯\\cdots⋯,  <𝚝𝚒𝚕𝚎⁢\_⁢𝟼𝚝𝚒𝚕𝚎\_6\\mathtt{tile\\\_6}typewriter\_tile \_ typewriter\_6>,  <𝚝𝚒𝚕𝚎⁢\_⁢𝚐𝚕𝚘𝚋𝚊𝚕𝚝𝚒𝚕𝚎\_𝚐𝚕𝚘𝚋𝚊𝚕\\mathtt{tile\\\_global}typewriter\_tile \_ typewriter\_global>.


Report issue for preceding element

3. c)


2-D grid tag: <𝚝𝚒𝚕𝚎⁢\_⁢𝚡𝟶⁢\_⁢𝚢𝟶𝚝𝚒𝚕𝚎\_𝚡𝟶\_𝚢𝟶\\mathtt{tile\\\_x0\\\_y0}typewriter\_tile \_ typewriter\_x0 \_ typewriter\_y0>,  <𝚝𝚒𝚕𝚎⁢\_⁢𝚡𝟷⁢\_⁢𝚢𝟶𝚝𝚒𝚕𝚎\_𝚡𝟷\_𝚢𝟶\\mathtt{tile\\\_x1\\\_y0}typewriter\_tile \_ typewriter\_x1 \_ typewriter\_y0>,  ⋯⋯\\cdots⋯,  <𝚝𝚒𝚕𝚎⁢\_⁢𝚡𝚆⁢\_⁢𝚢𝙷𝚝𝚒𝚕𝚎\_𝚡𝚆\_𝚢𝙷\\mathtt{tile\\\_xW\\\_yH}typewriter\_tile \_ typewriter\_xW \_ typewriter\_yH>,  <𝚝𝚒𝚕𝚎⁢\_⁢𝚐𝚕𝚘𝚋𝚊𝚕𝚝𝚒𝚕𝚎\_𝚐𝚕𝚘𝚋𝚊𝚕\\mathtt{tile\\\_global}typewriter\_tile \_ typewriter\_global>, where the {𝚒:𝚓:𝚒𝚓\\mathtt{i:j}typewriter\_i : typewriter\_j} of <𝚝𝚒𝚕𝚎⁢\_⁢𝚡𝚒⁢\_⁢𝚢𝚓𝚝𝚒𝚕𝚎\_𝚡𝚒\_𝚢𝚓\\mathtt{tile\\\_xi\\\_yj}typewriter\_tile \_ typewriter\_xi \_ typewriter\_yj\> can be in {1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 2:1, 2:2, 2:3, 3:1, 3:2, 4:1, 5:1, 6:1}.


Report issue for preceding element

4. d)


2-D bounding-box tag: <𝚋𝚘𝚡𝚋𝚘𝚡\\mathtt{box}typewriter\_box\> (x0subscript𝑥0x\_{0}italic\_x start\_POSTSUBSCRIPT 0 end\_POSTSUBSCRIPT, y0subscript𝑦0y\_{0}italic\_y start\_POSTSUBSCRIPT 0 end\_POSTSUBSCRIPT), (x1subscript𝑥1x\_{1}italic\_x start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT, y1subscript𝑦1y\_{1}italic\_y start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT) </𝚋𝚘𝚡𝚋𝚘𝚡\\mathtt{box}typewriter\_box>, ⋯⋯\\cdots⋯, <𝚋𝚘𝚡𝚋𝚘𝚡\\mathtt{box}typewriter\_box\> (xWsubscript𝑥𝑊x\_{W}italic\_x start\_POSTSUBSCRIPT italic\_W end\_POSTSUBSCRIPT, yHsubscript𝑦𝐻y\_{H}italic\_y start\_POSTSUBSCRIPT italic\_H end\_POSTSUBSCRIPT), (xW+1subscript𝑥𝑊1x\_{W+1}italic\_x start\_POSTSUBSCRIPT italic\_W + 1 end\_POSTSUBSCRIPT, yH+1subscript𝑦𝐻1y\_{H+1}italic\_y start\_POSTSUBSCRIPT italic\_H + 1 end\_POSTSUBSCRIPT) </𝚋𝚘𝚡𝚋𝚘𝚡\\mathtt{box}typewriter\_box>, where the (xisubscript𝑥𝑖x\_{i}italic\_x start\_POSTSUBSCRIPT italic\_i end\_POSTSUBSCRIPT, yjsubscript𝑦𝑗y\_{j}italic\_y start\_POSTSUBSCRIPT italic\_j end\_POSTSUBSCRIPT), (xi+1subscript𝑥𝑖1x\_{i+1}italic\_x start\_POSTSUBSCRIPT italic\_i + 1 end\_POSTSUBSCRIPT, yj+1subscript𝑦𝑗1y\_{j+1}italic\_y start\_POSTSUBSCRIPT italic\_j + 1 end\_POSTSUBSCRIPT) are the (left, top), (right, bottom) coordinates of that particular title within the whole high-resolution image.


Report issue for preceding element


From Table [1](https://arxiv.org/html/2409.11402v1#S4.T1 "Table 1 ‣ Tile Tag for Dynamic High-Resolution. ‣ 4.2 NVLM-D: Decoder-only Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs"), we can observe that:
1) The vanilla dynamic high-resolution method (DHR + No tag) significantly improves performance across all benchmarks, except for MMMU (50.0 vs. 50.9), compared to its low-resolution counterpart.
It is worth mentioning that previous DHR methods \[ [30](https://arxiv.org/html/2409.11402v1#bib.bib30 ""); [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\] also exhibit lower MMMU accuracy compared to their low-resolution counterparts.
2) Inserting all types of tile tags into the LLM decoder significantly outperforms simple concatenation with no tags.
In particular, we find that the introduction of tile tag greatly improves the performance on OCR-related tasks, including ChartQA \[ [93](https://arxiv.org/html/2409.11402v1#bib.bib93 "")\], DocVQA \[ [95](https://arxiv.org/html/2409.11402v1#bib.bib95 "")\] and OCRBench \[ [81](https://arxiv.org/html/2409.11402v1#bib.bib81 "")\].
3) 1-D tile tag <𝚝𝚒𝚕𝚎⁢\_⁢𝚔𝚝𝚒𝚕𝚎\_𝚔\\mathtt{tile\\\_k}typewriter\_tile \_ typewriter\_k\> performs generally better than other tags. We hypothesize that although the 1-D tile tag does not tell 2-D information (e.g., 2×\\times×3 vs. 3×\\times×2), it offers better generalization at test time.
Importantly, this tile tag design for dynamic high-resolution also offers moderate improvement on math and multidisciplinary reasoning tasks, including MathVista \[ [87](https://arxiv.org/html/2409.11402v1#bib.bib87 "")\] and MMMU \[ [166](https://arxiv.org/html/2409.11402v1#bib.bib166 "")\].

Report issue for preceding element

### 4.3 NVLM-X: X-attention Model

Report issue for preceding element

NVLM-X employs gated cross-attention to process image tokens and differs from the Flamingo model \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 "")\] in two key ways:

Report issue for preceding element

1. •


During our initial exploration, we found that while the _perceiver resampler_ is beneficial for natural image captioning, it negatively impacts dense OCR tasks, such as transcribing text from scanned documents (see Appendix [C](https://arxiv.org/html/2409.11402v1#A3 "Appendix C Perceiver Resampler in Flamingo Impacts OCR Performance ‣ NVLM: Open Frontier-Class Multimodal LLMs") for further details). The primary reason is that the cross-attention to latent array in the Perceiver \[ [48](https://arxiv.org/html/2409.11402v1#bib.bib48 "")\] mixes the input image tokens, potentially disrupting the spatial relationships between image patches, which are crucial for document OCR.
Based on this observation, our NVLM-X architecture does not use a perceiver resampler; instead, it relies solely on cross-attention to read image tokens directly from the vision encoder.

Report issue for preceding element

2. •


Freezing the LLM during the multimodal SFT stage compromises performance on vision-language tasks, as the multimodal LLM needs to quickly adapt to new tasks and novel instructions that were not encountered during text-only instruction tuning.
We illustrate this observation in Table [9](https://arxiv.org/html/2409.11402v1#S6.T9 "Table 9 ‣ 6.4 Text-only Performance ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs") in § [6](https://arxiv.org/html/2409.11402v1#S6 "6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs").
Thus, we unfreeze the LLM backbone of NVLM-X during multimodal SFT and blend in a high-quality text-only SFT dataset to maintain strong text-only performance.
Note that this also differs from Llama 3-V \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\], which freezes the LLM during multimodal training.

Report issue for preceding element


The model configuration and training details for NVLM-X models can be found in § [4.5](https://arxiv.org/html/2409.11402v1#S4.SS5 "4.5 Model Configurations and Training Method ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

#### Tile Tag for Dynamic High-Resolution.

Report issue for preceding element

NVLM-X uses the same dynamic high-resolution approach as NVLM-D to obtain image tokens from a global thumbnail tile and regular tiles.
As illustrated in Figure [3](https://arxiv.org/html/2409.11402v1#S3.F3 "Figure 3 ‣ 3.3 High-Resolution Inputs ‣ 3 Preliminaries ‣ NVLM: Open Frontier-Class Multimodal LLMs"), NVLM-X employs gated X𝑋Xitalic\_X-attention to process the flattened image tokens for each tile, rather than feeding them directly into the LLM decoder.
Similar to the design used in NVLM-D, we insert a sequence of text-based tile tags <𝚝𝚒𝚕𝚎⁢\_⁢𝟷𝚝𝚒𝚕𝚎\_1\\mathtt{tile\\\_1}typewriter\_tile \_ typewriter\_1\> ⋯⋯\\cdots⋯ <𝚝𝚒𝚕𝚎⁢\_⁢𝚔𝚝𝚒𝚕𝚎\_𝚔\\mathtt{tile\\\_k}typewriter\_tile \_ typewriter\_k\> in the LLM decoder, while allowing each tag <𝚝𝚒𝚕𝚎⁢\_⁢𝚔𝚝𝚒𝚕𝚎\_𝚔\\mathtt{tile\\\_k}typewriter\_tile \_ typewriter\_k\> to only attend to its corresponding image tokens by properly configuring the X𝑋Xitalic\_X-attention mask.
This approach ensures that the LLM is better informed about the tiling structure without needing to infer it from the content of thumbnail tile and regular tiles.

Report issue for preceding element

In Table [2](https://arxiv.org/html/2409.11402v1#S4.T2 "Table 2 ‣ Tile Tag for Dynamic High-Resolution. ‣ 4.3 NVLM-X: X-attention Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs"), we present an ablation study of NVLM-X with Yi-34B LLM backbone using low-resolution 4482superscript4482448^{2}448 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT input, dynamic high-resolution (DHR) without tile tags and with 1-D <𝚝𝚒𝚕𝚎⁢\_⁢𝚔𝚝𝚒𝚕𝚎\_𝚔\\mathtt{tile\\\_k}typewriter\_tile \_ typewriter\_k>  tags.
We find that:
1) The vanilla dynamic high-resolution approach (DHR + No tag) significantly outperforms its low-resolution counterpart across all benchmarks, except MMMU (53.0 vs. 53.2).
2) Adding tile tags further improves performance across all benchmarks, including multimodal reasoning (MMMU: 54.1 vs. 53.0, MathVista: 59.6 vs. 57.6) and OCR-related tasks.

Report issue for preceding element

Table 2: Ablation study of using tile tag  <𝚝𝚒𝚕𝚎⁢\_⁢𝚔𝚝𝚒𝚕𝚎\_𝚔\\mathtt{tile\\\_k}typewriter\_tile \_ typewriter\_k>  for dynamic high-resolution (DHR) using the cross-attention-based NVLM-X with Yi-34B as the backbone LLM.
All models are trained for 10K iterations with batch size 512 without checkpoint selection to ensure a straightforward comparison.

Tile-tag formatMMMU (val)MathVistaAI2D (test)ChartQADocVQATextVQAOCRBenchLow-resolution (4482superscript4482448^{2}448 start\_POSTSUPERSCRIPT 2 end\_POSTSUPERSCRIPT)53.257.469.865.662.363.2612DHR + No tag53.057.678.573.275.475.2682DHR + 1-D tag54.159.680.779.280.978.6744
Report issue for preceding element

Report issue for preceding element

#### Decoder-only vs. X-attention.

Report issue for preceding element

The pros and cons of cross-attention-based NVLM-X and decoder-only NVLM-D can be summarized in the following.
_i) Parameter efficiency:_ NVLM-D has fewer parameters than NVLM-X, as the latter has the newly introduced gated cross-attention layers.
The number of additional parameters becomes significant as the model scales up. For instance, Llama 3-V 405B added 100B parameters to the text-only Llama-3.1-405B-Instruct \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\].
_ii)_ _Training efficiency:_
NVLM-X enables more efficient processing of high-resolution images by eliminating the need to unroll all image tokens on the LLM decoder side. See Table [3](https://arxiv.org/html/2409.11402v1#S4.T3 "Table 3 ‣ Decoder-only vs. X-attention. ‣ 4.3 NVLM-X: X-attention Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs") for a comparison of training _throughput_ between 34B NVLM-D and NVLM-X models. Note that the decoder-only NVLM-D requires much longer sequence lengths, as all image tokens are concatenated and fed into the LLM decoder, leading to higher GPU memory consumption and lower training throughput.
_iii) Multimodal reasoning:_ NVLM-D performs unified processing of all tokens from different modalities, enabling joint multimodal reasoning at the LLM decoder. However, the long sequence of tokens for high-resolution images (e.g., 256×\\times×7 = 1792 tokens) may still make reasoning challenging, even with the assistance of tile tags.

Report issue for preceding element

Table 3: Training throughput (samples per second) of NVLM-X, NVLM-D, and NVLM-H with Yi-34B as the backbone LLM.
We use 128 H100 GPUs during supervised fine-tuning with unfrozen LLMs.
All three models are implemented in Megatron-LM with tensor parallelism set to 8 \[ [129](https://arxiv.org/html/2409.11402v1#bib.bib129 "")\]. Sequence lengths in the LLM decoder are set with 1,024 tokens for text, 256 image tokens for the thumbnail tile, and 256×6 for the 6 regular tiles.

ModelsBatch size\# of H100Sequence length\# of TilesElapsed time (ms)ThroughputGPUsin LLM decoderper iterationsamples / secNVLM-X 34B2561281,0246+15,06350.6NVLM-D 34B2561281,024 + 256×\\times×7 = 2,8166+18,88528.8NVLM-H 34B2561281,024 + 256 = 1,2806+17,07136.2
Report issue for preceding element

Report issue for preceding element

### 4.4 NVLM-H: Hybrid Model

Report issue for preceding element

Drawing inspiration from the comparison of NVLM-X and NVLM-D, we propose NVLM-H, a novel hybrid architecture that combines the best of both approaches.
As illustrated in Figure [3](https://arxiv.org/html/2409.11402v1#S3.F3 "Figure 3 ‣ 3.3 High-Resolution Inputs ‣ 3 Preliminaries ‣ NVLM: Open Frontier-Class Multimodal LLMs"), NVLM-H separates the processing of image tokens into two paths.
The thumbnail image tokens are fed into the LLM alongside text tokens and processed by self-attention layers, enabling joint multimodal reasoning.
Simultaneously, a dynamic number of regular tiles are processed through gated cross-attention, enabling the model to capture finer image details.
This approach enhances high-resolution capability compared to NVLM-X while significantly improving computational efficiency compared to NVLM-D.
Table [3](https://arxiv.org/html/2409.11402v1#S4.T3 "Table 3 ‣ Decoder-only vs. X-attention. ‣ 4.3 NVLM-X: X-attention Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs") demonstrates that NVLM-H has higher throughput than NVLM-D in training.

Report issue for preceding element

#### Tile Tag for Dynamic High-Resolution.

Report issue for preceding element

NVLM-H utilizes the same 1-D flattened tile tag <𝚝𝚒𝚕𝚎⁢\_⁢𝚔𝚝𝚒𝚕𝚎\_𝚔\\mathtt{tile\\\_k}typewriter\_tile \_ typewriter\_k\> introduced in § [4.2](https://arxiv.org/html/2409.11402v1#S4.SS2 "4.2 NVLM-D: Decoder-only Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs") for NVLM-D. The primary distinction lies in the processing location. As shown in Figure [3](https://arxiv.org/html/2409.11402v1#S3.F3 "Figure 3 ‣ 3.3 High-Resolution Inputs ‣ 3 Preliminaries ‣ NVLM: Open Frontier-Class Multimodal LLMs"), text embeddings of <𝚝𝚒𝚕𝚎⁢\_⁢𝚔𝚝𝚒𝚕𝚎\_𝚔\\mathtt{tile\\\_k}typewriter\_tile \_ typewriter\_k\> are integrated into the gated cross-attention layers alongside visual embeddings. This approach is effective because the text and visual embeddings are well-aligned during pre-training, enabling the model to seamlessly interpret tile tags within the cross-attention mechanism. Consistent with the results in Table [1](https://arxiv.org/html/2409.11402v1#S4.T1 "Table 1 ‣ Tile Tag for Dynamic High-Resolution. ‣ 4.2 NVLM-D: Decoder-only Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs") and Table [2](https://arxiv.org/html/2409.11402v1#S4.T2 "Table 2 ‣ Tile Tag for Dynamic High-Resolution. ‣ 4.3 NVLM-X: X-attention Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs"), adding tile tags enhances NVLM-H’s performance on OCR-related tasks compared to no tagging.

Report issue for preceding element

### 4.5 Model Configurations and Training Method

Report issue for preceding element

We provide the model configurations and training details for all NVLM-1.0 models below.

Report issue for preceding element

#### Backbone LLMs and Vision Encoder.

Report issue for preceding element

For the NVLM-D, NVLM-X, and NVLM-H 72B models, we use Qwen2-72B-Instruct \[ [119](https://arxiv.org/html/2409.11402v1#bib.bib119 "")\] as the backbone LLM.
For computational reasons, we also use the smaller Nous-Hermes-2-Yi-34B \[ [102](https://arxiv.org/html/2409.11402v1#bib.bib102 "")\] as the LLM backbone for faster ablation studies and experimentation.
After finalizing the optimized designs, we shifted our computational resources to improving the NVLM-1.0 72B models.
Across all NVLM models, InternViT-6B-448px-V1-5 \[ [113](https://arxiv.org/html/2409.11402v1#bib.bib113 "")\] serves as the vision encoder.

Report issue for preceding element

#### Modality-Alignment Module.

Report issue for preceding element

We include the details of modality-alignment modules for three NVLM architechtures in the following:

Report issue for preceding element

- •


For NVLM-D models, the LLM and vision encoder are connected by a two-layer MLP to align the modalities, with hidden dimensions of 12800→20480→7168→1280020480→716812800\\rightarrow 20480\\rightarrow 716812800 → 20480 → 7168 for 34B model, and 12800→29568→8192→1280029568→819212800\\rightarrow 29568\\rightarrow 819212800 → 29568 → 8192 for 72B model.
Note that InternViT-6B has a hidden dimension of 3200, which increases to 3200×4=1280032004128003200\\times 4=128003200 × 4 = 12800 after applying pixel shuffle.
Yi-34B \[ [102](https://arxiv.org/html/2409.11402v1#bib.bib102 "")\] has hidden dimension 7168716871687168, and Qwen2-72B has hideen dimension 8192819281928192.

Report issue for preceding element

- •


For NVLM-X models, the images features are first projected to LLMs’ hidden dimension with a one-layer MLP, 12800→7168→12800716812800\\rightarrow 716812800 → 7168 for 34B model, and 12800→8192→12800819212800\\rightarrow 819212800 → 8192 for 72B model.
We insert a gated X𝑋Xitalic\_X-attention layer every 6 and 8 LLM self-attention layers, respectively. This results in a total of 10 X𝑋Xitalic\_X-attention layers for both models.

Report issue for preceding element

- •


The NVLM-H 34B and 72B models utilize a two-layer MLP and X𝑋Xitalic\_X-attention layers as the modality-alignment module. The image tokens for both thumbnail and regular tiles are projected through the two-layer MLP, with hidden dimensions of 12800→20480→7168→1280020480→716812800\\rightarrow 20480\\rightarrow 716812800 → 20480 → 7168 for the 34B model, and 12800→29568→8192→1280029568→819212800\\rightarrow 29568\\rightarrow 819212800 → 29568 → 8192 for the 72B model. The projected thumbnail image tokens are then directly fed into the LLM decoder. The projected image tokens of regular tiles are cross-attended by the X𝑋Xitalic\_X-attention layers. As with NVLM-X, ten gated X𝑋Xitalic\_X-attention layers are inserted for both the 34B and 72B models.

Report issue for preceding element


#### Training Method.

Report issue for preceding element

We employ a unified training method for all NVLM models.
The training process involves two stages:
_i)_ Pretraining: we freeze both the LLM backbone and vision encoder for all models. We only train the modality-alignment modules, i.e., projector MLP or X𝑋Xitalic\_X-attention layers, using our pretraining dataset detailed in Table [4](https://arxiv.org/html/2409.11402v1#S5.T4 "Table 4 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs").
For pretraining hyperparameters, one can refer to Table [10](https://arxiv.org/html/2409.11402v1#A2.T10 "Table 10 ‣ Appendix B Training Hyperparameters ‣ NVLM: Open Frontier-Class Multimodal LLMs") in Appendix [B](https://arxiv.org/html/2409.11402v1#A2 "Appendix B Training Hyperparameters ‣ NVLM: Open Frontier-Class Multimodal LLMs").
We find a large batch size of 2048 improves the pretraining with frozen LLMs.
_ii)_ Supervised fine-tuning (SFT): we keep the vision encoder frozen while training both the LLM and modality-alignment modules with our multimodal SFT datasets detailed in Table [6](https://arxiv.org/html/2409.11402v1#S5.T6 "Table 6 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs"), along with a text-only SFT dataset.
For hyperparameters of SFT, one can refer to Table [11](https://arxiv.org/html/2409.11402v1#A2.T11 "Table 11 ‣ Appendix B Training Hyperparameters ‣ NVLM: Open Frontier-Class Multimodal LLMs") in Appendix [B](https://arxiv.org/html/2409.11402v1#A2 "Appendix B Training Hyperparameters ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

## 5 Training Data

Report issue for preceding element

In this section, we provide details of the pretraining and supervised fine-tuning (SFT) datasets. These curated training datasets are used across all three architectures in the NVLM family.
All datasets are formatted based on the task type and the chat template provided in Appendix [E](https://arxiv.org/html/2409.11402v1#A5 "Appendix E Data Formats and ChatML Tamplate ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

### 5.1 Multimodal Pretraining Data

Report issue for preceding element

We curate a diverse, high-quality multimodal pretraining dataset, all sourced from the open-source community.
We find that _the quality of the dataset matters more than its scale, even at the pretraining stage_.
In early exploration, we experimented with much larger but noisier datasets \[e.g., [124](https://arxiv.org/html/2409.11402v1#bib.bib124 ""); [33](https://arxiv.org/html/2409.11402v1#bib.bib33 "")\], commonly used for training CLIP-style vision encoders.
However, we found these unfiltered datasets to be less effective for training both decoder-only and X𝑋Xitalic\_X-attention-based multimodal LLMs, even with a frozen LLM during pretraining.
The potential reason could be that noisy text-image data leads to inaccurate alignment between the two modalities.
We also experimented with interleaved text-image datasets, including MMC4 \[ [176](https://arxiv.org/html/2409.11402v1#bib.bib176 "")\] and OBELICS \[ [63](https://arxiv.org/html/2409.11402v1#bib.bib63 "")\], and found that they had minimal impact on downstream vision-language tasks, even in few-shot settings, within the state-of-the-art NVLM framework.
We hypothesize more careful filtering and recaptioning are needed for such interleaved text-image datasets.

Report issue for preceding element

Table 4: Datasets used by NVLM-1.0 at the pretraining stage.

TaskDatasetCaptioningCOCO \[ [72](https://arxiv.org/html/2409.11402v1#bib.bib72 "")\], CC3M \[ [127](https://arxiv.org/html/2409.11402v1#bib.bib127 "")\], SBU \[ [114](https://arxiv.org/html/2409.11402v1#bib.bib114 "")\], LAION-115M (sanitized) \[ [123](https://arxiv.org/html/2409.11402v1#bib.bib123 ""); [66](https://arxiv.org/html/2409.11402v1#bib.bib66 "")\]VQA (natural image)VQAv2 \[ [38](https://arxiv.org/html/2409.11402v1#bib.bib38 "")\], Visual Genome \[ [59](https://arxiv.org/html/2409.11402v1#bib.bib59 "")\]ChartDVQA \[ [51](https://arxiv.org/html/2409.11402v1#bib.bib51 "")\]DocumentDocmatix \[ [90](https://arxiv.org/html/2409.11402v1#bib.bib90 "")\]OCR / Scene-TextOCR-VQA \[ [98](https://arxiv.org/html/2409.11402v1#bib.bib98 "")\], COCO-Text \[ [144](https://arxiv.org/html/2409.11402v1#bib.bib144 "")\], TextOCR \[ [132](https://arxiv.org/html/2409.11402v1#bib.bib132 "")\], ReCTs \[ [170](https://arxiv.org/html/2409.11402v1#bib.bib170 "")\], RRC-ArT \[ [22](https://arxiv.org/html/2409.11402v1#bib.bib22 "")\], RRC-LSVT \[ [134](https://arxiv.org/html/2409.11402v1#bib.bib134 "")\]RCTW \[ [128](https://arxiv.org/html/2409.11402v1#bib.bib128 "")\], synthdog-en \[ [57](https://arxiv.org/html/2409.11402v1#bib.bib57 "")\], pdfa-eng-wds \[ [117](https://arxiv.org/html/2409.11402v1#bib.bib117 "")\]MathCLEVR-Math \[ [73](https://arxiv.org/html/2409.11402v1#bib.bib73 "")\]
Report issue for preceding element

Report issue for preceding elementTable 5: An ablation study comparing the use of our pretraining data in Table [4](https://arxiv.org/html/2409.11402v1#S5.T4 "Table 4 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs") with the pretraining data from LLaVA-1.5 \[ [77](https://arxiv.org/html/2409.11402v1#bib.bib77 "")\] using decoder-only NVLM-D with Yi-34B as the backbone LLM. Both models are trained for 20K iterations with batch size 128 without checkpoint selection to ensure a straightforward comparison.

Pretraining dataMMMU (val)MathVistaAI2D (test)ChartQADocVQATextVQAOCRBenchLLaVA-1.5 data \[ [77](https://arxiv.org/html/2409.11402v1#bib.bib77 "")\]51.848.980.580.385.278.9760Our pretraining data52.053.882.181.187.479.9806
Report issue for preceding element

Report issue for preceding element

We provide a list of the pretraining datasets in Table [4](https://arxiv.org/html/2409.11402v1#S5.T4 "Table 4 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs").
These datasets cover different tasks:
1) Captioning. In particular, we use a filtered and recaptioned version of LAION-115M from Li et al. \[ [66](https://arxiv.org/html/2409.11402v1#bib.bib66 "")\]. We perform thorough data scanning and sanitization to ensure the dataset is free of any harmful or inappropriate content.
2) Visual question answering (VQA) on natural image.
3) VQA on chart and scanned document.
4) Math reasoning in a visual context.
5) OCR and scene-text recognition.
In addition to large-scale captioning datasets, we find that incorporating large task-oriented datasets during the pretraining stage enhances cross-modal alignment and leads to better final results. We also experimented with blending relatively small task-oriented datasets used in SFT into pretraining. However, this approach caused overfitting on these datasets and impaired the model’s reasoning ability when evaluated on zero-shot benchmarks such as MMMU and MathVista.

Report issue for preceding element

Previous work \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 "")\] has shown that abundant and diverse pretraining data is crucial for the success of cross-attention-based models.
In contrast, decoder-only models, such as LLaVA \[ [78](https://arxiv.org/html/2409.11402v1#bib.bib78 ""); [80](https://arxiv.org/html/2409.11402v1#bib.bib80 "")\], work well with smaller pretraining datasets \[ [77](https://arxiv.org/html/2409.11402v1#bib.bib77 "")\], which are simply filtered subsets of the captioning datasets including CC3M \[ [127](https://arxiv.org/html/2409.11402v1#bib.bib127 "")\], SBU \[ [114](https://arxiv.org/html/2409.11402v1#bib.bib114 "")\], and LAION-115M \[ [123](https://arxiv.org/html/2409.11402v1#bib.bib123 ""); [66](https://arxiv.org/html/2409.11402v1#bib.bib66 "")\].
In contrast, our findings demonstrate that the diverse pretraining data shown in Table [4](https://arxiv.org/html/2409.11402v1#S5.T4 "Table 4 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs")
can still significantly enhance the performance of decoder-only multimodal LLMs, even in state-of-the-art settings with highly curated SFT datasets.
We conducted an ablation study comparing our pretraining data with LLaVA-1.5’s pretraining data \[ [77](https://arxiv.org/html/2409.11402v1#bib.bib77 "")\] for the NVLM-D with Yi-34B as LLM backbone, as shown in Table [5](https://arxiv.org/html/2409.11402v1#S5.T5 "Table 5 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs"). The pretrained models are then fine-tuned on the same high-quality SFT dataset in Table [6](https://arxiv.org/html/2409.11402v1#S5.T6 "Table 6 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs").
One can see that our diverse pretraining data provide consistent improvements across all benchmarks, in particular a significant improvement in math reasoning and OCR-related tasks, as we add these types of data in pretraining.

Report issue for preceding element

Table 6: Datasets used by NVLM-1.0 at supervised-fine-tuning (SFT).

TaskDatasetCaptioningCOCO \[ [72](https://arxiv.org/html/2409.11402v1#bib.bib72 "")\], TextCaps \[ [130](https://arxiv.org/html/2409.11402v1#bib.bib130 "")\], ShareGPT-4o \[ [61](https://arxiv.org/html/2409.11402v1#bib.bib61 "")\]VQA (natural image)VQAv2 \[ [38](https://arxiv.org/html/2409.11402v1#bib.bib38 "")\], Visual Genome \[ [59](https://arxiv.org/html/2409.11402v1#bib.bib59 "")\], TallyQA \[ [2](https://arxiv.org/html/2409.11402v1#bib.bib2 "")\], Visual7W \[ [177](https://arxiv.org/html/2409.11402v1#bib.bib177 "")\],
Vizwiz \[ [39](https://arxiv.org/html/2409.11402v1#bib.bib39 "")\]General KnowledgeOK-VQA \[ [91](https://arxiv.org/html/2409.11402v1#bib.bib91 "")\], A-OKVQA \[ [125](https://arxiv.org/html/2409.11402v1#bib.bib125 "")\]Visual ReasoningGQA \[ [45](https://arxiv.org/html/2409.11402v1#bib.bib45 "")\], Super-CLEVR \[ [69](https://arxiv.org/html/2409.11402v1#bib.bib69 "")\], Raven \[ [168](https://arxiv.org/html/2409.11402v1#bib.bib168 "")\],
VSR \[ [74](https://arxiv.org/html/2409.11402v1#bib.bib74 "")\]Chart & DiagramDVQA \[ [51](https://arxiv.org/html/2409.11402v1#bib.bib51 "")\], PlotQA \[ [97](https://arxiv.org/html/2409.11402v1#bib.bib97 "")\], MMC-Instruction \[ [76](https://arxiv.org/html/2409.11402v1#bib.bib76 "")\], ChartQA \[ [93](https://arxiv.org/html/2409.11402v1#bib.bib93 "")\], InfographicVQA \[ [96](https://arxiv.org/html/2409.11402v1#bib.bib96 "")\]FigureQA \[ [52](https://arxiv.org/html/2409.11402v1#bib.bib52 "")\], IconQA \[ [84](https://arxiv.org/html/2409.11402v1#bib.bib84 "")\], Chart2Text \[ [103](https://arxiv.org/html/2409.11402v1#bib.bib103 "")\], Diagram Image2Text \[ [53](https://arxiv.org/html/2409.11402v1#bib.bib53 "")\]TableWikiTableQuestions \[ [116](https://arxiv.org/html/2409.11402v1#bib.bib116 "")\],
RobuT(WTQ, WikiSQL, SQA) \[ [173](https://arxiv.org/html/2409.11402v1#bib.bib173 "")\], HiTab \[ [20](https://arxiv.org/html/2409.11402v1#bib.bib20 "")\]DocumentDocVQA \[ [95](https://arxiv.org/html/2409.11402v1#bib.bib95 "")\], Docmatix \[ [90](https://arxiv.org/html/2409.11402v1#bib.bib90 "")\], DUDE \[ [142](https://arxiv.org/html/2409.11402v1#bib.bib142 "")\], VisualMRC \[ [135](https://arxiv.org/html/2409.11402v1#bib.bib135 "")\], TAT-DQA\[ [175](https://arxiv.org/html/2409.11402v1#bib.bib175 "")\]UReader IE \[ [160](https://arxiv.org/html/2409.11402v1#bib.bib160 "")\], UReader KG \[ [160](https://arxiv.org/html/2409.11402v1#bib.bib160 "")\], UReader QA \[ [160](https://arxiv.org/html/2409.11402v1#bib.bib160 "")\],OCR / Screen / Scene-TextOCR-VQA \[ [98](https://arxiv.org/html/2409.11402v1#bib.bib98 "")\],
TextVQA \[ [131](https://arxiv.org/html/2409.11402v1#bib.bib131 "")\], ST-VQA \[ [10](https://arxiv.org/html/2409.11402v1#bib.bib10 "")\], ScreenQA \[ [43](https://arxiv.org/html/2409.11402v1#bib.bib43 "")\], SlideQA \[ [136](https://arxiv.org/html/2409.11402v1#bib.bib136 "")\], PDF-VQA \[ [29](https://arxiv.org/html/2409.11402v1#bib.bib29 "")\]VQA-CD \[ [89](https://arxiv.org/html/2409.11402v1#bib.bib89 "")\], VQAonBD \[ [1](https://arxiv.org/html/2409.11402v1#bib.bib1 "")\], POIE \[ [60](https://arxiv.org/html/2409.11402v1#bib.bib60 "")\], SROIE \[ [44](https://arxiv.org/html/2409.11402v1#bib.bib44 "")\], ORAND \[ [28](https://arxiv.org/html/2409.11402v1#bib.bib28 "")\],
EST-VQA \[ [148](https://arxiv.org/html/2409.11402v1#bib.bib148 "")\]FUNSD \[ [49](https://arxiv.org/html/2409.11402v1#bib.bib49 "")\], SQuAD(rendering) \[ [121](https://arxiv.org/html/2409.11402v1#bib.bib121 "")\], WordArt \[ [155](https://arxiv.org/html/2409.11402v1#bib.bib155 "")\], IAM \[ [92](https://arxiv.org/html/2409.11402v1#bib.bib92 "")\], IIIT5K \[ [46](https://arxiv.org/html/2409.11402v1#bib.bib46 "")\], HME100K \[ [164](https://arxiv.org/html/2409.11402v1#bib.bib164 "")\]synthdog-en \[ [57](https://arxiv.org/html/2409.11402v1#bib.bib57 "")\], Bentham QA \[ [94](https://arxiv.org/html/2409.11402v1#bib.bib94 "")\], HW-SQuAD \[ [94](https://arxiv.org/html/2409.11402v1#bib.bib94 "")\],
WebSight \[ [64](https://arxiv.org/html/2409.11402v1#bib.bib64 "")\], ChromeWriting \[ [152](https://arxiv.org/html/2409.11402v1#bib.bib152 "")\]K12 Printing \[ [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\], COCO-Text \[ [144](https://arxiv.org/html/2409.11402v1#bib.bib144 "")\], TextOCR \[ [132](https://arxiv.org/html/2409.11402v1#bib.bib132 "")\], ReCTs \[ [170](https://arxiv.org/html/2409.11402v1#bib.bib170 "")\], pdfa-eng-wds \[ [117](https://arxiv.org/html/2409.11402v1#bib.bib117 "")\]MathCLEVR-Math \[ [73](https://arxiv.org/html/2409.11402v1#bib.bib73 "")\], GeoQA+ \[ [13](https://arxiv.org/html/2409.11402v1#bib.bib13 "")\], Geometry3K \[ [83](https://arxiv.org/html/2409.11402v1#bib.bib83 "")\], TabMWP \[ [86](https://arxiv.org/html/2409.11402v1#bib.bib86 "")\], GSM8K(rendering) \[ [25](https://arxiv.org/html/2409.11402v1#bib.bib25 "")\]MetaMathQA(rendering) \[ [162](https://arxiv.org/html/2409.11402v1#bib.bib162 "")\], MAVIS Data Engine \[ [171](https://arxiv.org/html/2409.11402v1#bib.bib171 "")\], MAVIS Manual Collection \[ [171](https://arxiv.org/html/2409.11402v1#bib.bib171 "")\]Geo170K Align \[ [34](https://arxiv.org/html/2409.11402v1#bib.bib34 "")\], Geo170K QA \[ [34](https://arxiv.org/html/2409.11402v1#bib.bib34 "")\], GeoMVerse \[ [54](https://arxiv.org/html/2409.11402v1#bib.bib54 "")\], GEOS \[ [126](https://arxiv.org/html/2409.11402v1#bib.bib126 "")\], UniGeo \[ [14](https://arxiv.org/html/2409.11402v1#bib.bib14 "")\]ScienceAI2D \[ [55](https://arxiv.org/html/2409.11402v1#bib.bib55 "")\], ScienceQA \[ [85](https://arxiv.org/html/2409.11402v1#bib.bib85 "")\], TQA \[ [56](https://arxiv.org/html/2409.11402v1#bib.bib56 "")\],
ArXivQA \[ [68](https://arxiv.org/html/2409.11402v1#bib.bib68 "")\], textbook dataVisual Instruction-TuningLRV-Instruction \[ [75](https://arxiv.org/html/2409.11402v1#bib.bib75 "")\], LLaVA-158K \[ [79](https://arxiv.org/html/2409.11402v1#bib.bib79 "")\], LLaVAR \[ [172](https://arxiv.org/html/2409.11402v1#bib.bib172 "")\]Text-only SFTSlimOrca \[ [70](https://arxiv.org/html/2409.11402v1#bib.bib70 "")\], ShareGPT \[ [138](https://arxiv.org/html/2409.11402v1#bib.bib138 "")\], EvolInstruct \[ [156](https://arxiv.org/html/2409.11402v1#bib.bib156 "")\], GPTeacher \[ [137](https://arxiv.org/html/2409.11402v1#bib.bib137 "")\], AlpacaGPT4 \[ [118](https://arxiv.org/html/2409.11402v1#bib.bib118 "")\],UltraInteract \[ [163](https://arxiv.org/html/2409.11402v1#bib.bib163 "")\], OrcaMathWordProblems \[ [99](https://arxiv.org/html/2409.11402v1#bib.bib99 "")\], MathInstruct \[ [165](https://arxiv.org/html/2409.11402v1#bib.bib165 "")\], MetaMath \[ [162](https://arxiv.org/html/2409.11402v1#bib.bib162 "")\],GlaiveCodeAssistant \[ [37](https://arxiv.org/html/2409.11402v1#bib.bib37 "")\], Magicoder \[ [151](https://arxiv.org/html/2409.11402v1#bib.bib151 "")\], WizardCoder \[ [88](https://arxiv.org/html/2409.11402v1#bib.bib88 "")\].
Report issue for preceding element

Report issue for preceding element

### 5.2 Multimodal SFT Data

Report issue for preceding element

We collected a diverse, high-quality, task-oriented SFT dataset to enhance NVLM’s capabilites on a wide range of visoin-language tasks.
A detailed list of SFT datasets is provided in Table [6](https://arxiv.org/html/2409.11402v1#S5.T6 "Table 6 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs").
In addition to high-quality datasets with short captions, such as COCO \[ [72](https://arxiv.org/html/2409.11402v1#bib.bib72 "")\] and TextCaps \[ [130](https://arxiv.org/html/2409.11402v1#bib.bib130 "")\], we also include ShareGPT-4o \[ [61](https://arxiv.org/html/2409.11402v1#bib.bib61 "")\], which provides detailed image descriptions.
Additionally, we have included several VQA datasets based on natural images \[ [38](https://arxiv.org/html/2409.11402v1#bib.bib38 "")\], with a focus on object layout \[ [59](https://arxiv.org/html/2409.11402v1#bib.bib59 "")\], counting \[ [2](https://arxiv.org/html/2409.11402v1#bib.bib2 "")\], object-level grounding \[ [177](https://arxiv.org/html/2409.11402v1#bib.bib177 "")\], mobile phone photo with varying quality \[ [39](https://arxiv.org/html/2409.11402v1#bib.bib39 "")\], visual reasoning \[ [45](https://arxiv.org/html/2409.11402v1#bib.bib45 ""); [69](https://arxiv.org/html/2409.11402v1#bib.bib69 ""); [168](https://arxiv.org/html/2409.11402v1#bib.bib168 ""); [74](https://arxiv.org/html/2409.11402v1#bib.bib74 "")\], and knowledge-based VQA \[ [91](https://arxiv.org/html/2409.11402v1#bib.bib91 ""); [125](https://arxiv.org/html/2409.11402v1#bib.bib125 "")\].
The ability to understand charts, diagrams, tables, document images is a critical real-world application of multimodal LLMs. To enhance this capability, we have incorporated a diverse set of datasets (e.g., DVQA \[ [51](https://arxiv.org/html/2409.11402v1#bib.bib51 "")\], PlotQA \[ [97](https://arxiv.org/html/2409.11402v1#bib.bib97 "")\], WikiTableQuestions \[ [116](https://arxiv.org/html/2409.11402v1#bib.bib116 "")\], DocVQA \[ [95](https://arxiv.org/html/2409.11402v1#bib.bib95 "")\]).
OCR is a fundamental capability of multimodal LLMs, as it is directly related to performance on tasks involving scene text \[ [131](https://arxiv.org/html/2409.11402v1#bib.bib131 "")\], screenshots \[ [43](https://arxiv.org/html/2409.11402v1#bib.bib43 "")\], charts \[ [93](https://arxiv.org/html/2409.11402v1#bib.bib93 "")\], tables \[ [20](https://arxiv.org/html/2409.11402v1#bib.bib20 "")\], document images \[ [95](https://arxiv.org/html/2409.11402v1#bib.bib95 "")\], and handwritten text.
As a result, we have incorporated a substantial amount of OCR-related datasets in our SFT blend.
Another important capability is mathematical reasoning within a visual context. To enhance this, we have incorporated many multimodal math reasoning datasets listed in Table [6](https://arxiv.org/html/2409.11402v1#S5.T6 "Table 6 ‣ 5.1 Multimodal Pretraining Data ‣ 5 Training Data ‣ NVLM: Open Frontier-Class Multimodal LLMs").
Interestingly, the abundant multimodal math data not only leads to significant improvements in vision-language tasks like MathVista \[ [87](https://arxiv.org/html/2409.11402v1#bib.bib87 "")\], but also results in substantial gains on text-only math benchmarks, including GSM8K \[ [24](https://arxiv.org/html/2409.11402v1#bib.bib24 "")\] and MATH \[ [41](https://arxiv.org/html/2409.11402v1#bib.bib41 "")\].

Report issue for preceding element

Following previous leading open-source work \[e.g., [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\], we incorporate the training splits of datasets including ChartQA \[ [93](https://arxiv.org/html/2409.11402v1#bib.bib93 "")\], DocVQA \[ [95](https://arxiv.org/html/2409.11402v1#bib.bib95 "")\], VQAv2 \[ [38](https://arxiv.org/html/2409.11402v1#bib.bib38 "")\], TextVQA \[ [131](https://arxiv.org/html/2409.11402v1#bib.bib131 "")\] and AI2D \[ [55](https://arxiv.org/html/2409.11402v1#bib.bib55 "")\] into the SFT blend. Their test sets are used as evaluation benchmarks in Section [6](https://arxiv.org/html/2409.11402v1#S6 "6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs"), meaning they are not evaluated in a zero-shot setting.
Note that it is unknown whether the proprietary multimodal LLMs are being evaluated on these benchmarks in a zero-shot or fine-tuning setting, as no information is provided regarding their training datasets.

Report issue for preceding element

### 5.3 Text-only SFT Data

Report issue for preceding element

We curated a high-quality text-only SFT dataset and incorporated it into the multimodal fine-tuning stage, effectively preserving the LLM backbone’s text-only performance and preventing catastrophic forgetting. Previous leading open-access multimodal LLMs \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 ""); [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\] also include text-only SFT datasets but still show significant performance degradation on text-only benchmarks (see Table [8](https://arxiv.org/html/2409.11402v1#S6.T8 "Table 8 ‣ 6.3 Main Results ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs") for details). The key difference between our recipe and theirs lies in the quality of the data.

Report issue for preceding element

Our text-only SFT dataset is built on top of open-source SFT datasets. We collect SFT datasets from general categories, including ShareGPT \[ [21](https://arxiv.org/html/2409.11402v1#bib.bib21 ""); [138](https://arxiv.org/html/2409.11402v1#bib.bib138 "")\], SlimOrca \[ [70](https://arxiv.org/html/2409.11402v1#bib.bib70 ""); [101](https://arxiv.org/html/2409.11402v1#bib.bib101 "")\], EvolInstruct \[ [156](https://arxiv.org/html/2409.11402v1#bib.bib156 "")\], GPTeacher \[ [137](https://arxiv.org/html/2409.11402v1#bib.bib137 "")\], AlpacaGPT4 \[ [118](https://arxiv.org/html/2409.11402v1#bib.bib118 "")\], and UltraInteract \[ [163](https://arxiv.org/html/2409.11402v1#bib.bib163 "")\]. Additionally, we collect datasets from math category, including OrcaMathWordProblems \[ [99](https://arxiv.org/html/2409.11402v1#bib.bib99 "")\], MathInstruct \[ [165](https://arxiv.org/html/2409.11402v1#bib.bib165 "")\], MetaMath \[ [162](https://arxiv.org/html/2409.11402v1#bib.bib162 "")\], and from code category, including Magicoder \[ [151](https://arxiv.org/html/2409.11402v1#bib.bib151 "")\], WizardCoder \[ [88](https://arxiv.org/html/2409.11402v1#bib.bib88 "")\], and GlaiveCodeAssistant \[ [37](https://arxiv.org/html/2409.11402v1#bib.bib37 "")\]. After that, we leverage OpenAI models, GPT-4o \[ [108](https://arxiv.org/html/2409.11402v1#bib.bib108 "")\] and GPT-4o-mini \[ [109](https://arxiv.org/html/2409.11402v1#bib.bib109 "")\], to further refine the responses of the prompts from these datasets to enhance the quality of our SFT dataset. Finally, we conduct data decontamination to make sure our dataset does not contain the prompts from all benchmark test datasets.

Report issue for preceding element

## 6 Results

Report issue for preceding element

In this section, we present a comprehensive evaluation of the NVLM-1.0 model family across a wide range of benchmarks to assess their multimodal capabilities, comparing them to other leading open-access and proprietary multimodal LLMs. Additionally, we evaluate the NVLM-1.0 models and other top open-access multimodal LLMs on key text-only benchmarks, demonstrating either no degradation or even improvements in the text-only performance of the NVLM-1.0 models, in sharp contrast to the significant degradation observed in other open-access multimodal LLMs.

Report issue for preceding element

### 6.1 Benchmarks

Report issue for preceding element

We first introduce the vision-language and text-only benchmarks used in this work.
Following previous frontier-class multimodal LLMs \[e.g., [108](https://arxiv.org/html/2409.11402v1#bib.bib108 ""); [6](https://arxiv.org/html/2409.11402v1#bib.bib6 ""); [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\], we evaluate NVLM on nine vision-language benchmarks, focusing on multimodal reasoning, math reasoning in visual context, natural image understanding, scene-text reading, chart understanding, document understanding, real-world perception, and OCR capabilities:

Report issue for preceding element

1. ⋄⋄\\diamond⋄


MMMU \[ [166](https://arxiv.org/html/2409.11402v1#bib.bib166 "")\] is one of the most popular multimodal reasoning benchmarks, covering multidisciplinary college-level problems. We do evaluations on both validation and test sets.

Report issue for preceding element

2. ⋄⋄\\diamond⋄


MathVista \[ [87](https://arxiv.org/html/2409.11402v1#bib.bib87 "")\] is a math reasoning benchmark that covers a variety of mathematical problems, e.g., geometry, function plot, table/chart related arithmetic, in visual contexts. We perform the evaluation on its testmini set.

Report issue for preceding element

3. ⋄⋄\\diamond⋄


VQAv2 \[ [38](https://arxiv.org/html/2409.11402v1#bib.bib38 "")\] is a natural image understanding benchmark. We evaluate NVLM models on the test-dev set.

Report issue for preceding element

4. ⋄⋄\\diamond⋄


AI2D \[ [55](https://arxiv.org/html/2409.11402v1#bib.bib55 "")\] is a multimodal reasoning dataset with Grade School Science diagrams.We evaluate the test set using two evaluation settings from VLMEvalKit \[ [31](https://arxiv.org/html/2409.11402v1#bib.bib31 "")\] (see Appendix [10](https://arxiv.org/html/2409.11402v1#A4.F10 "Figure 10 ‣ Appendix D Evaluation Details of AI2D ‣ NVLM: Open Frontier-Class Multimodal LLMs") for examples).
In the first setting (“test”), the text in the image is replaced with letter options from the answer choices. In the second setting (“test\_no\_mask”), the text in the image is replaced with both the letter option and the corresponding value of the answer choices, which we refer to as _no\_mask_. Note that the first setting is used as the default metric unless _no\_mask_ is explicitly stated.

Report issue for preceding element

5. ⋄⋄\\diamond⋄


TextVQA \[ [131](https://arxiv.org/html/2409.11402v1#bib.bib131 "")\] is a scene-text reading benchmark that includes various text-reading problems from natural images. We condudt evaluation on its validation set.

Report issue for preceding element

6. ⋄⋄\\diamond⋄


ChartQA \[ [93](https://arxiv.org/html/2409.11402v1#bib.bib93 "")\] is a chart understanding benchmark that involves visual and logical reasoning. We perform evaluation on its test set.

Report issue for preceding element

7. ⋄⋄\\diamond⋄


DocVQA \[ [95](https://arxiv.org/html/2409.11402v1#bib.bib95 "")\] is dataset for VQA on document images. We do evaluation on its test set.

Report issue for preceding element

8. ⋄⋄\\diamond⋄


RealWordQA \[ [153](https://arxiv.org/html/2409.11402v1#bib.bib153 "")\] is a benchmark focused on physical world perception and understanding.

Report issue for preceding element

9. ⋄⋄\\diamond⋄


OCRBench \[ [81](https://arxiv.org/html/2409.11402v1#bib.bib81 "")\] is a comprehensive benchmark created to evaluate the OCR capabilities of multimodal LLMs. It consists of five components: text recognition in images, scene text-centric VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.

Report issue for preceding element


To assess the degradation of text-only performance during multimodal training, we evaluate and compare the multimodal LLMs against their corresponding text-only LLM backbones across four key benchmarks, focusing on multidisciplinary knowledge reasoning, math reasoning, and coding capabilities:

Report issue for preceding element

1. ∘\\circ∘


MMLU \[ [40](https://arxiv.org/html/2409.11402v1#bib.bib40 "")\] is a multidisciplinary benchmark that covers 57 subjects, including elementary mathematics, U.S. history, computer science, law, and more


Report issue for preceding element

2. ∘\\circ∘


GSM8K \[ [24](https://arxiv.org/html/2409.11402v1#bib.bib24 "")\] is a benchmark consisting of grade school math word problems.

Report issue for preceding element

3. ∘\\circ∘


MATH \[ [41](https://arxiv.org/html/2409.11402v1#bib.bib41 "")\] is a math reasoning benchmark that covers math problems ranging across 5 levels of difficulty and 7 sub-disciplines.

Report issue for preceding element

4. ∘\\circ∘


HumanEval \[ [16](https://arxiv.org/html/2409.11402v1#bib.bib16 "")\] is a coding benchmark that measures functional correctness for synthesizing programs from docstrings.

Report issue for preceding element


### 6.2 Baseline Models

Report issue for preceding element

We compare our models to leading proprietary and open-access multimodal LLMs. The state-of-the-art (SOTA) proprietary models include GPT-4o \[ [108](https://arxiv.org/html/2409.11402v1#bib.bib108 "")\], Claude 3.5 \[ [6](https://arxiv.org/html/2409.11402v1#bib.bib6 "")\], Gemini Pro 1.5 \[ [36](https://arxiv.org/html/2409.11402v1#bib.bib36 "")\], and Grok-2 \[ [154](https://arxiv.org/html/2409.11402v1#bib.bib154 "")\].
The SOTA open-access models include InternVL-2-Llama3-76B \[ [112](https://arxiv.org/html/2409.11402v1#bib.bib112 "")\], InternVL 2-Pro \[ [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\], LLaVA-OneVision 72B \[ [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\], Llama 3-V 70B and 405B \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\]. Note that the model weights of top-performing InternVL 2-Pro (size unspecified) and Llama 3-V have not yet been made open-access.
We optimize and evaluate the following NVLM-1.0 models: _i)_ decoder-only NVLM-D 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B, which process image tokens within the LLM decoder,
_ii)_ cross-attention-based NVLM-X 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B, which handle image tokens through X𝑋Xitalic\_X-attention layers,
and _iii)_ hybrid NVLM-H 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B, which process global thumbnail image tokens using self-attention layers in the LLM decoder, and regular tile image tokens using X𝑋Xitalic\_X-attention layers.

Report issue for preceding element

Given limited computational resources and the goal of building a frontier-class multimodal LLM, we used the smaller 34B models for faster ablation studies and iterations, without focusing on careful checkpoint selection or hyperparameter optimization.
We include the unoptimized 34B results in Appendix [F](https://arxiv.org/html/2409.11402v1#A6 "Appendix F Unoptimized Results Using Yi-34B as the Backbone LLM ‣ NVLM: Open Frontier-Class Multimodal LLMs") for reference purpose.
Note that, although our 34B results are not optimized, the 34B NVLM models still significantly outperform other models, including VILA-1.5 40B \[ [71](https://arxiv.org/html/2409.11402v1#bib.bib71 "")\] and Cambrian-1 34B \[ [139](https://arxiv.org/html/2409.11402v1#bib.bib139 "")\].

Report issue for preceding element

Table 7: Evaluation on vision-language and text-only benchmarks. For vision-language benchmarks, all baseline model results are sourced from official reports and the benchmark hosts.
For open multimodal LLMs, we list the models that were open-access at the time of this report’s publication and mark \* for models not yet open-access.
We highlight the highest score for each benchmark in both the proprietary and open-access categories.
Text-only Avg. 4 represents the average accuracy degradation (–) or improvement (+) of the multimodal LLM compared to its backbone LLM on text-only benchmarks after multimodal training, measured across four key benchmarks: MMLU, GSM8K, MATH, and HumanEval (see Table [8](https://arxiv.org/html/2409.11402v1#S6.T8 "Table 8 ‣ 6.3 Main Results ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs") for full results).

TasksMMMUMathVistaVQAv2AI2DTextVQAChartQADocVQAReal-OCR-Text-onlytest / valtestminitest-devtest / _no\_mask_valtesttestWorldQABenchAvg. 4_Proprietary_GPT-4V \[ [107](https://arxiv.org/html/2409.11402v1#bib.bib107 "")\]56.1 / 56.849.977.278.278.078.588.461.4645-GPT-4-Turbo \[ [106](https://arxiv.org/html/2409.11402v1#bib.bib106 "")\]-  / 63.158.1-89.4-78.187.2-678-GPT-4o \[ [108](https://arxiv.org/html/2409.11402v1#bib.bib108 "")\]-  / 69.163.8-94.2-85.792.8-736-Claude 3 Sonnet \[ [5](https://arxiv.org/html/2409.11402v1#bib.bib5 "")\]-  / 53.147.9-88.7-81.189.551.9646-Claude 3 Opus \[ [5](https://arxiv.org/html/2409.11402v1#bib.bib5 "")\]-  / 59.450.5-88.1-80.889.349.8694-Claude 3.5 Sonnet \[ [6](https://arxiv.org/html/2409.11402v1#bib.bib6 "")\]-  / 68.367.7-94.7-90.895.2-788-Gemini Pro 1.0 \[ [35](https://arxiv.org/html/2409.11402v1#bib.bib35 "")\]-  / 47.945.271.273.974.674.188.1-659-Gemini Ultra 1.0 \[ [35](https://arxiv.org/html/2409.11402v1#bib.bib35 "")\]-  / 59.453.077.879.582.380.890.9---Gemini Pro 1.5 \[ [36](https://arxiv.org/html/2409.11402v1#bib.bib36 "")\]-  / 58.552.180.280.373.581.386.567.5--Gemini Pro 1.5 (Aug 2024)-  / 62.263.980.294.478.787.293.170.4754-Grok-1.5V \[ [153](https://arxiv.org/html/2409.11402v1#bib.bib153 "")\]-  / 53.652.8-88.378.176.185.668.7--Grok-2 \[ [154](https://arxiv.org/html/2409.11402v1#bib.bib154 "")\]-  / 66.169.0----93.6---_Others_QWen-VL-MAX46.8 / 51.451.078.879.379.579.893.1-723-Adept Fuyu-Heavy \[ [3](https://arxiv.org/html/2409.11402v1#bib.bib3 "")\]-  / 48.3-77.881.2-75.4----_Open-access_LLaVA-Next 34B \[ [80](https://arxiv.org/html/2409.11402v1#bib.bib80 "")\]44.7 / 51.146.5--69.5---574-VILA-1.5 40B \[ [71](https://arxiv.org/html/2409.11402v1#bib.bib71 "")\]46.9 / 51.9-84.3------– 6.9Cambrian-1 34B \[ [139](https://arxiv.org/html/2409.11402v1#bib.bib139 "")\] -  / 49.753.2-79.776.775.675.567.8600-LLaVA-OneVision 72B \[ [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\]-  / 56.867.5-85.6-83.791.3--– 6.3InternVL-1.2 40B \[ [19](https://arxiv.org/html/2409.11402v1#bib.bib19 "")\]-  / 51.647.7-79.072.568.057.767.5569-InternVL-1.5 26B \[ [18](https://arxiv.org/html/2409.11402v1#bib.bib18 "")\]-  / 45.253.5-80.780.683.890.966.0724-InternVL-2 40B \[ [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\]-  / 53.963.7-87.183.086.293.971.8837-InternVL-2-Llama3-76B-  / 55.265.5-87.6 / 94.884.488.494.172.2839– 6.7InternVL-2-Pro\[ [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\]-  / 58.966.3-87.3 / 96.0-87.195.1-837-\*Llama 3-V 70B\[ [32](https://arxiv.org/html/2409.11402v1#bib.bib32 "")\]-  / 60.6-79.193.083.483.292.2--0Llama 3-V 405B\[ [32](https://arxiv.org/html/2409.11402v1#bib.bib32 "")\]-  / 64.5-80.294.184.885.892.6--0NVLM-D 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B54.6 / 59.765.285.485.2 / 94.282.186.092.669.7853\+ 4.3NVLM-X 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B53.6 / 57.464.685.284.2 / 93.680.282.982.966.1828\+ 2.5NVLM-H 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B53.0 / 60.266.685.283.8 / 93.380.383.383.166.0831\+ 2.7
Report issue for preceding element

Report issue for preceding element

### 6.3 Main Results

Report issue for preceding element

The main results are presented in Table [7](https://arxiv.org/html/2409.11402v1#S6.T7 "Table 7 ‣ 6.2 Baseline Models ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs"), which includes outcomes from nine vision-language benchmarks and four text-only benchmarks.
Our NVLM-1.0 72B models rival the leading proprietary models (e.g., GPT-4o) and open-access models, including LLaMA 3V (not yet publicly available) and InternVL 2.
Specifically, the following observations can be drawn from Table [7](https://arxiv.org/html/2409.11402v1#S6.T7 "Table 7 ‣ 6.2 Baseline Models ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs"):

Report issue for preceding element

1. ∙∙\\bullet∙


NVLM-D1.0 72B achieves the highest scores on OCRBench (853) and VQAv2 (85.4) among all leading proprietary and open-access models. Its MMMU score (59.7) also significantly surpasses all leading open-access models at the time of this report’s publication, including LLaVA-OneVision 72B (56.8) \[ [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\] and InternVL-2-Llama3-76B (55.2) \[ [112](https://arxiv.org/html/2409.11402v1#bib.bib112 "")\].
On AI2D, TextVQA, ChartQA, and DocVQA, it performs only slightly worse than the best-performing InternVL-2-Llama3-76B, matches very strong GPT-4o \[ [108](https://arxiv.org/html/2409.11402v1#bib.bib108 "")\], and significantly outperforms other leading open-access models, including Cambrian-1 \[ [139](https://arxiv.org/html/2409.11402v1#bib.bib139 "")\] and LLaVA-OneVision 72B \[ [65](https://arxiv.org/html/2409.11402v1#bib.bib65 "")\].

Report issue for preceding element

2. ∙∙\\bullet∙


NVLM-H1.0 72B achieves the highest MMMU (Val) score (60.2) among all multimodal LLMs that are open-access at the time of this report’s publication. It also achieves the best MathVista score (66.6) within NVLM-1.0 family, which already outperforms many very strong models including GPT-4o \[ [108](https://arxiv.org/html/2409.11402v1#bib.bib108 "")\], Gemini Pro 1.5 (Aug 2024) \[ [36](https://arxiv.org/html/2409.11402v1#bib.bib36 "")\], InternVL-2-Pro \[ [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\].
This demonstrate its superb multimodal reasoning capability.

Report issue for preceding element

3. ∙∙\\bullet∙


NVLM-X1.0 72B also achieves frontier-class results and stands as the best-in-class cross-attention-based multimodal LLMs, rivaling the yet-to-be-released Llama 3-V 70B \[ [82](https://arxiv.org/html/2409.11402v1#bib.bib82 "")\].
One notable advantage of NVLM-X1.0 is its significantly faster training and inference speeds compared to its decoder-only counterpart, as demonstrated in Table [3](https://arxiv.org/html/2409.11402v1#S4.T3 "Table 3 ‣ Decoder-only vs. X-attention. ‣ 4.3 NVLM-X: X-attention Model ‣ 4 NVLM: Models and Training Methods ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

4. ∙∙\\bullet∙


Open-access multimodal LLMs, such as LLaVA-OneVision 72B and InternVL-2-Llama3-76B, show significant performance degradation on text-only tasks after multimodal training. In contrast, our NVLM-1.0 models exhibit even improved text-only performance, thanks to the inclusion of high-quality text-only SFT data. This demonstrates that unfreezing the LLM backbone during multimodal SFT does not compromise text performance, as long as high-quality text alignment data is incorporated.

Report issue for preceding element


Table 8: Evaluation on text benchmarks: MMLU, GSM8K, MATH and HumanEval.
For leading proprietary models, information about potential text performance degradation during multimodal training has not been disclosed.
The model weights of \*LLaMA 3-V had not been released at the time of this report.
Text-only Avg. 4 represents the average accuracy degradation (–) or improvement (+) of the multimodal LLM compared to its backbone LLM on text benchmarks after multimodal training.

TasksBackboneMMLUGSM8KMATHHumanEvalAvg.Text-onlyLLMAccuracyAvg. 4_Proprietary_GPT-4o \[ [108](https://arxiv.org/html/2409.11402v1#bib.bib108 "")\]N/A88.7-76.690.2-unknownGemini Pro 1.5 (Aug 2024) \[ [36](https://arxiv.org/html/2409.11402v1#bib.bib36 "")\]N/A85.990.867.784.182.1unknownClaude 3.5 Sonnet \[ [6](https://arxiv.org/html/2409.11402v1#bib.bib6 "")\]N/A88.796.471.192.087.0unknown_Open LLM__(a)_ Nous-Hermes-2-Yi-34B \[ [102](https://arxiv.org/html/2409.11402v1#bib.bib102 "")\]N/A75.578.621.843.354.8N/A_(b)_ Qwen2-72B-Instruct \[ [119](https://arxiv.org/html/2409.11402v1#bib.bib119 "")\]N/A82.391.159.786.079.8N/A_(c)_ Llama-3-70B-Instruct \[ [32](https://arxiv.org/html/2409.11402v1#bib.bib32 "")\]N/A82.093.051.081.776.6N/A_(d)_ Llama-3.1-70B-Instruct \[ [32](https://arxiv.org/html/2409.11402v1#bib.bib32 "")\]N/A83.695.168.080.581.8N/A_(e)_ Llama-3.1-405B-Instruct \[ [32](https://arxiv.org/html/2409.11402v1#bib.bib32 "")\]N/A87.396.873.889.086.7N/A_Open Multimodal LLM_VILA-1.5 40B \[ [71](https://arxiv.org/html/2409.11402v1#bib.bib71 "")\]_(a)_73.367.516.834.147.9– 6.9LLaVA-OneVision 72B \[ [80](https://arxiv.org/html/2409.11402v1#bib.bib80 "")\]_(b)_80.689.949.274.473.5– 6.3InternVL-2-Llama3-76B \[ [111](https://arxiv.org/html/2409.11402v1#bib.bib111 "")\]_(c)_78.587.142.571.369.9 – 6.7Llama 3-V 70B\[ [32](https://arxiv.org/html/2409.11402v1#bib.bib32 "")\]_(d)_83.695.168.080.581.80Llama 3-V 405B\[ [32](https://arxiv.org/html/2409.11402v1#bib.bib32 "")\]_(e)_87.396.873.889.086.70NVLM-D 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B_(b)_82.092.973.188.484.1\+ 4.3NVLM-X 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B_(b)_81.491.870.685.282.3\+ 2.5NVLM-H 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B_(b)_80.491.571.486.682.5\+ 2.7
Report issue for preceding element

Report issue for preceding element

### 6.4 Text-only Performance

Report issue for preceding element

We present detailed results of text-only performance for our NVLM models, along with leading proprietary and open-access multimodal LLMs, in Table [8](https://arxiv.org/html/2409.11402v1#S6.T8 "Table 8 ‣ 6.3 Main Results ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs").
It can be observed that all open-access models experience a significant drop in accuracy compared to their LLM backbones. For instance, VILA-1.5 40B sees a notable decrease of 6.9 points, from 54.8 to 47.9. Similarly, the average accuracy of LLaVA-OneVision 72B and InternVL-2-Llama3-76B drops by 6.3 and 6.9 points, respectively.
Llama 3-V experiences no degradation in text-only performance because the LLM is frozen during multimodal training.
However, as we will demonstrate in § [6.5](https://arxiv.org/html/2409.11402v1#S6.SS5 "6.5 Frozen versus Unfrozen LLM during Mutimodal SFT ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs"), this frozen LLM strategy may lead to an unnecessary trade-off in vision-language performance.

Report issue for preceding element

By incorporating high-quality text SFT data, both the NVLM-1.0 72B models achieve higher average accuracy than their respective LLM backbone, Qwen2-72B-Instruct \[ [119](https://arxiv.org/html/2409.11402v1#bib.bib119 "")\]. For example, the average accuracy across four benchmarks increases from 79.8 to 84.1 for the NVLM-D1.0 72B model.
It is particularly interesting that the math capabilities of the NVLM-1.0 models improve significantly compared to its text-only backbone. We attribute this to the high-quality text-only SFT data and the substantial amount of multimodal math data included in our training blend, which enhances math reasoning skills overall, regardless of the modality.

Report issue for preceding element

Table 9: Impact on vision-language performance with a frozen vs. unfrozen LLM backbone during multimodal SFT for the cross-attention model NVLM-X.

TasksMMMUMathVistaVQAv2AI2DTextVQAChartQADocVQARealWorld-OCR-test / valtestminitest-devtestvaltesttestQABenchNVLM-X 34B (frozen)43.2 / 51.651.883.872.472.474.473.263.4696NVLM-X 34B47.2 / 54.059.284.579.678.279.479.264.8802NVLM-X 72B (frozen)50.6 / 54.460.685.376.276.276.276.465.3722NVLM-X 1.0 1.0{}\_{\\text{ 1.0}}start\_FLOATSUBSCRIPT 1.0 end\_FLOATSUBSCRIPT 72B53.6 / 57.464.685.284.280.282.982.966.1828
Report issue for preceding element

Report issue for preceding element

### 6.5 Frozen versus Unfrozen LLM during Mutimodal SFT

Report issue for preceding element

In this subsection, we compare two methods for maintaining text-only performance in the cross-attention-based NVLM-X: _i)_ Freezing the LLM during multimodal SFT training, which ensures no degradation in text performance due to the gated X𝑋Xitalic\_X-attention layers, and _ii)_ our default approach, which incorporates a high-quality text-only dataset during multimodal SFT training.
It is important to note that freezing the LLM for decoder-only multimodal model during SFT leads to poor results on vision-language tasks (as demonstrated in a similar study by \[ [71](https://arxiv.org/html/2409.11402v1#bib.bib71 "")\]), due to the very limited capacity of the MLP projector module.

Report issue for preceding element

In Table [9](https://arxiv.org/html/2409.11402v1#S6.T9 "Table 9 ‣ 6.4 Text-only Performance ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs"), it can be seen that freezing the LLM yields reasonably good results. Notably, accuracy scales well as the model size increases from 34B to 72B, reaffirming findings from the original Flamingo study, which also froze the LLM during multimodal training. However, compared to the unfrozen setting, freezing the LLM still results in a moderate performance drop on vision-language tasks. For example, NVLM-X1.0 72B with a frozen LLM performs only comparably to the smaller NVLM-X-34B, showing slight improvements on multimodal reasoning tasks (MMMU and MathVista) and natural image understanding tasks (VQAv2 and RealWorldQA), while falling behind on OCR-related tasks (AI2D, TextVQA, ChartQA, DocVQA, and OCRBench).

Report issue for preceding element

## 7 Conclusion

Report issue for preceding element

We introduce NVLM-1.0, a family of frontier multimodal large language models that achieve state-of-the-art results on vision-language tasks, rivaling leading multimodal LLMs, without compromising text-only performance during multimodal training.
Furthermore, we provide key insights on architecture design, tile-based dynamic high-resolution input, multimodal training data curation, and how to achieve production-grade multimodality with even improved text-only performance after multimodal training.
We open-source the model weights and provide the full technical details to the community.

Report issue for preceding element

## 8 Acknowledgement

Report issue for preceding element

We would like to express our sincere gratitude to Jared Casper, Peter Dykas, and Mike Chrzanowski for their support with Megatron-LM.
Our thanks also go to Lukas Voegtle and Philipp Fischer for the implementation and debugging efforts with their Megatron-Energon dataloader, which is released early. We are also grateful to Hongxu Yin, Timo Roman, Karan Sapra and Valerii Iakovlev for their help and support.

Report issue for preceding element

## References

Report issue for preceding element

- VQA \[2023\]↑
VQAonBD 2023 Dataset, 2023.
URL [https://ilocr.iiit.ac.in/vqabd/dataset.html](https://ilocr.iiit.ac.in/vqabd/dataset.html "").

- Acharya et al. \[2019\]↑
Acharya, M., Kafle, K., and Kanan, C.
Tallyqa: Answering complex counting questions.
In _Proceedings of the AAAI conference on artificial intelligence_, 2019.

- Adept \[2024\]↑
Adept.
Adept Fuyu-Heavy: A new multimodal model, 2024.

- Alayrac et al. \[2022\]↑
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.
Flamingo: A visual language model for few-shot learning.
In _NeurIPS_, 2022.

- Anthropic \[2024a\]↑
Anthropic.
Introducing the next generation of Claude, 2024a.

- Anthropic \[2024b\]↑
Anthropic.
Claude 3.5 Sonnet, 2024b.

- Awadalla et al. \[2023\]↑
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L.
Openflamingo: An open-source framework for training large autoregressive vision-language models.
_arXiv preprint arXiv:2308.01390_, 2023.

- Bai et al. \[2023\]↑
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J.
Qwen-VL: A frontier large vision-language model with versatile abilities.
_arXiv preprint arXiv:2308.12966_, 2023.

- Bao et al. \[2021\]↑
Bao, H., Dong, L., Piao, S., and Wei, F.
Beit: Bert pre-training of image transformers.
_arXiv preprint arXiv:2106.08254_, 2021.

- Biten et al. \[2019\]↑
Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D.
Scene text visual question answering.
In _Proceedings of the IEEE/CVF international conference on computer vision_, pp.  4291–4301, 2019.

- Brown et al. \[2020\]↑
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.
Language models are few-shot learners.
_NeurIPS_, 2020.

- Byeon et al. \[2022\]↑
Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., and Kim, S.
Coyo-700m: Image-text pair dataset.
[https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset ""), 2022.

- Cao & Xiao \[2022\]↑
Cao, J. and Xiao, J.
An augmented benchmark dataset for geometric question answering through dual parallel text encoding.
In _Proceedings of the 29th International Conference on Computational Linguistics_, pp.  1511–1520, 2022.

- Chen et al. \[2022a\]↑
Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., and Liang, X.
Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression.
_arXiv preprint arXiv:2212.02746_, 2022a.

- Chen et al. \[2023\]↑
Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., and Elhoseiny, M.
MiniGPT-v2: Large language model as a unified interface for vision-language multi-task learning.
_arXiv preprint arXiv:2310.09478_, 2023.

- Chen et al. \[2021\]↑
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.
Evaluating large language models trained on code.
_arXiv preprint arXiv:2107.03374_, 2021.

- Chen et al. \[2022b\]↑
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al.
Pali: A jointly-scaled multilingual language-image model.
_arXiv preprint arXiv:2209.06794_, 2022b.

- Chen et al. \[2024a\]↑
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al.
How far are we to GTP-4V? closing the gap to commercial multimodal models with open-source suites.
_arXiv preprint arXiv:2404.16821_, 2024a.

- Chen et al. \[2024b\]↑
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.
InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp.  24185–24198, 2024b.

- Cheng et al. \[2021\]↑
Cheng, Z., Dong, H., Wang, Z., Jia, R., Guo, J., Gao, Y., Han, S., Lou, J.-G., and Zhang, D.
Hitab: A hierarchical table dataset for question answering and natural language generation.
_arXiv preprint arXiv:2108.06712_, 2021.

- Chiang et al. \[2023\]↑
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P.
Vicuna: An open-source chatbot impressing gpt-4 with 90%\* chatgpt quality, 2023.
URL [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/ "").

- Chng et al. \[2019\]↑
Chng, C. K., Liu, Y., Sun, Y., Ng, C. C., Luo, C., Ni, Z., Fang, C., Zhang, S., Han, J., Ding, E., et al.
Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art.
In _2019 International Conference on Document Analysis and Recognition (ICDAR)_, pp.  1571–1576. IEEE, 2019.

- Chung et al. \[2024\]↑
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al.
Scaling instruction-finetuned language models.
_Journal of Machine Learning Research_, 2024.

- Cobbe et al. \[2021a\]↑
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.
Training verifiers to solve math word problems.
_arXiv preprint arXiv:2110.14168_, 2021a.

- Cobbe et al. \[2021b\]↑
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.
Training verifiers to solve math word problems, 2021.
_URL https://arxiv. org/abs/2110.14168_, 2021b.

- Dai et al. \[2023\]↑
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S.
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning, 2023.

- Dehghani et al. \[2023\]↑
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al.
Scaling vision transformers to 22 billion parameters.
In _International Conference on Machine Learning_, pp.  7480–7512. PMLR, 2023.

- Diem et al. \[2014\]↑
Diem, M., Fiel, S., Kleber, F., Sablatnig, R., Saavedra, J. M., Contreras, D., Barrios, J. M., and Oliveira, L. S.
Icfhr 2014 competition on handwritten digit string recognition in challenging datasets (hdsrc 2014).
In _2014 14th International Conference on Frontiers in Handwriting Recognition_, pp.  779–784. IEEE, 2014.

- Ding et al. \[2023\]↑
Ding, Y., Luo, S., Chung, H., and Han, S. C.
Pdf-vqa: A new dataset for real-world vqa on pdf documents.
In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_, pp.  585–601. Springer, 2023.

- Dong et al. \[2024\]↑
Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Zhang, S., Duan, H., Zhang, W., Li, Y., et al.
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD.
_arXiv preprint arXiv:2404.06512_, 2024.

- Duan et al. \[2024\]↑
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., and Chen, K.
Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024.
URL [https://arxiv.org/abs/2407.11691](https://arxiv.org/abs/2407.11691 "").

- Dubey et al. \[2024\]↑
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.
The llama 3 herd of models.
_arXiv preprint arXiv:2407.21783_, 2024.

- Gadre et al. \[2024\]↑
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al.
Datacomp: In search of the next generation of multimodal datasets.
_Advances in Neural Information Processing Systems_, 36, 2024.

- Gao et al. \[2023\]↑
Gao, J., Pi, R., Zhang, J., Ye, J., Zhong, W., Wang, Y., Hong, L., Han, J., Xu, H., Li, Z., et al.
G-llava: Solving geometric problem with multi-modal large language model.
_arXiv preprint arXiv:2312.11370_, 2023.

- Gemini-Team \[2023\]↑
Gemini-Team.
Gemini: a family of highly capable multimodal models.
_arXiv preprint arXiv:2312.11805_, 2023.

- Gemini-Team \[2024\]↑
Gemini-Team.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
_arXiv preprint arXiv:2403.05530_, 2024.

- Glaive-AI \[2023\]↑
Glaive-AI.
GlaiveCodeAssistant, 2023.
URL [https://huggingface.co/datasets/glaiveai/glaive-code-assistant-v2](https://huggingface.co/datasets/glaiveai/glaive-code-assistant-v2 "").

- Goyal et al. \[2017\]↑
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D.
Making the v in VQA matter: Elevating the role of image understanding in visual question answering.
In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp.  6904–6913, 2017.

- Gurari et al. \[2018\]↑
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P.
Vizwiz grand challenge: Answering visual questions from blind people.
In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp.  3608–3617, 2018.

- Hendrycks et al. \[2020\]↑
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.
Measuring massive multitask language understanding.
_arXiv preprint arXiv:2009.03300_, 2020.

- Hendrycks et al. \[2021\]↑
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J.
Measuring mathematical problem solving with the math dataset.
_arXiv preprint arXiv:2103.03874_, 2021.

- Hoffmann et al. \[2022\]↑
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al.
Training compute-optimal large language models.
_arXiv preprint arXiv:2203.15556_, 2022.

- Hsiao et al. \[2024\]↑
Hsiao, Y.-C., Zubach, F., Wang, M., and Chen, J.
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots, 2024.

- Huang et al. \[2019\]↑
Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., and Jawahar, C.
Icdar2019 competition on scanned receipt ocr and information extraction.
In _2019 International Conference on Document Analysis and Recognition (ICDAR)_, pp.  1516–1520. IEEE, 2019.

- Hudson & Manning \[2019\]↑
Hudson, D. A. and Manning, C. D.
GQA: A new dataset for real-world visual reasoning and compositional question answering.
In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pp.  6700–6709, 2019.

- \[46\]↑
IIIT.
URL [http://cvit.iiit.ac.in/projects/SceneTextUnderstanding/IIIT5K.html](http://cvit.iiit.ac.in/projects/SceneTextUnderstanding/IIIT5K.html "").

- Ilharco et al. \[2021\]↑
Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L.
OpenCLIP, 2021.

- Jaegle et al. \[2021\]↑
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J.
Perceiver: General perception with iterative attention.
In _ICML_, 2021.

- Jaume et al. \[2019\]↑
Jaume, G., Ekenel, H. K., and Thiran, J.-P.
Funsd: A dataset for form understanding in noisy scanned documents.
In _2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)_, volume 2, pp.  1–6. IEEE, 2019.

- Jiang et al. \[2023\]↑
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.
Mistral 7b.
_arXiv preprint arXiv:2310.06825_, 2023.

- Kafle et al. \[2018\]↑
Kafle, K., Price, B., Cohen, S., and Kanan, C.
DVQA: Understanding data visualizations via question answering.
In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp.  5648–5656, 2018.

- Kahou et al. \[2017\]↑
Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., and Bengio, Y.
Figureqa: An annotated figure dataset for visual reasoning.
_arXiv preprint arXiv:1710.07300_, 2017.

- Kamizuru \[2023\]↑
Kamizuru, Y., 2023.
URL [https://huggingface.co/datasets/Kamizuru00/diagram\_image\_to\_text](https://huggingface.co/datasets/Kamizuru00/diagram_image_to_text "").

- Kazemi et al. \[2023\]↑
Kazemi, M., Alvari, H., Anand, A., Wu, J., Chen, X., and Soricut, R.
Geomverse: A systematic evaluation of large models for geometric reasoning.
_arXiv preprint arXiv:2312.12241_, 2023.

- Kembhavi et al. \[2016\]↑
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A.
A diagram is worth a dozen images.
In _ECCV_, 2016.

- Kembhavi et al. \[2017\]↑
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H.
Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension.
In _Proceedings of the IEEE Conference on Computer Vision and Pattern recognition_, 2017.

- Kim et al. \[2022\]↑
Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., and Park, S.
Ocr-free document understanding transformer.
In _European Conference on Computer Vision (ECCV)_, 2022.

- Kirillov et al. \[2023\]↑
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al.
Segment anything.
In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pp.  4015–4026, 2023.

- Krishna et al. \[2017\]↑
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al.
Visual genome: Connecting language and vision using crowdsourced dense image annotations.
_International journal of computer vision_, 123:32–73, 2017.

- Kuang et al. \[2023\]↑
Kuang, J., Hua, W., Liang, D., Yang, M., Jiang, D., Ren, B., and Bai, X.
Visual information extraction in the wild: practical dataset and end-to-end solution.
In _International Conference on Document Analysis and Recognition_, 2023.

- Laboratory \[2024\]↑
Laboratory, S. A.
Sharegpt-4o, 2024.
URL [https://sharegpt4o.github.io/](https://sharegpt4o.github.io/ "").

- Laurencon et al. \[2023\]↑
Laurencon, H., van Strien, D., Bekman, S., Tronchon, L., Saulnier, L., Wang, T., Karamcheti, S., Singh, A., Pistilli, G., Jernite, Y., et al.
Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023.

- Laurençon et al. \[2024a\]↑
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A., Kiela, D., et al.
Obelics: An open web-scale filtered dataset of interleaved image-text documents.
_Advances in Neural Information Processing Systems_, 2024a.

- Laurençon et al. \[2024b\]↑
Laurençon, H., Tronchon, L., and Sanh, V.
Unlocking the conversion of web screenshots into html code with the websight dataset.
_arXiv preprint arXiv:2403.09029_, 2024b.

- Li et al. \[2024a\]↑
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C.
Llava-onevision: Easy visual task transfer.
_arXiv preprint arXiv:2408.03326_, 2024a.

- Li et al. \[2022\]↑
Li, J., Li, D., Xiong, C., and Hoi, S.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.
In _ICML_, 2022.

- Li et al. \[2023a\]↑
Li, J., Li, D., Savarese, S., and Hoi, S.
BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In _ICML_, 2023a.

- Li et al. \[2024b\]↑
Li, L., Wang, Y., Xu, R., Wang, P., Feng, X., Kong, L., and Liu, Q.
Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models.
_arXiv preprint arXiv:2403.00231_, 2024b.

- Li et al. \[2023b\]↑
Li, Z., Wang, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., Van Durme, B., and Yuille, A. L.
Super-CLEVR: A virtual benchmark to diagnose domain robustness in visual reasoning.
In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp.  14963–14973, 2023b.

- Lian et al. \[2023\]↑
Lian, W., Wang, G., Goodson, B., Pentland, E., Cook, A., Vong, C., and "Teknium".
Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023.
URL [https://https://huggingface.co/Open-Orca/SlimOrca](https://https//huggingface.co/Open-Orca/SlimOrca "").

- Lin et al. \[2024\]↑
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S.
Vila: On pre-training for visual language models.
In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp.  26689–26699, 2024.

- Lin et al. \[2014\]↑
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.
Microsoft coco: Common objects in context.
In _ECCV 2014_, 2014.

- Lindström & Abraham \[2022\]↑
Lindström, A. D. and Abraham, S. S.
Clevr-math: A dataset for compositional language, visual and mathematical reasoning.
_arXiv preprint arXiv:2208.05358_, 2022.

- Liu et al. \[2023a\]↑
Liu, F., Emerson, G., and Collier, N.
Visual spatial reasoning.
_Transactions of the Association for Computational Linguistics_, 11:635–651, 2023a.

- Liu et al. \[2023b\]↑
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L.
Aligning large multi-modal model with robust instruction tuning.
_arXiv preprint arXiv:2306.14565_, 2023b.

- Liu et al. \[2023c\]↑
Liu, F., Wang, X., Yao, W., Chen, J., Song, K., Cho, S., Yacoob, Y., and Yu, D.
Mmc: Advancing multimodal chart understanding with large-scale instruction tuning.
_arXiv preprint arXiv:2311.10774_, 2023c.

- Liu \[2023\]↑
Liu, H.
Llava-pretrain.
[https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain ""), 2023.

- Liu et al. \[2023d\]↑
Liu, H., Li, C., Li, Y., and Lee, Y. J.
Improved baselines with visual instruction tuning, 2023d.

- Liu et al. \[2023e\]↑
Liu, H., Li, C., Wu, Q., and Lee, Y. J.
Visual instruction tuning, 2023e.

- Liu et al. \[2024a\]↑
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J.
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a.
URL [https://llava-vl.github.io/blog/2024-01-30-llava-next/](https://llava-vl.github.io/blog/2024-01-30-llava-next/ "").

- Liu et al. \[2024b\]↑
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X., lin Liu, C., Jin, L., and Bai, X.
Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024b.
URL [https://arxiv.org/abs/2305.07895](https://arxiv.org/abs/2305.07895 "").

- Llama-Team \[2023\]↑
Llama-Team.
The llama 3 herd of models.
_arXiv preprint arXiv:2302.04858_, 2023.

- Lu et al. \[2021a\]↑
Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.-C.
Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning.
_arXiv preprint arXiv:2105.04165_, 2021a.

- Lu et al. \[2021b\]↑
Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S.-C.
IconQA: A new benchmark for abstract diagram understanding and visual language reasoning.
_arXiv preprint arXiv:2110.13214_, 2021b.

- Lu et al. \[2022a\]↑
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A.
Learn to explain: Multimodal reasoning via thought chains for science question answering.
_Advances in Neural Information Processing Systems_, 35:2507–2521, 2022a.

- Lu et al. \[2022b\]↑
Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A.
Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.
_arXiv preprint arXiv:2209.14610_, 2022b.

- Lu et al. \[2024\]↑
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J.
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
In _International Conference on Learning Representations (ICLR)_, 2024.

- Luo et al. \[2023\]↑
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D.
Wizardcoder: Empowering code large language models with evol-instruct.
_arXiv preprint arXiv:2306.08568_, 2023.

- Mahamoud et al. \[2022\]↑
Mahamoud, I. S., Coustaty, M., Joseph, A., d’Andecy, V. P., and Ogier, J.-M.
Qalayout: Question answering layout based on multimodal attention for visual question answering on corporate document.
In _International Workshop on Document Analysis Systems_, pp.  659–673. Springer, 2022.

- Marafioti & Laurencon \[2024\]↑
Marafioti, A. and Laurencon, H.
Docmatix - a huge dataset for document visual question answering, 2024.

- Marino et al. \[2019\]↑
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R.
OK-VQA: A visual question answering benchmark requiring external knowledge.
In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_, pp.  3195–3204, 2019.

- Marti & Bunke \[2002\]↑
Marti, U.-V. and Bunke, H.
The iam-database: an english sentence database for offline handwriting recognition.
_International journal on document analysis and recognition_, 5:39–46, 2002.

- Masry et al. \[2022\]↑
Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E.
ChartQA: A benchmark for question answering about charts with visual and logical reasoning.
_arXiv preprint arXiv:2203.10244_, 2022.

- Mathew et al. \[2021a\]↑
Mathew, M., Gomez, L., Karatzas, D., and Jawahar, C.
Asking questions on handwritten document collections.
_International Journal on Document Analysis and Recognition (IJDAR)_, 24(3):235–249, 2021a.

- Mathew et al. \[2021b\]↑
Mathew, M., Karatzas, D., and Jawahar, C.
DocVQA: A dataset for VQA on document images.
In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_, pp.  2200–2209, 2021b.

- Mathew et al. \[2022\]↑
Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C.
InfographicVQA.
In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_, pp.  1697–1706, 2022.

- Methani et al. \[2020\]↑
Methani, N., Ganguly, P., Khapra, M. M., and Kumar, P.
Plotqa: Reasoning over scientific plots.
In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_, pp.  1527–1536, 2020.

- Mishra et al. \[2019\]↑
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A.
OCR-VQA: Visual question answering by reading text in images.
In _2019 international conference on document analysis and recognition (ICDAR)_, pp.  947–952. IEEE, 2019.

- Mitra et al. \[2024\]↑
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A.
Orca-math: Unlocking the potential of slms in grade school math.
_arXiv preprint arXiv:2402.14830_, 2024.

- mlfoundations \[2021\]↑
mlfoundations.
An open source implementation of CLIP, 2021.
URL [https://github.com/mlfoundations/open\_clip](https://github.com/mlfoundations/open_clip "").

- Mukherjee et al. \[2023\]↑
Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A.
Orca: Progressive learning from complex explanation traces of gpt-4.
_arXiv preprint arXiv:2306.02707_, 2023.

- Nous \[2024\]↑
Nous.
Nous-Hermes-2-Yi-34B, 2024.

- Obeid & Hoque \[2020\]↑
Obeid, J. and Hoque, E.
Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model.
_arXiv preprint arXiv:2010.09142_, 2020.

- OpenAI \[2022\]↑
OpenAI.
Introducing ChatGPT, 2022.

- OpenAI \[2023a\]↑
OpenAI.
GPT-4, 2023a.

- OpenAI \[2023b\]↑
OpenAI.
New models and developer products announced at DevDay, 2023b.

- OpenAI \[2023c\]↑
OpenAI.
GPT-4V(ision) System Card, 2023c.

- OpenAI \[2024a\]↑
OpenAI.
Hello GPT-4o, 2024a.

- OpenAI \[2024b\]↑
OpenAI.
GPT-4o mini: advancing cost-efficient intelligence, 2024b.

- OpenAI \[2024c\]↑
OpenAI.
GPT-4o System Card, 2024c.

- OpenGVLab \[2024a\]↑
OpenGVLab.
InternVL2: Better than the Best – Expanding Performance Boundaries of Open-Source Multimodal Models with the Progressive Scaling Strategy, 2024a.

- OpenGVLab \[2024b\]↑
OpenGVLab.
InternVL2-Llama3-76B, 2024b.
URL [https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B "").

- OpenGVLab \[2024c\]↑
OpenGVLab.
InternViT-6B-448px-V1-5, 2024c.
URL [https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5 "").

- Ordonez et al. \[2011\]↑
Ordonez, V., Kulkarni, G., and Berg, T.
Im2text: Describing images using 1 million captioned photographs.
_Advances in neural information processing systems_, 24, 2011.

- Ouyang et al. \[2022\]↑
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.
Training language models to follow instructions with human feedback.
_Advances in neural information processing systems_, 2022.

- Pasupat & Liang \[2015\]↑
Pasupat, P. and Liang, P.
Compositional semantic parsing on semi-structured tables.
_arXiv preprint arXiv:1508.00305_, 2015.

- PDF-Association \[2024\]↑
PDF-Association, 2024.
URL [https://huggingface.co/datasets/pixparse/pdfa-eng-wds](https://huggingface.co/datasets/pixparse/pdfa-eng-wds "").

- Peng et al. \[2023\]↑
Peng, B., Li, C., He, P., Galley, M., and Gao, J.
Instruction tuning with gpt-4.
_arXiv preprint arXiv:2304.03277_, 2023.

- Qwen-Team \[2024\]↑
Qwen-Team.
Qwen2 technical report, 2024.

- Radford et al. \[2021\]↑
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.
Learning transferable visual models from natural language supervision.
In _ICML_, 2021.

- Rajpurkar et al. \[2016\]↑
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P.
Squad: 100,000+ questions for machine comprehension of text.
_arXiv preprint arXiv:1606.05250_, 2016.

- Reka \[2024\]↑
Reka.
Reka Core: Our Frontier Class Multimodal Language Model, 2024.

- Schuhmann et al. \[2021\]↑
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A.
Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.
_arXiv preprint arXiv:2111.02114_, 2021.

- Schuhmann et al. \[2022\]↑
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.
Laion-5B: An open large-scale dataset for training next generation image-text models.
_Advances in Neural Information Processing Systems_, 35:25278–25294, 2022.

- Schwenk et al. \[2022\]↑
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R.
A-OKVQA: A benchmark for visual question answering using world knowledge.
In _European conference on computer vision_, pp.  146–162. Springer, 2022.

- Seo et al. \[2015\]↑
Seo, M., Hajishirzi, H., Farhadi, A., Etzioni, O., and Malcolm, C.
Solving geometry problems: Combining text and diagram interpretation.
In _Proceedings of the 2015 conference on empirical methods in natural language processing_, pp.  1466–1476, 2015.

- Sharma et al. \[2018\]↑
Sharma, P., Ding, N., Goodman, S., and Soricut, R.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.
In _ACL_, 2018.

- Shi et al. \[2017\]↑
Shi, B., Yao, C., Liao, M., Yang, M., Xu, P., Cui, L., Belongie, S., Lu, S., and Bai, X.
Icdar2017 competition on reading chinese text in the wild (rctw-17).
In _2017 14th iapr international conference on document analysis and recognition (ICDAR)_, volume 1, pp.  1429–1434. IEEE, 2017.

- Shoeybi et al. \[2019\]↑
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B.
Megatron-lm: Training multi-billion parameter language models using model parallelism.
_arXiv preprint arXiv:1909.08053_, 2019.
URL [https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM "").

- Sidorov et al. \[2020\]↑
Sidorov, O., Hu, R., Rohrbach, M., and Singh, A.
Textcaps: a dataset for image captioning with reading comprehension.
In _ECCV_, 2020.

- Singh et al. \[2019\]↑
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M.
Towards VQA models that can read.
In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pp.  8317–8326, 2019.

- Singh et al. \[2021\]↑
Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., and Hassner, T.
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text.
In _The Conference on Computer Vision and Pattern Recognition_, 2021.

- Smith et al. \[2022\]↑
Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al.
Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.
_arXiv preprint arXiv:2201.11990_, 2022.

- Sun et al. \[2019\]↑
Sun, Y., Ni, Z., Chng, C.-K., Liu, Y., Luo, C., Ng, C. C., Han, J., Ding, E., Liu, J., Karatzas, D., et al.
Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt.
In _2019 International Conference on Document Analysis and Recognition (ICDAR)_, pp.  1557–1562. IEEE, 2019.

- Tanaka et al. \[2021\]↑
Tanaka, R., Nishida, K., and Yoshida, S.
Visualmrc: Machine reading comprehension on document images.
In _Proceedings of the AAAI Conference on Artificial Intelligence_, 2021.

- Tanaka et al. \[2023\]↑
Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T., Saito, I., and Saito, K.
SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images.
In _AAAI_, 2023.

- Teknium \[2023\]↑
Teknium.
GPTeacher-General-Instruct, 2023.
URL [https://huggingface.co/datasets/teknium/GPTeacher-General-Instruct](https://huggingface.co/datasets/teknium/GPTeacher-General-Instruct "").

- The-Vicuna-Team \[2023\]↑
The-Vicuna-Team.
ShareGPT-Vicuna, 2023.
URL [https://huggingface.co/datasets/anon8231489123/ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered "").

- Tong et al. \[2024\]↑
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al.
Cambrian-1: A fully open, vision-centric exploration of multimodal llms.
_arXiv preprint arXiv:2406.16860_, 2024.

- Touvron et al. \[2023a\]↑
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.
Llama: Open and efficient foundation language models.
_arXiv preprint arXiv:2302.13971_, 2023a.

- Touvron et al. \[2023b\]↑
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.
Llama 2: Open foundation and fine-tuned chat models.
_arXiv preprint arXiv:2307.09288_, 2023b.

- Van Landeghem et al. \[2023\]↑
Van Landeghem, J., Tito, R., Borchmann, Ł., Pietruszka, M., Joziak, P., Powalski, R., Jurkiewicz, D., Coustaty, M., Anckaert, B., Valveny, E., et al.
Document understanding dataset and evaluation (dude).
In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pp.  19528–19540, 2023.

- Vaswani et al. \[2017\]↑
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.
Attention is all you need.
_Advances in Neural Information Processing Systems_, 2017.

- Veit et al. \[2016\]↑
Veit, A., Matera, T., Neumann, L., Matas, J., and Belongie, S.
Coco-text: Dataset and benchmark for text detection and recognition in natural images.
_arXiv preprint arXiv:1601.07140_, 2016.

- Wang et al. \[2022a\]↑
Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L.
Git: A generative image-to-text transformer for vision and language.
_arXiv preprint arXiv:2205.14100_, 2022a.

- Wang et al. \[2022b\]↑
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al.
Image as a foreign language: Beit pretraining for all vision and vision-language tasks.
_arXiv preprint arXiv:2208.10442_, 2022b.

- Wang et al. \[2023\]↑
Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al.
Cogvlm: Visual expert for pretrained language models.
_arXiv preprint arXiv:2311.03079_, 2023.

- Wang et al. \[2020\]↑
Wang, X., Liu, Y., Shen, C., Ng, C. C., Luo, C., Jin, L., Chan, C. S., Hengel, A. v. d., and Wang, L.
On the general value of evidence, and bilingual scene-text visual question answering.
In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp.  10126–10135, 2020.

- Wang et al. \[2022c\]↑
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y.
SimVLM: Simple visual language model pretraining with weak supervision.
In _ICLR_, 2022c.

- Wei et al. \[2021\]↑
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V.
Finetuned language models are zero-shot learners.
_arXiv preprint arXiv:2109.01652_, 2021.

- Wei et al. \[2024\]↑
Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L.
Magicoder: Empowering code generation with oss-instruct.
In _Forty-first International Conference on Machine Learning_, 2024.

- Wendler \[2023\]↑
Wendler, C., 2023.
URL [https://huggingface.co/datasets/wendlerc/RenderedText](https://huggingface.co/datasets/wendlerc/RenderedText "").

- xAI \[2024a\]↑
xAI.
Grok-1.5 vision preview: Connecting the digital and physical worlds with our first multimodal model, 2024a.

- xAI \[2024b\]↑
xAI.
Grok-2 beta release, 2024b.

- Xie et al. \[2022\]↑
Xie, X., Fu, L., Zhang, Z., Wang, Z., and Bai, X.
Toward understanding wordart: Corner-guided transformer for scene text recognition.
In _European conference on computer vision_, pp.  303–321. Springer, 2022.

- Xu et al. \[2024\]↑
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D.
WizardLM: Empowering large pre-trained language models to follow complex instructions.
In _The Twelfth International Conference on Learning Representations_, 2024.
URL [https://openreview.net/forum?id=CfXh93NDgH](https://openreview.net/forum?id=CfXh93NDgH "").

- Xu et al. \[2016\]↑
Xu, J., Mei, T., Yao, T., and Rui, Y.
MSR-VTT: A large video description dataset for bridging video and language.
In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp.  5288–5296, 2016.

- Yang et al. \[2023\]↑
Yang, Z., Ping, W., Liu, Z., Korthikanti, V., Nie, W., Huang, D.-A., Fan, L., Yu, Z., Lan, S., Li, B., et al.
Re-ViLM: Retrieval-augmented visual language model for zero and few-shot image captioning.
_arXiv preprint arXiv:2302.04858_, 2023.

- Yao et al. \[2024\]↑
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al.
Minicpm-v: A gpt-4v level mllm on your phone.
_arXiv preprint arXiv:2408.01800_, 2024.

- Ye et al. \[2023\]↑
Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al.
Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model.
_arXiv preprint arXiv:2310.05126_, 2023.

- Young et al. \[2024\]↑
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al.
Yi: Open foundation models by 01. ai.
_arXiv preprint arXiv:2403.04652_, 2024.

- Yu et al. \[2023\]↑
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W.
Metamath: Bootstrap your own mathematical questions for large language models.
_arXiv preprint arXiv:2309.12284_, 2023.

- Yuan et al. \[2024\]↑
Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., et al.
Advancing llm reasoning generalists with preference trees.
_arXiv preprint arXiv:2404.02078_, 2024.

- Yuan et al. \[2022\]↑
Yuan, Y., Liu, X., Dikubab, W., Liu, H., Ji, Z., Wu, Z., and Bai, X.
Syntax-aware network for handwritten mathematical expression recognition.
_arXiv preprint arXiv:2203.01601_, 2022.

- Yue et al. \[2023\]↑
Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W.
Mammoth: Building math generalist models through hybrid instruction tuning.
_arXiv preprint arXiv:2309.05653_, 2023.

- Yue et al. \[2024\]↑
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In _Proceedings of CVPR_, 2024.

- Zhai et al. \[2023\]↑
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.
Sigmoid loss for language image pre-training.
In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pp.  11975–11986, 2023.

- Zhang et al. \[2019a\]↑
Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C.
Raven: A dataset for relational and analogical visual reasoning.
In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pp.  5317–5327, 2019a.

- Zhang et al. \[2021\]↑
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J.
Vinvl: Revisiting visual representations in vision-language models.
In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp.  5579–5588, 2021.

- Zhang et al. \[2019b\]↑
Zhang, R., Zhou, Y., Jiang, Q., Song, Q., Li, N., Zhou, K., Wang, L., Wang, D., Liao, M., Yang, M., et al.
Icdar 2019 robust reading challenge on reading chinese text on signboard.
In _2019 international conference on document analysis and recognition (ICDAR)_, pp.  1577–1581. IEEE, 2019b.

- Zhang et al. \[2024\]↑
Zhang, R., Wei, X., Jiang, D., Zhang, Y., Guo, Z., Tong, C., Liu, J., Zhou, A., Wei, B., Zhang, S., et al.
Mavis: Mathematical visual instruction tuning.
_arXiv preprint arXiv:2407.08739_, 2024.

- Zhang et al. \[2023\]↑
Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T.
LLaVAR: Enhanced visual instruction tuning for text-rich image understanding.
_arXiv preprint arXiv:2306.17107_, 2023.

- Zhao et al. \[2023\]↑
Zhao, Y., Zhao, C., Nan, L., Qi, Z., Zhang, W., Tang, X., Mi, B., and Radev, D.
RobuT: A systematic study of table QA robustness against human-annotated adversarial perturbations.
_arXiv preprint arXiv:2306.14321_, 2023.

- Zhu et al. \[2023a\]↑
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M.
MiniGPT-4: Enhancing vision-language understanding with advanced large language models.
_arXiv preprint arXiv:2304.10592_, 2023a.

- Zhu et al. \[2022\]↑
Zhu, F., Lei, W., Feng, F., Wang, C., Zhang, H., and Chua, T.-S.
Towards complex document understanding by discrete reasoning.
In _Proceedings of the 30th ACM International Conference on Multimedia_, pp.  4857–4866, 2022.

- Zhu et al. \[2023b\]↑
Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., and Choi, Y.
Multimodal C4: An open, billion-scale corpus of images interleaved with text.
_arXiv preprint arXiv:2304.06939_, 2023b.

- Zhu et al. \[2016\]↑
Zhu, Y., Groth, O., Bernstein, M., and Fei-Fei, L.
Visual7W: Grounded question answering in images.
In _CVPR_, 2016.


## Appendix

Report issue for preceding element

## Appendix A Qualitative Examples from the NVLM-1.0-D 72B Model

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x4.png)

Report issue for preceding element

Figure 5: NVLM-1.0-D 72B model demonstrates good instruction-following capability. Based on the instructions, it appropriately controls the target generation’s length. It can also generate a very high-quality, detailed description of the provided image.
Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x5.png)

Report issue for preceding element

Figure 6: More examples of NVLM-1.0-D 72B model’s ability to understand memes, which is a challenging task that requires an understanding of humor and knowledge of important social trends, context, or events.
Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x6.png)

Report issue for preceding element

Figure 7: Demonstration of NVLM-1.0-D 72B model’s strong scene understanding capability. It has common-sense knowledge to recognize potential dangers or accidents and correctly suggests what immediate actions should be taken. Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/extracted/5857645/figures/qualitative_appendix/math_v4.png)

Report issue for preceding element

Figure 8: NVLM-1.0-D 72B model can solve math questions by providing step-by-step mathematical reasoning. We render latex equations for readability.
Report issue for preceding element

## Appendix B Training Hyperparameters

Report issue for preceding element

We provide the pretraining hyperparameters in Table [10](https://arxiv.org/html/2409.11402v1#A2.T10 "Table 10 ‣ Appendix B Training Hyperparameters ‣ NVLM: Open Frontier-Class Multimodal LLMs") and the SFT hyperparameters in Table [11](https://arxiv.org/html/2409.11402v1#A2.T11 "Table 11 ‣ Appendix B Training Hyperparameters ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

Table 10: Training hyper-parameters of NVLM models in the pretraining stage.

      Hyper-parameters      NVLM-D      NVLM-X      NVLM-H      Trainable weights      MLP      X-attn layers      X-attn layers & MLP      # of gated cross-attention layers      N/A      10      10      Global batch size      2,048      2048      2,048      Max learning rate      1e-4      1e-4      1e-4      Min learning rate      2.5e-5      1e-5      1e-5      Learning rate warmup steps      500      1,000      1,000      Scheduler      cosine      cosine      cosine      Optimizer      AdamW      AdamW      AdamW      Optimizer configβ1=0.9subscript𝛽10.9\\beta\_{1}=0.9italic\_β start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT = 0.9, β2=0.95subscript𝛽20.95\\beta\_{2}=0.95italic\_β start\_POSTSUBSCRIPT 2 end\_POSTSUBSCRIPT = 0.95β1=0.9subscript𝛽10.9\\beta\_{1}=0.9italic\_β start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT = 0.9, β2=0.95subscript𝛽20.95\\beta\_{2}=0.95italic\_β start\_POSTSUBSCRIPT 2 end\_POSTSUBSCRIPT = 0.95β1=0.9subscript𝛽10.9\\beta\_{1}=0.9italic\_β start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT = 0.9, β2=0.98subscript𝛽20.98\\beta\_{2}=0.98italic\_β start\_POSTSUBSCRIPT 2 end\_POSTSUBSCRIPT = 0.98      Weight decay      0.1      0.05      0.05      Gradient clipping      10      1.0      1.0      Sequence length in the LLM decoder      512      512      512      Downsampling of visual tokens      1024->256      1024->256      1024->256      # of visual token per tile      256      256      256      # of tiles      1      1      6+1      Tensor parallelism      8      8      8      Pipeline parallelism      1      1      1      # of training steps      20K      20K      20K
Report issue for preceding element

Report issue for preceding elementTable 11: Training hyper-parameters of NVLM models in the SFT stage.

Hyper-parametersNVLM-DNVLM-XNVLM-HTrainable weightsMLP & LLMX-attn layers & LLMX-attn layers & MLP & LLM\# of gated cross-attention layersN/A1010Global batch size128512 (34B), 256 (72B)256Max learning rate2⁢e−62superscript𝑒62e^{-6}2 italic\_e start\_POSTSUPERSCRIPT - 6 end\_POSTSUPERSCRIPT1⁢e−51superscript𝑒51e^{-5}1 italic\_e start\_POSTSUPERSCRIPT - 5 end\_POSTSUPERSCRIPT1⁢e−51superscript𝑒51e^{-5}1 italic\_e start\_POSTSUPERSCRIPT - 5 end\_POSTSUPERSCRIPTMin learning rate2.5⁢e−72.5superscript𝑒72.5e^{-7}2.5 italic\_e start\_POSTSUPERSCRIPT - 7 end\_POSTSUPERSCRIPT1⁢e−61superscript𝑒61e^{-6}1 italic\_e start\_POSTSUPERSCRIPT - 6 end\_POSTSUPERSCRIPT (34B), 1⁢e−71superscript𝑒71e^{-7}1 italic\_e start\_POSTSUPERSCRIPT - 7 end\_POSTSUPERSCRIPT (72B)1⁢e−71superscript𝑒71e^{-7}1 italic\_e start\_POSTSUPERSCRIPT - 7 end\_POSTSUPERSCRIPTLearning rate warmup steps1,0005001,000SchedulercosinecosinecosineOptimizerAdamWAdamWAdamWOptimizer configβ1=0.9subscript𝛽10.9\\beta\_{1}=0.9italic\_β start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT = 0.9, β2=0.98subscript𝛽20.98\\beta\_{2}=0.98italic\_β start\_POSTSUBSCRIPT 2 end\_POSTSUBSCRIPT = 0.98β1=0.9subscript𝛽10.9\\beta\_{1}=0.9italic\_β start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT = 0.9, β2=0.98subscript𝛽20.98\\beta\_{2}=0.98italic\_β start\_POSTSUBSCRIPT 2 end\_POSTSUBSCRIPT = 0.98β1=0.9subscript𝛽10.9\\beta\_{1}=0.9italic\_β start\_POSTSUBSCRIPT 1 end\_POSTSUBSCRIPT = 0.9, β2=0.98subscript𝛽20.98\\beta\_{2}=0.98italic\_β start\_POSTSUBSCRIPT 2 end\_POSTSUBSCRIPT = 0.98Weight decay0.10.050.05Gradient clipping101.01.0Sequence length in the LLM decoder3,2001,0241,280Downsampling of visual tokens1024->2561024->2561024->256\# of visual token per tile256256256\# of tiles6+16+16+1Tensor parallelism888Pipeline parallelism411\# of training steps40K20K40K
Report issue for preceding element

Report issue for preceding element

## Appendix C Perceiver Resampler in Flamingo Impacts OCR Performance

Report issue for preceding element

In this study, we utilize a pretrained Flamingo model \[ [4](https://arxiv.org/html/2409.11402v1#bib.bib4 "")\] from Yang et al. \[ [158](https://arxiv.org/html/2409.11402v1#bib.bib158 "")\], built on a 1.3B LLM, and fine-tune it on an internal document OCR dataset consisting of 30K samples.
In Figure [9](https://arxiv.org/html/2409.11402v1#A3.F9 "Figure 9 ‣ Appendix C Perceiver Resampler in Flamingo Impacts OCR Performance ‣ NVLM: Open Frontier-Class Multimodal LLMs"), we observe that the original Flamingo model, incorporating the _perceiver resampler_, struggles to overfit this OCR dataset. For instance, even after numerous epochs, the training loss remains around 0.4.
However, when we remove the _perceiver resampler_ and only train the cross-attention layer, the loss decreases to 0 at the same iteration.
We hypothesize that the X𝑋Xitalic\_X-attention operation to the _latent array_ in Perceiver \[ [48](https://arxiv.org/html/2409.11402v1#bib.bib48 "")\] may shuffle the spatial information among the image patches, making it challenging for the subsequent cross-attention layer to disentangle.

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/extracted/5857645/figures/flamingo_overfit/with_perceiver.png)(a)Training loss of Flamingo w/ perceiver.Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/extracted/5857645/figures/flamingo_overfit/without_perceiver.png)(b)Training loss of “Flamingo” w/o perceiver.Report issue for preceding element

Figure 9: An overfitting experiment on the Flamingo models with and without the perceiver resampler on a document OCR dataset.Report issue for preceding element

## Appendix D Evaluation Details of AI2D

Report issue for preceding element

We provide an illustration of two AI2D evaluation settings in Figure [10(a)](https://arxiv.org/html/2409.11402v1#A4.F10.sf1 "In Figure 10 ‣ Appendix D Evaluation Details of AI2D ‣ NVLM: Open Frontier-Class Multimodal LLMs") and [10(b)](https://arxiv.org/html/2409.11402v1#A4.F10.sf2 "In Figure 10 ‣ Appendix D Evaluation Details of AI2D ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/extracted/5857645/figures/ai2d_with_mask.png)(a)Evaluation Setting 1: With maskReport issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/extracted/5857645/figures/ai2d_with_no_mask.png)(b)Evaluation Setting 2: With no maskReport issue for preceding element

Figure 10: Illustration of two AI2D evaluation settings adopted from VLMEvalKit using a test sample with the question “Which is the leg closest to the head?”.Report issue for preceding element

## Appendix E Data Formats and ChatML Tamplate

Report issue for preceding element

We provide the examples of training data formats for various tasks in Figure [11](https://arxiv.org/html/2409.11402v1#A5.F11 "Figure 11 ‣ Appendix E Data Formats and ChatML Tamplate ‣ NVLM: Open Frontier-Class Multimodal LLMs") and ChatML template used in SFT in Figure [12](https://arxiv.org/html/2409.11402v1#A5.F12 "Figure 12 ‣ Appendix E Data Formats and ChatML Tamplate ‣ NVLM: Open Frontier-Class Multimodal LLMs").

Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x7.png)

Report issue for preceding elementFigure 11: Examples of training formats for various tasks used in pre-training. The emerald colored <image> tag indicates where to insert visual features. The blue colored text represents the ground truth associated with loss.
Report issue for preceding element

![Refer to caption](https://arxiv.org/html/2409.11402v1/x8.png)

Report issue for preceding elementFigure 12: An example of the ChatML template used in SFT. The emerald colored <image> tag indicates where to insert visual features. The blue colored text represents the ground truth associated with loss.
Report issue for preceding element

## Appendix F Unoptimized Results Using Yi-34B as the Backbone LLM

Report issue for preceding element

We only utilized the smaller 34B models for faster ablation studies and iterations, without detailed checkpoint selection or hyperparameter optimization. The unoptimized 34B results are provided in the following Tables for reference.
Note that, although our 34B results are not optimized, the 34B NVLM models still significantly outperform other models, including VILA-1.5 40B \[ [71](https://arxiv.org/html/2409.11402v1#bib.bib71 "")\] and Cambrian-1 34B \[ [139](https://arxiv.org/html/2409.11402v1#bib.bib139 "")\].

Report issue for preceding element

Table 12: Evaluation of 34B models on vision-language and text-only benchmarks.
Text-only Avg. 4 represents the average accuracy degradation or improvement of the multimodal LLM compared to its backbone text-only LLM after multimodal training, measured across four text benchmarks: MMLU, GSM8K, MATH, and HumanEval (see Table [8](https://arxiv.org/html/2409.11402v1#S6.T8 "Table 8 ‣ 6.3 Main Results ‣ 6 Results ‣ NVLM: Open Frontier-Class Multimodal LLMs") for full results).

TasksMMMUMathVistaVQAv2AI2DTextVQAChartQADocVQAReal-OCR-Text-onlytest / valtestminitest-devtest / _no\_mask_valtesttestWorldQABenchAvg. 4NVLM-D 34B48.7 / 52.159.984.382.6 / 93.080.084.089.167.3819+11.8NVLM-X 34B47.2 / 54.059.284.579.6 / 91.278.279.479.264.8802+11.2NVLM-H 34B46.3 / 53.059.484.881.5 / 93.079.082.080.566.3821+11.6
Report issue for preceding element

Report issue for preceding elementTable 13: Evaluation of 34B models on text benchmarks: MMLU, GSM8K, MATH and HumanEval.

TasksBackboneMMLUGSM8KMATHHumanEvalAvg.Text-onlyLLMAccuracyAvg. 4_(a)_ Nous-Hermes-2-Yi-34B \[ [102](https://arxiv.org/html/2409.11402v1#bib.bib102 "")\]N/A75.578.621.843.354.8N/ANVLM-D 34B_(a)_73.482.347.862.866.6+11.8NVLM-X 34B_(a)_73.282.246.462.266.0+11.2NVLM-H 34B_(a)_73.482.746.363.466.4+11.6
Report issue for preceding element

Report issue for preceding element

Report IssueReport Issue for Selection

Generated by
[L\\
A\\
T\\
Exml![[LOGO]](<Base64-Image-Removed>)](https://math.nist.gov/~BMiller/LaTeXML/)

</details>

<details>
<summary>Using files</summary>

# Using files

The Gemini API supports uploading media files separately from the prompt input, allowing your media to be reused across multiple requests and multiple prompts. For more details, check out the [Prompting with media](https://ai.google.dev/gemini-api/docs/prompting_with_media) guide.

## Method: media.upload

Creates a `File`.

### Endpoint

Upload URI, for media upload requests:

post  
`https://generativelanguage.googleapis.com/upload/v1beta/files`

Metadata URI, for metadata-only requests:

post  
`https://generativelanguage.googleapis.com/v1beta/files`

### Request body

The request body contains data with the following structure:

Fields

`file` object (File)

Optional. Metadata for the file to create.

### Example request

Python
```
from google import genai

client = genai.Client()
myfile = client.files.upload(file=media / "Cajun_instruments.jpg")
print(f"{myfile=}")

result = client.models.generate_content(
    model="gemini-2.0-flash",
    contents=[\
        myfile,\
        "\n\n",\
        "Can you tell me about the instruments in this photo?",\
    ],
)
print(f"{result.text=}")
```

JavaScript
```
// Make sure to include the following import:
// import {GoogleGenAI} from '@google/genai';
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const myfile = await ai.files.upload({
  file: path.join(media, "Cajun_instruments.jpg"),
  config: { mimeType: "image/jpeg" },
});
console.log("Uploaded file:", myfile);

const result = await ai.models.generateContent({
  model: "gemini-2.0-flash",
  contents: createUserContent([\
    createPartFromUri(myfile.uri, myfile.mimeType),\
    "\n\n",\
    "Can you tell me about the instruments in this photo?",\
  ]),
});
console.log("result.text=", result.text);
```

Go
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}
myfile, err := client.Files.UploadFromPath(
	ctx,
	filepath.Join(getMedia(), "Cajun_instruments.jpg"),
	&genai.UploadFileConfig{
		MIMEType : "image/jpeg",
	},
)
if err != nil {
	log.Fatal(err)
}
fmt.Printf("myfile=%+v\n", myfile)

parts := []*genai.Part{
	genai.NewPartFromURI(myfile.URI, myfile.MIMEType),
	genai.NewPartFromText("\n\n"),
	genai.NewPartFromText("Can you tell me about the instruments in this photo?"),
}

contents := []*genai.Content{
	genai.NewContentFromParts(parts, genai.RoleUser),
}

response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil)
if err != nil {
	log.Fatal(err)
}
text := response.Text()
fmt.Printf("result.text=%s\n", text)
```

Shell (resumable upload example)
```
MIME_TYPE=$(file -b --mime-type "${IMG_PATH_2}")
NUM_BYTES=$(wc -c < "${IMG_PATH_2}")
DISPLAY_NAME=TEXT

tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${IMG_PATH_2}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GEMINI_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"text": "Can you tell me about the instruments in this photo?"},\
          {"file_data":\
            {"mime_type": "image/jpeg",\
            "file_uri": '$file_uri'}\
        }]\
        }]
       }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json
```

Python audio example
```
from google import genai

client = genai.Client()
myfile = client.files.upload(file=media / "sample.mp3")
print(f"{myfile=}")

result = client.models.generate_content(
    model="gemini-2.0-flash", contents=[myfile, "Describe this audio clip"]
)
print(f"{result.text=}")
```

JavaScript audio example
```
// Make sure to include the following import:
// import {GoogleGenAI} from '@google/genai';
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const myfile = await ai.files.upload({
  file: path.join(media, "sample.mp3"),
  config: { mimeType: "audio/mpeg" },
});
console.log("Uploaded file:", myfile);

const result = await ai.models.generateContent({
  model: "gemini-2.0-flash",
  contents: createUserContent([\
    createPartFromUri(myfile.uri, myfile.mimeType),\
    "Describe this audio clip",\
  ]),
});
console.log("result.text=", result.text);
```

Go audio example
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}
myfile, err := client.Files.UploadFromPath(
	ctx,
	filepath.Join(getMedia(), "sample.mp3"),
	&genai.UploadFileConfig{
		MIMEType : "audio/mpeg",
	},
)
if err != nil {
	log.Fatal(err)
}
fmt.Printf("myfile=%+v\n", myfile)

parts := []*genai.Part{
	genai.NewPartFromURI(myfile.URI, myfile.MIMEType),
	genai.NewPartFromText("Describe this audio clip"),
}

contents := []*genai.Content{
	genai.NewContentFromParts(parts, genai.RoleUser),
}

response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil)
if err != nil {
	log.Fatal(err)
}
text := response.Text()
fmt.Printf("result.text=%s\n", text)
```

Shell audio example (resumable)
```
MIME_TYPE=$(file -b --mime-type "${AUDIO_PATH}")
NUM_BYTES=$(wc -c < "${AUDIO_PATH}")
DISPLAY_NAME=AUDIO

tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${AUDIO_PATH}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GEMINI_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"text": "Describe this audio clip"},\
          {"file_data":{"mime_type": "audio/mp3", "file_uri": '$file_uri'}}]\
        }]
       }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json
```

Python text file example
```
from google import genai

client = genai.Client()
myfile = client.files.upload(file=media / "poem.txt")
print(f"{myfile=}")

result = client.models.generate_content(
    model="gemini-2.0-flash",
    contents=[myfile, "\n\n", "Can you add a few more lines to this poem?"],
)
print(f"{result.text=}")
```

JavaScript text example
```
// Make sure to include the following import:
// import {GoogleGenAI} from '@google/genai';
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const myfile = await ai.files.upload({
  file: path.join(media, "poem.txt"),
});
console.log("Uploaded file:", myfile);

const result = await ai.models.generateContent({
  model: "gemini-2.0-flash",
  contents: createUserContent([\
    createPartFromUri(myfile.uri, myfile.mimeType),\
    "\n\n",\
    "Can you add a few more lines to this poem?",\
  ]),
});
console.log("result.text=", result.text);
```

Go text example
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}

myfile, err := client.Files.UploadFromPath(
	ctx,
	filepath.Join(getMedia(), "poem.txt"),
	&genai.UploadFileConfig{
		MIMEType : "text/plain",
	},
)
if err != nil {
	log.Fatal(err)
}
fmt.Printf("myfile=%+v\n", myfile)

parts := []*genai.Part{
	genai.NewPartFromURI(myfile.URI, myfile.MIMEType),
	genai.NewPartFromText("\n\n"),
	genai.NewPartFromText("Can you add a few more lines to this poem?"),
}

contents := []*genai.Content{
	genai.NewContentFromParts(parts, genai.RoleUser),
}

response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil)
if err != nil {
	log.Fatal(err)
}
text := response.Text()
fmt.Printf("result.text=%s\n", text)
```

Shell text example (resumable)
```
MIME_TYPE=$(file -b --mime-type "${TEXT_PATH}")
NUM_BYTES=$(wc -c < "${TEXT_PATH}")
DISPLAY_NAME=TEXT

tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${TEXT_PATH}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GEMINI_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"text": "Can you add a few more lines to this poem?"},\
          {"file_data":{"mime_type": "text/plain", "file_uri": '$file_uri'}}]\
        }]
       }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json

name=$(jq ".file.name" file_info.json)
# Get the file of interest to check state
curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json
# Print some information about the file you got
name=$(jq ".file.name" file_info.json)
echo name=$name
file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

curl --request "DELETE" https://generativelanguage.googleapis.com/v1beta/files/$name?key=$GEMINI_API_KEY
```

Python video example (upload + poll)
```
from google import genai
import time

client = genai.Client()
# Video clip (CC BY 3.0) from https://peach.blender.org/download/
myfile = client.files.upload(file=media / "Big_Buck_Bunny.mp4")
print(f"{myfile=}")

# Poll until the video file is completely processed (state becomes ACTIVE).
while not myfile.state or myfile.state.name != "ACTIVE":
    print("Processing video...")
    print("File state:", myfile.state)
    time.sleep(5)
    myfile = client.files.get(name=myfile.name)

result = client.models.generate_content(
    model="gemini-2.0-flash", contents=[myfile, "Describe this video clip"]
)
print(f"{result.text=}")
```

JavaScript video example
```
// Make sure to include the following import:
// import {GoogleGenAI} from '@google/genai';
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
let myfile = await ai.files.upload({
  file: path.join(media, "Big_Buck_Bunny.mp4"),
  config: { mimeType: "video/mp4" },
});
console.log("Uploaded video file:", myfile);

// Poll until the video file is completely processed (state becomes ACTIVE).
while (!myfile.state || myfile.state.toString() !== "ACTIVE") {
  console.log("Processing video...");
  console.log("File state: ", myfile.state);
  await sleep(5000);
  myfile = await ai.files.get({ name: myfile.name });
}

const result = await ai.models.generateContent({
  model: "gemini-2.0-flash",
  contents: createUserContent([\
    createPartFromUri(myfile.uri, myfile.mimeType),\
    "Describe this video clip",\
  ]),
});
console.log("result.text=", result.text);
```

Go video example
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}
myfile, err := client.Files.UploadFromPath(
	ctx,
	filepath.Join(getMedia(), "Big_Buck_Bunny.mp4"),
	&genai.UploadFileConfig{
		MIMEType : "video/mp4",
	},
)
if err != nil {
	log.Fatal(err)
}
fmt.Printf("myfile=%+v\n", myfile)

// Poll until the video file is completely processed (state becomes ACTIVE).
for myfile.State == genai.FileStateUnspecified || myfile.State != genai.FileStateActive {
	fmt.Println("Processing video...")
	fmt.Println("File state:", myfile.State)
	time.Sleep(5 * time.Second)

	myfile, err = client.Files.Get(ctx, myfile.Name, nil)
	if err != nil {
		log.Fatal(err)
	}
}

parts := []*genai.Part{
	genai.NewPartFromURI(myfile.URI, myfile.MIMEType),
	genai.NewPartFromText("Describe this video clip"),
}

contents := []*genai.Content{
	genai.NewContentFromParts(parts, genai.RoleUser),
}

response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil)
if err != nil {
	log.Fatal(err)
}
text := response.Text()
fmt.Printf("result.text=%s\n", text)
```

Shell video resumable example (polling)
```
MIME_TYPE=$(file -b --mime-type "${VIDEO_PATH}")
NUM_BYTES=$(wc -c < "${VIDEO_PATH}")
DISPLAY_NAME=VIDEO_PATH

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${VIDEO_PATH}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

state=$(jq ".file.state" file_info.json)
echo state=$state

# Ensure the state of the video is 'ACTIVE'
while [[ "($state)" = *"PROCESSING"* ]];
do
  echo "Processing video..."
  sleep 5
  # Get the file of interest to check state
  curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json
  state=$(jq ".file.state" file_info.json)
done

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GEMINI_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"text": "Describe this video clip"},\
          {"file_data":{"mime_type": "video/mp4", "file_uri": '$file_uri'}}]\
        }]
       }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json
```

Python PDF example
```
from google import genai

client = genai.Client()
sample_pdf = client.files.upload(file=media / "test.pdf")
response = client.models.generate_content(
    model="gemini-2.0-flash",
    contents=["Give me a summary of this pdf file.", sample_pdf],
)
print(response.text)
```

Go PDF example
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}
samplePdf, err := client.Files.UploadFromPath(
	ctx,
	filepath.Join(getMedia(), "test.pdf"),
	&genai.UploadFileConfig{
		MIMEType: "application/pdf",
	},
)
if err != nil {
	log.Fatal(err)
}

parts := []*genai.Part{
	genai.NewPartFromText("Give me a summary of this pdf file."),
	genai.NewPartFromURI(samplePdf.URI, samplePdf.MIMEType),
}

contents := []*genai.Content{
	genai.NewContentFromParts(parts, genai.RoleUser),
}

response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil)
if err != nil {
	log.Fatal(err)
}
text := response.Text()
fmt.Println(text)
```

### Response body

Response for `media.upload`.

If successful, the response body contains data with the following structure:

Fields

`file` object (File)

Metadata for the created file.

```json
{
  "file": {
    "object (File)"
  }
}
```

## Method: files.get

Gets the metadata for the given `File`.

### Endpoint

get  
`https://generativelanguage.googleapis.com/v1beta/{name=files/*}`

### Path parameters

`name` string

Required. The name of the `File` to get. Example: `files/abc-123` It takes the form `files/{file}`.

### Request body

The request body must be empty.

### Example request

Python
```
from google import genai

client = genai.Client()
myfile = client.files.upload(file=media / "poem.txt")
file_name = myfile.name
print(file_name)  # "files/*"

myfile = client.files.get(name=file_name)
print(myfile)
```

JavaScript
```
// Make sure to include the following import:
// import {GoogleGenAI} from '@google/genai';
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const myfile = await ai.files.upload({
  file: path.join(media, "poem.txt"),
});
const fileName = myfile.name;
console.log(fileName);

const fetchedFile = await ai.files.get({ name: fileName });
console.log(fetchedFile);
```

Go
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}
myfile, err := client.Files.UploadFromPath(
	ctx,
	filepath.Join(getMedia(), "poem.txt"),
	&genai.UploadFileConfig{
		MIMEType: "text/plain",
	},
)
if err != nil {
	log.Fatal(err)
}
fileName := myfile.Name
fmt.Println(fileName)
file, err := client.Files.Get(ctx, fileName, nil)
if err != nil {
	log.Fatal(err)
}
fmt.Println(file)
```

Shell
```
name=$(jq ".file.name" file_info.json)
# Get the file of interest to check state
curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json
# Print some information about the file you got
name=$(jq ".file.name" file_info.json)
echo name=$name
file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri
```

### Response body

If successful, the response body contains an instance of `File`.

## Method: files.list

Lists the metadata for `File`s owned by the requesting project.

### Endpoint

get  
`https://generativelanguage.googleapis.com/v1beta/files`

### Query parameters

`pageSize` integer

Optional. Maximum number of `File`s to return per page. If unspecified, defaults to 10. Maximum `pageSize` is 100.

`pageToken` string

Optional. A page token from a previous `files.list` call.

### Request body

The request body must be empty.

### Example request

Python
```
from google import genai

client = genai.Client()
print("My files:")
for f in client.files.list():
    print("  ", f.name)
```

JavaScript
```
// Make sure to include the following import:
// import {GoogleGenAI} from '@google/genai';
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
console.log("My files:");
// Using the pager style to list files
const pager = await ai.files.list({ config: { pageSize: 10 } });
let page = pager.page;
const names = [];
while (true) {
  for (const f of page) {
    console.log("  ", f.name);
    names.push(f.name);
  }
  if (!pager.hasNextPage()) break;
  page = await pager.nextPage();
}
```

Go
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}
fmt.Println("My files:")
page, err := client.Files.List(ctx, nil)
if err != nil {
	log.Fatal(err)
}
for _, f := range page.Items {
	fmt.Println("  ", f.Name)
}
```

Shell
```
echo "My files: "

curl "https://generativelanguage.googleapis.com/v1beta/files?key=${GEMINI_API_KEY}"
```

### Response body

Response for `files.list`.

If successful, the response body contains data with the following structure:

Fields

`files[]` object (File)

The list of `File`s.

`nextPageToken` string

A token that can be sent as a `pageToken` into a subsequent `files.list` call.

```json
{
  "files": [
    { "object (File)" }
  ],
  "nextPageToken": "string"
}
```

## Method: files.delete

Deletes the `File`.

### Endpoint

delete  
`https://generativelanguage.googleapis.com/v1beta/{name=files/*}`

### Path parameters

`name` string

Required. The name of the `File` to delete. Example: `files/abc-123` It takes the form `files/{file}`.

### Request body

The request body must be empty.

### Example request

Python
```
from google import genai

client = genai.Client()
myfile = client.files.upload(file=media / "poem.txt")

client.files.delete(name=myfile.name)

try:
    result = client.models.generate_content(
        model="gemini-2.0-flash", contents=[myfile, "Describe this file."]
    )
    print(result)
except genai.errors.ClientError:
    pass
```

Go (delete example + expect error)
```
ctx := context.Background()
client, err := genai.NewClient(ctx, &genai.ClientConfig{
	APIKey:  os.Getenv("GEMINI_API_KEY"),
	Backend: genai.BackendGeminiAPI,
})
if err != nil {
	log.Fatal(err)
}
myfile, err := client.Files.UploadFromPath(
	ctx,
	filepath.Join(getMedia(), "poem.txt"),
	&genai.UploadFileConfig{
		MIMEType: "text/plain",
	},
)
if err != nil {
	log.Fatal(err)
}
// Delete the file.
_, err = client.Files.Delete(ctx, myfile.Name, nil)
if err != nil {
	log.Fatal(err)
}
// Attempt to use the deleted file.
parts := []*genai.Part{
	genai.NewPartFromURI(myfile.URI, myfile.MIMEType,),
	genai.NewPartFromText("Describe this file."),
}

contents := []*genai.Content{
	genai.NewContentFromParts(parts, genai.RoleUser),
}

_, err = client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil)
// Expect an error when using a deleted file.
if err != nil {
	return nil
}
return fmt.Errorf("expected an error when using deleted file")
```

Shell
```
curl --request "DELETE" https://generativelanguage.googleapis.com/v1beta/files/$name?key=$GEMINI_API_KEY
```

### Response body

If successful, the response body is an empty JSON object.

## REST Resource: files

- Resource: File
- VideoFileMetadata
- State
- Source
- Methods

## Resource: File

A file uploaded to the API. Next ID: 15

Fields

`name` string

Immutable. Identifier. The `File` resource name. The ID (name excluding the "files/" prefix) can contain up to 40 characters that are lowercase alphanumeric or dashes (-). The ID cannot start or end with a dash. If the name is empty on create, a unique name will be generated. Example: `files/123-456`

`displayName` string

Optional. The human-readable display name for the `File`. The display name must be no more than 512 characters in length, including spaces. Example: "Welcome Image"

`mimeType` string

Output only. MIME type of the file.

`sizeBytes` string (int64 format)

Output only. Size of the file in bytes.

`createTime` string (Timestamp format)

Output only. The timestamp of when the `File` was created.

Uses RFC 3339. Examples: `"2014-10-02T15:01:23Z"`, `"2014-10-02T15:01:23.045123456Z"`

`updateTime` string (Timestamp format)

Output only. The timestamp of when the `File` was last updated.

`expirationTime` string (Timestamp format)

Output only. The timestamp of when the `File` will be deleted. Only set if the `File` is scheduled to expire.

`sha256Hash` string (bytes format)

Output only. SHA-256 hash of the uploaded bytes. A base64-encoded string.

`uri` string

Output only. The uri of the `File`.

`downloadUri` string

Output only. The download uri of the `File`.

`state` enum (State)

Output only. Processing state of the File.

`source` enum (Source)

Source of the File.

`error` object (Status)

Output only. Error status if File processing failed.

`metadata` Union type

Metadata for the File. `metadata` can be only one of the following:

`videoMetadata` object (VideoFileMetadata)

Output only. Metadata for a video.

## VideoFileMetadata

Metadata for a video `File`.

Fields

`videoDuration` string (Duration format)

Duration of the video. Example: `"3.5s"`

```json
{
  "videoDuration": "string"
}
```

## State

States for the lifecycle of a File.

Enums:
- `STATE_UNSPECIFIED` — The default value. This value is used if the state is omitted.
- `PROCESSING` — File is being processed and cannot be used for inference yet.
- `ACTIVE` — File is processed and available for inference.
- `FAILED` — File failed processing.

## Source

Enums:
- `SOURCE_UNSPECIFIED` — Used if source is not specified.
- `UPLOADED` — Indicates the file is uploaded by the user.
- `GENERATED` — Indicates the file is generated by Google.

## Status

The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. Each `Status` message contains three pieces of data: error code, error message, and error details.

Fields

`code` integer

The status code, which should be an enum value of `google.rpc.Code`.

`message` string

A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the `google.rpc.Status.details` field, or localized by the client.

`details[]` object

A list of messages that carry the error details. An object containing fields of an arbitrary type. An additional field `"@type"` contains a URI identifying the type. Example: `{ "id": 1234, "@type": "types.example.com/standard/id" }`.

```json
{
  "code": integer,
  "message": "string",
  "details": [
    {
      "@type": "string"
    }
  ]
}
```

</details>


## Code Sources

<details>
<summary>Repository analysis for https://github.com/towardsai/course-ai-agents/blob/dev/lessons/11_multimodal/notebook.ipynb</summary>

# Repository analysis for https://github.com/towardsai/course-ai-agents/blob/dev/lessons/11_multimodal/notebook.ipynb

## Summary
Repository: towardsai/course-ai-agents
Branch: dev
Commit: 74b11704cb644f8a395ef3b5e23b33c453d8fe77
File: notebook.ipynb
Lines: 1,414

Estimated tokens: 11.9k

## File tree
```Directory structure:
└── notebook.ipynb

```

## Extracted content
================================================
FILE: lessons/11_multimodal/notebook.ipynb
================================================
# Jupyter notebook converted to Python script.

"""
# Lesson 11: Multimodal

This notebook demonstrates how to build multimodal AI systems that can process and understand multimodal data such text, images and documents using Google's Gemini models.

We will use the `google-genai` library to interact with Google's Gemini models.

**Learning Objectives:**

1. **Process multimodal content**: Learn to handle images and PDFs in different formats (bytes, base64, URLs) with Gemini models
2. **Implement object detection**: Use multimodal LLMs for visual analysis and structured output generation
3. **Build multimodal RAG systems**: Create and index embeddings for images, documents and text to enable semantic search across multimodal content
4. **Develop multimodal AI agents**: Construct ReAct agents that can search through and reason about multimodal information
"""

"""
## 1. Setup

First, we define some standard Magic Python commands to autoreload Python packages whenever they change:
"""

%load_ext autoreload
%autoreload 2

"""
### Set Up Python Environment

To set up your Python virtual environment using `uv` and load it into the Notebook, follow the step-by-step instructions from the `Course Admin` lesson from the beginning of the course.

**TL/DR:** Be sure the correct kernel pointing to your `uv` virtual environment is selected.
"""

"""
### Configure Gemini API

To configure the Gemini API, follow the step-by-step instructions from the `Course Admin` lesson.

But here is a quick check on what you need to run this Notebook:

1.  Get your key from [Google AI Studio](https://aistudio.google.com/app/apikey).
2.  From the root of your project, run: `cp .env.example .env` 
3.  Within the `.env` file, fill in the `GOOGLE_API_KEY` variable:

Now, the code below will load the key from the `.env` file:
"""

from lessons.utils import env

env.load(required_env_vars=["GOOGLE_API_KEY"])
# Output:
#   Trying to load environment variables from `/Users/pauliusztin/Documents/01_projects/TAI/course-ai-agents/.env`

#   Environment variables loaded successfully.


"""
### Import Key Packages
"""

import base64
import io
from pathlib import Path
from typing import Literal

from google import genai
from google.genai import types
from IPython.display import Image as IPythonImage
from PIL import Image as PILImage

from lessons.utils import pretty_print

"""
### Initialize the Gemini Client
"""

client = genai.Client()

"""
### Define Constants

We will use the `gemini-2.5-flash` model, which is fast and cost-effective:
"""

MODEL_ID = "gemini-2.5-flash"

"""
## 2. Applying multimodal LLMs to images and PDFs

There are three core ways we can process images and PDFs with multimodal LLMs:
1. As raw bytes
2. As base64 encoded strings
3. As URLs

We will first look into how we can process images and then PDFs.

Now, let's look at our test image:

"""

def display_image(image_path: Path) -> None:
    """
    Display an image from a file path in the notebook.

    Args:
        image_path: Path to the image file to display

    Returns:
        None
    """

    image = IPythonImage(filename=image_path, width=400)
    display(image)


display_image(Path("images") / "image_1.jpeg")
# Output:
#   <IPython.core.display.Image object>

"""
### 2.1 As raw bytes
"""

def load_image_as_bytes(
    image_path: Path, format: Literal["WEBP", "JPEG", "PNG"] = "WEBP", max_width: int = 600, return_size: bool = False
) -> bytes | tuple[bytes, tuple[int, int]]:
    """
    Load an image from file path and convert it to bytes with optional resizing.

    Args:
        image_path: Path to the image file to load
        format: Output image format (WEBP, JPEG, or PNG). Defaults to "WEBP"
        max_width: Maximum width for resizing. If image width exceeds this, it will be resized proportionally. Defaults to 600
        return_size: If True, returns both bytes and image size tuple. Defaults to False

    Returns:
        bytes: Image data as bytes, or tuple of (bytes, (width, height)) if return_size is True
    """

    image = PILImage.open(image_path)
    if image.width > max_width:
        ratio = max_width / image.width
        new_size = (max_width, int(image.height * ratio))
        image = image.resize(new_size)

    byte_stream = io.BytesIO()
    image.save(byte_stream, format=format)

    if return_size:
        return byte_stream.getvalue(), image.size

    return byte_stream.getvalue()

"""
Load image:
"""

image_bytes = load_image_as_bytes(image_path=Path("images") / "image_1.jpeg", format="WEBP")
pretty_print.wrapped([f"Bytes `{image_bytes[:30]}...`", f"Size: {len(image_bytes)} bytes"], title="Image as Bytes")
# Output:
#   [93m------------------------------------------ Image as Bytes ------------------------------------------[0m

#     Bytes `b'RIFF`\xad\x00\x00WEBPVP8 T\xad\x00\x00P\xec\x02\x9d\x01*X\x02X\x02'...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#     Size: 44392 bytes

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Compute captions:
"""

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=image_bytes,
            mime_type="image/webp",
        ),
        "Tell me what is in this image in one paragraph.",
    ],
)
pretty_print.wrapped(response.text, title="Image 1 Caption")

# Output:
#   [93m----------------------------------------- Image 1 Caption -----------------------------------------[0m

#     This image presents a striking contrast between a large, imposing humanoid robot and a small, delicate kitten within an industrial setting. The robot is crafted from dark, intricate metallic plates, featuring a head adorned with circuit-like patterns and intensely glowing red eyes that convey a powerful, almost menacing presence. Perched playfully on the robot's heavily armored right arm, with its front paws resting gently on the shoulder, is a fluffy grey tabby kitten, looking out with an expression of curious innocence. The background reveals a sparse workshop or factory environment with metallic structures and a soft, diffused light source from the upper left, creating a scene that beautifully juxtaposes advanced technology with the simple charm of nature.

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Using the same approach, we can easily pass multiple images simultaneously. For example, the previous one plus the one below, and compare them:
"""

display_image(Path("images") / "image_2.jpeg")
# Output:
#   <IPython.core.display.Image object>

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=load_image_as_bytes(image_path=Path("images") / "image_1.jpeg", format="WEBP"),
            mime_type="image/webp",
        ),
        types.Part.from_bytes(
            data=load_image_as_bytes(image_path=Path("images") / "image_2.jpeg", format="WEBP"),
            mime_type="image/webp",
        ),
        "What's the difference between these two images? Describe it in one paragraph.",
    ],
)
pretty_print.wrapped(response.text, title="Differences between images")

# Output:
#   [93m------------------------------------ Differences between images ------------------------------------[0m

#     The primary difference between the two images is the nature of the interaction and the overall mood. The first image depicts a small, grey kitten playfully climbing on the arm of a large, imposing robot in what appears to be an industrial setting, suggesting curiosity, gentle engagement, or even budding companionship. In stark contrast, the second image portrays a large, fluffy white dog and a sleek, black robot locked in aggressive, confrontational stances in a dilapidated urban alleyway, conveying a clear sense of hostility, imminent conflict, and tension rather than interaction.

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
### 2.2 As base64 encoded strings

Now, let's load the same image as base64:
"""

from typing import cast


def load_image_as_base64(
    image_path: Path, format: Literal["WEBP", "JPEG", "PNG"] = "WEBP", max_width: int = 600, return_size: bool = False
) -> str:
    """
    Load an image and convert it to base64 encoded string.

    Args:
        image_path: Path to the image file to load
        format: Output image format (WEBP, JPEG, or PNG). Defaults to "WEBP"
        max_width: Maximum width for resizing. If image width exceeds this, it will be resized proportionally. Defaults to 600
        return_size: Parameter passed to load_image_as_bytes function. Defaults to False

    Returns:
        str: Base64 encoded string representation of the image
    """

    image_bytes = load_image_as_bytes(image_path=image_path, format=format, max_width=max_width, return_size=False)

    return base64.b64encode(cast(bytes, image_bytes)).decode("utf-8")

image_base64 = load_image_as_base64(image_path=Path("images") / "image_1.jpeg", format="WEBP")
pretty_print.wrapped(
    [f"Base64: {image_base64[:100]}...`", f"Size: {len(image_base64)} characters"], title="Image as Base64"
)
# Output:
#   [93m----------------------------------------- Image as Base64 -----------------------------------------[0m

#     Base64: UklGRmCtAABXRUJQVlA4IFStAABQ7AKdASpYAlgCPm0ylEekIqInJnQ7gOANiWdtk7FnEo2gDknjPixW9SNSb5P7IbBNhLn87Vtp...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#     Size: 59192 characters

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
On average base64 format is 33% larger than raw bytes. As we can see in this use case as well:
"""

print(f"Image as Base64 is {(len(image_base64) - len(image_bytes)) / len(image_bytes) * 100:.2f}% larger than as bytes")
# Output:
#   Image as Base64 is 33.34% larger than as bytes


"""
Now, let's recompute the image caption using this method:
"""

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(data=image_base64, mime_type="image/webp"),
        "Tell me what is in this image in one paragraph.",
    ],
)
response.text
# Output:
#   'The image presents a striking juxtaposition of a powerful, metallic robot and a small, curious kitten. The robot, dominating the right side of the frame, features a dark, intricately detailed head resembling circuit board patterns, with intense, glowing red eyes that pierce forward. Its massive, articulated arm is extended, and a fluffy grey tabby kitten is playfully perched on its forearm and shoulder, looking towards the robot with a raised paw as if exploring or batting it. The background suggests an industrial or workshop environment, with metallic structures and diffuse light from a window on the left, highlighting the unexpected encounter between advanced technology and innocent nature.'

"""
### 2.3 As public URLs

Using Gemini `url_context` out-of-the-box tool, we can automatically visit and parse webpages, PDFs, and images from the open internet. You only have to provide the direct URL in the prompt and configure the `url_context` tool. This makes it a no-brainer to parse multiple data formats when available online:
"""

response = client.models.generate_content(
    model=MODEL_ID,
    contents="Based on the provided paper as a PDF, tell me how ReAct works: https://arxiv.org/pdf/2210.03629",
    config=types.GenerateContentConfig(tools=[{"url_context": {}}]),
)
pretty_print.wrapped(response.text, title="How ReAct works")
# Output:
#   [93m----------------------------------------- How ReAct works -----------------------------------------[0m

#     

#   The ReAct (Reasoning and Acting) paradigm is a method that combines verbal reasoning traces with task-specific actions in an interleaved manner within large language models (LLMs). This approach allows for greater synergy between reasoning and acting, enabling LLMs to perform more dynamic reasoning and interact with external environments.

#   

#   Here's a breakdown of how ReAct works:

#   *   **Interleaved Reasoning and Acting:** ReAct prompts LLMs to generate both "thoughts" (verbal reasoning traces) and "actions" related to a task. These are produced in an alternating sequence, mimicking human cognition where inner speech (reasoning) guides actions, and actions provide new information for further reasoning.

#   *   **Reasoning to Act:** The reasoning traces help the model to induce, track, and update action plans, as well as handle exceptions. For example, a thought might involve decomposing a task goal into smaller subgoals, injecting commonsense knowledge, or extracting important information from observations.

#   *   **Acting to Reason:** Actions allow the LLM to interface with and gather additional information from external sources, such as knowledge bases (like a Wikipedia API) or interactive environments. The observations received after an action then inform subsequent reasoning steps.

#   *   **Augmented Action Space:** ReAct augments the agent's action space to include a language space (`L`) for "thoughts" in addition to the traditional action space (`A`) for external interactions. Thoughts do not directly affect the external environment but update the agent's internal context to support future reasoning or acting.

#   *   **Few-shot Prompting:** ReAct primarily utilizes few-shot in-context examples to prompt frozen large language models to generate both domain-specific actions and free-form language thoughts. These examples are human-created trajectories of actions, thoughts, and environment observations.

#   

#   **Benefits of ReAct:**

#   *   **Improved Performance:** ReAct has shown to outperform state-of-the-art baselines that only perform reasoning (Chain-of-Thought) or acting alone in various tasks, including question answering, fact verification, and interactive decision-making benchmarks.

#   *   **Reduced Hallucination and Error Propagation:** By interacting with external sources, ReAct can overcome issues of hallucination and error propagation that can occur in reasoning-only methods.

#   *   **Human Interpretability and Trustworthiness:** The interleaved reasoning traces make the model's task-solving process more interpretable and trustworthy for humans, as they can inspect the reasoning and factual correctness.

#   *   **Flexibility and Generalizability:** The flexible thought space and thought-action occurrence format allow ReAct to be applied to diverse tasks with different action spaces and reasoning needs.

#   

#   In essence, ReAct allows LLMs to think and act in a more integrated and adaptive way, leading to more robust and explainable performance across a range of complex tasks.The ReAct (Reasoning and Acting) paradigm enhances large language models (LLMs) by synergizing verbal reasoning traces with task-specific actions in an interleaved manner. This approach fosters a dynamic interaction where internal reasoning guides external actions, and observations from those actions inform subsequent reasoning steps.

#   

#   Key aspects of how ReAct works include:

#   *   **Interleaved Thoughts and Actions** The core of ReAct involves LLMs generating both "thoughts" (verbal reasoning) and "actions" in an alternating sequence. This mimics human problem-solving, where internal deliberation (e.g., planning, self-correction) is intertwined with physical or communicative actions.

#   *   **Reasoning to Guide Actions** Reasoning traces help the model to formulate, monitor, and adjust action plans, as well as manage unexpected situations. These thoughts can involve breaking down complex goals, incorporating commonsense knowledge, or extracting relevant information from observations.

#   *   **Actions to Inform Reasoning** Actions enable the LLM to interact with external environments, such as a Wikipedia API for knowledge retrieval or simulated environments for decision-making. The observations generated by these actions then provide new information that can be integrated into the reasoning process.

#   *   **Augmented Action Space** ReAct expands the agent's available actions to include a language space for "thoughts," in addition to domain-specific actions that affect the external environment. Thoughts do not alter the environment but update the agent's internal context to support future reasoning or actions.

#   *   **Few-Shot Prompting** The ReAct framework primarily relies on few-shot in-context examples to prompt LLMs. These examples demonstrate a human-like trajectory of thoughts, actions, and environmental observations to guide the model in solving tasks.

#   

#   This integrated approach offers several advantages:

#   *   **Enhanced Performance** ReAct has demonstrated superior performance compared to methods that only employ reasoning (Chain-of-Thought) or acting in isolation, across various tasks like question answering, fact verification, and interactive decision-making.

#   *   **Reduced Errors and Hallucinations** By interacting with external information sources, ReAct helps mitigate issues such as factual hallucinations and error propagation often seen in reasoning-only methods.

#   *   **Improved Interpretability and Trustworthiness** The explicit reasoning traces make the model's decision-making process more transparent and understandable to humans, increasing interpretability and trustworthiness.

#   *   **Flexibility and Generalizability** ReAct's flexible design, accommodating diverse reasoning types and action spaces, allows it to be applied effectively to a wide range of complex tasks.

#   

#   In essence, ReAct empowers LLMs to engage in a more integrated and adaptive problem-solving process by continuously leveraging reasoning to guide actions and using observations from those actions to refine its reasoning.

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
### 2.4 As URLs from private data lakes

At the time of writing this notebook, Gemini works well primarily with GCP Cloud Storage links and not with other buckets such as S3. Buckets are excellent for production use cases, but they complicate our simple demonstration. Therefore, we will show you a mocked example.

The code would look like this, where you have to change the `uri` and ensure the LLM has the right permissions to your GCS bucket:
```python
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_uri(uri="gs://gemini-images/image_1.jpeg", mime_type="image/webp"),
        "Tell me what is in this image in one paragraph.",
    ],
)
```
"""

"""
### 2.5 Object detection with LLMs

As a more exciting example, let's do object detection with multimodal LLMs.

First, let's define the output Pydantic models:
"""

from pydantic import BaseModel, Field


class BoundingBox(BaseModel):
    ymin: float
    xmin: float
    ymax: float
    xmax: float
    label: str = Field(
        default="The category of the object found within the bounding box. For example: cat, dog, diagram, robot."
    )


class Detections(BaseModel):
    bounding_boxes: list[BoundingBox]

"""
Then the prompt and image:
"""

prompt = """
Detect all of the prominent items in the image. 
The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000.
Also, output the label of the object found within the bounding box.
"""

image_bytes, image_size = load_image_as_bytes(
    image_path=Path("images") / "image_1.jpeg", format="WEBP", return_size=True
)

"""
Now, let's call the LLM:
"""

config = types.GenerateContentConfig(
    response_mime_type="application/json",
    response_schema=Detections,
)

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=image_bytes,
            mime_type="image/webp",
        ),
        prompt,
    ],
    config=config,
)

detections = cast(Detections, response.parsed)
pretty_print.wrapped([f"Image size: {image_size}", *detections.bounding_boxes], title="Detections")
# Output:
#   [93m-------------------------------------------- Detections --------------------------------------------[0m

#     Image size: (600, 600)

#   [93m----------------------------------------------------------------------------------------------------[0m

#     ymin=2.0 xmin=517.0 ymax=997.0 xmax=1000.0 label='robot'

#   [93m----------------------------------------------------------------------------------------------------[0m

#     ymin=272.0 xmin=28.0 ymax=801.0 xmax=535.0 label='kitten'

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Let's also visualize the bounding boxes: 
"""

import matplotlib.pyplot as plt
import matplotlib.patches as patches
import numpy as np


def visualize_detections(detections: Detections, image_path: Path) -> None:
    """
    Visualize detected bounding boxes on an image with red rectangles and labels.

    Args:
        detections: Detections object containing bounding boxes in [ymin, xmin, ymax, xmax] format normalized to 0-1000
        image_path: Path to the image file to visualize

    Returns:
        None: Displays the image with bounding boxes in the notebook
    """

    # Clear any existing plots to prevent overlapping
    plt.clf()

    image = PILImage.open(image_path)
    image_array = np.array(image)
    img_height, img_width = image_array.shape[:2]

    fig, ax = plt.subplots(1, 1, figsize=(8, 6))
    ax.imshow(image_array)

    for bbox in detections.bounding_boxes:
        # Convert normalized coordinates (0-1000) to pixel coordinates
        xmin = (bbox.xmin / 1000) * img_width
        ymin = (bbox.ymin / 1000) * img_height
        xmax = (bbox.xmax / 1000) * img_width
        ymax = (bbox.ymax / 1000) * img_height

        # Calculate box dimensions (matplotlib uses bottom-left corner + width/height)
        width = xmax - xmin
        height = ymax - ymin

        # Create rectangle patch (x, y is bottom-left corner)
        rect = patches.Rectangle((xmin, ymin), width, height, linewidth=3, edgecolor="red", facecolor="none")

        # Add rectangle to the plot
        ax.add_patch(rect)

        # Add label text (positioned at top-left of bounding box)
        ax.text(
            xmin,
            ymin + 5,  # Slightly above the box
            bbox.label[:15],
            fontsize=12,
            color="red",
            fontweight="bold",
            bbox=dict(boxstyle="round,pad=0.3", facecolor="white", alpha=0.8),
        )

    # Remove axis ticks and labels for cleaner display
    ax.set_xticks([])
    ax.set_yticks([])
    ax.set_title(f"Object Detection Results: {image_path.name}", fontsize=14, fontweight="bold")

    plt.tight_layout()
    plt.show()

visualize_detections(detections, Path("images") / "image_1.jpeg")
# Output:
#   <Figure size 640x480 with 0 Axes>
#   <Figure size 800x600 with 1 Axes>

"""
### 2.6 Working with PDFs

Ultimately, let's see how we can work with PDFs. We will use the legendary `Attention Is All You Need` Paper as an example. 

To display it, we extracted the first 3 pages of the PDF as images. For example, this is how the page looks:

"""

display_image(Path("images") / "attention_is_all_you_need_0.jpeg")
# Output:
#   <IPython.core.display.Image object>

"""
We can treat PDFs similarly to images. Therefore, we can pass PDFs as bytes:
"""

pdf_bytes = (Path("pdfs") / "attention_is_all_you_need_paper.pdf").read_bytes()
pretty_print.wrapped(f"Bytes: {pdf_bytes[:40]}...", title="PDF bytes")
# Output:
#   [93m-------------------------------------------- PDF bytes --------------------------------------------[0m

#     Bytes: b'%PDF-1.7\n%\xe2\xe3\xcf\xd3\n24 0 obj\n<<\n/Filter /Flat'...

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Call the LLM:
"""

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(data=pdf_bytes, mime_type="application/pdf"),
        "What is this document about? Provide a brief summary of the main topics.",
    ],
)
pretty_print.wrapped(response.text, title="PDF Summary (as bytes)")
# Output:
#   [93m-------------------------------------- PDF Summary (as bytes) --------------------------------------[0m

#     This document introduces the **Transformer**, a novel neural network architecture for **sequence transduction** (e.g., machine translation) that relies **solely on attention mechanisms**, entirely **dispensing with recurrent and convolutional layers**.

#   

#   The main topics covered include:

#   

#   1.  **The Transformer Architecture:** A detailed description of its encoder-decoder structure, utilizing stacked multi-head self-attention and position-wise fully connected layers.

#   2.  **Attention Mechanisms:** Explanation of scaled dot-product attention, multi-head attention, and their application within the encoder and decoder.

#   3.  **Positional Encoding:** How sequence order information is integrated into the model, given the absence of recurrence or convolution.

#   4.  **Advantages of Self-Attention:** A comparison highlighting self-attention's benefits in computational complexity, parallelization, and handling long-range dependencies.

#   5.  **Performance and Results:** Experimental findings demonstrating the Transformer's superior quality (achieving state-of-the-art BLEU scores) and significantly faster training times on WMT 2014 English-to-German and English-to-French machine translation tasks.

#   6.  **Generalization:** Successful application of the Transformer to English constituency parsing, showcasing its versatility.

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Alternatively, as base64 encoded strings:
"""

def load_pdf_as_base64(pdf_path: Path) -> str:
    """
    Load a PDF file and convert it to base64 encoded string.

    Args:
        pdf_path: Path to the PDF file to load

    Returns:
        str: Base64 encoded string representation of the PDF
    """

    with open(pdf_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

"""
Load the PDF:
"""

pdf_base64 = load_pdf_as_base64(pdf_path=Path("pdfs") / "attention_is_all_you_need_paper.pdf")
pretty_print.wrapped(f"Base64: {pdf_base64[:40]}...", title="PDF as Base64")
# Output:
#   [93m------------------------------------------ PDF as Base64 ------------------------------------------[0m

#     Base64: JVBERi0xLjcKJeLjz9MKMjQgMCBvYmoKPDwKL0Zp...

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Call the LLM:
"""

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        "What is this document about? Provide a brief summary of the main topics.",
        types.Part.from_bytes(data=pdf_base64, mime_type="application/pdf"),
    ],
)

pretty_print.wrapped(response.text, title="PDF Summary (as base64)")
# Output:
#   [93m------------------------------------- PDF Summary (as base64) -------------------------------------[0m

#     This document introduces **The Transformer**, a novel neural network architecture designed for sequence transduction tasks, such as machine translation.

#   

#   Here's a brief summary of the main topics:

#   

#   1.  **Introduction of the Transformer Architecture:** The paper proposes a new model that completely dispenses with recurrent neural networks (RNNs) and convolutional neural networks (CNNs), relying entirely on **attention mechanisms**.

#   2.  **Self-Attention and Multi-Head Attention:** The core of the Transformer is its self-attention mechanism, specifically a "Scaled Dot-Product Attention" which is further enhanced by "Multi-Head Attention" to allow the model to jointly attend to information from different representation subspaces.

#   3.  **Encoder-Decoder Structure with Positional Encodings:** The Transformer maintains an encoder-decoder structure, with each stack composed of multiple identical layers. Since it lacks recurrence or convolutions, "positional encodings" are added to the input embeddings to inject information about the sequence order.

#   4.  **Performance and Efficiency:** The model is shown to be superior in quality, more parallelizable, and requires significantly less training time compared to existing dominant models based on RNNs or CNNs.

#   5.  **State-of-the-Art Results:** It achieves new state-of-the-art BLEU scores on WMT 2014 English-to-German and English-to-French machine translation tasks.

#   6.  **Generalization to Other Tasks:** The Transformer's effectiveness is also demonstrated on English constituency parsing, indicating its potential for broader applicability.

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Now, let's do a more interesting example and detect the diagrams from a page of the transformers paper, such as the one below:
"""

display_image(Path("images") / "attention_is_all_you_need_1.jpeg")
# Output:
#   <IPython.core.display.Image object>

"""
Define the object detection prompt to detect diagrams (similar to how we did for images):
"""

prompt = """
Detect all the diagrams from the provided image as 2d bounding boxes. 
The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000.
Also, output the label of the object found within the bounding box.
"""

image_bytes, image_size = load_image_as_bytes(
    image_path=Path("images") / "attention_is_all_you_need_1.jpeg", format="WEBP", return_size=True
)

"""
Call the LLM:
"""

config = types.GenerateContentConfig(
    response_mime_type="application/json",
    response_schema=Detections,
)
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=image_bytes,
            mime_type="image/webp",
        ),
        prompt,
    ],
    config=config,
)
detections = cast(Detections, response.parsed)
pretty_print.wrapped([f"Image size: {image_size}", *detections.bounding_boxes], title="Detections")
# Output:
#   [93m-------------------------------------------- Detections --------------------------------------------[0m

#     Image size: (600, 776)

#   [93m----------------------------------------------------------------------------------------------------[0m

#     ymin=80.0 xmin=280.0 ymax=470.0 xmax=720.0 label='diagram'

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Visualize the detections:
"""

visualize_detections(detections, Path("images") / "attention_is_all_you_need_1.jpeg")
# Output:
#   <Figure size 640x480 with 0 Axes>
#   <Figure size 800x600 with 1 Axes>

"""
## 3. Implementing multimodal RAG for images, PDFs and text

To bring everything we did in this course together, let's implement a multimodal RAG system that works with text, images, and PDFs.

These are the images and PDF pages (as images) we will index for semantic search:
"""

def display_image_grid(image_paths: list[Path], rows: int = 2, cols: int = 2, figsize: tuple = (8, 6)) -> None:
    """
    Display a grid of images.

    Args:
        image_paths: List of paths to images to display
        rows: Number of rows in the grid
        cols: Number of columns in the grid
        figsize: Figure size as (width, height)
    """

    fig, axes = plt.subplots(rows, cols, figsize=figsize)
    axes = axes.ravel()

    for idx, img_path in enumerate(image_paths[: rows * cols]):
        img = PILImage.open(img_path)
        axes[idx].imshow(img)
        axes[idx].axis("off")

    plt.tight_layout()
    plt.show()


display_image_grid(
    image_paths=[
        Path("images") / "image_1.jpeg",
        Path("images") / "image_2.jpeg",
        Path("images") / "image_3.jpeg",
        Path("images") / "image_4.jpeg",
        Path("images") / "attention_is_all_you_need_1.jpeg",
        Path("images") / "attention_is_all_you_need_2.jpeg",
    ],
    rows=2,
    cols=3,
)
# Output:
#   <Figure size 800x600 with 6 Axes>

"""
Now, let's define the core functions.

First, one that creates image descriptions:
"""

from io import BytesIO
from typing import Any

import numpy as np


def generate_image_description(image_bytes: bytes) -> str:
    """
    Generate a detailed description of an image using Gemini Vision model.

    Args:
        image_bytes: Image data as bytes

    Returns:
        str: Generated description of the image
    """

    try:
        # Convert bytes back to PIL Image for vision model
        img = PILImage.open(BytesIO(image_bytes))

        # Use Gemini Vision model to describe the image
        prompt = """
        Describe this image in detail for semantic search purposes. 
        Include objects, scenery, colors, composition, text, and any other visual elements that would help someone find 
        this image through text queries.
        """

        response = client.models.generate_content(
            model=MODEL_ID,
            contents=[prompt, img],
        )

        if response and response.text:
            description = response.text.strip()

            return description
        else:
            print("❌ No description generated from vision model")

            return ""

    except Exception as e:
        print(f"❌ Failed to generate image description: {e}")

        return ""


"""
Another one that creates embedding using `gemini_embedding-001`, based on the given input:
"""

def embed_text_with_gemini(content: str) -> np.ndarray | None:
    """
    Embed text content using Gemini's text embedding model.

    Args:
        content: Text string to embed

    Returns:
        np.ndarray | None: Embedding vector as numpy array or None if failed
    """

    try:
        result = client.models.embed_content(
            model="gemini-embedding-001",  # Gemini's text embedding model
            contents=[content],
        )
        if not result or not result.embeddings:
            print("❌ No embedding data found in response")
            return None

        return np.array(result.embeddings[0].values)

    except Exception as e:
        print(f"❌ Failed to embed text: {e}")
        return None

"""
Let's see how this works:
"""

embedding = embed_text_with_gemini("This is a test")
embedding
# Output:
#   array([-0.02252334, -0.00076438,  0.00240217, ..., -0.00574729,

#          -0.00052345, -0.00213343], shape=(3072,))

"""
As we can see below, it creates a 3072 embedding from the input text:
"""

embedding.shape
# Output:
#   (3072,)

"""
Let's glue these functions and create the vector index out of our test images and PDF pages:
"""

from typing import cast


def create_vector_index(image_paths: list[Path]) -> list[dict]:
    """
    Create embeddings for images by generating descriptions and embedding them.

    This function processes a list of image paths by:
    1. Loading each image as bytes
    2. Generating a text description using Gemini Vision
    3. Creating an embedding of that description using Gemini Embeddings

    Args:
        image_paths (list[Path]): List of paths to image files to process

    Returns:
        list[dict]: List of dictionaries with the following keys:
            - content (bytes): Raw image bytes
            - type (str): Always "image"
            - filename (Path): Original image path
            - description (str): Generated image description
            - embedding (np.ndarray): Vector embedding of the description
    """

    vector_index = []
    for image_path in image_paths:
        image_bytes = cast(bytes, load_image_as_bytes(image_path, format="WEBP", return_size=False))

        image_description = generate_image_description(image_bytes)
        pretty_print.wrapped(f"`{image_description[:500]}...`", title="Generated image description:")

        # IMPORTANT NOTE: When working with multimodal embedding models, we can directly embed the
        # `image_bytes` instead of generating and embedding the description. Otherwise, everything
        # else remains the same within the whole RAG system.
        image_embedding = embed_text_with_gemini(image_description)

        vector_index.append(
            {
                "content": image_bytes,
                "type": "image",
                "filename": image_path,
                "description": image_description,
                "embedding": image_embedding,
            }
        )

    return vector_index

"""
We call the `create_vector_index` function on all the images from the `images` dir:
"""

image_paths = list(Path("images").glob("*.jpeg"))
vector_index = create_vector_index(image_paths)
# Output:
#   [93m----------------------------------- Generated image description: -----------------------------------[0m

#     `This image is a page from a technical document, likely a research paper or textbook, focusing on the "Attention" mechanism in machine learning, particularly for Transformer models.

#   

#   **Composition and Layout:**

#   The page has a clean, professional layout with black text on a white background. The top half is dominated by two distinct block diagrams side-by-side, visually explaining complex concepts. The bottom half is dense with technical text, including headings, paragraphs, a mathematical equatio...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------- Generated image description: -----------------------------------[0m

#     `This image features a striking juxtaposition of a futuristic, menacing robot and an adorable, curious kitten in an industrial setting.

#   

#   **Objects:**

#   

#   1.  **Robot:** Occupying the right and center-right of the frame, a large, powerful, humanoid robot is depicted from the chest up.

#       *   **Head/Helmet:** The robot's head is designed like a sleek, dark metallic helmet, featuring intricate circuit board patterns or etched lines across its surface, giving it a high-tech, integrated look.

#       *   **...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------- Generated image description: -----------------------------------[0m

#     `This image depicts a dramatic and unusual confrontation between a fluffy white dog and a sleek, dark humanoid robot in a gritty urban alleyway.

#   

#   **Objects:**

#   *   **Dog:** A large, white, long-haired dog, strongly resembling a Samoyed or a white husky mix, is positioned on the left side of the frame. It stands on its hind legs, with its front paws raised and mouth open in a snarl or aggressive bark, revealing sharp teeth. Its fluffy tail is curled over its back. A dark collar is visible around it...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------- Generated image description: -----------------------------------[0m

#     `This image captures a close-up, highly detailed view of a focused African American man engaged in building or repairing a desktop computer, illuminated by internal LED lights.

#   

#   **Objects:**

#   *   **Person:** A mature African American man with a well-groomed, greying beard and mustache, wearing black-framed glasses. He is dressed in a dark, possibly teal or greyish-blue, crew-neck t-shirt. His brow is furrowed in concentration, and his gaze is directed downwards into the computer case.

#   *   **Comput...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------- Generated image description: -----------------------------------[0m

#     `This image displays a detailed technical diagram of the **Transformer model architecture**, a fundamental neural network structure in deep learning, particularly for natural language processing (NLP) tasks. The diagram, labeled "Figure 1: The Transformer - model architecture," is centrally placed on a white page, with descriptive text below it.

#   

#   **Diagram Details:**

#   

#   The architecture is presented as an **encoder-decoder stack**, depicted in two main vertical columns.

#   

#   1.  **Input and Embedding L...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------- Generated image description: -----------------------------------[0m

#     `This image depicts a highly dynamic and intense futuristic robot battle in a sleek, dimly lit arena. Two humanoid robots are engaged in a powerful fistfight, with one delivering a devastating blow to the other.

#   

#   **Objects:**

#   *   **Left Robot:** This robot is sleek and agile, predominantly metallic silver or chrome with bright blue glowing accents. Blue light emanates from its head (acting as a visor or eyes) and a strip across its chest. Its armor appears streamlined and polished. It is shown in...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------- Generated image description: -----------------------------------[0m

#     `This image is a detailed depiction of the first page of a seminal academic paper titled "Attention Is All You Need." It features a clean, professional layout on a white background with black text, and a distinct reddish-brown disclaimer at the top.

#   

#   **Overall Composition & Layout:**

#   The page is organized in a single-column format, characteristic of research papers or preprints. Text is the dominant visual element, structured into distinct sections: a copyright notice, the main title, author and ...`

#   [93m----------------------------------------------------------------------------------------------------[0m


if len(vector_index) == 0:
    pretty_print.wrapped("Could not create the vector index.", title="❌")
else:
    pretty_print.wrapped(
        f"Successfully created {len(vector_index)} embeddings under the `vector_index` variable", title="✅"
    )
# Output:
#   [93m------------------------------------------------ ✅ ------------------------------------------------[0m

#     Successfully created 7 embeddings under the `vector_index` variable

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
This is how an element from the `vector_index` looks like:
"""

vector_index[0].keys()
# Output:
#   dict_keys(['content', 'type', 'filename', 'description', 'embedding'])

vector_index[0]["embedding"].shape
# Output:
#   (3072,)

print(f"{vector_index[0]['description'][:150]}...")
# Output:
#   This image is a page from a technical document, likely a research paper or textbook, focusing on the "Attention" mechanism in machine learning, partic...


"""
Now let's define a function that finds `top_k` most similar items from the vector_index based on a user query:
"""

from sklearn.metrics.pairwise import cosine_similarity


def search_multimodal(query_text: str, vector_index: list[dict], top_k: int = 3) -> list[Any]:
    """
    Search for most similar documents to query using direct Gemini client.

    This function embeds the query text and compares it against pre-computed embeddings
    of document descriptions to find the most semantically similar matches.

    Args:
        query_text: Text query to search for
        docs: List of document dictionaries containing embeddings and metadata
        top_k: Number of top results to return. Defaults to 3

    Returns:
        list[Any]: List of document dictionaries with similarity scores, sorted by relevance
    """

    print(f"\n🔍 Embedding query: '{query_text}'")

    query_embedding = embed_text_with_gemini(query_text)

    if query_embedding is None:
        print("❌ Failed to embed query")
        return []
    else:
        print("✅ Query embedded successfully")

    # Calculate similarities using our custom function
    embeddings = [doc["embedding"] for doc in vector_index]
    similarities = cosine_similarity([query_embedding], embeddings).flatten()

    # Get top results
    top_indices = np.argsort(similarities)[::-1][:top_k]  # type: ignore

    results = []
    for idx in top_indices.tolist():
        results.append({**vector_index[idx], "similarity": similarities[idx]})

    return results

"""
Let's test this with an example:
"""

query = "what is the architecture of the transformer neural network?"
results = search_multimodal(query, vector_index, top_k=1)

if not results:
    pretty_print.wrapped("❌ No results found", title="❌")
else:
    result = results[0]

    pretty_print.wrapped(
        [
            f"Similarity {result['similarity']:.3f}",
            f"Filename {result['filename']}",
            f"Description `{result['description'][:1000]}...`",
        ],
        title=f"Results for query = {query}",
    )
    display_image(Path(result["filename"]))
# Output:
#   

#   🔍 Embedding query: 'what is the architecture of the transformer neural network?'

#   ✅ Query embedded successfully

#   [93m--------- Results for query = what is the architecture of the transformer neural network? ---------[0m

#     Similarity 0.786

#   [93m----------------------------------------------------------------------------------------------------[0m

#     Filename images/attention_is_all_you_need_1.jpeg

#   [93m----------------------------------------------------------------------------------------------------[0m

#     Description `This image displays a detailed technical diagram of the **Transformer model architecture**, a fundamental neural network structure in deep learning, particularly for natural language processing (NLP) tasks. The diagram, labeled "Figure 1: The Transformer - model architecture," is centrally placed on a white page, with descriptive text below it.

#   

#   **Diagram Details:**

#   

#   The architecture is presented as an **encoder-decoder stack**, depicted in two main vertical columns.

#   

#   1.  **Input and Embedding Layers (Bottom):**

#       *   At the very bottom left, a black arrow points to a label "Inputs."

#       *   Directly above it is a light pink rectangular block labeled "Input Embedding."

#       *   A circular light gray block labeled "Positional Encoding" is connected via a black arrow and a "+" sign to the "Input Embedding" block's output.

#       *   Similarly, on the bottom right, a label "Outputs (shifted right)" has an arrow pointing to a light pink rectangular block labeled "Output Embedding."

#       *   A...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   <IPython.core.display.Image object>

"""
...and another example:
"""

query = "a kitten with a robot"
results = search_multimodal(query, vector_index, top_k=1)

if not results:
    pretty_print.wrapped("❌ No results found", title="❌")
else:
    result = results[0]

    pretty_print.wrapped(
        [
            f"Similarity {result['similarity']:.3f}",
            f"Filename {result['filename']}",
            f"Description `{result['description'][:1000]}...`",
        ],
        title=f"Results for query = {query}",
    )
    display_image(Path(result["filename"]))
# Output:
#   

#   🔍 Embedding query: 'a kitten with a robot'

#   ✅ Query embedded successfully

#   [93m---------------------------- Results for query = a kitten with a robot ----------------------------[0m

#     Similarity 0.814

#   [93m----------------------------------------------------------------------------------------------------[0m

#     Filename images/image_1.jpeg

#   [93m----------------------------------------------------------------------------------------------------[0m

#     Description `This image features a striking juxtaposition of a futuristic, menacing robot and an adorable, curious kitten in an industrial setting.

#   

#   **Objects:**

#   

#   1.  **Robot:** Occupying the right and center-right of the frame, a large, powerful, humanoid robot is depicted from the chest up.

#       *   **Head/Helmet:** The robot's head is designed like a sleek, dark metallic helmet, featuring intricate circuit board patterns or etched lines across its surface, giving it a high-tech, integrated look.

#       *   **Eyes:** The most prominent feature of the robot's face are its intensely glowing, bright red eyes, which emit a subtle light flare, creating a sense of power or surveillance. These eyes are set within dark, shadowed sockets.

#       *   **Body/Armor:** The robot's body is clad in heavy, articulated dark gray and silver metallic armor with visible rivets, bolts, and panel lines. Exposed mechanical elements, wires, and pistons can be seen in the joints, particularly around the neck and shoulder area.

#   ...`

#   [93m----------------------------------------------------------------------------------------------------[0m

#   <IPython.core.display.Image object>

"""
## 4. Building multimodal AI agents
"""

"""
The last step is to hook our RAG `search_multimodal` function to a ReAct agent to create an agentic RAG system.

First, we define the `multimodal_search_tool` using LangGraph:
"""

from langchain_core.tools import tool
from langchain_google_genai import ChatGoogleGenerativeAI
from langgraph.prebuilt import create_react_agent


@tool
def multimodal_search_tool(query: str) -> dict[str, Any]:
    """
    Search through a collection of images and their text descriptions to find relevant content.

    This tool searches through a pre-indexed collection of image-text pairs using the query
    and returns the most relevant match. The search uses multimodal embeddings to find
    semantic matches between the query and the content.

    Args:
        query: Text query describing what to search for (e.g., "cat", "kitten with robot")

    Returns:
        A formatted string containing the search result with description and similarity score
    """

    pretty_print.wrapped(query, title="🔍 Tool executing search for:")

    results = search_multimodal(query, vector_index, top_k=1)

    if not results:
        return {"role": "tool_result", "content": "No relevant content found for your query."}
    else:
        pretty_print.wrapped(str(results[0]["filename"]), title="🔍 Found results:")
    result = results[0]

    content = [
        {
            "type": "text",
            "text": f"Image description: {result['description']}",
        },
        types.Part.from_bytes(
            data=result["content"],
            mime_type="image/jpeg",
        ),
    ]

    return {
        "role": "tool_result",
        "content": content,
    }

"""
Next, we create a ReAct agent using LangGraph's `create_react_agent` function and the RAG tool defined above:
"""

def build_react_agent() -> Any:
    """
    Build a ReAct agent with multimodal search capabilities.

    This function creates a LangGraph ReAct agent that can search through images
    and text using the multimodal_search_tool. The agent uses Gemini 2.5 Pro
    for reasoning and tool execution.

    Returns:
        Any: A LangGraph ReAct agent instance configured with multimodal search tools
    """

    tools = [multimodal_search_tool]

    system_prompt = """You are a helpful AI assistant that can search through images and text to answer questions.
    
    When asked about visual content like animals, objects, or scenes:
    1. Use the multimodal_search_tool to find relevant images and descriptions
    2. Carefully analyze the image or image descriptions from the search results
    3. Look for specific details like colors, features, objects, or characteristics
    4. Provide a clear, direct answer based on the search results
    5. If you can't find the specific information requested, be honest about limitations
    
    Pay special attention to:
    - Colors and visual characteristics
    - Animal features and breeds
    - Objects and their properties
    - Scene descriptions and context
    
    Always search first using your tools before attempting to answer questions about specific images or visual content.
    """

    agent = create_react_agent(
        model=ChatGoogleGenerativeAI(model="gemini-2.5-pro", temperature=0.1),
        tools=tools,
        prompt=system_prompt,
    )

    return agent


react_agent = build_react_agent()
react_agent
# Output:
#   <langgraph.graph.state.CompiledStateGraph object at 0x31147b8c0>

"""
Now, let's test it and make the ReAct agent find the color of our kitten from the indexed dataset:
"""

try:
    test_question = "what color is my kitten?"
    pretty_print.wrapped(test_question, title="🧪 Asking question:")

    response = react_agent.invoke(input={"messages": test_question})
    messages = response.get("messages", [])
    if messages:
        final_message = messages[-1].content
    else:
        final_message = "No response from the agent"
    pretty_print.wrapped(final_message, title="🤖 Agent response")
except Exception as e:
    print(f"❌ Error in ReAct agent: {e}")
# Output:
#   [93m---------------------------------------- 🧪 Asking question: ----------------------------------------[0m

#     what color is my kitten?

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------- 🔍 Tool executing search for: -----------------------------------[0m

#     my kitten

#   [93m----------------------------------------------------------------------------------------------------[0m

#   

#   🔍 Embedding query: 'my kitten'

#   ✅ Query embedded successfully

#   [93m----------------------------------------- 🔍 Found results: -----------------------------------------[0m

#     images/image_1.jpeg

#   [93m----------------------------------------------------------------------------------------------------[0m

#   [93m----------------------------------------- 🤖 Agent response -----------------------------------------[0m

#     Based on the image, your kitten is a gray tabby. It has soft, short gray fur with darker tabby stripe patterns.

#   [93m----------------------------------------------------------------------------------------------------[0m


"""
Based on the image from the previous section (the one with the kitten and robot), the answer is correct.
"""

</details>


## YouTube Video Transcripts

<details>
<summary>[ 00:00 ] (A man in a white t-shirt and a cap speaks directly to the camera. The background shows a modern kitchen and living room area.) Although AI research is traditionally split into distinct fields like (A graphic of a robot saying "Hello" with "NLP" above it appears next to the speaker.) NLP and computer vision, (A graphic of a laptop with eyes and "CV" above it appears on the right side of the screen.) countless real-world problems require solutions that integrate information across these modalities. (A thought bubble with a banana and the text "Where can I find this?" appears on the left. On the right, a laptop with eyes and a thought bubble saying "Aisle 2" appears.) In this video, I'll discuss how we can solve such problems using (The text "Multimodal Embeddings" appears over the lower third of the screen.) multimodal embeddings. Then, show how to use them to do things like (A graphic showing a banana image with a list of fruits "Apple, Banana, Papaya" and a checkmark next to "Banana" appears on the left.) zero-shot image classification and (A graphic of a search bar with "Papaya" and a magnifying glass, showing images of "Apple, Banana, Papaya" with a green box around "Papaya" appears on the right.) image search. And if you're new here, welcome. (The text "Shaw Talebi" and "Data Scientist, Bread Enthusiast" appears on a lower third graphic.) I'm Shaw, I make videos about the things I'm learning about and building in AI. And if you enjoyed this content, please consider clicking the subscribe button. (A red "SUBSCRIBE" button with a hand clicking it appears.) That's a great no-cost way you can support me in all the videos that I make. (The video transitions to a title slide.)</summary>

[ 00:00 ] (A man in a white t-shirt and a cap speaks directly to the camera. The background shows a modern kitchen and living room area.) Although AI research is traditionally split into distinct fields like (A graphic of a robot saying "Hello" with "NLP" above it appears next to the speaker.) NLP and computer vision, (A graphic of a laptop with eyes and "CV" above it appears on the right side of the screen.) countless real-world problems require solutions that integrate information across these modalities. (A thought bubble with a banana and the text "Where can I find this?" appears on the left. On the right, a laptop with eyes and a thought bubble saying "Aisle 2" appears.) In this video, I'll discuss how we can solve such problems using (The text "Multimodal Embeddings" appears over the lower third of the screen.) multimodal embeddings. Then, show how to use them to do things like (A graphic showing a banana image with a list of fruits "Apple, Banana, Papaya" and a checkmark next to "Banana" appears on the left.) zero-shot image classification and (A graphic of a search bar with "Papaya" and a magnifying glass, showing images of "Apple, Banana, Papaya" with a green box around "Papaya" appears on the right.) image search. And if you're new here, welcome. (The text "Shaw Talebi" and "Data Scientist, Bread Enthusiast" appears on a lower third graphic.) I'm Shaw, I make videos about the things I'm learning about and building in AI. And if you enjoyed this content, please consider clicking the subscribe button. (A red "SUBSCRIBE" button with a hand clicking it appears.) That's a great no-cost way you can support me in all the videos that I make. (The video transitions to a title slide.)
[ 00:45 ] (The title slide reads: "Multimodal Embeddings: An introduction with example code (ft. CLIP)". Below the title, there's a diagram showing a book and an image going into a "CLIP" model, which then outputs "x_txt" and "x_img" vectors. At the bottom left, it says "Shaw Talebi".) Here we're going to talk about multimodal embeddings. Although the discussion here will focus around CLIP, which works with text and image data, this is a much more general idea that can be extended to many other modalities. (The video transitions to the next slide.)
[ 01:01 ] (The slide reads: "What are Embeddings? Useful numerical representations of data learned via model training".) Before talking about multimodal embeddings, it's worth answering the question what are embeddings? The way I'll define embeddings here are useful numerical representations of data learned through model training. (A brown square with the text "BERT" in the center appears.) A classic example of this is BERT, which is a popular language model before the era of GPT-3 and all the modern large language models. BERT used to be state of the art, and one of the things that it does [ 01:30 ] is masked language modeling. In other words, you can give it a sequence of text where one of the tokens in that sequence is masked, meaning that it's not visible. (A text input box appears on the left side of the BERT square, with "Listen to your [MASK]." written inside. An arrow points from this input box to the BERT square.) And BERT will predict the most likely token that goes in the place of that mask. (An arrow points from the BERT square to a text output box with "Listen to your instincts." written inside.) So if you pass in the sequence listen to your, the most likely token that goes in the sequence is instincts. So it turns out that through learning how to do this prediction, BERT learns useful representations of text, which can be extended to other NLP tasks. The basic idea here is that you'll take BERT and you'll drop its head, so the classification head, which is doing this masked language modeling, (A downward arrow labeled "Drop head" appears below the BERT square.) and you'll have this mutilated version of BERT, (The BERT square deforms into a trapezoidal shape, labeled "BERT (mutilated)".) which instead of doing this token prediction, it'll take an input sequence of text (An arrow points to the left of the "BERT (mutilated)" shape, with "Listen to your instincts." written next to it.) and return a numerical representation of it. (An arrow points to the right of the "BERT (mutilated)" shape, with a matrix of numbers appearing next to it, labeled "n x d".) Where each row in this matrix corresponds to each token in this text sequence, and then each of these columns corresponds to the embedding dimension, [ 02:40 ] the dimension of this internal representation of text that BERT uses in order to do masked language modeling. (The numbers in the matrix are highlighted in rows, then in columns.) We can go one step further and go from token-level representations to sequence-level representations. (The matrix transforms into a 1 x d vector of numbers, labeled "1 x d".) So we could do something like take the average across all these tokens in the sequence, and we're left with a 1 by D matrix, which represents the entire sequence. And of course, to get these embeddings to be a bit more practical, people will often do additional fine-tuning on top of these embeddings, but this is the basic idea of where they are coming from. A key point about embeddings is that these aren't just arbitrary numerical representations. They are typically semantically meaningful, such that if we were to look at how text were organized in this embedding space, (The slide transitions to a 2D Cartesian plane with "Dim 1" and "Dim 2" axes. Several text labels like "A cute puppy", "A good boy", "Best pet in the world", "A cute cat", and "Funny cat meme" are plotted on the plane.) similar concepts would tend to be located close together, while dissimilar concepts will tend to be far apart. For example, the sequence a cute puppy might be relatively close to the sequence a good boy, while that same sequence a cute puppy might be relatively far from a sequence like funny cat meme. However, this isn't limited to just text. We can generate embeddings for any type of data. (Another 2D Cartesian plane appears next to the first one, also with "Dim 1" and "Dim 2" axes. Several images of animals are plotted on it, including cats, dogs, a lion, and a goat.) Another popular type of data we might work with are images. So if we had some image embeddings, the space might be structured like this, where we tend to have cats in the top left part, the dogs tend to be in the bottom right part, and then we have a goat further away from these. [ 04:15 ] Although text embeddings and image embeddings are super helpful in that they can be adapted and repurposed to solve other either NLP tasks or computer vision tasks. One major limitation here is that any random text embedding space we might be working with, and any random image embedding space we might be working in, don't have any relationship to one another. There's no way out of the box to directly map this text embedding space to this image embedding space, and vice versa. Even if they are semantically meaningful in of themselves. (The text "Text and image embedding spaces are not aligned!" appears at the bottom right of the image.) And that's something we can plainly see here in that the text and image embedding spaces are not aligned, because for this text embedding space, the puppies tend to be in the top right, the cats tend to be at the bottom, while in our image embedding space, the cats tend to be up here, and the dogs tend to be down here. But what if there was a way we could merge these two embedding spaces together? (The slide transitions to the next slide.)
[ 05:08 ] (The slide reads: "Multimodal Embeddings: Embeddings which align representations of different data modalities".) That's exactly the key idea behind multimodal embeddings, which are embeddings which align representations of different data modalities. (A 2D Cartesian plane appears again, now with both text labels and animal images plotted on it. Semantically similar text and images are clustered together.) And the basic intuition here is that if we had a multimodal embedding space, we could represent text and images in the same vector space. So now, indeed, texts like a cute puppy will be close to images of cute puppies, the text a cute cat will be close to images of a cute cat, and the same thing will hold for other concepts. However, this idea is not limited to just images and text. (Sound wave graphics appear over some of the text labels and animal images, indicating audio embeddings.) We could just as easily embed audio and images together. Maybe this is a audio file that is a cat meowing, this is a goat making goat noises, we have a puppy with like a cute bark, and then maybe we have like a funny shrieking sound associated with this cat meme. (The slide transitions to the next slide.)
[ 05:59 ] (The slide title remains the same, but the graphic shows brain activity maps instead of audio waves, corresponding to images and text.) Another application of this is aligning representations of brain signals with images and text. What this means is if we were to record someone's brain activity and then represent it in this embedding space, we could in principle decode the brain information to generate images and text. So, in essence, reading people's thoughts. Actually, in reference number four, they are aiming to do exactly this with large language models. Intuitively, this idea of multimodal embeddings is pretty simple to understand. We have this embedding space, which is agnostic to modality. So it doesn't matter if it's a image of a cat, a text description of a cat, or the brain signals of someone looking at a picture of a cat, these numerical representations will be relatively similar. But how do we create these aligned numerical representations? In other words, how does this work under the hood? (The slide transitions to the next slide.)
[ 06:56 ] (The slide reads: "Contrastive Learning: Learning approach that seeks to represent different views of the same information similarly".) So the key technique behind creating these types of embeddings is contrastive learning, which is an approach that seeks to represent different views of the same underlying information similarly. And the way this works is that we'll train a model on two things: One, positive pairs (The text "Positive Pairs" appears in green on the left side of the screen.) of data, and two, negative pairs. (The text "Negative Pairs" appears in red on the right side of the screen.) So, in the case of aligning image and text representations, positive pairs might be a picture of a cute cat and a textual caption for this image. (An image of a cat and the text "A cute cat" with a green checkmark next to it appears under "Positive Pairs".) And then we might have the text and an image of a cute puppy. (An image of a puppy and the text "A cute puppy" with a green checkmark next to it appears.) And then we might have the same thing for a baby goat. (An image of a baby goat and the text "Cute baby goat" with a green checkmark next to it appears.) On the other hand, negative captions might look something like this, (An image of a cat and the text "A cute puppy" with a red X next to it appears under "Negative Pairs".) where you have the image of a cat, but the caption is a cute puppy. (An image of a puppy and the text "Cute baby goat" with a red X next to it appears.) Image of a puppy, and the caption is a goat. (An image of a baby goat and the text "A cute cat" with a red X next to it appears.) And you have a goat and the caption is a cat. So the intuition here is that we train a model to maximize the similarity between these positive pairs, and minimize the similarity between these negative pairs. That's the key intuition. In the following slides, we're going to go one level deeper and look at the loss function and the math behind how this is accomplished. If you don't care about the math and how this is working under the hood, feel free to skip ahead to the example code, but if you're interested in the math, we're about to jump right into it. (The slide transitions to the next slide.)
[ 08:17 ] (The slide reads: "Contrastive Learning" and shows a diagram. Three images (cat, puppy, goat) are on the left, each with an arrow pointing to a vector of numbers (embedding). These vectors combine to form a matrix "I_e". On the right, three text captions ("A cute cat", "A cute puppy", "Cute baby goat") also point to vectors, forming a matrix "T_e".) The way we can use contrastive learning to align image and text representations is we can take images, generate image embeddings using an image encoder. So basically we take our images and generate a single numerical representation for them. And then we can take all these image embeddings, and we can concatenate them into a matrix that I'll call I sub E. So this will be an N by D matrix, where N is the number of images, so if you have three images here, it will be 1, 2, 3. And then D, so the number of columns will be the embedding dimension. Then we can do a similar thing for text. So we can get a text encoder from off the shelf. We can generate these text embeddings, and then we can concatenate them into a matrix that I'll call T sub E, and that will have the same shape. So we'll have N captions, and then they'll have some embedding dimension D. (The diagram updates with text, showing the multiplication of raw embeddings by learnable weights, normalized, to produce the matrices I_e and T_e.) Just to point out here that the representations that we would put into these matrices won't directly come from an image encoder and text encoder. Instead, these will be multiplied by some learnable weight matrix, and then normalized before being organized in this matrix. So that weight matrix that we multiply the original embeddings by are the learnable parameters. (The slide transitions to the next slide.)
[ 09:40 ] (The slide now displays the formula: "logits_{i,j} = sim(I_e[i], T_e[j]) / t". Below it, a 3x3 matrix is shown, with rows labeled by the animal images and columns by the text captions, each cell containing "logits_{i,j}". Diagonal elements are green, off-diagonal elements are red.) Once we have these matrices I and T, we can construct this logits matrix. Basically what that means is we're going to take each image in our image embedding matrix, and then each text sequence in our text embedding matrix, and we're going to compute their similarity. Typically this is just a cosine similarity, so you do the dot product between these two matrices, and then you'll divide it by a temperature parameter. That's what this tau parameter is representing. [ 10:04 ] And the reason we call them logits is because at some point, it's going to be the argument in an exponential function. And we'll see that in a little bit here. So taking just those three examples from the previous slide, the similarity between the first image and the first text sequence will be in this 1-1 position of the matrix. And then the similarity between this cat image and the sequence a cute puppy will be represented by this value here. Then the similarity between this cat image and the text sequence a cute baby goat will be represented by this value here, and so on and so forth. Just looking at this, we can see that what we want is to make the logits on the diagonal of this matrix as big as possible. So in other words, we want to maximize the similarity between the positive pairs. And then we want to minimize the off-diagonal elements, which correspond to negative pairs. (The slide updates to include the "Contrastive Loss (Images)" formula: "ℓ_i = -log (exp(logits_{i,i}) / Σ_{j=1}^{n} exp(logits_{i,j}))".) One way we can do this is via the contrastive loss. So this might take slightly different forms depending on the context or the paper that you're reading. But here I'm going to follow what was done in developing CLIP, which is reference number three here. And so basically, one way we can achieve this goal of maximizing the similarity of these on-diagonal elements, and minimizing the similarity between these off-diagonal elements is via this equation here, which is basically saying for the Ith image, so let's say this cat image here, we want the numerator to be as big as possible. So the numerator will be the II element, so this will be either 1-1, or 2-2, or 3-3. And then we want the denominator to be as small as possible. So if the numerator is big, the denominator is small, that means this fraction becomes big. And then if we take the log of that, we'll still have a big number. And then we want this number to be as big as possible because the goal of training is to minimize the loss. And then if this number is big, and we have a minus sign next to it, then this will be as minimal as possible. That was probably a bit abstract, so let's walk through this step by step. (The slide updates with a visual breakdown of the "Contrastive Loss (Images)" formula applied to the first row of the logits matrix.) Let's look at just the first image first. With this notation, I call the loss associated with the first image L1. This will consist of taking this 1-1 logit, and then summing over all the logits in this first row. So we're basically taking this image and comparing it to every single caption. (The slide visually breaks down the formula for the second and third images.) Then we do the same thing for the second image. We have the positive pair similarity here. And then we sum over all the logits in this row. And then we do a similar thing for image number three. So we look at the positive pair similarity, and then we sum over all the logits or similarities in this row. (The slide updates with the "Contrastive Loss (Images)" formula for L_I, which is the average of individual losses.) We can do this for every single image in our batch, or even in our whole training dataset, and then we can aggregate them to get the final contrastive loss. What that'll look like is we'll take the loss according to the first image, the loss according to the second image, and the loss corresponding to the third image, and then we can just take their average. And that'll give us the contrastive loss for the images. But we can do the same exact thing for text. (The slide updates with the "Contrastive Loss (Text)" formula for L_j, which is similar to the image loss but sums over images for each text.) This is how I'm noting contrastive loss for the text. I've switched the index here from J to I, and then I've changed the summation here to I. I feel this notation might be a bit too subtle, but hopefully explaining it step by step it makes sense what I mean here. So let's see what this looks like for the first text sequence. We're going to be evaluating a cute cat. So we'll look at logits 1-1 here, and then we'll sum over the logits in this first column. We'll do the same thing for this second text sequence, a cute puppy. We'll sum over all the logits in this column. And then finally, we do it for the final text sequence. It's important to note here that generally this logits matrix is asymmetric, because the similarity between the text a cute puppy and this image of a cat is, in general, different than the similarity between this image of a puppy and the text sequence a cute cat. That's an important thing to note here, and that's the reason why we go through this whole procedure for the images and the text sequences separately. And then we can aggregate the loss over all the text examples just like we did for the images like this, and then we'll get a total text loss by taking the average of all the examples in our mini-batch. (The slide updates with the "Final Loss" formula: "L = (L_I + L_T) / 2".) We can then combine the image loss and text loss together by taking their average. And then we can write it all out to have this big monstrosity all on one page. (The formula expands to show the full expressions for L_I and L_T.) But basically, this first term here corresponds to the image loss, the second term here corresponds to the text loss, and this is how we train the weights which translate the raw image and text encodings into our multimodal embedding space. This will give us a training signal which we can use to update the weights of these projection matrices. We can just keep doing that until we're satisfied with the loss. (The slide transitions to the next slide.)
[ 15:13 ] (The slide reads: "Example 1: Using CLIP for 0-shot Image Classification". A Python logo and a hugging face emoji appear on the right side.) So if that was much more math than you were hoping to get out of this video, I apologize for that, but let's jump to practical applications of multimodal embeddings. Here I'm going to use CLIP for two different use cases. This first use case is zero-shot image classification. The meaning of that is we're going to do image classification without explicitly training CLIP to distinguish between the different image classes that we're considering. (The slide updates with Python code to import libraries.) The first step is to import transformers. I'm going to bring in these two things. And then I'm importing this PIL library, which will allow us to work with images in Python. (The slide updates with Python code to load the model and processor.) Next, we'll load in the model and the data processor. The image preprocessing is important because images could be any type of size and shape and all that. The CLIP processor is an abstraction that ensures the data are in a suitable format to be passed through the model. (The slide updates with Python code to load an image and define text classes. An image of a cat appears on the right.) Next, we're going to load in our image. So I'm going to load in this image of a cute cat. So it's the same one we've seen so far. Then I'm going to define the text classes. So this is a really interesting aspect of using CLIP for zero-shot image classification, because before, if you wanted to do image classification, traditionally that was something that was set at model training. It was [ 16:30 ] implicitly coded into the architecture of the model in that you had this classification head, and each value in the output layer corresponded to the probability of class one versus class two versus class three, so on and so forth. But now, when using CLIP, which is trained via contrastive learning, we actually pass these classes as text inputs. So with our text and image inputs defined, we can pass these through our processor to put them in a suitable format as CLIP. (The slide updates with Python code to pass image and text to CLIP.) And then we can just pass it to the model. Then with this one line of code, we'll generate these outputs. (The slide updates with Python code to convert image logits to probabilities.) Then we can extract the logits per image. Recall the logits matrix that we saw a few slides ago, (The logits matrix appears again next to the image.) where we had an image, and then we had logit values or similarity values between that image and every single piece of text that we passed into the model. That's exactly what we're extracting here. We're extracting the similarity score of the input image to both the text inputs. (The section of code highlights the extraction of similarity scores.) We can then convert these logits to probabilities via the softmax. (The slide updates with Python code to print results.) And then, this will give us a prediction. What I'm doing here is I'm just doing argmax of the probabilities tensor, and using that to pick out the predicted class from this original list that I created. And then I'm just going to print everything like this. So I'll print the predicted class as well as a rounded probability corresponding to that class. (The results "a photo of a cat | Probability = 0.9979" appear below the code.) With that, the most probable class is a photo of a cat with an associated probability of 99.79%. So basically nails the classification of this image. (The slide updates with a new test of classes: "ugly cat" vs "cute cat".) But let's see what happens when we use different text classes. Instead of passing in a photo of a cat and a photo of a dog, which are pretty easy classes to distinguish between, let's try something a bit more nuanced like a ugly cat versus cute cat. And again, here the model basically nails it with a 97% probability of this cute cat class. (The slide updates with another new test of classes: "cat meme" vs "not cat meme".) And then we can try something even more challenging, like trying to distinguish if this is a cat meme or not a cat meme. And it indeed gets that it's not a cat meme, but we can see that the probability dropped significantly. (The image on the right changes to the "cat meme" image.) Then as a final test of this model, let's see what happens when we pass in an actual cat meme and give it the class choices of cat meme versus not cat meme. (The results "cat meme | Probability = 0.8338" appear below the code.) And so here the model again nails it. It correctly classifies this as a cat meme with a probability of [ 19:00 ] 83%. And so again, what we're doing here, using CLIP, is we're taking these three entities. We're taking the text sequence of cat meme, the text sequence of not cat meme, and this image of a cat, encoding them in a shared embedding space, and we're evaluating the similarity between this image of a cat and the text sequence cat meme, and the similarity between this image of a cat and the text sequence not a cat meme. And then we can convert that similarity into a probability, as well as a class prediction. The key unlock here is that you are not restricted or limited in the different class labels you can use for image classification. You can be as detailed or vague as you like. You can adapt this to endless different use cases, which is pretty amazing. (The slide transitions to the next slide.)
[ 19:50 ] (The slide reads: "Example 2: Using CLIP for Image Search". A Python logo and a hand emoji appear on the right side.) This second example is basically the inverse of zero-shot image classification. There, we had an input image, and we wanted to match it with one of the input text sequences. Here, in example two, we're going to do the exact opposite. So instead of starting with an image, we're going to start with a piece of text, in other words, a search query, and then we're going to match it to a set of images. So essentially, what we're doing is we're doing a search over a set of images. (The slide updates with Python code to load images.) The way this looks is we'll first load in our images. Here we have a picture of a cute cat, picture of a dog, and a picture of a goat. We'll store them in this image list. We're using the PIL library to open the images and just store them in this list. (The slide updates with Python code to define the query and process inputs.) Then we're going to define a query and process the inputs. Here our query will be a cute dog. And then we'll pass this query along with our image list through the processor, so it's in the appropriate format for CLIP. (The slide updates with Python code to compute image probabilities.) Then we'll run these inputs through our model, get these outputs, extract the logits per text now. Before we did logits per image, now we're doing logits per text. So these are going to be the similarity scores between the input text and all the images that we inputted. And then we'll convert these logits into probabilities. (The slide updates with Python code to evaluate the best match.) So with that, we can evaluate the best match. So I'm doing that again in a similar way. So we have these probabilities, doing argmax, which will give us an integer 0, 1, or 2. We can use that to pick out the best matched image. And then we can take the probability associated with that image. And then we can just print everything. (The results "Match probability: 0.9817" and an image of a dog appear.) So again, the query here was a cute dog, and this is the best matched image with a probability of about 98%. But again, that was a super easy example. So let's try a trickier query like something cute but metal. (The results "Match probability: 0.7715" and an image of a baby goat appear.) In this case, the model returns the goat, which is indeed cute, but also goats are associated with heavy metal music, and it got a 77% match probability. (The slide updates with a new query: "a good boy". The goat image remains.) Reading this, a good boy, the text itself doesn't have anything to do with animals. You know, maybe it's a human boy and he's well-behaved. But a good boy is a colloquialism for dogs that we use often. (The results "Match probability: 0.8248" and an image of a dog appear.) And the model can pick that up quite easily. So it matches it with a dog with an 82% probability. It would be interesting to see if we threw in a picture of a human boy to see how the model would handle that case. This could be something that you do with the example code from the GitHub. (The slide updates with a new query: "the best pet in the world". The dog image remains.) And then we can try an extremely controversial query like the best pet in the world. (The results "Match probability: 0.5664" and an image of a cat appear.) For this, the model returns a cat with a 56% match probability. This is likely indicating that on average, people on the internet love cats more than they love dogs. [ 22:42 ] Nevertheless, it's super interesting how we can use this model in order to do search like this. (The video transitions to the next slide.)
[ 22:47 ] (The slide reads: "What's Next?".) So those were the two examples. Code is on the GitHub, link in the description below. Let's look ahead to the next video of this series. In the previous video, so part one, (A box appears labeled "Multimodal LLM" with arrows pointing to it from "book", "image", and "audio" icons, and an arrow pointing away to a "book" icon.) we talked about multimodal large language models. So basically, large language models that can process or generate multiple data modalities. In this video, (A box appears labeled "Multimodal Embedding" with arrows pointing to it from "book" and "image" icons, and an arrow pointing away to "book" and "image" icons.) we talked about multimodal embeddings, like those generated by CLIP, which can be used to do things like image search. So we pass in a query and a set of potential images, and then it'll spit out the best matched image. (The slide transitions to the next slide.)
[ 23:25 ] (The slide reads: "What's Next? Multimodal RAG".) In the next video of this series, we're going to bring these two ideas together to create a multimodal RAG system. The basic flow here will be to take a user query, like what's there to do in Bali? (A box appears labeled "MME" with an arrow pointing to it from the "What's there to do in Bali?" query. Inside the box, book and image icons are scattered.) We'll pass the query into a multimodal retrieval system, which involves using a multimodal embedding model to pick out the documents and images that are most relevant to this query. (The relevant documents and images move to the right, to a step labeled "Create Prompt".) We'll take the user query and relevant documents and images to generate a prompt. (The prompt moves to the right, into a "Multimodal LLM" box.) And then we'll pass that prompt into a multimodal large language model, (The model outputs a document icon, labeled "Model Response".) which can process the user query, relevant text documents, and relevant images to generate a helpful response. (The video transitions to the final slide.)
[ 24:10 ] (The slide shows a screenshot of an article titled "Multimodal Embeddings: An introduction" and a picture of building blocks spelling "WORDS". The text of the article reads: "This is the 2nd article in a larger series on multimodal AI. In the previous post, we saw how to augment large language models (LLMs) to understand new data modalities (e.g., images, audio, video). One such approach relied on encoders that generate vector representations (i.e. embeddings) of non-text data. In this article, I will discuss multimodal embeddings and share what they can do via two practical use cases.") And as a final note, if you enjoyed this video and you want to learn more, check out the blog published in Towards Data Science. There I went into some details that I probably missed here. And as always, even though this is going to be a member-only story, you can access it completely for free using the friend link in the description below. And with that, thank you so much for your time and thanks for watching. (The screen fades to black with a simple graphic and "Thanks for watching".)
_The video explains multimodal embeddings, focusing on how they align different data types like text and images in a shared vector space, and demonstrates their practical application through zero-shot image classification and image search using the CLIP model._

</details>


## Additional Sources Scraped

<details>
<summary>complex-document-recognition-ocr-doesn-t-work-and-here-s-how</summary>

# Complex Document Recognition: OCR Doesn’t Work and Here’s How You Fix It

by Oleg Kokorin

October 12th, 2023

</details>

<details>
<summary>google-generative-ai-embeddings-ai-studio-gemini-api-langcha</summary>

Connect to Google's generative AI embeddings service using the `GoogleGenerativeAIEmbeddings` class, found in the [langchain-google-genai](https://pypi.org/project/langchain-google-genai/) package.

This will help you get started with Google's Generative AI embedding models (like Gemini) using LangChain. For detailed documentation on `GoogleGenerativeAIEmbeddings` features and configuration options, please refer to the [API reference](https://python.langchain.com/v0.2/api_reference/google_genai/embeddings/langchain_google_genai.embeddings.GoogleGenerativeAIEmbeddings.html).

## Overview [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#overview "Direct link to Overview")

### Integration details [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#integration-details "Direct link to Integration details")

| Provider | Package |
| --- | --- |
| [Google Gemini](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai) | [langchain-google-genai](https://python.langchain.com/api_reference/google_genai/embeddings/langchain_google_genai.embeddings.GoogleGenerativeAIEmbeddings.html) |

## Setup [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#setup "Direct link to Setup")

To access Google Generative AI embedding models you'll need to create a Google Cloud project, enable the Generative Language API, get an API key, and install the `langchain-google-genai` integration package.

### Credentials [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#credentials "Direct link to Credentials")

To use Google Generative AI models, you must have an API key. You can create one in Google AI Studio. See the [Google documentation](https://ai.google.dev/gemini-api/docs/api-key) for instructions.

Once you have a key, set it as an environment variable `GOOGLE_API_KEY`:

```codeBlockLines_e6Vv
import getpass
import os

if not os.getenv("GOOGLE_API_KEY"):
    os.environ["GOOGLE_API_KEY"] = getpass.getpass("Enter your Google API key: ")

```

To enable automated tracing of your model calls, set your [LangSmith](https://docs.smith.langchain.com/) API key:

```codeBlockLines_e6Vv
# os.environ["LANGSMITH_TRACING"] = "true"
# os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")

```

## Installation [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#installation "Direct link to Installation")

```codeBlockLines_e6Vv
%pip install --upgrade --quiet  langchain-google-genai

```

## Usage [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#usage "Direct link to Usage")

```codeBlockLines_e6Vv
from langchain_google_genai import GoogleGenerativeAIEmbeddings

embeddings = GoogleGenerativeAIEmbeddings(model="models/gemini-embedding-001")
vector = embeddings.embed_query("hello, world!")
vector[:5]

```

```codeBlockLines_e6Vv
[-0.024917153641581535,\
 0.012005362659692764,\
 -0.003886754624545574,\
 -0.05774897709488869,\
 0.0020742062479257584]

```

## Batch [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#batch "Direct link to Batch")

You can also embed multiple strings at once for a processing speedup:

```codeBlockLines_e6Vv
vectors = embeddings.embed_documents(
    [\
        "Today is Monday",\
        "Today is Tuesday",\
        "Today is April Fools day",\
    ]
)
len(vectors), len(vectors[0])

```

```codeBlockLines_e6Vv
(3, 3072)

```

## Indexing and Retrieval [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#indexing-and-retrieval "Direct link to Indexing and Retrieval")

Embedding models are often used in retrieval-augmented generation (RAG) flows, both as part of indexing data as well as later retrieving it. For more detailed instructions, please see our [RAG tutorials](https://python.langchain.com/docs/tutorials/rag/).

Below, see how to index and retrieve data using the `embeddings` object we initialized above. In this example, we will index and retrieve a sample document in the `InMemoryVectorStore`.

```codeBlockLines_e6Vv
# Create a vector store with a sample text
from langchain_core.vectorstores import InMemoryVectorStore

text = "LangChain is the framework for building context-aware reasoning applications"

vectorstore = InMemoryVectorStore.from_texts(
    [text],
    embedding=embeddings,
)

# Use the vectorstore as a retriever
retriever = vectorstore.as_retriever()

# Retrieve the most similar text
retrieved_documents = retriever.invoke("What is LangChain?")

# show the retrieved document's content
retrieved_documents[0].page_content

```

**API Reference:** [InMemoryVectorStore](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.in_memory.InMemoryVectorStore.html)

```codeBlockLines_e6Vv
'LangChain is the framework for building context-aware reasoning applications'

```

## Task type [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#task-type "Direct link to Task type")

`GoogleGenerativeAIEmbeddings` optionally support a `task_type`, which currently must be one of:

- `SEMANTIC_SIMILARITY`: Used to generate embeddings that are optimized to assess text similarity.
- `CLASSIFICATION`: Used to generate embeddings that are optimized to classify texts according to preset labels.
- `CLUSTERING`: Used to generate embeddings that are optimized to cluster texts based on their similarities.
- `RETRIEVAL_DOCUMENT`, `RETRIEVAL_QUERY`, `QUESTION_ANSWERING`, and `FACT_VERIFICATION`: Used to generate embeddings that are optimized for document search or information retrieval.
- `CODE_RETRIEVAL_QUERY`: Used to retrieve a code block based on a natural language query, such as sort an array or reverse a linked list. Embeddings of the code blocks are computed using `RETRIEVAL_DOCUMENT`.

By default, we use `RETRIEVAL_DOCUMENT` in the `embed_documents` method and `RETRIEVAL_QUERY` in the `embed_query` method. If you provide a task type, we will use that for all methods.

```codeBlockLines_e6Vv
%pip install --upgrade --quiet  matplotlib scikit-learn

```

```codeBlockLines_e6Vv
from langchain_google_genai import GoogleGenerativeAIEmbeddings
from sklearn.metrics.pairwise import cosine_similarity

query_embeddings = GoogleGenerativeAIEmbeddings(
    model="models/gemini-embedding-001", task_type="RETRIEVAL_QUERY"
)
doc_embeddings = GoogleGenerativeAIEmbeddings(
    model="models/gemini-embedding-001", task_type="RETRIEVAL_DOCUMENT"
)

q_embed = query_embeddings.embed_query("What is the capital of France?")
d_embed = doc_embeddings.embed_documents(
    ["The capital of France is Paris.", "Philipp is likes to eat pizza."]
)

for i, d in enumerate(d_embed):
    print(f"Document {i + 1}:")
    print(f"Cosine similarity with query: {cosine_similarity([q_embed], [d])[0][0]}")
    print("---")

```

```codeBlockLines_e6Vv
Document 1
Cosine similarity with query: 0.7892893360164779
---
Document 2
Cosine similarity with query: 0.5438283285204146
---

```

## API Reference [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#api-reference "Direct link to API Reference")

For detailed documentation on `GoogleGenerativeAIEmbeddings` features and configuration options, please refer to the [API reference](https://python.langchain.com/api_reference/google_genai/embeddings/langchain_google_genai.embeddings.GoogleGenerativeAIEmbeddings.html).

## Additional Configuration [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#additional-configuration "Direct link to Additional Configuration")

You can pass the following parameters to ChatGoogleGenerativeAI in order to customize the SDK's behavior:

- `client_options`: [Client Options](https://googleapis.dev/python/google-api-core/latest/client_options.html#module-google.api_core.client_options) to pass to the Google API Client, such as a custom `client_options["api_endpoint"]`
- `transport`: The transport method to use, such as `rest`, `grpc`, or `grpc_asyncio`.

## Related [​](https://python.langchain.com/docs/integrations/text_embedding/google_generative_ai/\#related "Direct link to Related")

- Embedding model [conceptual guide](https://python.langchain.com/docs/concepts/embedding_models/)
- Embedding model [how-to guides](https://python.langchain.com/docs/how_to/#embedding-models)

</details>

<details>
<summary>image-understanding-gemini-api-google-ai-for-developers</summary>

# Image understanding

Gemini models are built to be multimodal from the ground up, unlocking a wide range of image processing and computer vision tasks including but not limited to image captioning, classification, and visual question answering without having to train specialized ML models.

## Passing images to Gemini

You can provide images as input to Gemini using two methods:

- Passing inline image data: Ideal for smaller files (total request size less than 20MB, including prompts).
- Uploading images using the File API: Recommended for larger files or for reusing images across multiple requests.

### Passing inline image data

You can pass inline image data in the request to `generateContent`. You can provide image data as Base64 encoded strings or by reading local files directly (depending on the language).

The following example shows how to read an image from a local file and pass it to `generateContent` API for processing.

Python
```
  from google.genai import types

  with open('path/to/small-sample.jpg', 'rb') as f:
      image_bytes = f.read()

  response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[\
      types.Part.from_bytes(\
        data=image_bytes,\
        mime_type='image/jpeg',\
      ),\
      'Caption this image.'\
    ]
  )

  print(response.text)

```

JavaScript
```
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";

const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
  encoding: "base64",
});

const contents = [\
  {\
    inlineData: {\
      mimeType: "image/jpeg",\
      data: base64ImageFile,\
    },\
  },\
  { text: "Caption this image." },\
];

const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: contents,
});
console.log(response.text);

```

Go
```
bytes, _ := os.ReadFile("path/to/small-sample.jpg")

parts := []*genai.Part{
  genai.NewPartFromBytes(bytes, "image/jpeg"),
  genai.NewPartFromText("Caption this image."),
}

contents := []*genai.Content{
  genai.NewContentFromParts(parts, genai.RoleUser),
}

result, _ := client.Models.GenerateContent(
  ctx,
  "gemini-2.5-flash",
  contents,
  nil,
)

fmt.Println(result.Text())

```

curl / REST
```
IMG_PATH="/path/to/your/image1.jpg"

if [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then
B64FLAGS="--input"
else
B64FLAGS="-w0"
fi

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-X POST \
-d '{
    "contents": [{\
    "parts":[\
        {\
            "inline_data": {\
            "mime_type":"image/jpeg",\
            "data": "'"$(base64 $B64FLAGS $IMG_PATH)"'"\
            }\
        },\
        {"text": "Caption this image."},\
    ]\
    }]
}' 2> /dev/null

```

You can also fetch an image from a URL, convert it to bytes, and pass it to `generateContent` as shown in the following examples.

Python
```
from google import genai
from google.genai import types

import requests

image_path = "https://goo.gle/instrument-img"
image_bytes = requests.get(image_path).content
image = types.Part.from_bytes(
  data=image_bytes, mime_type="image/jpeg"
)

client = genai.Client()

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=["What is this image?", image],
)

print(response.text)

```

JavaScript
```
import { GoogleGenAI } from "@google/genai";

async function main() {
  const ai = new GoogleGenAI({});

  const imageUrl = "https://goo.gle/instrument-img";

  const response = await fetch(imageUrl);
  const imageArrayBuffer = await response.arrayBuffer();
  const base64ImageData = Buffer.from(imageArrayBuffer).toString('base64');

  const result = await ai.models.generateContent({
    model: "gemini-2.5-flash",
    contents: [\
    {\
      inlineData: {\
        mimeType: 'image/jpeg',\
        data: base64ImageData,\
      },\
    },\
    { text: "Caption this image." }\
  ],
  });
  console.log(result.text);
}

main();

```

Go
```
package main

import (
  "context"
  "fmt"
  "os"
  "io"
  "net/http"
  "google.golang.org/genai"
)

func main() {
  ctx := context.Background()
  client, err := genai.NewClient(ctx, nil)
  if err != nil {
      log.Fatal(err)
  }

  // Download the image.
  imageResp, _ := http.Get("https://goo.gle/instrument-img")

  imageBytes, _ := io.ReadAll(imageResp.Body)

  parts := []*genai.Part{
    genai.NewPartFromBytes(imageBytes, "image/jpeg"),
    genai.NewPartFromText("Caption this image."),
  }

  contents := []*genai.Content{
    genai.NewContentFromParts(parts, genai.RoleUser),
  }

  result, _ := client.Models.GenerateContent(
    ctx,
    "gemini-2.5-flash",
    contents,
    nil,
  )

  fmt.Println(result.Text())
}

```

curl / REST
```
IMG_URL="https://goo.gle/instrument-img"

MIME_TYPE=$(curl -sIL "$IMG_URL" | grep -i '^content-type:' | awk -F ': ' '{print $2}' | sed 's/\r$//' | head -n 1)
if [[ -z "$MIME_TYPE" || ! "$MIME_TYPE" == image/* ]]; then
  MIME_TYPE="image/jpeg"
fi

# Check for macOS
if [[ "$(uname)" == "Darwin" ]]; then
  IMAGE_B64=$(curl -sL "$IMG_URL" | base64 -b 0)
elif [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then
  IMAGE_B64=$(curl -sL "$IMG_URL" | base64)
else
  IMAGE_B64=$(curl -sL "$IMG_URL" | base64 -w0)
fi

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent" \
    -H "x-goog-api-key: $GEMINI_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
            {\
              "inline_data": {\
                "mime_type":"'"$MIME_TYPE"'",\
                "data": "'"$IMAGE_B64"'"\
              }\
            },\
            {"text": "Caption this image."}\
        ]\
      }]
    }' 2> /dev/null

```

### Uploading images using the File API

For large files or to be able to use the same image file repeatedly, use the Files API. The following code uploads an image file and then uses the file in a call to `generateContent`. See the Files API guide for more information and examples.

Python
```
from google import genai

client = genai.Client()

my_file = client.files.upload(file="path/to/sample.jpg")

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=[my_file, "Caption this image."],
)

print(response.text)

```

JavaScript
```
import {
  GoogleGenAI,
  createUserContent,
  createPartFromUri,
} from "@google/genai";

const ai = new GoogleGenAI({});

async function main() {
  const myfile = await ai.files.upload({
    file: "path/to/sample.jpg",
    config: { mimeType: "image/jpeg" },
  });

  const response = await ai.models.generateContent({
    model: "gemini-2.5-flash",
    contents: createUserContent([\
      createPartFromUri(myfile.uri, myfile.mimeType),\
      "Caption this image.",\
    ]),
  });
  console.log(response.text);
}

await main();

```

Go
```
package main

import (
  "context"
  "fmt"
  "os"
  "google.golang.org/genai"
)

func main() {
  ctx := context.Background()
  client, err := genai.NewClient(ctx, nil)
  if err != nil {
      log.Fatal(err)
  }

  uploadedFile, _ := client.Files.UploadFromPath(ctx, "path/to/sample.jpg", nil)

  parts := []*genai.Part{
      genai.NewPartFromText("Caption this image."),
      genai.NewPartFromURI(uploadedFile.URI, uploadedFile.MIMEType),
  }

  contents := []*genai.Content{
      genai.NewContentFromParts(parts, genai.RoleUser),
  }

  result, _ := client.Models.GenerateContent(
      ctx,
      "gemini-2.5-flash",
      contents,
      nil,
  )

  fmt.Println(result.Text())
}

```

curl / REST
```
IMAGE_PATH="path/to/sample.jpg"
MIME_TYPE=$(file -b --mime-type "${IMAGE_PATH}")
NUM_BYTES=$(wc -c < "${IMAGE_PATH}")
DISPLAY_NAME=IMAGE

tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "https://generativelanguage.googleapis.com/upload/v1beta/files" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${IMAGE_PATH}" 2> /dev/null > file_info.json

file_uri=$(jq -r ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent" \
    -H "x-goog-api-key: $GEMINI_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"file_data":{"mime_type": "'"${MIME_TYPE}"'", "file_uri": "'"${file_uri}"'"}},\
          {"text": "Caption this image."}]\
        }]
      }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json

```

## Prompting with multiple images

You can provide multiple images in a single prompt by including multiple image `Part` objects in the `contents` array. These can be a mix of inline data (local files or URLs) and File API references.

Python
```
from google import genai
from google.genai import types

client = genai.Client()

# Upload the first image
image1_path = "path/to/image1.jpg"
uploaded_file = client.files.upload(file=image1_path)

# Prepare the second image as inline data
image2_path = "path/to/image2.png"
with open(image2_path, 'rb') as f:
    img2_bytes = f.read()

# Create the prompt with text and multiple images
response = client.models.generate_content(

    model="gemini-2.5-flash",
    contents=[\
        "What is different between these two images?",\
        uploaded_file,  # Use the uploaded file reference\
        types.Part.from_bytes(\
            data=img2_bytes,\
            mime_type='image/png'\
        )\
    ]
)

print(response.text)

```

JavaScript
```
import {
  GoogleGenAI,
  createUserContent,
  createPartFromUri,
} from "@google/genai";
import * as fs from "node:fs";

const ai = new GoogleGenAI({});

async function main() {
  // Upload the first image
  const image1_path = "path/to/image1.jpg";
  const uploadedFile = await ai.files.upload({
    file: image1_path,
    config: { mimeType: "image/jpeg" },
  });

  // Prepare the second image as inline data
  const image2_path = "path/to/image2.png";
  const base64Image2File = fs.readFileSync(image2_path, {
    encoding: "base64",
  });

  // Create the prompt with text and multiple images

  const response = await ai.models.generateContent({

    model: "gemini-2.5-flash",
    contents: createUserContent([\
      "What is different between these two images?",\
      createPartFromUri(uploadedFile.uri, uploadedFile.mimeType),\
      {\
        inlineData: {\
          mimeType: "image/png",\
          data: base64Image2File,\
        },\
      },\
    ]),
  });
  console.log(response.text);
}

await main();

```

Go
```
// Upload the first image
image1Path := "path/to/image1.jpg"
uploadedFile, _ := client.Files.UploadFromPath(ctx, image1Path, nil)

// Prepare the second image as inline data
image2Path := "path/to/image2.jpeg"
imgBytes, _ := os.ReadFile(image2Path)

parts := []*genai.Part{
  genai.NewPartFromText("What is different between these two images?"),
  genai.NewPartFromBytes(imgBytes, "image/jpeg"),
  genai.NewPartFromURI(uploadedFile.URI, uploadedFile.MIMEType),
}

contents := []*genai.Content{
  genai.NewContentFromParts(parts, genai.RoleUser),
}

result, _ := client.Models.GenerateContent(
  ctx,
  "gemini-2.5-flash",
  contents,
  nil,
)

fmt.Println(result.Text())

```

curl / REST
```
# Upload the first image
IMAGE1_PATH="path/to/image1.jpg"
MIME1_TYPE=$(file -b --mime-type "${IMAGE1_PATH}")
NUM1_BYTES=$(wc -c < "${IMAGE1_PATH}")
DISPLAY_NAME1=IMAGE1

tmp_header_file1=upload-header1.tmp

curl "https://generativelanguage.googleapis.com/upload/v1beta/files" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -D upload-header1.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM1_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME1_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME1}'}}" 2> /dev/null

upload_url1=$(grep -i "x-goog-upload-url: " "${tmp_header_file1}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file1}"

curl "${upload_url1}" \
  -H "Content-Length: ${NUM1_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${IMAGE1_PATH}" 2> /dev/null > file_info1.json

file1_uri=$(jq ".file.uri" file_info1.json)
echo file1_uri=$file1_uri

# Prepare the second image (inline)
IMAGE2_PATH="path/to/image2.png"
MIME2_TYPE=$(file -b --mime-type "${IMAGE2_PATH}")

if [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then
  B64FLAGS="--input"
else
  B64FLAGS="-w0"
fi
IMAGE2_BASE64=$(base64 $B64FLAGS $IMAGE2_PATH)

# Now generate content using both images
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent" \
    -H "x-goog-api-key: $GEMINI_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{\
        "parts":[\
          {"text": "What is different between these two images?"},\
          {"file_data":{"mime_type": "'"${MIME1_TYPE}"'", "file_uri": '$file1_uri'}},\
          {\
            "inline_data": {\
              "mime_type":"'"${MIME2_TYPE}"'",\
              "data": "'"$IMAGE2_BASE64"'"\
            }\
          }\
        ]\
      }]
    }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json

```

## Object detection

From Gemini 2.0 onwards, models are further trained to detect objects in an image and get their bounding box coordinates. The coordinates, relative to image dimensions, scale to [0, 1000]. You need to descale these coordinates based on your original image size.

Python
```
from google import genai
from google.genai import types
from PIL import Image
import json

client = genai.Client()
prompt = "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000."

image = Image.open("/path/to/image.png")

config = types.GenerateContentConfig(
  response_mime_type="application/json"
  )

response = client.models.generate_content(model="gemini-2.5-flash",
                                          contents=[image, prompt],
                                          config=config
                                          )

width, height = image.size
bounding_boxes = json.loads(response.text)

converted_bounding_boxes = []
for bounding_box in bounding_boxes:
    abs_y1 = int(bounding_box["box_2d"][0]/1000 * height)
    abs_x1 = int(bounding_box["box_2d"][1]/1000 * width)
    abs_y2 = int(bounding_box["box_2d"][2]/1000 * height)
    abs_x2 = int(bounding_box["box_2d"][3]/1000 * width)
    converted_bounding_boxes.append([abs_x1, abs_y1, abs_x2, abs_y2])

print("Image size: ", width, height)
print("Bounding boxes:", converted_bounding_boxes)

```

For more examples, check following notebooks in the Gemini Cookbook:

- 2D spatial understanding notebook
- Experimental 3D pointing notebook

## Segmentation

Starting with Gemini 2.5, models not only detect items but also segment them and provide their contour masks.

The model predicts a JSON list, where each item represents a segmentation mask. Each item has a bounding box ("`box_2d`") in the format `[y0, x0, y1, x1]` with normalized coordinates between 0 and 1000, a label ("`label`") that identifies the object, and finally the segmentation mask inside the bounding box, as base64 encoded png that is a probability map with values between 0 and 255. The mask needs to be resized to match the bounding box dimensions, then binarized at your confidence threshold (127 for the midpoint).

Python
````
from google import genai
from google.genai import types
from PIL import Image, ImageDraw
import io
import base64
import json
import numpy as np
import os

client = genai.Client()

def parse_json(json_output: str):
  # Parsing out the markdown fencing
  lines = json_output.splitlines()
  for i, line in enumerate(lines):
    if line == "```json":
      json_output = "\n".join(lines[i+1:])  # Remove everything before "```json"
      output = json_output.split("```")[0]  # Remove everything after the closing "```"
      break  # Exit the loop once "```json" is found
  return json_output

def extract_segmentation_masks(image_path: str, output_dir: str = "segmentation_outputs"):
  # Load and resize image
  im = Image.open(image_path)
  im.thumbnail([1024, 1024], Image.Resampling.LANCZOS)

  prompt = """
  Give the segmentation masks for the wooden and glass items.
  Output a JSON list of segmentation masks where each entry contains the 2D
  bounding box in the key "box_2d", the segmentation mask in key "mask", and
  the text label in the key "label". Use descriptive labels.
  """

  config = types.GenerateContentConfig(
    thinking_config=types.ThinkingConfig(thinking_budget=0) # set thinking_budget to 0 for better results in object detection
  )

  response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=[prompt, im], # Pillow images can be directly passed as inputs (which will be converted by the SDK)
    config=config
  )

  # Parse JSON response
  items = json.loads(parse_json(response.text))

  # Create output directory
  os.makedirs(output_dir, exist_ok=True)

  # Process each mask
  for i, item in enumerate(items):
      # Get bounding box coordinates
      box = item["box_2d"]
      y0 = int(box[0] / 1000 * im.size[1])
      x0 = int(box[1] / 1000 * im.size[0])
      y1 = int(box[2] / 1000 * im.size[1])
      x1 = int(box[3] / 1000 * im.size[0])

      # Skip invalid boxes
      if y0 >= y1 or x0 >= x1:
          continue

      # Process mask
      png_str = item["mask"]
      if not png_str.startswith("data:image/png;base64,"):
          continue

      # Remove prefix
      png_str = png_str.removeprefix("data:image/png;base64,")
      mask_data = base64.b64decode(png_str)
      mask = Image.open(io.BytesIO(mask_data))

      # Resize mask to match bounding box
      mask = mask.resize((x1 - x0, y1 - y0), Image.Resampling.BILINEAR)

      # Convert mask to numpy array for processing
      mask_array = np.array(mask)

      # Create overlay for this mask
      overlay = Image.new('RGBA', im.size, (0, 0, 0, 0))
      overlay_draw = ImageDraw.Draw(overlay)

      # Create overlay for the mask
      color = (255, 255, 255, 200)
      for y in range(y0, y1):
          for x in range(x0, x1):
              if mask_array[y - y0, x - x0] > 128:  # Threshold for mask
                  overlay_draw.point((x, y), fill=color)

      # Save individual mask and its overlay
      mask_filename = f"{item['label']}_{i}_mask.png"
      overlay_filename = f"{item['label']}_{i}_overlay.png"

      mask.save(os.path.join(output_dir, mask_filename))

      # Create and save overlay
      composite = Image.alpha_composite(im.convert('RGBA'), overlay)
      composite.save(os.path.join(output_dir, overlay_filename))
      print(f"Saved mask and overlay for {item['label']} to {output_dir}")

# Example usage
if __name__ == "__main__":
  extract_segmentation_masks("path/to/image.png")

````

https://ai.google.dev/static/gemini-api/docs/images/segmentation.jpg  
An example segmentation output with objects and segmentation masks

## Supported image formats

Gemini supports the following image format MIME types:

- PNG - `image/png`
- JPEG - `image/jpeg`
- WEBP - `image/webp`
- HEIC - `image/heic`
- HEIF - `image/heif`

## Capabilities

All Gemini model versions are multimodal and can be utilized in a wide range of image processing and computer vision tasks including but not limited to image captioning, visual question and answering, image classification, object detection and segmentation.

Gemini can reduce the need to use specialized ML models depending on your quality and performance requirements.

Some later model versions are specifically trained improve accuracy of specialized tasks in addition to generic capabilities:

- **Gemini 2.0 models** are further trained to support enhanced object detection.
- **Gemini 2.5 models** are further trained to support enhanced segmentation in addition to object detection.

## Limitations and key technical information

### File limit

Gemini 2.5 Pro/Flash, 2.0 Flash, 1.5 Pro, and 1.5 Flash support a maximum of 3,600 image files per request.

### Token calculation

- **Gemini 1.5 Flash and Gemini 1.5 Pro**: 258 tokens if both dimensions <= 384 pixels. Larger images are tiled (min tile 256px, max 768px, resized to 768x768), with each tile costing 258 tokens.
- **Gemini 2.0 Flash and Gemini 2.5 Flash/Pro**: 258 tokens if both dimensions <= 384 pixels. Larger images are tiled into 768x768 pixel tiles, each costing 258 tokens.

A rough formula for calculating the number of tiles is as follows:

- Calculate the crop unit size which is roughly: floor(min(width, height) / 1.5).
- Divide each dimension by the crop unit size and multiply together to get the number of tiles.

For example, for an image of dimensions 960x540 would have a crop unit size of 360. Divide each dimension by 360 and the number of tile is 3 * 2 = 6.

## Tips and best practices

- Verify that images are correctly rotated.
- Use clear, non-blurry images.
- When using a single image with text, place the text prompt _after_ the image part in the `contents` array.

## What's next

This guide shows you how to upload image files and generate text outputs from image inputs. To learn more, see the following resources:

- Files API: Learn more about uploading and managing files for use with Gemini.
- System instructions: System instructions let you steer the behavior of the model based on your specific needs and use cases.
- File prompting strategies: The Gemini API supports prompting with text, image, audio, and video data, also known as multimodal prompting.
- Safety guidance: Sometimes generative AI models produce unexpected outputs, such as outputs that are inaccurate, biased, or offensive. Post-processing and human evaluation are essential to limit the risk of harm from such outputs.

</details>

<details>
<summary>image</summary>

# ColPali: EFFICIENT DOCUMENT RETRIEVAL WITH VISION LANGUAGE MODELS

Manuel Faysse∗1,3 Hugues Sibille∗1,4 Tony $\\mathbf { W \_ { u } } ^ { \* 1 }$ Bilel Omrani1

Gautier Viaud1 C´eline Hudelot3 Pierre Colombo2,3

1Illuin Technology 2Equall.ai 3CentraleSupe´lec, Paris-Saclay 4ETH Zu¨rich

[manuel.faysse@centralesupelec.fr](mailto:manuel.faysse@centralesupelec.fr)

# ABSTRACT

Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at [https://hf.co/vidore](https://hf.co/vidore).

# 1 INTRODUCTION

Document Retrieval consists of matching a user query to relevant documents in a given corpus. It is central to many widespread industrial applications, either as a standalone ranking system (search engines) or as part of more complex information extraction or Retrieval Augmented Generation (RAG) pipelines.

Over recent years, pretrained language models have enabled large improvements in text embedding models. In practical industrial settings, however, the primary performance bottleneck for efficient document retrieval stems not from embedding model performance but from the prior data ingestion pipeline. Indexing a standard PDF document involves several steps. First, PDF parsers or Optical Character Recognition (OCR) systems are used to extract words from the pages. Document layout detection models can then be run to segment paragraphs, titles, and other page objects such as tables, figures, and headers. A chunking strategy is then defined to group text passages with some semantical coherence, and modern retrieval setups may even integrate a captioning step to describe visually rich elements in a natural language form, more suitable for embedding models. In our experiments (Table 2), we typically find that optimizing the ingestion pipeline yields much better performance on visually rich document retrieval than optimizing the text embedding model.

Contribution 1: ViDoRe. In this work, we argue that document retrieval systems should not be evaluated solely on the capabilities of text embedding models (Bajaj et al., 2016; Thakur et al., 2021; Muennighoff et al., 2022), but should also consider the context and visual elements of the documents to be retrieved. To this end, we create and openly release ViDoRe, a comprehensive benchmark to evaluate systems on page-level document retrieval with a wide coverage of domains, visual elements, and languages. ViDoRe addresses practical document retrieval scenarios, where queries often necessitate both textual and visual understanding for accurate document matching. We highlight the shortcomings of current text-centric systems in these settings.1

![](https://arxiv.org/pdf/images/391f32efa12ee5d8c95be1f641c318cd9e821711f7d2f28610e9c54a25d13db6.jpg)

Figure 1: ColPali simplifies document retrieval w.r.t. standard retrieval methods while achieving stronger performances with better latencies. Latencies and results are detailed in section 5 and subsection B.4.

Contribution 2: ColPali. We propose a novel concept and model architecture based on Vision Language Models (VLMs) to efficiently index documents purely from their visual features, allowing for subsequent fast query matching with late interaction mechanisms (Khattab & Zaharia, 2020). Our method, ColPali, significantly outperforms all other retrieval systems on $V i D o R e$ while being fast and end-to-end trainable. These results demonstrate the potential and the many benefits of this novel Retrieval in Vision Space concept, which could significantly alter the way document retrieval is approached in the industry moving forward. We release all resources at [https://hf.co/vidore](https://hf.co/vidore).

# 2 PROBLEM FORMULATION & RELATED WORK

Problem Setting. In our setting, a retrieval system scores how relevant a document $d$ from corpus $\\mathcal { D }$ is with respect to a query $q$ . Computing the similarity score $s ( q , d ) \\in \\mathbb { R }$ for each of the $\| \\mathcal { D } \|$ documents in the corpus creates a ranking we can use to extract the most relevant documents. In this work, we focus on page-level retrieval: given a query, is the correct document page retrieved by the system? For coherence with existing literature, we further use the term document to refer to individual pages, i.e. the atomic retrieved elements in our setting. As we focus on practical industrial retrieval applications (RAG, search engines) with potentially large corpora sizes, latency constraints are imposed on scoring systems. Most current retrieval systems can be decomposed into (1) an offline indexation phase in which a document index is built and (2) an online querying phase in which a query is matched to documents from the index and where low latency is vital to the user experience.

Under these industrial constraints, we identify three main properties an efficient document retrieval systems should exhibit: (R1) strong retrieval performance, as measured by standard retrieval metrics; $( R 2 )$ fast online querying, measured through average latencies; (R3) high throughput corpus indexation, ie. the number of pages that can be embedded in a given timeframe.

2.1 TEXTUAL RETRIEVAL METHODS

# Document Retrieval in Text Space.

Statistical methods based on word frequency like TF-IDF (Sparck Jones, 1972) and BM25 (Robertson et al., 1994) are still widely used due to their simplicity and efficiency. More recently, neural embedding models based on fine-tuned large language models display state-of-the-art performance on a variety of text embedding tasks and top the retrieval leaderboards (Muennighoff et al., 2022).

Neural Retrievers. In bi-encoder models (Reimers & Gurevych, 2019; Karpukhin et al., 2020; Wang et al., 2022), documents are independently mapped offline to a dense vector space. Queries are embedded online and matched to documents through a fast cosine distance computation. A slower, but slightly more performant alternative, cross-encoder systems (Wang et al., 2020; Cohere, 2024) concatenate query and document as a single input sequence and iteratively attribute matching scores to each possible combination. This enables full attention computation between query and document terms but comes at the cost of computational efficiency, as $\| \\mathcal D \|$ encoding passes must be done online.

Multi-Vector retrieval via late interaction. In the late interaction paradigm introduced by ColBERT (Khattab & Zaharia, 2020), an embedding is pre-computed and indexed per document token. At runtime, similarity can be computed with individual query token embeddings. The idea is to benefit from the rich interaction between individual query and document terms while taking advantage of the offline computation and fast query matching enabled by bi-encoders. See section E for more details.

Retrieval Evaluation. Although benchmarks and leaderboards have been developed to evaluate text embedding models (Thakur et al., 2021; Muennighoff et al., 2022), much of the performance improvements in industrial use cases of embedding models stem from the prior data ingestion pipeline. While documents often rely on visual elements to more efficiently convey information to human readers, text-only systems barely tap into these visual cues. Other work has also independently studied table or chart retrieval systems through repurposed Question Answering datasets (Zhang et al., 2019; Nowak et al., 2024) but only assessing specialized methods for each task.

To our knowledge, no benchmark evaluates document retrieval systems in practical settings; in an end-to-end manner, across several document types and topics, and by evaluating the use of both textual and visual document features.

# 2.2 INTEGRATING VISUAL FEATURES

Contrastive Vision Language Models. Mapping latent representations of textual content to corresponding representations of visual content has been done by aligning disjoint visual and text encoders through contrastive losses (Radford et al., 2021; Zhai et al., 2023). While some OCR capabilities exist in these models, the visual component is often not optimized for text understanding.

The Fine-grained Interactive Language-Image Pre-training (Yao et al., 2021) framework extends the late interaction mechanism to cross-modal Vision Language Models, relying on max similarity operations between text tokens and image patches.

Visually Rich Document Understanding. To go beyond text, some document-focused models jointly encode text tokens alongside visual or document layout features (Appalaraju et al., 2021; Kim et al., 2021; Huang et al., 2022; Tang et al., 2022). Large Language transformer Models (LLMs) with strong reasoning capabilities have recently been combined with Vision Transformers (ViTs) (Dosovitskiy et al., 2020) to create VLMs (Alayrac et al., 2022; Liu et al., 2023; Bai et al., 2023; Lauren¸con et al., 2024b) where image patch vectors from contrastively trained ViT models (Zhai et al., 2023) are fed as input embeddings to the LLM and concatenated with the text-token embeddings.

PaliGemma. The PaliGemma-3B model (Beyer et al., 2024) extends concepts from Pali3 (Chen et al., 2023), and projects SigLIP-So400m/14 (Alabdulmohsin et al., 2023) patch embeddings into Gemma-2B’s text vector space (Gemma Team et al., 2024). Along with its reasonable size w.r.t. other performant VLMs, an interesting property of PaliGemma’s text model is that it is fine-tuned with full-block attention on the prefix (instruction text and image tokens). See Appendix E for more details.

VLMs display enhanced capabilities in Visual Question Answering, captioning, and document understanding (Yue et al., 2023), but are not optimized for retrieval tasks.

# 3 THE ViDoRe BENCHMARK

Existing benchmarks for contrastive vision-language models primarily evaluate retrieval for natural images (Lin et al., 2014; Borchmann et al., 2021; Thapliyal et al., 2022). On the other hand, textual retrieval benchmarks (Muennighoff et al., 2022) are evaluated at at textual passage level and are not tailored for document retrieval tasks. We fill the gap with ViDoRe, a comprehensive benchmark for document retrieval using visual features.

# 3.1 BENCHMARK DESIGN

ViDoRe is designed to comprehensively evaluate retrieval systems on their capacity to match queries to relevant documents at the page level. This benchmark encompasses multiple orthogonal subtasks, with focuses on various modalities - text, figures, infographics, tables; thematic domains - medical, business, scientific, administrative; or languages - English, French. Tasks also span varying levels of complexity, in order to capture signals from both weaker and stronger systems. As many systems require large amounts of time to index pages (captioning-based approaches can take dozens of seconds per page for instance), we limit the number of candidate documents for each retrieval task in order to evaluate even complex systems in a reasonable timeframe without sacrificing quality. For trainable retrieval systems, we provide a reference training set that can be used to facilitate comparisons.

Table 1: ViDoRe comprehensively evaluates multimodal retrieval methods.

|     |     |     |     |     |
| --- | --- | --- | --- | --- |
| Dataset | Language | \# Queries | \# Documents | Description |
| Academic Tasks |
| DocVQA | English | 500 | 500 | Scanned documents from UCSF Industry |
| InfoVQA | English | 500 | 500 | Infographics scrapped from the web |
| TAT-DQA | English | 1600 | 1600 | High-quality financial reports |
| arXiVQA | English | 500 | 500 | Scientific Figures from arXiv |
| TabFQuAD | French | 210 | 210 | Tables scrapped from the web |
| Practical Tasks |
| Energy | English | 100 | 1000 | Documents about energy |
| Government | English | 100 | 1000 | Administrative documents |
| Healthcare | English | 100 | 1000 | Medical documents |
| AI | English | 100 | 1000 | Scientific documents related to AI |
| Shift Project | French | 100 | 1000 | Environmental reports |

Academic Tasks. We repurpose widely used visual question-answering benchmarks for retrieval tasks: for each page-question-answer triplet, we use the question as the query, and the associated page as the gold document (Table 1). These academic datasets either focus on single specific modalities (Mathew et al., 2020; 2021; Li et al., 2024) or target more varied visually rich documents (Zhu et al., 2022). Moreover, we consider TabFQuAD, a human-labeled dataset on tables extracted from French industrial PDF documents released with this work. Details can be found in subsection A.1.

Practical tasks. We construct topic-specific retrieval benchmarks spanning multiple domains to go beyond repurposed QA datasets and evaluate retrieval in more realistic industrial situations (e.g. RAG). To achieve this, we collect publicly accessible PDF documents and generate queries pertaining to document pages using Claude-3 Sonnet, a high-quality proprietary vision-language model (Anthropic, 2024). In total, we collect 1,000 document pages per topic, which we associate with 100 queries extensively filtered for quality and relevance by human annotators. The corpus topics are intentionally specific to maximize syntactic proximity between documents, creating more challenging retrieval tasks and covering an array of orthogonal domains (Table 1). 2

Evaluation Metrics. We evaluate performance on our benchmark (Requirement R1) using standard metrics from the retrieval literature (nDCG, Recall@K, MRR). We report $\\mathrm { n D C G } @ 5$ values as the main performance metric in this work and release the complete sets of results along with the models3.

To validate compliance with practical industrial requirements (section 2), we also consider query latencies (R2) and indexing throughputs (R3).

# 3.2 ASSESSING CURRENT SYSTEMS

Unstructured. We evaluate retrieval systems representative of those found in standard industrial RAG pipelines. As is common practice, we rely on the Unstructured4 off-the-shelf tool in the highest resolution settings to construct high-quality text chunks from PDF documents. Unstructured orchestrates the document parsing pipeline, relying on deep learning vision models to detect titles and document layouts (Ge et al., 2021), OCR engines (Smith, 2007) to extract text in non-native PDFs, specialized methods or models to detect and reconstruct tables, and implements a chunking strategy (by-title) that leverages the detected document structure to preserve section boundaries when concatenating texts. As is common practice, in our simplest Unstructured configuration (text-only), only textual elements are kept and figures, images, and tables are considered noisy information and are filtered out.

Unstructured $+ \\textbf { X }$ . While Unstructured is a strong baseline by itself, we further augment Unstructured’s output by integrating the visual elements. In $( + \ O C R )$ , tables, charts, and images are run through an OCR engine, processed by Unstructured, and chunked independently. In ( $+$ Captioning), we set up a fully-fledged captioning strategy (Zhao et al., 2023), in which we feed visual elements to a strong proprietary Vision Language Model (Claude-3 Sonnet (Anthropic, 2024)) to obtain highly detailed textual descriptions of the elements. Both strategies aim to integrate visual elements in the retrieval pipeline but incur significant latency and resource costs (subsection 5.2).

Embedding Model. To embed textual chunks, we evaluate Okapi BM25, the de facto standard sparse statistical retrieval method, and the dense encoder of BGE-M3 (Chen et al., 2024), a multilingual neural method with SOTA performance in its size category. Chunks are embedded and scored independently, and page-level scores are obtained by max-pooling over the page’s chunk scores.5

Contrastive VLMs. We also evaluate the strongest available vision-language embedding models; Jina CLIP (Koukounas et al., 2024), Nomic Embed Vision (Nomic, 2024), and SigLIP-So400m/14 (Alabdulmohsin et al., 2023).

Results. From a performance perspective, best results are obtained by combining the Unstructured parser with visual information, either from captioning strategies or by running OCR on the visual elements (Table 2). Little difference is seen between BM25 and BGE-M3 embeddings highlighting the visual information bottleneck. Contrastive VLMs lag behind. Beyond retrieval performance (R1), the indexing latencies (R2) reported in Figure 2 illustrate that PDF parsing pipelines can be very lengthy, especially when incorporating OCR or captioning strategies. Querying latencies at runtime (R3) are very good for all evaluated systems $\\leq 2 2 \\mathrm { m s }$ on a NVIDIA L4) due to fast query encoding and cosine similarity matching.

![](https://arxiv.org/pdf/images/9b42e697ca163d7e29300ce3893bfbb565058abb5496f1a4433ddbc0420c9cb1.jpg)

Figure 2: Offline document indexing with ColPali is much simpler and faster compared to standard retrieval methods. The PDF Parser results are obtained following the Unstructured settings with BGE-M3 detailed in subsection 3.2. All indexing speeds are averaged per-page latencies. More details in subsection B.4

# 4 LATE INTERACTION BASED VISION RETRIEVAL

# 4.1 ARCHITECTURE

Vision-Language Models. Encouraged by their strong document understanding capabilities, we propose adapting recent VLMs for retrieval. The key concept is to leverage the alignment between output embeddings of text and image tokens acquired during multi-modal fine-tuning. To this extent, we introduce ColPali, a Paligemma-3B extension that is capable of generating ColBERT-style multivector representations of text and images (Figure 1). PaliGemma-3B is a strong candidate due to its small size, the many released checkpoints fine-tuned for different image resolutions and tasks, and the promising performances on various document understanding benchmarks. We add a projection layer to map each of the language model’s output token embeddings (whether from text or image tokens) to a vector space of reduced dimension $D = 1 2 8$ as used in the ColBERT paper (Khattab & Zaharia, 2020) to keep lightweight bag-of-embedding representations.

Late Interaction. Given query $q$ and document $d$ , we denote as $\\mathbf { E \_ { q } } \\in \\mathbb { R } ^ { N \_ { q } \\times D }$ and $\\mathbf { E \_ { d } } \\in \\mathbb { R } ^ { N \_ { d } \\times D }$ their respective multi-vector representation in the common embedding space $\\mathbb { R } ^ { D }$ , where $N \_ { q }$ and $N \_ { d }$ are respectively the number of vectors in the query and in the document page embeddings. The late interaction operator, $\\operatorname { L I } \\left( q , d \\right)$ , is the sum over all query vectors $\\mathbf { E \_ { q } } ^ { ( j ) }$ , of its maximum dot product $\\langle \\cdot \| \\cdot \\rangle$ with each of the $N \_ { d }$ document embedding vectors $\\mathbf { E \_ { d \\left( 1 : N \_ { d } \\right) } }$ .

$$
\\mathbf { L } \\left( q , d \\right) = \\sum \_ { i \\in \\left\[ \\left\| 1 , N \_ { q } \\right\| \\right\] } \\operatorname\* { m a x } \_ { j \\in \\left\[ \\left\| 1 , N \_ { d } \\right\| \\right\] } \\langle \\mathbf { E \_ { q } } ^ { ( i ) } \| \\mathbf { E \_ { d } } ^ { ( j ) } \\rangle
$$

Contrastive Loss. The Late Interaction operation is fully differentiable, enabling backpropagation. Let a batch ${ q \_ { k } , d \_ { k } } \_ { k \\in \[ \| 1 , b \| \] }$ composed of $b$ query-page pairs, where for all $k \\in \[ \| \\bar { 1 , \\upsilon } \| \]$ , the document page $d \_ { k }$ is the document corresponding to query $q \_ { k }$ . Following Khattab & Zaharia (2020), we define our in-batch contrastive loss $\\mathcal { L }$ as the softmaxed cross-entropy of the positive scores $s \_ { k } ^ { + } = \\mathrm { L I } \\left( q \_ { k } , d \_ { k } \\right)$ w.r.t. to the maximal in-batch negative scores $s \_ { k } ^ { - } = \\operatorname\* { m a x } \_ { l , l \\neq k } \\quad \\mathrm { L I } ( q \_ { k } , d \_ { l } ) ^ { 6 }$ :

$$
\\mathcal { L } = - \\frac { 1 } { b } \\sum \_ { k = 1 } ^ { b } \\log \\left\[ \\frac { \\exp \\left( s \_ { k } ^ { + } \\right) } { \\exp \\left( s \_ { k } ^ { + } \\right) + \\exp \\left( s \_ { k } ^ { - } \\right) } \\right\] = \\frac { 1 } { b } \\sum \_ { k = 1 } ^ { b } \\log \\left( 1 + \\exp \\left( s \_ { k } ^ { - } - s \_ { k } ^ { + } \\right) \\right)
$$

# 4.2 MODEL TRAINING

Dataset. Our training dataset of 118,695 query-page pairs is comprised of train sets of openly available academic datasets $( 6 3 % )$ and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions $( 3 7 % )$ . Dataset split details are given in subsection A.3. Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages7. We explicitly verify no multi-page PDF document is used both ViDoRe and in the train set to prevent evaluation contamination. A validation set is created with $2 %$ of the samples to tune hyperparameters. We openly release the training dataset8 for reproducibility and to encourage further research.

Parameters. All models are trained for 1 epoch on the train set. Unless specified otherwise, we train models in bfloat16 format, use low-rank adapters (LoRA, Hu et al. (2021)) with $\\alpha = 3 2$ and $r = 3 2$ on the transformer layers from the language model, as well as the final randomly initialized projection layer, and use a paged adamw 8bit optimizer. We train on an 8 GPU setup with data parallelism, a learning rate of $5 e - 5$ with linear decay with $2 . 5 %$ warmup steps, and a batch size of 32.

Query Augmentation. As in Khattab & Zaharia (2020), we append 5  tokens to the query tokens to serve as a soft, differentiable query expansion or re-weighting mechanism.

# 5 RESULTS

Table 2: Comprehensive evaluation of baseline models and our proposed method on ViDoRe. Results are presented using $\\mathrm { n D C G } @ 5$ metrics, and illustrate the impact of different components. Text-only metrics are not computed for benchmarks with only visual elements.

|     |     |     |     |     |     |     |     |     |     |     |     |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ArxivQ DocQ |  | InfoQ | TabF | TATQ | Shift | AI | Energy | Gov. | Health. | Avg. |
| Unstructured text-only |  |  |  |  |  |  |  |  |  |  |  |
| \- BM25 |  | 34.1 |  |  | 44.0 | 59.6 | 90.4 | 78.3 | 78.8 | 82.6 |  |
| \- BGE-M3 |  | 28.4↓5.7 |  |  | 36.1↓7.9 | 68.5†8.9 | 88.4↓2.0 | 76.8↓1.5 | 77.7↓1.1 | 84.6↑2.0 |  |
| Unstructured +oCR |  |  |  |  |  |  |  |  |  |  |  |
| \- BM25 | 31.6 | 36.8 | 62.9 | 46.5 | 62.7 | 64.3 | 92.8 | 85.9 | 83.9 | 87.2 | 65.5 |
| \- BGE-M3 | 31.4↓0.2 | 25.7↓11.1 60.1↓2.8 |  | 70.8↑24.3 50.5↓12.2 73.2↑8.9 |  |  | 90.2↓2.6 | 83.6↓2.3 | 84.9↑1.0 | 91.1↑3.9 | 66.1↑0.6 |
| Unstructured + Captioning |  |  |  |  |  |  |  |  |  |  |  |
| \- BM25 | 40.1 | 38.4 | 70.0 | 35.4 | 61.5 | 60.9 | 88.0 | 84.7 | 82.7 | 89.2 | 65.1 |
| \- BGE-M3 | 35.7↓4.4 | 32.9↓5.4 | 71.9†1.9 |  |  |  | 69.1†↑33.7 43.8↓17.7 73.1↑12.2 88.810.8 | 83.3↓1.4 | 80.4↓2.3 | 91.3↑2.1 | 67.0个1.9 |
| ContrastiveVLMs |  |  |  |  |  |  |  |  |  |  |  |
| Jina-CLIP | 25.4 | 11.9 | 35.5 | 20.2 | 3.3 | 3.8 | 15.2 | 19.7 | 21.4 | 20.8 | 17.7 |
| Nomic-vision | 17.1 | 10.7 | 30.1 | 16.3 | 2.7 | 1.1 | 12.9 | 10.9 | 11.4 | 15.7 | 12.9 |
| SigLIP (Vanilla) | 43.2 | 30.3 | 64.1 | 58.1 | 26.2 | 18.7 | 62.5 | 65.7 | 66.1 | 79.1 | 51.4 |
| Ours |  |  |  |  |  |  |  |  |  |  |  |
| SigLIP (vanilla) | 43.2 | 30.3 | 64.1 | 58.1 | 26.2 | 18.7 | 62.5 | 65.7 | 66.1 | 79.1 | 51.4 |
| BiSigLIP (+fine-tuning) |  | 58.5↑15.3 32.9↑2.6 | 70.5↑6.4 | 62.7↑4.6 | 30.5↑4.3 | 26.5↑7.8 | 74.3†11.8 73.7†8.0 |  | 74.2†8.1 | 82.3↑3.2 | 58.6↑7.2 |
| BiPali (+LLM) |  | 56.5↓-2.0 30.0↓-2.9 67.4↓-3.1 |  | 76.9↑14.2 | 33.4↑2.9 |  |  | 43.7↑17.2 71.2↓-3.1 61.9↓-11.7 | 73.8↓-0.4 | 73.6↓-8.8 | 58.8↑0.2 |
| ColPali (+Late Iter.) |  | 79.1↑22.6 54.424.5 81.8个14.4 83.9†7.0 |  |  |  |  | 65.8†32.4 73.2↑29.5 96.2↑25.0 91.0个29.1 |  |  | 92.7†18.9 94.4个20.8 | 81.3†22.5 |

# 5.1 PERFORMANCE (R1)

We show performance is achieved iteratively through the combination of three factors; (1) a carefully crafted task-specific dataset, (2) pairing a pretrained LLM to a vision model to better leverage text semantics from the image, and (3) using multi-vector embeddings rather than a single vector representation to better capture the vast amount of visual information present in a document.

Fine-tuning a Vision Model on a document retrieval oriented dataset: BiSigLIP. SigLIP9 is a strong vision-language bi-encoder producing single vector embeddings, and pretrained on billions of image-text pairs from the English split of WebLI (Chen et al., 2023). Further fine-tuning the textual component of this model on our document-oriented dataset (BiSigLIP) yields clear improvements across the board, particularly on figure retrieval (ArxivQA) and table retrieval tasks (TabFQuAD).

Feeding image patches to a LLM: BiPali. In the PaliGemma model architecture, SigLIP-generated patch embeddings are fed to a text language model and we can obtain LLM contextualized output patch embeddings.10 This technique aligns the image token representations with the text token embeddings in the LLM’s embeddings space, and augments the vision model embeddings with the language model’s text understanding capabilities. We average pool these representations to obtain a single dense vector, effectively creating a PaliGemma bi-encoder model (BiPali). After fine-tuning on the training dataset, we obtain a model that performs slightly worse in English than the tuned BiSigLIP variant.11 However, we see notable improvements in French tasks, indicating that BiPali’s LLM (Gemma 2B) helps multilingual text understanding. This is particularly notable as our training dataset does not contain non-English samples.

Leveraging Multi-Vector Embeddings through Late Interaction: ColPali. One benefit of inputting image patch embeddings through a language model is that they are natively mapped to a latent space similar to the textual input (query). This enables leveraging the ColBERT strategy to construct one embedding per image patch token, and at inference compute all interactions between text tokens and image patches, resulting in a step-change improvement in performance compared to BiPali. Results in Table 2 show that our ColPali model also largely outperforms the strong baselines based on Unstructured and captioning, as well as all evaluated text-image embedding models. The difference is particularly stark on the more visually complex benchmark tasks, such as InfographicVQA, ArxivQA, and TabFQuAD, respectively representing infographics, figures, and tables. However, text-centric documents are also better retrieved by the ColPali models across all evaluated domains and languages, making our approach the overall best-performing document-retrieval model.

Negative Results. For extensiveness, we also train ColSigLIP, a late interaction variant of the BiSigLIP model but obtain abysmal performances. We attribute this to the large gaps w.r.t. SigLIP’s pre-training, in which only a pooled latent representation is used in the contrastive loss, which does not optimize the representations of individual patch and token embeddings. Similarly, we train a $\\mathrm { B i S i g L I P } \_ { P a l i G e m m a }$ variant, in which we retrieve the image representations from the SigLIP model that has been further updated by PaliGemma fine-tuning, and use the text representations from PaliGemma’s text model. After fine-tuning on our dataset, performance is severely inferior to $\\mathrm { S i g L I P } \_ { V a n i l l a }$ which simply encodes with SigLIP’s original text and vision components. This indicates a logical misalignment between SigLIP embeddings, and Gemma embeddings after PaliGemma training. We detail these results in subsection C.1.

# 5.2 LATENCIES & MEMORY FOOTPRINT

Online Querying. (R2) Logically, querying latencies differ between ColPali and a BGE-M3 embedding model. For BGE, encoding takes about $2 2 ~ \\mathrm { m s }$ for 15 tokens, while encoding a query with ColPali’s language model takes about $\\mathrm { 3 0 ~ m s ^ { 1 2 } }$ . For smaller corpus sizes, computing the late interaction operation induces marginally small overheads $\\approx 1$ ms per 1000 pages in the corpus), and the cosine similarity computation between bi-encoder vectors is even faster. Optimized late interaction engines (Santhanam et al., 2022; Lee et al., 2023) enable to easily scale corpus sizes to millions of documents with reduced latency degradations.

Offline Indexing. (R3) Standard retrieval methods using bi-encoders represent each chunk as a single vector embedding, which is easy to store and fast to compute. However, processing a PDF to get the different chunks is the most time-consuming part (layout detection, OCR, chunking), and using captioning to handle multimodal data will only exacerbate this already lengthy process. On the other hand, ColPali directly encodes pages from their image representation. Although the model is larger than standard retrieval encoders, skipping the preprocessing allows large speedups at indexing13 (Figure 2). As pages are embedded end-to-end in single forward pass, the VRAM usage depends exclusively on the sequence length (number of patches per image) which is fixed as well, enabling efficient batching strategies to fully leverage hardware acceleration. ColPali also benefits from most LLM efficiency improvements introduced in the ecosystem such as Flash Attention (Dao, 2023).

Storage Footprint. Our method requires storing a vector per image patch, along with 6 extra text tokens “Describe the image” concatenated to image patches. We project each PaliGemma vector to a lower dimensional space $D = 1 2 8 ,$ ) to maximize efficiency, leading to a memory footprint of 257.5 KB per page (subsection B.3). Importantly, the memory footprint of the naive ColBERT indexing strategy can be drastically improved through compression and clustering mechanisms (Santhanam et al., 2022; Clavie´ et al., 2024).

Token pooling. Token pooling (Clavi´e et al., 2024) is a CRUDE-compliant method (document addition/deletion-friendly) that aims amountto reduce the amount of multi-vector embeddings. For ColPali, many image patches share redundant information, e.g. white background patches. By pooling these patches together, we can reduce the amount of embeddings while retaining most information. Retrieval performance with hierarchical mean token pooling on image embeddings is shown in Figure 3 (left). With a pool factor of 3, the total number of vectors is reduced by $6 6 . 7 %$ while $9 7 . 8 %$ of the original performance is maintained. We note that the Shift dataset—composed of the most text-dense documents—is a clear outlier, showcasing more information dense documents contain less redundant patches and may be prone to worse performance degradation with such pooling techniques.

![](https://arxiv.org/pdf/images/8293b990ecdbbef3709cfbff1889d6bbaef5395f6aee03741bd76b28b993209f.jpg)

Figure 3: (Left: Token Pooling) Relative performance degradation when reducing the number of stored embeddings per document. (Right: Interpretability) For each term in a user query, ColPali identifies the most relevant document image patches (highlighted zones) and computes a query-topage matching score.

# 5.3 INTERPRETABILITY

By superimposing the late interaction heatmap on top of the original image, we can visualize the most salient image patches with respect to each term of the query, yielding interpretable insights into model focus zones. As epitomized in Figure 3 (right), we observe ColPali exhibits strong OCR capabilities as both the words “hourly” and “hours” present a high similarity score with the query token < hour>. We also note particular focus on other non-trivial image features such as the x-axis representing hours being salient. Other visualization examples are shown in Appendix D.

# 6 ABLATION STUDY

We run various ablations to better understand the mechanisms at play. By default, result deltas reported below refer to $\\mathrm { n D C G } @ 5$ values averaged over all ViDoRe tasks. Detailed results in C.2.

Tradeoffs between model size and the number of image patches. We train a variant of PaliGemma with half the number of image patches (512). While we observe a clear performance degradation with respects to to the 1024-patch ColPali model $( - 2 4 . 8 \\mathrm { n D C G } @ 5 )$ , memory usage is much lower. As an alternative to PaliGemma, we train Idefics2-8B (Lauren¸con et al., 2024b), a VLM with a similar architecture and based on a Mistral-7B (Jiang et al., 2023) language backbone and a SigLIP vision encoder paired with a perceiver resampler. The most notable differences with PaliGemma lie in the size of the language model (2B and 7B resp.) and the number of image patches (between 512 and 2048 for PaliGemma, and 64 post-resampling for Idefic $\\cdot 2 ^ { 1 4 }$ ). Our results suggest better language models enable more efficient representations of image embeddings - ColIdefics2 with 64 patches largely outperforms out ColPali with 512 patches $( + 2 0 . 1 \ \\mathrm { n D C G } @ 5 )$ . However ColIdefics2 (64) remains less accurate than ColPali (1024) $( - 4 . 7 \\mathrm { n D C G } @ 5 )$ while being about twice as slow in terms of training and inference latency. These results suggest there are tradeoffs between performance (R1), latencies during online querying (R2) and offline indexation phases (R3), and index memory size.

Unfreezing the vision component. We train a ColPali variant by also backpropagating through and updating the vision encoder and the projection layer. This leads to a slight performance degradation $( \\bar { - } 0 . 7 \\mathrm { n D C G } @ 5 )$ . These conclusions may change with larger scales of training data.

Impact of “query augmentation” tokens. In ColBERT, special tokens are concatenated to the input query to serve as soft query augmentation buffers. Training without these tokens, we observe no significant performance difference in the English benchmarks. However, performance on the French tasks seems to improve $\\left( + 9 . 8 \\mathrm { n D C G } @ \\mathrm { G } \\right.$ on Shift, $+ 6 . 3 \\mathrm { n D C G } @ 5 $ on TabFQuAD, Table 7).

Impact of the Pairwise CE loss. Training with an in-batch negative contrastive loss, instead of the pairwise CE loss that only considers the hardest negative sample, leads to a slight performance degradation $( - 1 . 6 \\mathrm { n D C G } @ 5 ) ,$ ) on the aggregated benchmark.

Adapting models to new tasks. Contrary to more complex multi-step retrieval pipelines, ColPali can be trained end-to-end, directly optimizing the downstream retrieval task which greatly facilitates fine-tuning to boost performance on specialized domains, multilingual retrieval, or specific visual elements the model struggles with. To demonstrate, we add 1552 samples representing French tables and associated queries to the training set. This represents the only French data in the training set, with all other examples being kept unchanged. We see clear $\\mathrm { n D C G } @ 5$ improvements $\\left( + 2 . 6 \\right)$ and even starker Recall $@ 1$ gains $( + 5 )$ on the TabFQuAD benchmark, with no performance degradation on the rest of the benchmark tasks $( + 0 . 4 \\mathrm { n D C G } @ 5 $ overall).

Better VLMs lead to better visual retrievers. As improved VLMs are released, it is interesting to observe if improved performances on generative tasks translate once these models are adapted for image retrieval tasks through ColPali training strategies. We train the recently released Qwen2-VL 2B (Wang et al., 2024b), a SOTA 2 billion parameter generative VLM, with the same data and training strategy, obtaining ColQwen2-VL. To approximately match ColPali’s memory requirements, we limit the number of image patches to 768, slightly less than ColPali’s 1024 patches. We observe clear performance improvements of $+ 5 . 3 \\mathrm { n D C G } @ 5 $ values over ColPali showcasing clear performance correlations between generative benchmarks performance and retrieving metrics.

Out-of-domain generalization. Some of the datasets in the ViDoRe benchmark have train sets, which we have integrated within the ColPali train set (eg. academic tasks). This is standard in embedding models (Wang et al., 2024a; Lee et al., 2024), and while ColPali also exhibits strong performance on tasks in which this is not the case (French data is never seen by the model during training for instance), it remains interesting to evaluate model performance when training is done on a fully disjoint data distribution. We train a ColPali variant solely using the recent DocMatix dataset (Lauren¸con et al., 2024a), a large scale, synthetically annotated visual document question answering dataset, which we subsample to obtain a comparably-sized train set. Results on ViDoRe show the performance drop is minor $( - 2 . 2 \\operatorname { n D C G } \ @ 5 )$ , still outperforming the closest baseline method by over 12 points. These results showcase ColPali generalizes well outside of its training distribution, and demonstrate that our results are not unreasonably boosted with respect to baselines (BGE-M3) that cannot be fine-tuned on the same data15

# 7 CONCLUSIONS

In this work, we introduced the Visual Document Retrieval Benchmark (ViDoRe), which evaluates document retrieval systems in realistic settings involving visually complex documents. We demonstrated that current retrieval pipelines and contrastive vision-language models struggle to efficiently exploit visual information embedded in documents, leading to suboptimal performance. To address this, we presented ColPali, a novel retrieval method that leverages Vision-Language Models to create high-quality, multi-vector embeddings purely from visual document features. ColPali largely outperforms the best existing document retrieval methods while enabling faster corpus indexing times and maintaining low querying latencies, thus circumventing many pain points of modern document retrieval applications. We hope to drive industrial adoption, and to encourage future work by publicly releasing the ViDoRe benchmark, the data, the codebase, and all models and baselines from our work.

Future Work. Beyond performance improvements that could be obtained through better data, backbone models or training strategies, our vision at term is to combine visual retrieval systems and visually grounded query answering to create end-to-end RAG systems that purely function from image features. This idea is supported by concurrent work (Ma et al., 2024) showcasing the strong promises of VLMs for visual QA, and may eventually become a new industrial standard for document processing. In this line of work, reliability is key, and confidence estimation techniques for Information Retrieval methods could become central to implement abstention mechanisms (GisserotBoukhlef et al., 2024), and are particularly interesting given the information rich multi-vector scoring mechanisms of late interaction systems. Expanding benchmarking efforts to cover more languages, modalities, and tasks is also a crucial future research direction (Jiang et al., 2024).

# REPRODUCIBILITY STATEMENT

For transparency, reproducibility and to foster future work, we release our training data, model checkpoints (adapters), entire codebase, and complete evaluation benchmark under MIT licenses as detailed in the main paper. We also host a public ViDoRe leaderboard16 to foster concurrent work in the space. The supplementary material further details training configurations for our models (also specified in HuggingFace model repositories), and dives into the process we used to generate synthetic data, how latency computations are performed, as well as provides further detailed evaluation results.

# ACKNOWLEDGEMENTS

This work is partially supported by Illuin Technology, and by a grant from ANRT France. This work was performed using HPC resources from the CINES ADASTRA through Grant 2024-AD011015443 and from IDRIS with grant 2024-AD011015724R1. We extend our warm thanks to Jonathan Dong, Caio Corro, Victor Pellegrain and Ender Konukoglu for their valuable feedback on the paper.

# REFERENCES

Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design. 2023. doi: 10.48550/ARXIV.2305. 13035. URL [https://arxiv.org/abs/2305.13035](https://arxiv.org/abs/2305.13035). Publisher: arXiv Version Number: 5.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a Visual Language Model for Few-Shot Learning. 2022. doi: 10.48550/ARXIV.2204.14198. URL [https://arxiv](https://arxiv/). org/abs/2204.14198. Publisher: arXiv Version Number: 2.

Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024. URL [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/) Model Card Claude 3.pdf.

Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. DocFormer: End-to-End Transformer for Document Understanding, 2021. URL [https://arxiv.org/abs/](https://arxiv.org/abs/) 2106.11539. Version Number: 2.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. 2023. doi: 10.48550/ARXIV.2308.12966. URL [https://arxiv.org/abs/2308.12966](https://arxiv.org/abs/2308.12966). Publisher: arXiv Version Number: 3.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset, 2016. URL [https://arxiv.org/abs/1611.09268](https://arxiv.org/abs/1611.09268). Version Number: 3.

Lucas Beyer, Andreas Steiner, Andre´ Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bosˇnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2407.07726.

Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Commun. ACM, 13(7): 422–426, July 1970. ISSN 0001-0782. doi: 10.1145/362686.362692. URL [https://doi.org/](https://doi.org/) 10.1145/362686.362692. Place: New York, NY, USA Publisher: Association for Computing Machinery.

Łukasz Borchmann, Michał Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michał Turski, Karolina Szyndler, and Filip Grali´nski. DUE: End-to-End Document Understanding Benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL [https://openreview.net/forum?id=rNs2FvJGDK](https://openreview.net/forum?id=rNs2FvJGDK).

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation, 2024. URL [https://arxiv.org/abs/2402.03216](https://arxiv.org/abs/2402.03216). Version Number: 3.

Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. PaLI-3 Vision Language Models: Smaller, Faster, Stronger, 2023. URL [https://arxiv](https://arxiv/). org/abs/2310.09199. Version Number: 2.

Benjamin Clavi´e, Antoine Chaffin, and Griffin Adams. Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2409.14683. Version Number: 1.

Cohere. Introducing Rerank 3: A New Foundation Model for Efficient Enterprise Search & Retrieval, April 2024. URL [https://cohere.com/blog/rerank-3](https://cohere.com/blog/rerank-3).

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL [https://arxiv.org/abs/2307.08691](https://arxiv.org/abs/2307.08691).

Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers Need Registers. 2023. doi: 10.48550/ARXIV.2309.16588. URL [https://arxiv.org/abs/2309](https://arxiv.org/abs/2309). 16588\. Publisher: \[object Object\] Version Number: 2.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018. URL [https://arxiv.org/abs/](https://arxiv.org/abs/) 1810.04805. Version Number: 2.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. 2020. doi: 10.48550/ARXIV.2010.11929. URL [https://arxiv.org/abs/2010.11929](https://arxiv.org/abs/2010.11929). Publisher: arXiv Version Number: 2.

Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: Exceeding YOLO Series in 2021, 2021. URL [https://arxiv.org/abs/2107.08430](https://arxiv.org/abs/2107.08430). Version Number: 2.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivie\`re, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Le´onard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Ame´lie He´liou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Cl´ement Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Bu hatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian o des venskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Au banowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, J Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Mi ixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael , Olivier Bachem, Oscar Chang, Oscar Wahltinez, ahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan ullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girg ndya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Cl´ement Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open Models Based on Gemini Research and Technology, 2024. URL [https://arxiv.org/abs/2403.08295](https://arxiv.org/abs/2403.08295). Version Number: 4.

Hippolyte Gisserot-Boukhlef, Manuel Faysse, Emmanuel Malherbe, C´eline Hudelot, and Pierre Colombo. Towards trustworthy reranking: A simple yet effective abstention mechanism, 2024. URL [https://arxiv.org/abs/2402.12997](https://arxiv.org/abs/2402.12997).

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. 2021. doi: 10.48550/ ARXIV.2106.09685. URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685). Publisher: arXiv Version Number: 2.

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. 2022. doi: 10.48550/ARXIV.2204.08387. URL [https://arxiv.org/abs/2204.08387](https://arxiv.org/abs/2204.08387). Publisher: arXiv Version Number: 3.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Le´lio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth´ee Lacroix, and William El Sayed. Mistral 7B. 2023. doi: 10.48550/ARXIV.2310. 06825. URL [https://arxiv.org/abs/2310.06825](https://arxiv.org/abs/2310.06825). Publisher: arXiv Version Number: 1.

Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks, 2024. URL https: //arxiv.org/abs/2410.05160.

Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering, 2020. URL [https://arxiv.org/abs/2004.04906](https://arxiv.org/abs/2004.04906). Version Number: 3.

Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. 2020. doi: 10.48550/ARXIV.2004.12832. URL [https://arxiv.org/abs/2004.12832](https://arxiv.org/abs/2004.12832).

Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. OCR-free Document Understanding Transformer, 2021. URL [https://arxiv.org/abs/2111.15664](https://arxiv.org/abs/2111.15664). Version Number: 5.

Andreas Koukounas, Georgios Mastrapas, Michael Gu¨nther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Mart´ınez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan Wang, and Han Xiao. Jina CLIP: Your CLIP Model Is Also Your Text Retriever, 2024. URL [https://arxiv.org/abs/2405.20204](https://arxiv.org/abs/2405.20204). Version Number: 1.

Hugo Lauren¸con, Andre´s Marafioti, Victor Sanh, and Le´o Tronchon. Building and better understanding vision-language models: insights and future directions., 2024a.

Hugo Laurenc¸on, Le´o Tronchon, Matthieu Cord, and Victor Sanh. What matters when building visionlanguage models?, May 2024b. URL [http://arxiv.org/abs/2405.02246](http://arxiv.org/abs/2405.02246). arXiv:2405.02246 \[cs\].

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models, 2024. URL [https://arxiv.org/abs/2405.17428](https://arxiv.org/abs/2405.17428).

Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, and Vincent Y. Zhao. Rethinking the Role of Token Retrieval in Multi-Vector Retrieval, 2023. URL [https://arxiv.org/abs/2304.01982](https://arxiv.org/abs/2304.01982). Version Number: 3.

Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll´ar. Microsoft COCO: Common Objects in Context, 2014. URL [https://arxiv.org/abs/1405.0312](https://arxiv.org/abs/1405.0312). Version Number: 3.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. 2023. doi: 10.48550/ARXIV.2304.08485. URL [https://arxiv.org/abs/2304.08485](https://arxiv.org/abs/2304.08485). Publisher: arXiv Version Number: 1.

Lucas Beyer\*, Andreas Steiner\*, Andr´e Susano Pinto\*, Alexander Kolesnikov\*, Xiao Wang\*, Xiaohua Zhai\*, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Jeremiah Harmsen, Daniel Keysers, Neil Houlsby, Xi Chen, Emanuele Bugliarello, Thomas Unterthiner, Keran Rong, Matthias Minderer, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Julian Eisenschlos, Manoj Kumar, Matko Bosˇnjak, Matthias Bauer, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Paul Voigtlaender, Pinelopi Papalampidi, Olivier Henaff, Skanda Koppula, Xi Xiong, Radu Soricut, Model release contributors and general support, Tris Warkentin, Kat Black, Luiz Gustavo Martins, Glenn Cameron, Raj Gundluru, Manvinder Singh, Meg Risdal, Nilay Chauhan, Nate Keating, Nesh Devanathan, Elisa Bandy, Joe Fernandez, Antonia Paterson, Jenny Brennan, Tom Eccles, Pankil Botadra, Ben Bariach, Lav Rai, Minwoo Park, Dustin Luong, Daniel Vlasic, Bo Wu, Wenming Ye, Divyashree Sreepathihalli, Kiranbir Sodhia, Alek Andreev, Armand Joulin, Surya Bhupatiraju, Minh Giang, Joelle Barral, and Zoubin Ghahramani. PaliGemma, 2024. URL [https://www.kaggle.com/m/23393](https://www.kaggle.com/m/23393).

Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations, 2024. URL [https://arxiv.org/abs/2407.01523](https://arxiv.org/abs/2407.01523).

Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A Dataset for VQA on Document Images. 2020. doi: 10.48550/ARXIV.2007.00398. URL [https://arxiv.org/abs/](https://arxiv.org/abs/) 2007.00398.

Minesh Mathew, Viraj Bagal, Rub\`en P´erez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. InfographicVQA, 2021. URL [https://arxiv.org/abs/2104.12756](https://arxiv.org/abs/2104.12756). Version Number: 2.

Niklas Muennighoff, Nouamane Tazi, Loı¨c Magne, and Nils Reimers. MTEB: Massive Text Embedding Benchmark, 2022. URL [https://arxiv.org/abs/2210.07316](https://arxiv.org/abs/2210.07316). Version Number: 3.

Nomic. Nomic Embed Vision: Expanding The Nomic Latent Space, June 2024. URL https: //blog.nomic.ai/posts/nomic-embed-vision.

Averi Nowak, Francesco Piccinno, and Yasemin Altun. Multimodal chart retrieval: A comparison of text, table and image based approaches. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5488–5505, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.307. URL [https://aclanthology.org/2024.naacl-long.307](https://aclanthology.org/2024.naacl-long.307).

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. 2021. doi: 10.48550/ ARXIV.2103.00020. URL [https://arxiv.org/abs/2103.00020](https://arxiv.org/abs/2103.00020). Publisher: arXiv Version Number: 1.

Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERTNetworks, 2019. URL [https://arxiv.org/abs/1908.10084](https://arxiv.org/abs/1908.10084). Version Number: 1.

Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Donna K. Harman (ed.), Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication, pp. 109–126. National Institute of Standards and Technology (NIST), 1994. URL [http://trec.nist.gov/pubs/trec3/papers/city.ps.gz](http://trec.nist.gov/pubs/trec3/papers/city.ps.gz).

Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. PLAID: An Efficient Engine for Late Interaction Retrieval, 2022. URL [https://arxiv.org/abs/2205.09707](https://arxiv.org/abs/2205.09707). Version Number: 1.

R. Smith. An Overview of the Tesseract OCR Engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007) Vol 2, pp. 629–633, Curitiba, Parana, Brazil, September 2007. IEEE. ISBN 978-0-7695-2822-9. doi: 10.1109/ICDAR.2007.4376991. URL [http://ieeexplore.ieee.org/document/4376991/](http://ieeexplore.ieee.org/document/4376991/). ISSN: 1520-5363.

Karen Sparck Jones. A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL. Journal of Documentation, 28(1):11–21, January 1972. ISSN 0022-0418. doi: 10.1108/eb026526. URL [https://www.emerald.com/insight/content/doi/](https://www.emerald.com/insight/content/doi/) 10.1108/eb026526/full/html.

Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying Vision, Text, and Layout for Universal Document Processing, 2022. URL [https://arxiv.org/abs/2212.02623](https://arxiv.org/abs/2212.02623). Version Number: 3.

Nandan Thakur, Nils Reimers, Andreas Ru¨ckle´, Abhishek Srivastava, and Iryna Gurevych. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, 2021. URL [https://arxiv.org/abs/2104.08663](https://arxiv.org/abs/2104.08663). Version Number: 4.

Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset, 2022. URL [https://arxiv.org/abs/2205.12522](https://arxiv.org/abs/2205.12522). Version Number: 2.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text Embeddings by Weakly-Supervised Contrastive Pre-training, 2022. URL [https://arxiv.org/abs/2212.03533](https://arxiv.org/abs/2212.03533). Version Number: 2.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models, 2024a. URL [https://arxiv.org/abs/2401](https://arxiv.org/abs/2401). 00368.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024b. URL [https://arxiv.org/abs/2409.12191](https://arxiv.org/abs/2409.12191).

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers, April 2020. URL [http://arxiv.org/abs/2002.10957](http://arxiv.org/abs/2002.10957). arXiv:2002.10957 \[cs\].

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained Interactive Language-Image Pre-Training, 2021. URL [https://arxiv.org/abs/2111.07783](https://arxiv.org/abs/2111.07783). Version Number: 1.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, 2023. URL [https://arxiv.org/abs/2311.16502](https://arxiv.org/abs/2311.16502). Version Number: 3.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre-Training. 2023. doi: 10.48550/ARXIV.2303.15343. URL [https://arxiv.org/abs/](https://arxiv.org/abs/) 2303.15343. Publisher: \[object Object\] Version Number: 4.

Li Zhang, Shuo Zhang, and Krisztian Balog. Table2vec: Neural word and entity embeddings for table population and retrieval. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’19. ACM, July 2019. doi: 10.1145/3331184.3331333. URL [http://dx.doi.org/10.1145/3331184.3331333](http://dx.doi.org/10.1145/3331184.3331333).

Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. Retrieving Multimodal Information for Augmented Generation: A Survey, 2023. URL [https://arxiv.org/abs/2303.10868](https://arxiv.org/abs/2303.10868). Version Number: 3.

Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. Towards Complex Document Understanding By Discrete Reasoning. 2022. doi: 10.48550/ARXIV.2207. 11871. URL [https://arxiv.org/abs/2207.11871](https://arxiv.org/abs/2207.11871). Publisher: arXiv Version Number: 3.

# A BENCHMARK DATASETS

A.1 ACADEMIC DATASETS

DocVQA (Mathew et al., 2020) includes collected images from the UCSF Industry Documents Library. Questions and answers were manually annotated.

InfoVQA (Mathew et al., 2021) includes infographics collected from the Internet using the search query “infographics”. Questions and answers were manually annotated.

TAT-DQA (Zhu et al., 2022) is a large-scale Document VQA dataset that was constructed from publicly available real-world financial reports. It focuses on rich tabular and textual content requiring numerical reasoning. Questions and answers were manually annotated by human experts in finance.

arXivQA (Li et al., 2024) is a VQA dataset based on figures extracted from arXiv publications. The questions were generated synthetically using GPT-4 Vision.

TabFQuAD (Table French Question Answering Dataset) is designed to evaluate TableQA models in realistic industry settings. We create additional queries to augment the existing human-annotated ones using the same method described in subsection A.2.

# A.2 PRACTICAL DATASETS

Methodology. Creating a relevant retrieval dataset close to real use cases is a major challenge as the dataset needs to be both sufficiently large for effective fine-tuning and sufficiently diverse to cover a broad range of modalities (full text, tables, charts, ...), domains (industry, healthcare, ...), and query-document interactions (extractive questions, open-ended questions, ...). Our approach to building this dataset involves several steps: (1) we use a web crawler to collect publicly available documents on various themes and sources, (2) we convert these PDFs into a series of images, one per page, and (3) we generate queries related to each image using a VLM.

Web-Crawler. We implemented a web crawler to efficiently collect large volumes of documents related to a given topic. The crawler is seeded with a user-defined query (e.g. “artificial intelligence”) and then uses GPT-3.5 Turbo to brainstorm related topics and subtopics. This query augmentation strategy aims at both broadening and deepening the search. GPT-3.5 Turbo is further used to generate diverse search queries from each subtopic. This query set is then consumed by a pool of parallel workers whose job is to fetch the associated most relevant documents. We use SerpAPI17 along with a filetype filter (PDF documents only) to programmatically scrape Google Search rankings. Each file is hashed and stored in a Bloom filter (Bloom, 1970) shared among workers to avoid duplicate documents in the final corpus. Unique scraped files are downloaded, and inserted into a SQLite database along with additional metadata.

Datamix. Using the web crawler, we collected approximately 100 documents for each of the following four seeds: “energy”, “government reports”, “healthcare industry”, and “artificial intelligence”. These seeds were meticulously hand-picked to align with real-use cases for retrieval models and visually rich pages. We also removed all documents containing any private information.

Query Generation. To increase the efficiency of our query generation scheme and to limit API calls, we generate at most 3 questions per image. From all the documents collected, we randomly sample 10,000 images per theme and call Claude-3 Sonnet with the following prompt:

You are an assistant specialized in Multimodal RAG tasks.

The task is the following: given an image from a pdf page, you will have to generate questions that can be asked by a user to retrieve information from a large documentary corpus.

The question should be relevant to the page, and should not be too specific or too general. The question should be about the subject of the page, and the answer needs to be found in the page.

Remember that the question is asked by a user to get some information from a

large documentary corpus that contains multimodal data. Generate a question

that could be asked by a user without knowing the existence and the content

of the corpus.

Generate as well the answer to the question, which should be found in the

page. And the format of the answer should be a list of words answering the

question.

Generate at most THREE pairs of questions and answers per page in a dictionary

with the following format, answer ONLY this dictionary NOTHING ELSE:

{ "questions": \[ { "question": "XXXXXX", "answer": \["YYYYYY"\] }, { "question": "XXXXXX", "answer": \["YYYYYY"\] }, { "question": "XXXXXX", "answer": \["YYYYYY"\] }, \]

}

where XXXXXX is the question and \['YYYYYY'\] is the corresponding list of answers

that could be as long as needed. Note: If there are no questions to ask about the page, return an empty list. Focus on making relevant questions concerning the page.

Here is the page:

Human Validation. We manually validate every single synthetically created query in ViDoRe to ensure quality, query relevance, and consistency with the benchmark objective of evaluating retrieval in practical industrial settings. During this step, we randomly assign document-pair queries to 4 volunteer annotators and instruct them to filter out queries that do not fit the above-listed criteria. We also instruct annotators to flag any documents they deem to contain PII information or content not suited for an academic benchmark. No flag was raised during the entirety of the process, validating our prior PDF collection strategy. 100 queries per topic are collected in this manner. Annotators are colleagues and collaborators of the authors who volunteered to help. Each annotator spent approximately 3 hours filtering the larger query set down to 100 high-quality queries per topic.

# A.3 TRAINING DATASET

The statistics of the train set are given in the following table. The creation of the train set follows the same methodology as in subsection A.2. We made sure that a PDF document cannot have pages in both the training set and the test set to prevent data leakage and that there are no duplicate documents in each split.

Table 3: Details on the different splits in the dataset used to train ColPali.

|     |     |     |     |
| --- | --- | --- | --- |
| Dataset Split | Split Size | Language | Domain |
| DocVQA | 39,463 | English | Scanned documents from UCSF Industry |
| InfoVQA | 10,074 | English | Infographics scrapped from the web |
| TATDQA | 13,251 | English | High-quality financial reports |
| arXivQA | 10,000 | English | Scientific Scientific Figures from arXiv |
| Scrapped PDFs | 45,940 | English | Varied PDFs from 3885 distinct URL domains |
| TOTAL | 118,695 | English-only | Mixed |

# B IMPLEMENTATION DETAILS

# B.1 CODEBASE

The codebase is written in PyTorch18 and leverages HuggingFace tooling for model implementations and trainers19.

# B.2 HYPERPARAMETERS

Hyperparameters are tuned on a validation split composed of $2 %$ of the training dataset. We find bi-encoder methods to be more sensible to learning rate variations than late interaction-based models and achieve the best performance for all models with a learning rate of $5 e \\mathrm { ~ - ~ } 5$ . We experiment with LoRA rank and $\\alpha$ values and do not notice particular improvements past $r = \\alpha = 3 2$ . Per-device batch sizes are kept small due to long sequence lengths that complicate scaling past $b = 4$ . We simulate larger batch sizes with multi-GPU training and train with a total batch size $b = 3 2$ with no accumulation, for 1 epoch on our training set.

# B.3 EMBEDDING SIZE

Minimizing storage footprint can be essential to industrial retrieval systems if databases contain millions of documents. With this criterion in view, we have compared the embedding sizes of the models in our study. As shown in Table 4, ColPali’s embedding size is an order of magnitude larger than BM25 and two orders of magnitude larger than BGE-M3. However, in practical scenarios, pooling multi-vector embeddings by centroid cluster, or quantizing embeddings to binary representations 20 can reduce storage costs by two orders of magnitude (Santhanam et al., 2022) with minimal performance hits, and make storage costs competitive with other systems.

|     |     |
| --- | --- |
| Model | Embeddingsize (KB) |
| BGE-M3 | 8.60 |
| BM25 (dense emb.) | 3.00 |
| BM25 (sparse emb.) | 1.56 ± 0.51 |
| ColPali (float16) | 257.5 |

Table 4: Comparison of the embedding sizes for the DocVQA test set from ViDoRe w.r.t. different retrieval models. The mean $\\pm$ std size is given for the sparse embeddings. In general multiple vectors (2-5) per page are used for BGE-M3 and BM25.

# B.4 LATENCY COMPUTATIONS

To ensure comparison fairness, the latencies of the different retrieval systems shown in Figure 2 are measured on the same g2-standard-8 GCP VM with a NVIDIA L4 GPU. Document pages are embedded using the highest settings of Unstructured with captioning (see subsection 3.2). SigLIP and ColPali are both loaded with bfloat16 parameter dtypes. The reported times in Table 5 are the average per-page latencies for each indexing operation on 1000 randomly chosen documents across all splits of the ViDoRe benchmark test set. A batch size of 8 was used for the BGE-M3 model used with Unstructured, and a batch size of 4 was used for SigLIP and ColPali.

|     |     |     |     |
| --- | --- | --- | --- |
| Indexing operation | Latency (s) |
| Unstructured | SigLIP | ColPali |
| Layout detection | 0.81 | NA | NA |
| OCR | 2.67 | NA | NA |
| Captioning | 3.71 | NA | NA |
| Page encoding | 0.03 | 0.12 | 0.39 |
| Total | 7.22 | 0.12 | 0.39 |

Table 5: Page-level latencies for document indexing using various retrieval systems. SigLIP and ColPali are much faster than Unstructured because they don’t require the layout detection, OCR, and captioning operations.

# B.5 CAPTIONING

Examples of captions generated for visually rich document chunks with Claude-3 Sonnet are shown in Figure 5 and Figure 4. The prompt used for generating the description is the following:

You are an assistant specialized in document analysis. Given a table or a figure, you have to provide a detailed summary of the content in maximum 3000 characters. Your summary should be qualitative and not quantitative. Here is the table/figure to analyze: {image}. Answer ONLY with the caption of the table/figure.

![](https://arxiv.org/pdf/images/b578972a7404cd92e3c49325f2b39546dbecd19e0ac452bc06ace2578cb6a1ba.jpg)

Figure 4: Example from the “Energy” test set.

Caption: The image depicts the hourly energy generation profile, illustrating the contributions of various energy sources over 24 hours. The data is presented as a stacked bar chart, with the $x$ -axis representing the hours of the day from 1 to 2, and the y-axis showing the average hourly generation in MW. The bars are segmented into different colors, each representing a distinct energy source: nuclear, bio, geothermal, solar, wind, hydro, natural gas, and other imports. The chart provides insights into the temporal variations in energy generation across different sources, highlighting the interplay between baseload and intermittent sources throughout the day.

![](https://arxiv.org/pdf/images/5e9e3c0f9ff527eae81290a8397593ca7163bf60bed81795640a8ebdc03d7a3c.jpg)

Figure 5: Example from the “Government Reports” test set.

Caption: The image shows a table titled “System of Record” which outlines the different types of documents or records maintained across various systems or departments within an organization related to project management and construction. The rows list documents like project plans, budgets, schedules, contracts, purchase orders, invoices, change requests, bid submissions, drawings, manuals, meeting minutes, and reports. The columns indicate the system or department responsible for maintaining each record, such as County Servers, Project View, OnBase, CGI Advantage Financial System, and Purchasing Department. The table uses ”W” and ”T” markers to denote which system or department serves as the primary source (writer) or storage location (trailer) for each type of document.

# C ADDITIONAL RESULTS

C.1 OTHER METRICS

|     |     |     |     |     |     |     |     |     |     |     |     |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ArxivQ DocQ |  | InfoQ | TabF | TATQ | Shift | AI | Energy | Gov. | Health. | Avg. |
| Unstructured text-only |  |  |  |  |  |  |  |  |  |  |  |
| BM25 |  | 26.6 |  |  | 34.6 | 45.0 | 86.0 | 70.0 | 68.0 | 74.0 |  |
| BGE-M3 |  | 22.8↓3.8 |  |  | 26.1↓8.5 | 51.0†6.0 | 81.0↓5.0 | 72.0↑2.0 | 67.0↓1.0 | 77.0↑3.0 |  |
| Unstructured + ocR |  |  |  |  |  |  |  |  |  |  |  |
| BM25 | 26.7 | 28.9 | 54.0 | 30.4 | 50.0 | 52.0 | 86.0 | 77.0 | 74.0 | 80.0 | 55.9 |
| BGE-M3 | 28.1†1.4 | 22.9↓6.0 | 53.8↓0.2 |  |  | 55.7†25.3 38.6↓11.4 56.0↑4.0 | 82.0↓4.0 | 79.0↑2.0 | 76.0↑2.0 | 83.0↑3.0 | 57.5↑1.6 |
| Unstructured + Captioning |  |  |  |  |  |  |  |  |  |  |  |
| BM25 | 35.5 | 30.2 | 61.5 | 24.3 | 49.0 | 47.0 | 79.0 | 76.0 | 75.0 | 81.0 | 55.9 |
| BGE-M3 | 29.3↓6.2 | 26.0↓4.2 | 62.1 t0.6 |  |  | 58.6†34.3 30.618.4 55.0个8.0 | 80.0†1.0 | 78.0+2.0 | 69.0↓6.0 | 83.0+2.0 | 57.2↑1.3 |
| Contrastive VLMs |  |  |  |  |  |  |  |  |  |  |  |
| Jina-CLIP | 19.4 | 7.3 | 26.7 | 12.5 | 1.6 | 2.0 | 11.0 | 13.0 | 15.0 | 17.0 | 12.6 |
| Nomic-vision | 10.4 | 6.7 | 22.1 | 9.6 | 1.6 | 0.0 | 9.0 | 9.0 | 7.0 | 13.0 | 8.8 |
| SigLIP (vanilla) | 34.2 | 21.3 | 51.8 | 46.1 | 17.9 | 13.0 | 50.0 | 51.0 | 47.0 | 65.0 | 39.7 |
| Ours |  |  |  |  |  |  |  |  |  |  |  |
| SigLIP (vanilla) | 34.2 | 21.3 | 51.8 | 46.1 | 17.9 | 13.0 | 50.0 | 51.0 | 47.0 | 65.0 | 39.7 |
| BiSigLIP (+fine-tuning) |  | 49.2↑15.0 23.8↑2.5 | 59.0↑7.2 | 52.1↑6.0 | 20.7†2.8 | 16.0↑3.0 |  | 62.0↑12.0 61.0↑10.0 55.0↑8.0 |  | 72.0†7.0 | 47.1↑7.4 |
| BiPali (+LLM) |  |  |  |  |  |  |  |  | 46.4↓-2.820.0↓-3.854.6↓-4.4 63.2↑1120.4-0.434.0↑18.059.0-3.0 45.0-16.057.02.0 | 56.0↓-16.0 | 45.6↓-1.5 |
| ColPali (+Late Inter.) |  | 72.4个26.045.6↑25.6 74.6↑20.0 75.4个12.1 |  |  |  |  |  | 53.1↑32.7 55.0个21.0 93.0↑34.0 85.0个40.0 | 85.0↑28.0 | 88.0↑32.0 | 72.7↑27.1 |

Table 6: Comprehensive evaluation of baseline models and our proposed method on ViDoRe. Results are presented using Recall $@$ 1 metrics. Text-only metrics are not computed for benchmarks with only visual elements.

C.2 MODEL VARIANTS

Table 7: Benchmark scores for the “negative results” and various ablations on ViDoRe; ColPali for reference. Results are presented using nDCG $\\textcircled { a } 5$ metrics. Text-only metrics are not computed for benchmarks with only visual elements.

|     |     |     |     |     |     |     |     |     |     |     |     |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ArxivQ DocQ InfoQ TabF TATQ |  |  |  |  | Shift AI |  | Energy |  | Gov. Health. | Avg. |
| ColSigLIP (PaliGemma) | 3.1 | 3.0 | 5.1 | 6.2 | 2.5 | 1.0 | 3.4 | 3.4 | 2.3 | 2.2 | 3.2 |
| BiSigLIP (PaliGemma) | 18.5 | 14.6 | 33.4 | 39.5 | 16.1 | 5.2 |  | 27.6 32.6 | 36.6 | 35.7 | 26.0 |
| ColSigLIP (Original) | 2.6 | 2.2 | 2.3 | 5.7 | 1.8 | 1.0 | 2.6 | 4.1 | 1.4 | 1.5 | 2.5 |
| ColPali (No Q.A. Tokens) | 80.4 | 53.2 | 82.4 | 77.4 | 65.7 | 63.4 |  | 97.0 89.9 | 93.6 | 92.4 | 79.6 |
| ColPali (Docmatix) | 71.3 | 48.0 | 80.0 | 83.9 | 59.1 | 73.8 |  | 95.7 93.8 | 92.5 | 93.1 | 79.1 |
| ColPali (224) | 71.0 | 37.4 | 62.3 | 65.7 | 28.6 | 20.4 |  | 65.7 66.8 | 73.9 | 73.0 | 56.5 |
| ColPali (Vision Trained) | 78.8 | 53.9 | 81.3 | 81.7 | 64.4 | 70.6 |  | 95.3 91.7 | 93.5 | 94.7 | 80.6 |
| ColPali (No Pairwise) | 79.0 | 53.0 | 82.1 | 85.3 | 63.2 | 66.2 |  | 94.9 88.9 | 92.7 | 92.1 | 79.7 |
| ColPali (+TabFQuAD training) | 77.6 | 54.7 | 82.6 | 86.5 | 65.4 | 73.9 |  | 94.8 92.4 | 94.2 | 94.8 | 81.7 |
| ColIdefics2 (64) | 73.6 | 48.0 | 82.4 | 81.6 | 63.0 | 57.2 |  | 95.5 86.9 | 86.6 | 91.2 | 76.6 |
| ColQwen2 (768) | 86.4 | 56.2 | 89.8 | 88.7 | 75.2 | 85.7 |  | 98.8 94.8 | 93.6 | 97.3 | 86.6 |
| ColPali (Reference: 448) | 79.1 | 54.4 | 81.8 | 83.9 | 65.8 | 73.2 |  | 96.2 91.0 | 92.7 | 94.4 | 81.3 |

![](https://arxiv.org/pdf/images/0217658dc526953c07436f5e23c0b69a3a6f330c58603950c07fdda9a8a2836d.jpg)

Query: "Quelle partie de la production pétroliere du Kazakhstan provient de champs en mer ?"

Figure 6: Similarity of the image patches w.r.t. the underlined token in the user query. This example is from the Shift test set.

# D MORE SIMILARITY MAPS

In Figure 6, ColPali assigns a high similarity to all patches with the word “Kazakhstan” when given the token < Kazakhstan>. Moreover, our model seems to exhibit world knowledge capabilities as the patch around the word ”Kashagan”—an offshore oil field in Kazakhstan—also shows a high similarity score.

It is also interesting to highlight that both this similarity map and the one displayed in Figure 3 (right) showcase a few white patches with high similarity scores. This behavior might first seem surprising as the white patches should not carry a meaningful signal from the original images. We believe the vectors associated with these patches share a similar role with the ViT registers (Darcet et al., 2023), i.e. these patches were repurposed for internal computations and stored the global information from the whole image.

# E MODEL GLOSSARY

SIGLIP

SigLIP (Sigmoid Loss for Language Image Pre-Training) builds upon CLIP (Contrastive LanguageImage Pretraining)—a foundational model that aligns images and text by maximizing the similarity between correct image-text pairs while minimizing it for incorrect ones, leveraging a contrastive loss (Zhai et al., 2023). Unlike CLIP (Radford et al., 2021), which applies the softmax function to the logits, SigLIP uses the sigmoid activation function. This innovation eliminates the need for a global view of all pairwise similarities between images and texts within a batch, enabling more flexible batch size scaling (up to 1M items per batch, with an effective optimal batch size of 32k). This approach allows SigLIP to achieve state-of-the-art performance in zero-shot image classification tasks.

# PALIGEMMA

PaliGemma is a 3B-parameter vision-language model. It integrates the SigLIP vision encoder with a Gemma-2B language decoder, connected via a multimodal linear projection layer (Lucas Beyer\* et al.,

2024). The model processes images by segmenting them into a fixed number of Vision Transformer (Dosovitskiy et al., 2020) tokens, which are prepended to an optional text prompt.

A distinguishing feature of PaliGemma is its operation as a Prefix-Language Model (Prefix-LM). This design ensures full attention between image tokens and the user-provided input (prefix) while generating outputs auto-regressively (suffix). This architecture allows image tokens to access the task-specific query during processing, facilitating more effective task-dependent reasoning.

PaliGemma was trained in four stages: unimodal pretraining with existing components, extended multimodal pretraining, short high-resolution pretraining, and task-specific fine-tuning.

# COLBERT

ColBERT (Contextualized Late Interaction over BERT) is a retrieval model designed to balance speed and effectiveness in information retrieval tasks (Khattab & Zaharia, 2020). Traditional retrieval models are typically categorized based on their type of interaction: either processing queries and documents independently for efficiency (bi-encoders) or jointly to capture rich contextual relationships (crossencoders). ColBERT combines the advantages of both approaches through a novel late interaction mechanism.

Queries and documents are encoded separately using BERT, enabling offline pre-computation of document representations for scalability. Instead of pooling embeddings into a single vector, ColBERT retains token-level embeddings and employs a MaxSim operator to compute fine-grained similarity scores. For each query token, the model determines the maximum similarity with document tokens, summing these scores to compute relevance.

This architecture preserves the contextual richness of deep language models while significantly improving computational efficiency. By delaying the interaction step, ColBERT supports vector similarity indexing, facilitating end-to-end retrieval from large collections without prohibitive costs. Empirical evaluations on passage search datasets demonstrate that ColBERT achieves competitive effectiveness compared to existing BERT-based models (Devlin et al., 2018), while executing queries orders of magnitude faster and with drastically reduced computational requirements.

# F EXAMPLES FROM THE ViDoRe BENCHMARK

# Energy

Query: What types of accounts or products allow investors to defer paying taxes?

Query: What is the estimated to-Query: What is the projected

tal savings for a PV system in peak electricity demand in Cali

Durham under the net metering fornia for the year 2030?

(flat rate) billing option over the stem’s useful life of 25 years?

![](https://arxiv.org/pdf/images/1f9bb209c65efe578ba50868a9c38ce8cb6f246966ae148e4620d11a0e607212.jpg)

![](https://arxiv.org/pdf/images/3960e4108709e4059f26cfe162eb3d936c3067e514cb08ca74f06162a8133d67.jpg)

![](https://arxiv.org/pdf/images/e42b7fab5707392bc051a93970281ac113009ce37c322c63a68c4abebcd55ef7.jpg)

# Artificial Intelligence

Query: What are some common outcome areas targeted by TAII for different age groups?

Query: What did the robot moni- Query: What is the key approach

tor to determine when to activate used in the PDP architecture?

or deactivate the blower motor

and blinker?

![](https://arxiv.org/pdf/images/9016549ff01b9fbab36065594cd1ee58efc6b7d6f0e48d20e3de677225676ae7.jpg)

![](https://arxiv.org/pdf/images/bd0530bcbb330f02782de1f5c1297af8b783f13733b31cc36cd41350f36dfa27.jpg)

# Healthcare Industry

Query: What is the chemical formula for the ferroelectric material Lead Zirconium Titanate (PZT)?

Query: What government entities are involved in public financing for healthcare in the US?

Query: What does the AVPU scale stand for in assessing the level of consciousness of a seriously ill child?

![](https://arxiv.org/pdf/images/1444da993a6ee3b1f59c12ed057e76987d6d04ec703b309af1f57ca2a86b3e76.jpg)

![](https://arxiv.org/pdf/images/38372b0dbe77a01b8b69787f56cb4099efbb3dcb110561bbe169fd3229ace1cb.jpg)

![](https://arxiv.org/pdf/images/91878a732ed3319672984664297e6dc86f53c974a9af005b68884f221dcec18c.jpg)

# Government Reports

# Query: What are some mandates for the EPA under the Pollution Prevention Act?

Query: What is the strategy of KPMG Hazem Hassan?

![](https://arxiv.org/pdf/images/adbe49c71bb05e93ba1043d1c60f64fa6debd89e45f60c536c414803465d5c0f.jpg)

Query: What is the trust signal score for the consumer industry best-in-class archetype?

![](https://arxiv.org/pdf/images/346df2ccef49c90eab7668c90dbca02d5f207f8c91880b4b9767e25c61c384aa.jpg)

![](https://arxiv.org/pdf/images/d05ca9155dc858c869d56a125f9ebfcf2b158cb761d7e7acddc5f4f01f51b550.jpg)

![](https://arxiv.org/pdf/images/73685f10f6769ee7ce32ed6752357e4d12e341abf7bd8c352f5decc74797d729.jpg)

# Shift

Query: Selon le graphique, Query: Quelle partie de la proquelle est la capacit´e d’import et duction pe´trolie\`re du Kazakhstan la consommation r´eelle de carbu- provient de champs en mer ?

rants SAF (biocarburants durables pour l’aviation) pr´evues en 2050 ?

Query: Quels sont les pays ayant la plus grande part des d´ecouvertes cumul´ees de p´etrole brut en 2020 (en milliers de barils, hors d´ecouvertes cumule´es) ?

![](https://arxiv.org/pdf/images/58ff928d61e2c6526f0ab107700f7119a4d1e1524fab70beb3e26f175712cb60.jpg)

![](https://arxiv.org/pdf/images/32a1b7a13ef9773e94478799bd2ca046af67fa42f07bd30746fe0e962ef4e9dd.jpg)

![](https://arxiv.org/pdf/images/2289f86f67ae91ed3003029d330ee9e6b6ec529bfa838ea82ced746417f7b1f1.jpg)

</details>

<details>
<summary>multi-modal-ml-with-openai-s-clip-pinecone</summary>

# Multi-modal ML with OpenAI's CLIP

* * *

Language models (LMs) can not rely on language alone. That is the idea behind the “Experience Grounds Language” paper, that proposes a framework to measure LMs' current and future progress. A key idea is that, beyond a certain threshold LMs need other forms of data, such as visual input \[1\] \[2\].

https://www.pinecone.io/_next/image/?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fvr8gru94%2Fproduction%2F25e7f2f54b543af8c34c143448a4b0c55f77c6b5-2360x854.png&w=3840&q=75

World Scopes (WS), as datasets become larger in scope and span multiple modalities, the capabilities of models trained with them increase.

The next step beyond well-known language models; BERT, GPT-3, and T5 is _”World Scope 3”_. In World Scope 3, we move from large text-only datasets to large multi-modal datasets. That is, datasets containing information from multiple forms of media, like _both_ images and text.

The world, both digital and real, is multi-modal. We perceive the world as an orchestra of language, imagery, video, smell, touch, and more. This chaotic ensemble produces an inner state, our “model” of the outside world.

AI must move in the same direction. Even specialist models that focus on language or vision must, at some point, have input from the other modalities. How can a model fully understand the concept of the word “person” without _seeing_ a person?

OpenAI **C** ontrastive **L** earning **I** n **P** retraining (CLIP) is a world scope three model. It can comprehend concepts in both text and image and even connect concepts between the two modalities. In this chapter we will learn about multi-modality, how CLIP works, and how to use CLIP for different use cases like encoding, classification, and object detection.

* * *

## Multi-modality

The multi-modal nature of CLIP is powered by two encoder models trained to “speak the same language”. Text inputs are passed to a text encoder, and image inputs to an image encoder \[3\]. These models then create a _vector representation_ of the respective input.

Both models “speak the same language” by encoding similar concepts in text and images into similar vectors. That means that the text “two dogs running across a frosty field” would output a vector similar to an _image_ of two dogs running across a frosty field.

https://www.pinecone.io/_next/image/?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fvr8gru94%2Fproduction%2Fa54a2f1fa0aeac03748c09df0fdfbb42aadc96b7-2430x1278.png&w=3840&q=75

Similar text and images will be encoded into a similar vector space. Dissimilar text and images do not share a similar vector space.

We can think of the language these models speak as the vector space in which they encode vectors. These two models can express nuanced information about text and images through this vector space. However, this “vector language” is far too abstract for us to directly understand.

Rather than directly reading this “language”, we can train other simple neural networks to understand it and make predictions that we can understand. Or we use vector search to identify similar concepts and patterns across text and image domains.

Let’s take a look at an example of CLIP in action.

### Text-to-Image Search

Entering a prompt in the search bar above allows us to search through images based on their _content_ rather than any attached textual metadata. We call this **C** ontent **B** ased **I** mage **R** etrieval (CBIR).

With CBIR, we can search for specific phrases such as “two dogs running across a frosty field”. We can even drop the word “dogs” and replace it with everyday slang for dogs like “good boy” or “mans best friend”, and we return the same images showing dogs running across fields.

CLIP can accurately understand language. It understands that _in the context_ of running across a field, we are likely referring to dogs and do not literally mean good children or someone’s “human” best friend.

Amusingly, the dataset contains no images of the food hot dogs (other than one). So, suppose we search for “hot dogs”. In that case, we first get an image containing a hot dog (and a dog), a dog looking toasty in a warm room, another dog looking warm with wooly clothing, and another dog posing for the camera. All of these portray a hot dog in one sense or another.

* * *

_After being processed by CLIP’s text or image encoder, we are left with vectors. That means we can search across_ **_any_** _modality with_ **_any_** _modality; we can search in either direction. We can also stick to a single modality, like text-to-text or image-to-image._

* * *

Now that we’ve seen what CLIP can do, let’s take a look at _how_ it can do this.

## CLIP

CLIP actually consists of two models trained in parallel. A 12-layer text transformer for building text embeddings and a ResNet or vision transformer (ViT) for building image embeddings \[3\].

https://www.pinecone.io/_next/image/?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fvr8gru94%2Fproduction%2F539716ea1571e459908c1fdc5a898fea239d8243-2803x1672.png&w=3840&q=75

Architecture diagram of CLIP with the text encoder and ViT or ResNet as the image encoder.

The text encoder and image encoder (ResNet _or_ ViT) output single vector embeddings for each text/image record fed into the encoders. All vectors are 512 dimensional and can be represented in the same vector space, meaning similar images and text produce vectors that appear near each other.

### Contrastive Pretraining

Across both [**N** atural](https://www.pinecone.io/learn/series/nlp/) [**L** anguage](https://www.pinecone.io/learn/series/nlp/) [**P** rocessing (NLP)](https://www.pinecone.io/learn/series/nlp/) and computer vision (CV), large pretrained models dominate the SotA. The idea is that by giving a big model a lot of data, they can learn general patterns from the dataset.

For language models, that may be the general rules and patterns in the English language. For vision models, that may be the characteristics of different scenes or objects.

The problem with multi-modality is that these models are trained separately and, by default, have no understanding of one another. CLIP solves this thanks to image-text _contrastive pretraining_. With CLIP, text and image encoders are trained while considering the other modality and context. Meaning that the text and image encoders share an “indirect understanding” of patterns in both modalities; language and vision.

Contrastive pretraining works by taking a _(text, image)_ pair – where the text describes the image – and learning to encode the pairs as closely as possible in vector space.

For this to work well, we also need negative pairs to provide a contrastive comparison. We need positive pairs that should output similar vectors and negative pairs that should output dissimilar vectors.

This is the general idea behind contrastive learning, which can be found in the training functions of many models, particularly those that produce embedding vectors.

The negative pairs can be extracted directly from positive pairs. If we have positive pairs (T1,I1)(T\_1,I\_1)(T1​,I1​) and (T2,I2)(T\_2,I\_2)(T2​,I2​), we simply swap the components, giving us the negative pairs (T1,I2)(T\_1,I\_2)(T1​,I2​) and (T2,I1)(T\_2,I\_1)(T2​,I1​).

With this, we can apply a loss function that maximizes the similarity between (T1,I1)(T\_1,I\_1)(T1​,I1​) and (T2,I2)(T\_2,I\_2)(T2​,I2​), and minimizes the similarity between (T1,I2)(T\_1,I\_2)(T1​,I2​) and (T2,I1)(T\_2,I\_1)(T2​,I1​). Altogether, this looks like this:

https://www.pinecone.io/_next/image/?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fvr8gru94%2Fproduction%2Fd6868e6dae721512fed8f1287fc9ffe6b6a2cddd-2332x1342.png&w=3840&q=75

Contrastive pretraining with CLIP.

In this image, we can see a single pretraining step on a single batch. The loss function assumes pairs in the diagonal should have a maximized dot product score, and all other pairs should have a minimized dot product score. Both text and image encoder models are optimized for this.

A fundamental assumption is that there are no other positive pairs within a single batch. For example, we assume that “two dogs running across a frosty field” is only relevant to the image it is paired with. We assume there are no other texts or images with similar meanings.

This assumption is possible because the datasets used for pretraining are diverse and large enough that the likelihood of two similar pairs appearing in a single batch is negligible. Therefore, rare enough to have a little-to-no negative impact on pretraining performance.

## Using CLIP

We have a good idea of what CLIP can be used for and how it is trained. With that, how can we get started with it?

OpenAI released a few implementations of CLIP via the Hugging Face library; this is the fastest way to get started. First, we need to install the necessary libraries.

`pip install transformers torch datasets`

Before we can do anything with CLIP, we need some text and images. The `jamescalam/image-text-demo` dataset contains a small number of image-text pairs we can use in our examples.

```python
from datasets import load_dataset

data = load_dataset(
    "jamescalam/image-text-demo",
    split="train"
)
```

https://www.pinecone.io/_next/image/?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fvr8gru94%2Fproduction%2Fa40f673ed52e07f497c7a39b032c27b33ce9f565-1128x761.png&w=3840&q=75

Example of text-image pair found in the dataset. Text is stored in the "text" feature and images in the "image" feature.

With these sample records ready, we can move on to initializing CLIP and an image/text preprocessor like so:

```python
from transformers import CLIPProcessor, CLIPModel
import torch

model_id = "openai/clip-vit-base-patch32"

processor = CLIPProcessor.from_pretrained(model_id)
model = CLIPModel.from_pretrained(model_id)

# move model to device if possible
device = 'cuda' if torch.cuda.is_available() else 'cpu'

model.to(device)
```

The `model` is CLIP itself. Note that we use the ViT image encoder (the model is `clip-vit`). Text and image data cannot be fed directly into CLIP. The text must be preprocessed to create “tokens IDs”, and images must be resized and normalized. The `processor` handles both of these functions.

### Encoding Text

We will start with encoding text using the CLIP text transformer. Before feeding text into CLIP, it must be preprocessed and converted into token IDs. Let’s take a batch of sentences from the `unsplash` data and encode them.

In\[5\]:

```python
text = data['text']  # 21

tokens = processor(
    text=text,
    padding=True,
    images=None,
    return_tensors='pt'
).to(device)
tokens.keys()
```

Out\[5\]:

```
dict_keys(['input_ids', 'attention_mask'])
```

This returns the typical text transformer inputs of `input_ids` and `attention_mask`.

The `input_ids` are token ID values where each token ID is an integer value ID that maps to a specific word or sub-word. For example the phrase _“multi-modality”_ may be split into tokens _\[“multi”, “-”, “modal”, “ity”\]_, which are then mapped to IDs _\[1021, 110, 2427, 425\]_.

A text transformer maps these token IDs to semantic vector embeddings that the model learned during pretraining.

The `attention_mask` is a tensor of 1s and 0s used by the model’s internal mechanisms to “pay attention” to real token IDs and ignore padding tokens.

* * *

_Padding tokens are a special type of token used by text transformers to create input sequences of a fixed length from sentences of varying length. They are appended to the end of shorter sentences, so “hello world” may become “hello world \[PAD\] \[PAD\] \[PAD\]”._

* * *

We then use CLIP to encode all of these text descriptions with `get_text_features` like so:

```python
text_emb = model.get_text_features(
    **tokens
)
```

One important thing to note here is that these embeddings are _not_ normalized. If we plan on using a similarity metric like the dot product, we must normalize the embeddings:

In\[9\]:

```python
print(text_emb.shape)
print(text_emb.min(), text_emb.max())
```

Out\[9\]:

```
torch.Size([21, 512])
tensor(-1.1893, grad_fn=<MinBackward1>) tensor(4.8015, grad_fn=<MaxBackward1>)

```

In\[40\]:

```python
# IF using dot product similarity, must normalize vectors like so...
import numpy as np

# detach text emb from graph, move to CPU, and convert to numpy array
text_emb = text_emb.detach().cpu().numpy()

# calculate value to normalize each vector by
norm_factor = np.linalg.norm(text_emb, axis=1)
norm_factor.shape
```

Out\[40\]:

```
(21,)
```

In\[41\]:

```python
text_emb = text_emb.T / norm_factor
# transpose back to (21, 512)
text_emb = text_emb.T
print(text_emb.shape)
print(text_emb.min(), text_emb.max())
```

Out\[41\]:

```
(21, 512)
-0.1526844 0.53449875

```

Alternatively, we can use cosine similarity as our metric as this only considers angular similarity and not vector magnitude (like dot product). For our examples, we will normalize and use dot product similarity.

We now have our text embeddings; let’s see how to do the same for images.

### Encoding Images

Images will be encoded using the ViT portion of CLIP. Similar to text encoding, we need to preprocess these images using the `preprocessor` like so:

In\[42\]:

```python
data['image'][0].size
```

Out\[42\]:

```
(6000, 3376)
```

In\[43\]:

```python
image_batch = data['image']

images = processor(
    text=None,
    images=image_batch,
    return_tensors='pt'
)['pixel_values'].to(device)
images.shape
```

Out\[43\]:

```
torch.Size([21, 3, 224, 224])
```

Preprocessing images does _not_ produce token IDs like those we saw from preprocessing our text. Instead, preprocessing images consists of resizing the image to a 244x244 array with three color channels (red, green, and blue) and normalizing pixel values into a \[0,1\]\[0,1\] range.

After preprocessing our images, we get the image features with `get_image_features` and normalize them as before:

In\[44\]:

```python
img_emb = model.get_image_features(images)
print(img_emb.shape)
print(img_emb.min(), img_emb.max())
```

Out\[44\]:

```
torch.Size([21, 512])
tensor(-8.6533, grad_fn=<MinBackward1>) tensor(2.6551, grad_fn=<MaxBackward1>)

```

In\[45\]:

```python
# NORMALIZE
# detach text emb from graph, move to CPU, and convert to numpy array
img_emb = img_emb.detach().cpu().numpy()

img_emb = img_emb.T / np.linalg.norm(img_emb, axis=1)
# transpose back to (21, 512)
img_emb = img_emb.T
print(img_emb.shape)
print(img_emb.min(), img_emb.max())
```

Out\[45\]:

```
(21, 512)
-0.7275361 0.23383287

```

With this, we have created CLIP embeddings for both text and images. We can move on to comparing items across the two modalities.

### Calculating Similarity

CLIP embedding similarities are represented by their angular similarity. Meaning we can identify similar pairs using cosine similarity:

cossim(A,B)=A⋅B∣∣A∣∣∗∣∣B∣∣=∑inAiBi∑inAi2∑inBi2cossim(A, B) = \\frac{A \\cdot B}{\|\|A\|\| \* \|\|B\|\|} = \\frac{\\sum\_i^nA\_iB\_i}{\\sqrt{\\sum\_i^nA\_i^2} \\sqrt{\\sum\_i^nB\_i^2}}cossim(A,B)=∣∣A∣∣∗∣∣B∣∣A⋅B​=∑in​Ai2​​∑in​Bi2​​∑in​Ai​Bi​​

Or, if we have normalized the embeddings, we can use dot product similarity:

dotproduct(A,B)=A⋅B=∑i=0n−1AiBidotproduct(A, B) = A \\cdot B = \\sum\_{i=0}^{n-1}A\_iB\_idotproduct(A,B)=A⋅B=i=0∑n−1​Ai​Bi​

Let’s try both. First, for cosine similarity, we do:

In\[46\]:

```python
from numpy.linalg import norm

cos_sim = np.dot(text_emb, img_emb.T) / (
    norm(text_emb, axis=1) * norm(img_emb, axis=1)
)
cos_sim.shape
```

Out\[46\]:

```
(21, 21)
```

In\[47\]:

```python
import matplotlib.pyplot as plt

plt.imshow(cos_sim)
plt.show()
```

Out\[47\]:

```
<Figure size 432x288 with 1 Axes>
```

And if we perform the same operation for dot product similarity, we should return the same results:

In\[48\]:

```python
dot_sim = np.dot(text_emb, img_emb.T)

plt.imshow(cos_sim)
plt.show()
```

Out\[48\]:

```
<Figure size 432x288 with 1 Axes>
```

Both of these similarity score arrays look the same, and if we check for the difference between the two arrays, we will see that the scores are the same. We see some slight differences due to floating point errors.

In\[51\]:

```python
diff = cos_sim - dot_sim
diff.min(), diff.max()
```

Out\[51\]:

```
(0.0, 2.9802322e-08)
```

Using the embedding functions of CLIP in this way, we can perform a semantic search across the modalities of text and image in any direction. We can search for images with text, text with images, text with text, and images with images.

These use cases are great, but we can make slight modifications to this for many other tasks.

### Classification

One of the most impressive demonstrations of CLIP is its unparalleled zero-shot performance on various tasks. For example, given the `fragment/imagenette` dataset from Hugging Face _Datasets_, we can write a list of brief sentences that align with the ten class labels.

https://www.pinecone.io/_next/image/?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fvr8gru94%2Fproduction%2Ff841984e7617686f5041ca95797498e2b0b085b5-1348x542.png&w=3840&q=75

We take the original imagenette labels and preappend "a photo of a ..." to each to create a set of CLIP-friendly sentence representations.

From this, we can calculate the cosine similarity between the text embeddings of these ten labels against an image we’d like to classify. The text that returns the highest similarity is our predicted class.

### Object Detection

Another compelling use case of zero-shot CLIP is object detection. We can do this by splitting our images into smaller patches and running each patch through the image encoder of CLIP. We then compare these patch embeddings to a text encoding describing what we are looking for. After calculating the similarity scores for all patches, we can collate them into a map of relevance.

For example, given an image of a butterfly and a cat, we could break it into many small patches. Given the prompt `"a fluffy cat"`, we will return an outline of the cat, whereas the prompt `"a butterfly"` will produce an outline of the butterfly.

https://www.pinecone.io/_next/image/?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fvr8gru94%2Fproduction%2Fbe4800918976efd9d974d9e5453985a5106f2558-2389x1455.png&w=3840&q=75

Zero-shot object detection with CLIP allows us to find specific objects with natural language prompts.

These are only a few of the use cases of CLIP and only scratch the surface of what is possible with this model and others in the scope of multi-modal ML.

* * *

That’s it for this introduction to multi-modal ML with OpenAI’s CLIP. The past years since the CLIP release have seen ever more fascinating applications of the model.

DALL-E 2 is a well-known example of CLIP. The incredible images generated by DALL-E 2 start by embedding the user’s text prompt with CLIP \[4\]. That text embedding is then passed to the diffusion model, which generates some mind-blowing images.

The fields of NLP and CV have mainly progressed independently of each other for the past decade. However, with the introduction of world scope three models, they’re becoming more entwined into a majestic multi-modal field of Machine Learning.

## Resources

\[1\] Y. Bisk et al., [Experience Grounds Language](https://arxiv.org/abs/2004.10151) (2020), EMNLP

\[2\] J. Alammar, [Experience Grounds Language: Improving language models beyond the world of text](https://www.youtube.com/watch?v=WQm7-X4gts4) (2022), YouTube

\[3\] A. Radford et al., [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020) (2021), arXiv

\[4\] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, M. Chen, [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://arxiv.org/abs/2204.06125) (2022), arXiv

</details>

<details>
<summary>multimodal-embeddings-an-introduction-towards-data-science</summary>

# Multimodal Embeddings: An Introduction

Mapping text and images into a common space

[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)

Nov 29, 2024

8 min read

This is the 2nd article in a [larger series](https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a) on multimodal AI. In the [previous post](https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3), we saw how to augment [large language models (LLMs)](https://shawhin.medium.com/list/large-language-models-llms-8e009ae3054c) to understand new data modalities (e.g., images, audio, video). One such approach relied on encoders that generate vector representations (i.e. embeddings) of non-text data. In this article, I will discuss _multimodal_ embeddings and share what they can do via two practical use cases.

https://towardsdatascience.com/wp-content/uploads/2024/11/1a6BF-kEeo8rd7OW2a3JYGA.pngImage from Canva.

---

AI research is traditionally split into distinct fields: NLP, computer vision (CV), robotics, human-computer interface (HCI), etc. However, countless practical tasks require the **integration of these different research areas** e.g. autonomous vehicles (CV + robotics), AI agents (NLP + CV + HCI), personalized learning (NLP + HCI), etc.

Although these fields aim to solve different problems and work with different data types, they all share a fundamental process. Namely, **generating useful numerical representations of real-world phenomena**.

Historically, this was done by hand. This means that researchers and practitioners would use their (or other people’s) expertise to explicitly transform data into a more helpful form. Today, however, _these can be derived another way_.

## **Embeddings**

**Embeddings** are **(useful) numerical representations of data learned implicitly through model training**. For example, through learning how to predict text, BERT learned representations of text, which are helpful for many NLP tasks \[1\]. Another example is the Vision Transformer (ViT), trained for image classification on Image Net, which can be repurposed for other applications \[2\].

A key point here is that these learned embedding spaces will have some underlying structure so that **similar concepts are located close together**. As shown in the toy examples below.

https://towardsdatascience.com/wp-content/uploads/2024/11/1jpmC6Kx7DxVeikEr15vooA.pngToy represetation of text and image embeddings, respectively. Image by author.

One **key limitation** of the previously mentioned models is they are restricted to a single data modality, e.g., text or images. Preventing cross-modal applications like image captioning, content moderation, image search, and more. _But what if we could merge these two representations?_

## **Multimodal Embeddings**

Although text and images may look very different to us, in a neural network, these are **represented via the same mathematical object**, i.e., a vector. Therefore, in principle, text, images, or any other data modality can processed by a single model.

This fact underlies **multimodal embeddings**, which **represent multiple data modalities in the same vector space** such that similar concepts are co-located (independent of their original representations).

https://towardsdatascience.com/wp-content/uploads/2024/11/15d3HBNjNIXLy0oMIvJjxWw.pngToy representation of multimodal embedding space. Image by author.

For example, CLIP encodes text and images into a shared embedding space \[3\]. A key insight from CLIP is that by aligning text and image representations, the **model is capable of 0-shot image classification on an arbitrary set of target classes** since any input text can be treated as a class label (we will see a concrete example of this later).

However, this idea is not limited to text and images. Virtually any data modalities can be aligned in this way e.g., text-audio, audio-image, text-EEG, image-tabular, and text-video. Unlocking use cases such as video captioning, advanced OCR, audio transcription, video search, and EEG-to-text \[4\].

## **Contrastive Learning**

The standard approach to aligning disparate embedding spaces is **contrastive learning (CL)**. A key intuition of CL is to **represent different views of the same _information_ similarly** \[5\].

This consists of learning representations that **maximize the similarity between positive pairs** and **minimize the similarity of negative pairs**. In the case of an image-text model, a positive pair might be an image with an appropriate caption, while a negative pair would be an image with an irrelevant caption (as shown below).

https://towardsdatascience.com/wp-content/uploads/2024/11/1AGHBVjzwjXapJSe4aUPrjg.pngExample positive and negative pairs used in contrastive training. Image by author.

**Two key aspects** **of CL** contribute to its effectiveness

1. Since positive and negative pairs can be curated from the data’s inherent structure (e.g., metadata from web images), CL training data **do not require manual labeling**, which unlocks larger-scale training and more powerful representations \[3\].
2. It simultaneously maximizes positive and minimizes negative pair similarity via a special loss function, as demonstrated by CLIP \[3\].

![CLIP's contrastive loss for text-image representation alignment [3]. Image by author.](https://towardsdatascience.com/wp-content/uploads/2024/11/12X1aT8fzFsgbqn23zXmmAA.png)CLIP’s contrastive loss for text-image representation alignment \[3\]. Image by author.

## **Example Code:** Using CLIP for 0-shot classification and image search

With a high-level understanding of how multimodal embeddings work, let’s see two concrete examples of what they can do. Here, I will use the open-source [CLIP model](https://huggingface.co/openai/clip-vit-base-patch16) to perform two tasks: 0-shot image classification and image search.

The **code for these examples** is freely available on the [GitHub repository](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/2-mm-embeddings).

---

### Use case 1: 0-shot Image Classification

The basic idea behind using CLIP for 0-shot image classification is to pass an image into the model along with a set of possible class labels. Then, a classification can be made by **evaluating which text input is most similar to the input image**.

We’ll start by importing the [Hugging Face Transformers library](https://huggingface.co/docs/transformers/en/installation) so that the CLIP model can be downloaded locally. Additionally, the PIL library is used to load images in Python.

```python
from transformers import CLIPProcessor, CLIPModel
from PIL import Image
```

Next, we can import a version of the clip model and its associated data processor. _Note: the processor handles tokenizing input text and image preparation._

```ini
# import model
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch16")

# import processor (handles text tokenization and image preprocessing)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch16")
```

We load in the below image of a cat and create a list of two possible class labels: " _a photo of a cat_" or " _a photo of a dog_".

```ini
# load image
image = Image.open("images/cat_cute.png")

# define text classes
text_classes = ["a photo of a cat", "a photo of a dog"]
```

https://towardsdatascience.com/wp-content/uploads/2024/11/1Nzo536sqahqm1Q24Ms2vmA.pngInput cat photo. Image from Canva.

Next, we’ll preprocess the image/text inputs and pass them into the model.

```ini
# pass image and text classes to processor
inputs = processor(text=text_classes, images=image, return_tensors="pt",
                                                    padding=True)

# pass inputs to CLIP
outputs = model(**inputs) # note: "**" unpacks dictionary items
```

To make a class prediction, we must extract the image logits and evaluate which class corresponds to the maximum.

```makefile
# image-text similarity score
logits_per_image = outputs.logits_per_image
# convert scores to probs via softmax
probs = logits_per_image.softmax(dim=1)

# print prediction
predicted_class = text_classes[probs.argmax()]
print(predicted_class, "| Probability = ",
                       round(float(probs[0][probs.argmax()]),4))
```

```language-none
>> a photo of a cat | Probability =  0.9979
```

The model nailed it with a 99.79% probability that it’s a cat photo. However, this was a super easy one. Let’s see what happens when we change the class labels to: " _ugly cat_" and " _cute cat_" for the same image.

```language-none
>> cute cat | Probability =  0.9703
```

The model easily identified that the image was indeed a cute cat. Let’s do something more challenging like the labels: " _cat meme_" or " _not cat meme_".

```language-none
>> not cat meme | Probability =  0.5464
```

While the model is less confident about this prediction with a 54.64% probability, it correctly implies that the image is not a meme.

### Use case 2: Image Search

Another application of CLIP is essentially the inverse of Use Case 1. Rather than identifying which text label matches an input image, we can evaluate **which image (in a set) best matches a text input (i.e. query)**—in other words, performing a search over images.

We start by storing a set of images in a list. Here, I have three images of a cat, dog, and goat, respectively.

```python
# create list of images to search over
image_name_list = ["images/cat_cute.png", "images/dog.png", "images/goat.png"]

image_list = []
for image_name in image_name_list:
    image_list.append(Image.open(image_name))
```

Next, we can define a query like " _a cute dog_" and pass it and the images into CLIP.

```python
# define a query
query = "a cute dog"

# pass images and query to CLIP
inputs = processor(text=query, images=image_list, return_tensors="pt",
                                                  padding=True)
```

We can then match the best image to the input text by extracting the text logits and evaluating the image corresponding to the maximum.

```python
# compute logits and probabilities
outputs = model(**inputs)
logits_per_text = outputs.logits_per_text
probs = logits_per_text.softmax(dim=1)

# print best match
best_match = image_list[probs.argmax()]
prob_match = round(float(probs[0][probs.argmax()]),4)

print("Match probability: ",prob_match)
display(best_match)
```

```language-none
>> Match probability:  0.9817
```

https://towardsdatascience.com/wp-content/uploads/2024/11/14wnqr5p_7N3QD5EkXIQeew.pngBest match for query "a cute dog". Image from Canva.

We see that (again) the model nailed this simple example. But let’s try some trickier examples.

```python
query = "something cute but metal 🤘"
```

```language-none
>> Match probability:  0.7715
```

https://towardsdatascience.com/wp-content/uploads/2024/11/1tIY3_ONQQT_cracAPWm8NQ.pngBest match for query "something cute but metal 🤘". Image from Canva.

```python
query = "a good boy"
```

```language-none
>> Match probability:  0.8248
```

https://towardsdatascience.com/wp-content/uploads/2024/11/14wnqr5p_7N3QD5EkXIQeew.pngBest match for query "a good boy". Image from Canva.

```python
query = "the best pet in the world"
```

```language-none
>> Match probability:  0.5664
```

https://towardsdatascience.com/wp-content/uploads/2024/11/1Nzo536sqahqm1Q24Ms2vmA.pngBest match for query "the best pet in the world". Image from Canva.

Although this last prediction is quite controversial, all the other matches were spot on! This is likely since images like these are ubiquitous on the internet and thus were seen many times in CLIP’s pre-training.

> [**YouTube-Blog/multimodal-ai/2-mm-embeddings at main · ShawhinT/YouTube-Blog**](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/2-mm-embeddings)

## What’s Next?

Multimodal embeddings unlock countless AI use cases that involve multiple data modalities. Here, we saw two such use cases, i.e., 0-shot image classification and image search using CLIP.

Another practical application of models like CLIP is multimodal RAG, which consists of the automated retrieval of multimodal context to an LLM. In the [next article](https://medium.com/towards-data-science/multimodal-rag-process-any-file-type-with-ai-e6921342c903) of this [series](https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a), we will see how this works under the hood and review a concrete example.

**More on Multimodal models 👇**

> [**Multimodal AI**](https://shawhin.medium.com/list/fe9521d0e77a)

---

**My website**: [https://www.shawhintalebi.com/](https://www.shawhintalebi.com/)

- \[1\] [BERT](https://arxiv.org/abs/1810.04805)
- \[2\] [ViT](https://arxiv.org/abs/2010.11929)
- \[3\] [CLIP](https://arxiv.org/abs/2103.00020)
- \[4\] [Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)](https://arxiv.org/abs/2410.07507)
- \[5\] [A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/abs/2002.05709)

</details>

<details>
<summary>multimodal-rag-with-colpali-milvus-and-vlms</summary>

# Multimodal RAG with Colpali, Milvus and VLMs

In this post, we will see how to doIn this post, we will see how to do multimodal RAG with [colpali](https://arxiv.org/abs/2407.01449), [milvus](https://milvus.io/) and a visual language model (gemini/gpt-4o).

We will build an application to upload a PDF and then do Q&A queries on it. Q&A can be done on both text and visual elements of the PDF. We will not extract text from the PDF; instead, we will treat it as an image and use colpali to get embeddings for the PDF pages. These embeddings will be indexed to Milvus, and then we will use a VLM to do Q&A queries on the PDF pages.

> If you just want to see the code in action, there is a demo at [https://huggingface.co/spaces/saumitras/colpali-milvus](https://huggingface.co/spaces/saumitras/colpali-milvus/). Code for the same is [here](https://github.com/saumitras/colpali-milvus-multimodal-rag/).

**TOC**:

1. [Problem](#problem)
2. [Why colpali?](#why-colpali)
3. [Understanding how colpali works](#understanding-how-colpali-works)
4. [Code to upload a PDF, get embedding using colpali, index it to Milvus, then do Q&A queries using a vision language model (gemini/openai)](#code)

## Problem

Let's say a company wants to build a Q&A/search interface for its internal documents, which include PDFs, word files, wikis, images, and text files. The traditional approach involves extracting text and media, detecting layout for structure, and indexing the information in a vector store for semantic search. However, this method often falls short for complex documents containing images, tables, and graphs. Let's look at an example below:

We have a [PDF with stats on covid](https://saumitra.me/2024/covid-slides.pdf) in the form of charts and tables. We want to answer the queries below:

```markdown
1. What is the correlation between the samples tested and the positivity rate?
2. When and what was the highest number of cases and TPR?
3. Which country had the highest omicron cases?
```

These queries can be answered by using data from following 3 pages:

**Page 4: A chart showing stats on samples and positivity rate**

[https://saumitra.me/2024/covid-page-4.png](https://saumitra.me/2024/covid-page-4.png)

**Page 8: A table showing cases and TPR**

[https://saumitra.me/2024/covid-page-8.png](https://saumitra.me/2024/covid-page-8.png)

**Page 9: A table showing cases by country**

[https://saumitra.me/2024/covid-page-9.png](https://saumitra.me/2024/covid-page-9.png)

It would be difficult to extract data from these pages as text in a manner which can be used for querying.
We want to show user the answer and source page(s) from the PDF which contains the answer, like below:

[https://saumitra.me/2024/rag-demo-screenshot.png](https://saumitra.me/2024/rag-demo-screenshot.png)

Let's understand how colpali can help us here.

## Why colpali?

Document retrieval has always been a key component of systems like search engines and information retrieval. Traditional document retrieval methods rely heavily on text-based methods (like OCR and text segmentation), often missing crucial visual cues like layouts, images, and tables.

Colpali addresses this by using Vision-Language Models (VLMs) to understand and retrieve visually rich documents, capturing both textual and visual information. Colpali's architecture allows direct encoding of document images into a common embedding space, eliminating the need for time-consuming text extraction and segmentation.

## Understanding how colpali works

Colpali works in the following steps:

### Step 1: Treating the Document as an Image

Imagine we have a PDF document. Normally, we would extract text from the document using OCR (Optical Character Recognition), segment it into different sections, and then use these segments for searching. colpali simplifies this process by treating the entire document page as an image, bypassing the need for complex text extraction, layout detection, or OCR.

### Step 2: Splitting the Image into Patches

Once colpali has this "image" of the document, it divides the page into small, uniform pieces called patches. Each patch captures a tiny portion of the page. It might contain a few words, a piece of a graph, or part of an image. This division helps the model focus on the document's small, detailed parts rather than trying to understand the whole page at once.

At first glance, it might seem like dividing an image into patches is similar to breaking text into chunks. However, these two methods have several key differences, especially in how they handle and preserve context. Let’s dive deeper into these differences to understand why patch-based processing in colpali is more effective for document retrieval compared to traditional text chunking.

#### Understanding Context Loss in Text Chunking

In traditional text chunking, text is split into smaller chunks based on certain tokens since many models limit the number of tokens they can process at once.

Problem with Context Loss:

- Chunking can split sentences or paragraphs midway, causing crucial context to be lost. It can also result in incomplete information in one chunk and missing context in another.
- Chunking doesn't preserve visual or structural information, such as the relationship between headings and their corresponding content or the placement of text in tables or figures.

For example, If you have a document with a heading followed by a table, text chunking might separate the heading and the table, losing the context that the table belongs to that heading.

#### Patch-Based Image Processing in colpali

Colpali divides the document image into patches, much like dividing a photo into small squares. Each patch is a fixed-size portion of the image, like a mini-snapshot of that part of the page.

Patches are more effective due to the following reasons:

- **No Loss of Structure:** The patches retain the document's visual structure, preserving its spatial layout. For instance, if a page has two columns of text or a table with rows and columns, each patch maintains its relative position, ensuring that the model understands the overall arrangement of the elements.
- **Multi-Modal Context:** Patches capture both textual and visual information. This includes both visual features (e.g., font styles, colors, boldness) and non-text elements (e.g., figures and graphs).
- **Positional Awareness:** Each patch has a positional embedding that tells the model where it is located on the page, helping the model understand the overall layout.

### Step 3: Embedding Creation and Aligning Visual and Textual Information

Each patch is then passed through a Vision Transformer (ViT), which converts them into unique embeddings. Next, colpali aligns these visual embeddings with the text of the query by transforming the query into its own set of embeddings. colpali uses a process called `alignment` that aligns image path embeddings and text embeddings in the same vector space. Only then can we compare the similarity between query and document embeddings.

### Step 4: Scoring the Relevance - Late Interaction Mechanism

At this point, colpali has embeddings for both the query and the document. The next challenge is to identify the relevant parts of the document. colpali uses a process called the `Late Interaction Mechanism`, where each piece of the query is finely matched against every part of the document, scoring and ranking their relevance.

Colpali highlights the most relevant pieces of the document, focusing on the patches that best match the query. This approach enables colpali to efficiently retrieve relevant information from visually rich documents, capturing both visual and textual data without losing context.

* * *

## Code

Full code at [https://github.com/saumitras/colpali-milvus-rag/](https://github.com/saumitras/colpali-milvus-rag/)

### 1. Add colpali processor

```python
model_name = "vidore/colpali-v1.2"
device = get_torch_device("cuda")

model = colpali.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map=device,
).eval()

processor = cast(colpaliProcessor, colpaliProcessor.from_pretrained(model_name))
```

### 2. Use colpali to get embeddings for image (pdf pages)

```python
def process_images(self, image_paths:list[str], batch_size=5):

    print(f"Processing {len(image_paths)} image_paths")

    images = self.get_images(image_paths)

    dataloader = DataLoader(
        dataset=ListDataset[str](images),
        batch_size=batch_size,
        shuffle=False,
        collate_fn=lambda x: processor.process_images(x),
    )

    ds: List[torch.Tensor] = []
    for batch_doc in tqdm(dataloader):
        with torch.no_grad():
            batch_doc = {k: v.to(model.device) for k, v in batch_doc.items()}
            embeddings_doc = model(**batch_doc)
        ds.extend(list(torch.unbind(embeddings_doc.to(device))))

    ds_np = [d.float().cpu().numpy() for d in ds]

    return ds_np
```

### 3. Use colpali to get embeddings for text (user query)

```python
def process_text(self, texts: list[str]):
    print(f"Processing {len(texts)} texts")

    dataloader = DataLoader(
        dataset=ListDataset[str](texts),
        batch_size=1,
        shuffle=False,
        collate_fn=lambda x: processor.process_queries(x),
    )

    qs: List[torch.Tensor] = []
    for batch_query in dataloader:
        with torch.no_grad():
            batch_query = {k: v.to(model.device) for k, v in batch_query.items()}
            embeddings_query = model(**batch_query)

        qs.extend(list(torch.unbind(embeddings_query.to(device))))

    qs_np = [q.float().cpu().numpy() for q in qs]

    return qs_np
```

### 4. Code to create collection, index and query in milvus

```python
class MilvusManager:
    def __init__(self, milvus_uri, collection_name, create_collection, dim=128):
        self.client = MilvusClient(uri=milvus_uri)
        self.collection_name = collection_name
        if self.client.has_collection(collection_name=self.collection_name):
            self.client.load_collection(collection_name)
        self.dim = dim

        if create_collection:
            self.create_collection()
            self.create_index()

    def create_collection(self):
        if self.client.has_collection(collection_name=self.collection_name):
            self.client.drop_collection(collection_name=self.collection_name)
        schema = self.client.create_schema(
            auto_id=True,
            enable_dynamic_fields=True,
        )
        schema.add_field(field_name="pk", datatype=DataType.INT64, is_primary=True)
        schema.add_field(
            field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=self.dim
        )
        schema.add_field(field_name="seq_id", datatype=DataType.INT16)
        schema.add_field(field_name="doc_id", datatype=DataType.INT64)
        schema.add_field(field_name="doc", datatype=DataType.VARCHAR, max_length=65535)

        self.client.create_collection(
            collection_name=self.collection_name, schema=schema
        )

    def create_index(self):
        self.client.release_collection(collection_name=self.collection_name)
        self.client.drop_index(
            collection_name=self.collection_name, index_name="vector"
        )
        index_params = self.client.prepare_index_params()
        index_params.add_index(
            field_name="vector",
            index_name="vector_index",
            index_type="HNSW",
            metric_type="IP",
            params={
                "M": 16,
                "efConstruction": 500,
            },
        )

        self.client.create_index(
            collection_name=self.collection_name, index_params=index_params, sync=True
        )

    def create_scalar_index(self):
        self.client.release_collection(collection_name=self.collection_name)

        index_params = self.client.prepare_index_params()
        index_params.add_index(
            field_name="doc_id",
            index_name="int32_index",
            index_type="INVERTED",
        )

        self.client.create_index(
            collection_name=self.collection_name, index_params=index_params, sync=True
        )

    def search(self, data, topk):
        search_params = {"metric_type": "IP", "params": {}}
        results = self.client.search(
            self.collection_name,
            data,
            limit=int(50),
            output_fields=["vector", "seq_id", "doc_id"],
            search_params=search_params,
        )
        doc_ids = set()
        for r_id in range(len(results)):
            for r in range(len(results[r_id])):
                doc_ids.add(results[r_id][r]["entity"]["doc_id"])

        scores = []

        def rerank_single_doc(doc_id, data, client, collection_name):
            doc_colbert_vecs = client.query(
                collection_name=collection_name,
                filter=f"doc_id in [{doc_id}, {doc_id + 1}]",
                output_fields=["seq_id", "vector", "doc"],
                limit=1000,
            )
            doc_vecs = np.vstack(
                [doc_colbert_vecs[i]["vector"] for i in range(len(doc_colbert_vecs))]
            )
            score = np.dot(data, doc_vecs.T).max(1).sum()
            return (score, doc_id)

        with concurrent.futures.ThreadPoolExecutor(max_workers=300) as executor:
            futures = {
                executor.submit(
                    rerank_single_doc, doc_id, data, self.client, self.collection_name
                ): doc_id
                for doc_id in doc_ids
            }
            for future in concurrent.futures.as_completed(futures):
                score, doc_id = future.result()
                scores.append((score, doc_id))

        scores.sort(key=lambda x: x[0], reverse=True)
        if len(scores) >= topk:
            return scores[:topk]
        else:
            return scores

    def insert(self, data):
        colbert_vecs = [vec for vec in data["colbert_vecs"]]
        seq_length = len(colbert_vecs)
        doc_ids = [data["doc_id"] for i in range(seq_length)]
        seq_ids = list(range(seq_length))
        docs = [""] * seq_length
        docs[0] = data["filepath"]

        self.client.insert(
            self.collection_name,
            [\
                {\
                    "vector": colbert_vecs[i],\
                    "seq_id": seq_ids[i],\
                    "doc_id": doc_ids[i],\
                    "doc": docs[i],\
                }\
                for i in range(seq_length)\
            ],
        )

    def get_images_as_doc(self, images_with_vectors:list):

        images_data = []

        for i in range(len(images_with_vectors)):
            data = {
                "colbert_vecs": images_with_vectors[i]["colbert_vecs"],
                "doc_id": i,
                "filepath": images_with_vectors[i]["filepath"],
            }
            images_data.append(data)

        return images_data

    def insert_images_data(self, image_data):
        data = self.get_images_as_doc(image_data)

        for i in range(len(data)):
            self.insert(data[i])
```

### 5. Save pdf as individual images

```python
class PdfManager:
    def __init__(self):
        pass

    def clear_and_recreate_dir(self, output_folder):
        print(f"Clearing output folder {output_folder}")

        if os.path.exists(output_folder):
            shutil.rmtree(output_folder)

        os.makedirs(output_folder)

    def save_images(self, id, pdf_path, max_pages, pages: list[int] = None) -> list[str]:
        output_folder = f"pages/{id}/"
        images = convert_from_path(pdf_path)

        print(f"Saving images from {pdf_path} to {output_folder}. Max pages: {max_pages}")

        self.clear_and_recreate_dir(output_folder)

        num_page_processed = 0

        for i, image in enumerate(images):
            if max_pages and num_page_processed >= max_pages:
                break

            if pages and i not in pages:
                continue

            full_save_path = f"{output_folder}/page_{i + 1}.png"

            image.save(full_save_path, "PNG")

            num_page_processed += 1

        return [f"{output_folder}/page_{i + 1}.png" for i in range(num_page_processed)]
```

### 6. Middleware to index and search Milvus for embeddings generated from colpali

```python
class Middleware:
    def __init__(self, id:str, create_collection=True):
        hashed_id = hashlib.md5(id.encode()).hexdigest()[:8]
        milvus_db_name = f"milvus_{hashed_id}.db"
        self.milvus_manager = MilvusManager(milvus_db_name, "colpali", create_collection)

    def index(self, pdf_path: str, id:str, max_pages: int, pages: list[int] = None):

        print(f"Indexing {pdf_path}, id: {id}, max_pages: {max_pages}")

        image_paths = pdf_manager.save_images(id, pdf_path, max_pages)

        print(f"Saved {len(image_paths)} images")

        colbert_vecs = colpali_manager.process_images(image_paths)

        images_data = [{\
            "colbert_vecs": colbert_vecs[i],\
            "filepath": image_paths[i]\
        } for i in range(len(image_paths))]

        print(f"Inserting {len(images_data)} images data to Milvus")

        self.milvus_manager.insert_images_data(images_data)

        print("Indexing completed")

        return image_paths


    def search(self, search_queries: list[str]):
        print(f"Searching for {len(search_queries)} queries")

        final_res = []

        for query in search_queries:
            print(f"Searching for query: {query}")
            query_vec = colpali_manager.process_text([query])[0]
            search_res = self.milvus_manager.search(query_vec, topk=1)
            print(f"Search result: {search_res} for query: {query}")
            final_res.append(search_res)

        return final_res
```

### 7. Use Gemini or gpt-4o to do Q&A on pdf page(s) matching user query

```python
class Rag:

    def get_answer_from_gemini(self, query, imagePaths):

        print(f"Querying Gemini for query={query}, imagePaths={imagePaths}")

        try:
            genai.configure(api_key=os.environ['GEMINI_API_KEY'])
            model = genai.GenerativeModel('gemini-1.5-flash')

            images = [Image.open(path) for path in imagePaths]

            chat = model.start_chat()

            response = chat.send_message([*images, query])

            answer = response.text

            print(answer)

            return answer

        except Exception as e:
            print(f"An error occurred while querying Gemini: {e}")
            return f"Error: {str(e)}"


    def get_answer_from_openai(self, query, imagesPaths):
        print(f"Querying OpenAI for query={query}, imagesPaths={imagesPaths}")

        try:
            payload = self.__get_openai_api_payload(query, imagesPaths)

            headers = {
                "Content-Type": "application/json",
                "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"
            }

            response = requests.post(
                url="https://api.openai.com/v1/chat/completions",
                headers=headers,
                json=payload
            )
            response.raise_for_status()  # Raise an HTTPError for bad responses

            answer = response.json()["choices"][0]["message"]["content"]

            print(answer)

            return answer

        except Exception as e:
            print(f"An error occurred while querying OpenAI: {e}")
            return None

    def __get_openai_api_payload(self, query:str, imagesPaths:List[str]):
        image_payload = []

        for imagePath in imagesPaths:
            base64_image = encode_image(imagePath)
            image_payload.append({
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{base64_image}"
                }
            })

        payload = {
            "model": "gpt-4o",
            "messages": [\
                {\
                    "role": "user",\
                    "content": [\
                        {\
                            "type": "text",\
                            "text": query\
                        },\
                        *image_payload\
                    ]\
                }\
            ],
            "max_tokens": 1024
        }

        return payload
```

In the next post, we will understand the limitations of colpali and a workaround for them.

## References

1. [https://milvus.io/docs/use_colpali_with_milvus.md](https://milvus.io/docs/use_colpali_with_milvus.md)
2. [https://arxiv.org/abs/2407.01449](https://arxiv.org/abs/2407.01449)

</details>

<details>
<summary>start-with-a-prebuilt-agent</summary>

# LangGraph quickstart

This guide shows you how to set up and use LangGraph's **prebuilt**, **reusable** components, which are designed to help you construct agentic systems quickly and reliably.

## Prerequisites

Before you start this tutorial, ensure you have the following:

- An [Anthropic](https://console.anthropic.com/settings/keys) API key

## 1. Install dependencies

If you haven't already, install LangGraph and LangChain:

```md-code__content
pip install -U langgraph "langchain[anthropic]"

```

Info

`langchain[anthropic]` is installed so the agent can call the [model](https://python.langchain.com/docs/integrations/chat/).

## 2. Create an agent

To create an agent, use [`create_react_agent`](https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent):

_API Reference: [create_react_agent](https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent)_

```md-code__content
from langgraph.prebuilt import create_react_agent

def get_weather(city: str) -> str:
    """Get weather for a given city."""
    return f"It's always sunny in {city}!"

agent = create_react_agent(
    model="anthropic:claude-3-7-sonnet-latest",
    tools=[get_weather],
    prompt="You are a helpful assistant"
)

# Run the agent
agent.invoke(
    {"messages": [{"role": "user", "content": "what is the weather in sf"}]}
)

```

## 3. Configure an LLM

To configure an LLM with specific parameters, such as temperature, use [init_chat_model](https://python.langchain.com/api_reference/langchain/chat_models/langchain.chat_models.base.init_chat_model.html):

_API Reference: [init_chat_model](https://python.langchain.com/api_reference/langchain/chat_models/langchain.chat_models.base.init_chat_model.html) \| [create_react_agent](https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent)_

```md-code__content
from langchain.chat_models import init_chat_model
from langgraph.prebuilt import create_react_agent

model = init_chat_model(
    "anthropic:claude-3-7-sonnet-latest",
    temperature=0
)

agent = create_react_agent(
    model=model,
    tools=[get_weather],
)

```

For more information on how to configure LLMs, see Models.

## 4. Add a custom prompt

Prompts instruct the LLM how to behave. Add one of the following types of prompts:

- **Static**: A string is interpreted as a **system message**.
- **Dynamic**: A list of messages generated at **runtime**, based on input or configuration.

Define a fixed prompt string or list of messages:

```md-code__content
from langgraph.prebuilt import create_react_agent

agent = create_react_agent(
    model="anthropic:claude-3-7-sonnet-latest",
    tools=[get_weather],
    # A static prompt that never changes
    prompt="Never answer questions about the weather."
)

agent.invoke(
    {"messages": [{"role": "user", "content": "what is the weather in sf"}]}
)

```

Define a function that returns a message list based on the agent's state and configuration:

```md-code__content
from langchain_core.messages import AnyMessage
from langchain_core.runnables import RunnableConfig
from langgraph.prebuilt.chat_agent_executor import AgentState
from langgraph.prebuilt import create_react_agent

def prompt(state: AgentState, config: RunnableConfig) -> list[AnyMessage]:
    user_name = config["configurable"].get("user_name")
    system_msg = f"You are a helpful assistant. Address the user as {user_name}."
    return [{"role": "system", "content": system_msg}] + state["messages"]

agent = create_react_agent(
    model="anthropic:claude-3-7-sonnet-latest",
    tools=[get_weather],
    prompt=prompt
)

agent.invoke(
    {"messages": [{"role": "user", "content": "what is the weather in sf"}]},
    config={"configurable": {"user_name": "John Smith"}}
)

```

For more information, see Context.

## 5. Add memory

To allow multi-turn conversations with an agent, you need to enable persistence by providing a checkpointer when creating an agent. At runtime, you need to provide a config containing `thread_id` — a unique identifier for the conversation (session):

_API Reference: [create_react_agent](https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent) \| [InMemorySaver](https://langchain-ai.github.io/langgraph/reference/checkpoints/#langgraph.checkpoint.memory.InMemorySaver)_

```md-code__content
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import InMemorySaver

checkpointer = InMemorySaver()

agent = create_react_agent(
    model="anthropic:claude-3-7-sonnet-latest",
    tools=[get_weather],
    checkpointer=checkpointer
)

# Run the agent
config = {"configurable": {"thread_id": "1"}}
sf_response = agent.invoke(
    {"messages": [{"role": "user", "content": "what is the weather in sf"}]},
    config
)
ny_response = agent.invoke(
    {"messages": [{"role": "user", "content": "what about new york?"}]},
    config
)

```

When you enable the checkpointer, it stores agent state at every step in the provided checkpointer database (or in memory, if using `InMemorySaver`).

Note that in the above example, when the agent is invoked the second time with the same `thread_id`, the original message history from the first conversation is automatically included, together with the new user input.

For more information, see Memory.

## 6. Configure structured output

To produce structured responses conforming to a schema, use the `response_format` parameter. The schema can be defined with a `Pydantic` model or `TypedDict`. The result will be accessible via the `structured_response` field.

_API Reference: [create_react_agent](https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent)_

```md-code__content
from pydantic import BaseModel
from langgraph.prebuilt import create_react_agent

class WeatherResponse(BaseModel):
    conditions: str

agent = create_react_agent(
    model="anthropic:claude-3-7-sonnet-latest",
    tools=[get_weather],
    response_format=WeatherResponse
)

response = agent.invoke(
    {"messages": [{"role": "user", "content": "what is the weather in sf"}]}
)

response["structured_response"]

```

LLM post-processing

Structured output requires an additional call to the LLM to format the response according to the schema.

</details>

<details>
<summary>the-8-best-ai-image-generators-in-2025-zapier</summary>

# The 8 best AI image generators in 2025

## Get the best AI-generated images using text-to-image AI.

By Harry Guinness · May 23, 2025

https://images.ctfassets.net/lzny33ho1g45/2olcy4TVSWAjqy5dsxLNZd/09b4a18346af97076615d5f1d1407c39/best-ai-image-generator-hero.jpg?fm=jpg&q=31&fit=thumb&w=1520&h=760

AI image generators have been brewing (generating?) up a storm for the last couple of years. If you've been on social media, watched prime time news shows, or read a magazine, AI-generated images have been impossible to miss. These kinds of AI-generated images are everywhere, and sometimes you won't even realize. If you want to join in the fun, or [add some AI-powered features to your business workflows](https://zapier.com/blog/ai-image-examples-for-business/), the apps on this list will give you what you're looking for.

I've been writing about AI image generators [since Google Deep Dream in 2015](https://photography.tutsplus.com/articles/brave-new-camera-computational-photography--cms-23438). That's about as long as anyone outside of a computer science lab has realistically been thinking about these tools, and I'm really excited by how far they've come.

I'm going to try to avoid the thorny discussions around artistic merit, whether or not these tools are replacing or augmenting artists, and copyright infringement in training data, at least where I can. Instead, I'll focus on the fact that these AI image generators can now produce excellent results from a wide range of text and image prompts.

It's worth taking a few hours to play around with one of these text-to-image AI apps—even just so you can appreciate them from a technical perspective. Whether you like it or not, we're all seeing a lot of their output at the moment. And there will only be more to come.

## The best AI image generators

- [ChatGPT (GPT-4o)](https://zapier.com/blog/best-ai-image-generator/#gpt-4o) for the best AI image generator overall
- [Midjourney](https://zapier.com/blog/best-ai-image-generator/#midjourney) for artistic results
- [Reve](https://zapier.com/blog/best-ai-image-generator/#reve) for overall prompt adherence
- [Ideogram](https://zapier.com/blog/best-ai-image-generator/#ideogram) for accurate text
- [Stable Diffusion](https://zapier.com/blog/best-ai-image-generator/#stable-diffusion) for customization and control of your AI images
- [FLUX.1](https://zapier.com/blog/best-ai-image-generator/#flux) for a Stable Diffusion alternative
- [Adobe Firefly](https://zapier.com/blog/best-ai-image-generator/#firefly) for integrating AI-generated images into photos
- [Recraft](https://zapier.com/blog/best-ai-image-generator/#recraft) for graphic design

## How do AI image generators work?

All these AI image generators take a text prompt and then turn it—as best they can—into a matching image. This opens up some wild possibilities, since your prompt can be anything from "an impressionist oil painting of a Canadian man riding a moose through a forest of maple trees" to "a painting in the style of Vermeer of a large fluffy Irish wolfhound enjoying a pint of beer in a traditional pub" or "a photograph of a donkey on the moon."

https://images.ctfassets.net/lzny33ho1g45/2udOp4paDgOh5HpqG5JRAQ/18abc9476c4705aacf3609edcec4f945/image8.jpeg?

I made this with Midjourney using the prompt "an impressionist oil painting of a Canadian man riding a moose through a forest of maple trees"

Seriously, the only real limits are your imagination, the AI image generator's ability to [comprehend your prompt](https://zapier.com/blog/natural-language-processing/), and any content filters put in place to stop plagiarism, copyright infringement, and bad actors flooding the internet with AI-generated violence or other NSFW content. (That Vermeer prompt used to work reliably, but some more restrictive image generators now block it because it uses a named artist.)

Most AI image generators work in a pretty similar way. [Millions or billions](https://laion.ai/blog/laion-5b/) of image-text pairs are used to train a neural network (basically, a very fancy computer algorithm [modeled loosely on the human brain](https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414)) on _what things are_. By allowing it to process near-countless images, it learns what dogs, the color red, Vermeers, and everything else are. Once this is done, you have an AI that can interpret almost any prompt—though [there is a skill in setting things up](https://zapier.com/blog/ai-art-prompts/) so it can do so accurately.

The next step is to actually render the AI-generated image. The latest generation of AI image generators typically uses a [process called diffusion](https://www.assemblyai.com/blog/diffusion-models-for-machine-learning-introduction/)—though OpenAI's latest foray into image generation uses a slightly different [process called autoregression](https://arxiv.org/abs/2404.02905). In essence, the image generators start with a random field of noise and then edit it in a series of steps to match their interpretation of the prompt. It's kind of like looking up at a cloudy sky, finding a cloud that looks kind of like a dog, and then being able to snap your fingers to keep making it more and more dog-like.

https://images.ctfassets.net/lzny33ho1g45/1LHdvgxMxOKcgWqC2yzoKh/ff7194426828d81a2d8437f4f9c38132/ai-image-generator-dogs.png?

A dog-shaped cloud floating in a clear blue sky—from top-left, going clockwise, at 10 steps, 20 steps, 40 steps, and 120 steps.

Before we dive in: I don't want to oversell things. What these text-to-image generators can do is very impressive, but they aren't likely to save you from ever having to do a product photoshoot again. If you just need some weird or unique images, they can really help. But if you're looking for something super specific, you're better off hiring a photographer—or licensing the exact image you want. Similarly, trying to use one to [make a header image for a blog post](https://zapier.com/blog/generate-blog-images-with-dall-e-and-zapier/) can take a lot more time than just finding a header image for your blog through a stock photo site.

## What makes the best AI image generator?

##### How we evaluate and test apps

Our best apps roundups are written by humans who've spent much of their careers using, testing, and writing about software. Unless explicitly stated, we spend dozens of hours researching and testing apps, using each app as it's intended to be used and evaluating it against the criteria we set for the category. We're never paid for placement in our articles from any app or for links to any site—we value the trust readers put in us to offer authentic evaluations of the categories and apps we review. For more details on our process, read the full rundown of [how we select apps to feature on the Zapier blog](https://zapier.com/blog/selecting-apps/).

There's a reason that AI image generators have become incredibly popular over the past few years: before that, they were pretty bad. The technology underlying them was incredibly cool and impressive, at least to research scientists, but [the images they could output](https://www.theguardian.com/artanddesign/2016/mar/28/google-deep-dream-art) were underwhelming. Even the original DALL·E was more of a fun novelty than a world-shaking revelation [when it launched in 2021](https://openai.com/research/dall-e).

Now that these text-to-image generators have been around for a while, there's some real competition between the different models. They've really increased in quality and can now even generate text somewhat accurately. If all you care about is the current "best" model, check out [Artificial Analysis's Image Arena](https://artificialanalysis.ai/text-to-image/arena?tab=Leaderboard). But we've reached the stage where the top dozen or more models are all excellent, so other features and usability matter more than they used to.

So, to find the best AI art generators, I set some pretty strict criteria:

- I was looking for apps that allowed you to generate AI images from a text prompt (and to a lesser degree, an image prompt). Tools that have you upload a dozen of your photos and then [spit out AI-generated portraits](https://land.prisma-ai.com/magic-avatars/) are fun (and normally built using Stable Diffusion), but they aren't the kind of general-purpose image generators I was considering.

- I was looking at the AI image generators themselves, not [tools built on top of them](https://zapier.com/blog/ai-art-generator/). For example, [NightCafe](https://nightcafe.studio/) is an AI picture generator that has a great community and app, but it just enables you to use [open source models](https://zapier.com/blog/open-source-ai/) like FLUX and Stable Diffusion, fine-tuned models based on various versions of them, the DALL·E 3 and Google Imagen APIs, as well as a handful of older generative models. It's worth checking out, but it doesn't meet my criteria for its own entry on this list.

Aside from all that, I also considered how easy each AI image creator is to use, what kinds of controls and customization options it provides (for things like AI image upscale), what pricing model it has, and most important of all: how good were the results? The best AI image generators are now far less likely to create weird or impossible-looking things.

I've been using and writing about text-to image generators since the original DALL·E launched, and about photography and art for more than a decade, so I'm pretty familiar with how all these tools work—and their various pros, cons, and bonkers behaviors. But writing this article was actually the first time I've put so many AI image generators head-to-head with the _same prompts_. The results were fascinating, and I'm delighted to say all the apps on the list offer genuine reasons to use them.

**How to use AI image generation at work**

Interested in AI, but not quite sure how you'd use it at work? Here are a few of the ways people are turning to AI image generation in their roles:

- Generating hero images for blog posts
- Creating social media posts
- Generating slide decks and storyboards
- Creating personalized images for customers

Learn more about [how to use AI image generation at work](https://zapier.com/blog/ai-image-examples-for-business/).

## The best AI image generators at a glance

|  | **Best for** | **Access options** | **Price** | **Parent company** |
| --- | --- | --- | --- | --- |
| [ChatGPT (GPT-4o)](https://zapier.com/blog/best-ai-image-generator/#gpt-4o) | Ease of use and overall quality | ChatGPT; API | Free with ChatGPT; fewer restrictions with ChatGPT Plus at $20/month | OpenAI |
| [Midjourney](https://zapier.com/blog/best-ai-image-generator/#midjourney) | Artistic results | Web app; Discord | From $10/month for ~200 images/month and commercial usage rights | Midjourney |
| [Reve](https://zapier.com/blog/best-ai-image-generator/#reve) | Adhering to prompts | Web app | 20 free credits/day; $5 for 500 credits | Reve |
| [Ideogram](https://zapier.com/blog/best-ai-image-generator/#ideogram) | Accurate text | Web app | Limited free plan; from $8/month for full-resolution download and 400 monthly priority credits | Ideogram AI |
| [Stable Diffusion](https://zapier.com/blog/best-ai-image-generator/#stable-diffusion) | Customization and control | NightCafe, Tensor.Art, Civitai, and lots of other apps; API; downloading it to a local server | Depends on the platform | Stability AI |
| [FLUX.1](https://zapier.com/blog/best-ai-image-generator/#flux) | Stable Diffusion alternative | NightCafe, Tensor.Art, Civitai, and lots of other apps; API; downloading it to a local server | Depends on the platform | Black Forest Labs |
| [Adobe Firefly](https://zapier.com/blog/best-ai-image-generator/#firefly) | Using AI-generated images in photos | firefly.adobe.com, Photoshop, Express, and other Adobe tools | Limited free credits; from $9.99 for 2,000 credits/month | Adobe |
| [Recraft](https://zapier.com/blog/best-ai-image-generator/#recraft) | Graphic design | Web app | Free for 50 credits/day; from $12/month for full features | Recraft |

## The best AI image generator overall

### [GPT-4o](https://chat.com/)(ChatGPT)

https://images.ctfassets.net/lzny33ho1g45/75DSS8gsgXORvalbs3MCyE/e5c337007c370f28ba0e27584234c762/image13.jpg?

**GPT-4o pros:**

- Incredibly easy to use and a best-in-class model
- Included with ChatGPT Plus, so you get a lot of AI for your money
- Integrates with Zapier

**GPT-4o cons:**

- Very slow
- Controls can be hit and miss
- $20/month is pricey if you don't want the rest of ChatGPT with it

After OpenAI's [DALL·E](https://zapier.com/blog/dall-e-3/) model kickstarted the text-to-image boom, it seemed to take a backseat to the company's language models. DALL·E 2 and DALL·E 3 were good when they debuted, but were both quickly overtaken by other models. But now OpenAI is back with a bang. GPT-4o, the [multimodal](https://zapier.com/blog/multimodal-ai/) model that powers [ChatGPT](https://zapier.com/blog/how-to-use-chatgpt/), can now [natively generate images](https://zapier.com/blog/chatgpt-image-generation/).

GPT-4o is one of the best image generators available. It's also ridiculously easy to use: tell ChatGPT what you want to see, and it'll create the image. Unfortunately, because GPT-4o uses an autoregression model instead of diffusion, it's much slower than the other image generators on this list—and it only generates a single image. If you're only occasionally generating a few images, this isn't a big deal, but it's worth noting.

It's really solid across the board: accurate text rendering, easy editing, understanding of numbers and position, the list goes on. GPT-4o's best feature, though, is what's caused it to go viral. It's great adhering to image prompts (and it's pretty good at adhering to regular prompts, too). If you upload a photo and direct it to create the image in the style of Picasso, Vermeer, or, yes, Studio Ghibli, it will do an exceptional job. It's also pretty good at incorporating feedback—ask it to change just one element of your image and it generally will. Compared to DALL·E 3 (which you can still use as a [GPT](https://chatgpt.com/g/g-2fkFE8rbu-dall-e)), it's a huge improvement.

In addition to GPT-4o image generation through ChatGPT, [OpenAI offers an API](https://zapier.com/blog/openai-api/), which means you can [connect ChatGPT to Zapier](https://zapier.com/apps/chatgpt/integrations) to do things like automatically create images from Google Forms or HubSpot responses—or any other apps you use. Learn more about [how to automate ChatGPT](https://zapier.com/blog/automate-chatgpt/).

**GPT-4o pricing:** Free users can access it, but if you don't want to run into limits, GPT-4o image generation is included as part of ChatGPT Plus at $20/month.

## The best AI image generator for artistic results

### Midjourney

https://images.ctfassets.net/lzny33ho1g45/5c2lxK4vhLWzfata4t1eul/5037e39582914b8b1f4be36d945085e3/image12.jpg?

**Midjourney pros:**

- Consistently produces some of the best looking AI-generated images
- The community is a great way to get inspiration

**Midjourney cons:**

- Images you generate are public by default
- [Free trials are currently suspended](https://help.midjourney.com/en/articles/8150088-is-there-a-free-trial)

For a long time, [Midjourney](https://zapier.com/blog/how-to-use-midjourney/) produced my favorite results of all of the image generators on this list. Other apps have finally caught up with its quality, but it still produces some of the most coherent, visually appealing, and interesting results with great textures and colors. It's telling that it was [the first AI image generator to win an art competition](https://www.nytimes.com/2022/09/02/technology/ai-artificial-intelligence-artists.html).

Best of all, Midjourney now has an actual web app. You no longer have to access it through Discord—though you can if you want.

Still, as you can probably guess, Midjourney isn't totally free of quirks: by default, every image you generate is posted publicly on Midjourney's Explore page and can be viewed on your profile. It gives everything a cool community aspect, but it means that anyone who cares to look can see what you're creating. While not necessarily a problem for artists, this might be a dealbreaker if you're looking to use Midjourney for business purposes.

If things still sound a bit confusing, don't worry. [Midjourney's help docs](https://docs.midjourney.com/docs/quick-start) are really good and walk you through getting started with both the web app and Discord, and they show you how to control all its various features, from selecting model versions and upscaling to using character references and its personalization tools. Once you understand the different options, the results you can get are genuinely amazing.

Midjourney's free trials are currently suspended because of [the overwhelming number of people trying to use it](https://www.theverge.com/2023/3/30/23662940/deepfake-viral-ai-misinformation-midjourney-stops-free-trials), but they're occasionally reinstated for a few days. If you miss a free trial window, the Basic Plan starts at $10/month and comes with 3.3 hours of GPU time per month, or around 200 images. You also get the option to buy additional GPU time, and you can use your images commercially.

**Midjourney pricing:** From $10/month for the Basic Plan that allows you to generate ~200 images/month and provides commercial usage rights.

**Read more:** [Midjourney vs. DALL·E 3](https://zapier.com/blog/midjourney-vs-dalle/)

## The best AI image generator for adhering to prompts

### Reve

https://images.ctfassets.net/lzny33ho1g45/1rErUICKuzBtIoT0x1EmHf/4e9492bc64da35bec554e2eb16f4ca02/image7.jpg?

**Reve Image 1.0 pros:**

- Great prompt adherence
- Free plan plus affordable credit system

**Reve Image 1.0 cons:**

- Images you generate are public by default

Reve Image 1.0 is a new image model that essentially came out of nowhere in March 2025. It instantly jumped to the top of Artificial Analysis's leaderboard—until it was replaced by GPT-4o a few days later. Still, Image 1.0 is an incredibly powerful image generator with best-in-class prompt adherence.

In plain English, that means Reve Image 1.0 is able to stick closely to the prompt you give it. If you ask for, say, an image with a warrior holding a sword and a wizard holding a staff, that's what you'll get—not a warrior with a staff and a wizard with a sword. This kind of adherence has been a struggle for image generators, especially as prompts get longer and more complicated. I was pretty blown away by just how many details Image 1.0 could manage.

On top of that, Image 1.0 is great with text, different styles, and photorealism. Really, the only area it falls short is with editing. While you can edit a prompt or instruct the model to do something differently, it isn't as effective as GPT-4o or Midjourney at incorporating these changes.

Reve Image 1.0 also represents a return to credit-based pricing, which had fallen out of vogue. You get 100 free credits to start and 20 credits per day. Packs of 500 credits cost $5. Each credit is good for one image, though be warned: on the default settings, you generate four images for every prompt.

**Reve Image 1.0 pricing:** Free for 20 credits per day; additional credits are $5/500

## Best AI image generator for accurate text

### Ideogram

https://images.ctfassets.net/lzny33ho1g45/7xaiByWYInfO3qQnxkpn9O/05f734289aa1b0f517b3f43eb74f9680/image15-ideogram.jpg?

**Ideogram pros:**

- Great looking AI-generated images—and among the most accurate text of any app
- There's a free plan

**Ideogram cons:**

- Images you generate are public by default

Although they're getting better, most AI image generators still struggle to generate text correctly—the diffusion process just doesn't lend itself to precisely rendering letters. Ideogram, though, has cracked it. Its latest 3.0 algorithm is able to accurately and reliably include text along with any generated image.

What makes this more impressive is Ideogram is also one of the best image generators overall. It has an intuitive web app and some nice features like an [image editor](https://docs.ideogram.ai/using-ideogram/ideogram-features/ideogram-editor) and the ability to [use any image as the basis for a new one](https://docs.ideogram.ai/using-ideogram/ideogram-features/remix). There's a new Batch Generator that allows you to upload a spreadsheet with a list of prompts, and it's beta testing a canvas feature that allows for more complex designs. In my testing, it was up there with Midjourney in terms of quality.

Ideogram even has a free plan. With it, you're limited to 10 credits a week, you have to wait a few minutes for a generation to start, and you only get Ideogram's basic features, but it's still a great way to get a feel for one of the best AI image generators available.

**Ideogram pricing:** Limited free plan; from $8/month for full-resolution download and 400 monthly priority credits.

## Best AI image generator for customization and control

### Stable Diffusion

https://images.ctfassets.net/lzny33ho1g45/4Az7EJ5gtpVyQYpyk3J7AX/5077102edb0e3f841c0d3160aeae1bd0/image3.jpeg?

**Stable Diffusion pros:**

- Widely available across AI art generator platforms
- Affordable, customizable, and super powerful with generally great results

**Stable Diffusion cons:**

- The company behind it is maybe collapsing
- There's no one easy option for using it

Unlike Midjourney and Ideogram, [Stable Diffusion](https://zapier.com/blog/how-to-use-stable-diffusion) [has an open license](https://zapier.com/blog/open-source-ai/). This means anyone with the requisite technical skills can [download some versions of it](https://github.com/CompVis/stable-diffusion) and run them locally on their own computer. It also means that you can train and fine-tune the model for specific purposes. For the past couple of years, almost all the services that use AI to generate [artistic portraits](https://land.prisma-ai.com/magic-avatars/), [historical portraits](https://www.myheritage.com/ai-time-machine), [architectural renders](https://interiorai.com/), and everything else use Stable Diffusion this way.

But this kind of open setup can also mean chaos. And that's exactly what's happened with Stability.ai, the company that was formed by some of the researchers who developed Stable Diffusion. In 2024, it was [on the verge of collapse](https://futurism.com/the-byte/stability-ai-collapsing-considering-sale), its latest model and licensing terms [had been heavily criticized](https://www.zeniteq.com/blog/stability-ais-july-2024-license-update), and most of the research team had left to form a new company (which I'll talk about next).

While Stability AI seems to have weathered the crisis for now, all this puts Stable Diffusion in a weird place. The existing versions are still some of the best models available, there are countless fine-tuned versions that make it even better for specific uses, and it's wildly popular—but I'm not sure for how much longer any of this will remain true. The latest version, Stable Diffusion 3.5, is a great model, but it's not as popular or widely available as the earlier models.

The best (or at least most stable) way to use the most popular versions of Stable Diffusion is through an image generation tool like [NightCafe](https://creator.nightcafe.studio/), [Tensor.Art](https://tensor.art/), or [Civitai](https://civitai.com/)—though you can find [lots of other apps](https://zapier.com/blog/ai-art-generator/) that will give you access to it. A lot of these platforms even give you a few free credits so you can try it out before paying. One word of warning, though: some of these platforms don't have the kind of content moderation that's typical on larger social sites. You might see some weird and NSFW things.

If you want to avoid all that or have total control, you can always download Stable Diffusion and run it locally.

**Stable Diffusion pricing:** Depends on the platform, but many offer free credits so you can try them out.

**Read more:** [Stable Diffusion vs. DALL·E](https://zapier.com/blog/stable-diffusion-vs-dalle) and [Midjourney vs. Stable Diffusion](https://zapier.com/blog/midjourney-vs-stable-diffusion).

## Best Stable Diffusion alternative

### FLUX.1

https://images.ctfassets.net/lzny33ho1g45/5xAzjYy11xVmiruodsWtSo/d7a171fd60b1ffd639829b40a87e8cc7/image5.jpg?

**FLUX.1 pros:**

- From the team behind Stable Diffusion—but without the drama
- Powerful and open

**FLUX.1 cons:**

- New and not as widely available as Stable Diffusion

As Stability.ai started collapsing, a significant portion of the team left the company to found [Black Forest Labs](https://blackforestlabs.ai/). Now, they've released their first series of text-to-image models: [FLUX.1](https://zapier.com/blog/flux-ai-image/).

In my testing, FLUX.1 is better than any version of Stable Diffusion that's widely available. It's also increasing in popularity and being embraced by the AI art community.

Right now, if you're looking to get into open AI image generation rather than just using one of the simpler text-to-image tools, I'd suggest experimenting with FLUX.1 over Stable Diffusion. FLUX.1 Schnell is released under an open Apache 2.0 license, while the larger FLUX.1 is open for non-commercial use.

Like Stable Diffusion, the simplest way to use FLUX.1 is through online AI art generators like NightCafe, Tensor.Art, and Civitai. Sign up for a free account, give it a go, and compare it side by side with some of the other models. But again, be warned that the content on these sites may not be entirely SFW.

**FLUX.1 pricing:** Depends on the platform, but many offer free credits so you can try them out.

## Best AI image generator for integrating AI-generated images into photos

### Adobe Firefly

https://images.ctfassets.net/lzny33ho1g45/4awQwjmU6tXZ9zR8TvLFCm/3835c17c8b44c9a9396d57e77701fdd5/best-ai-image-generator-image2.jpeg

**Adobe Firefly pros:**

- Integrates well with Adobe's apps, especially Photoshop
- Powerful when it's matching an image

**Adobe Firefly cons:**

- Not the best as a pure text-to-image model

Adobe has been building AI tools into its apps for more than 15 years, so it should be no surprise that it has one of the most powerful text-to-image generators—at least in terms of how it integrates with other tools. You can try its AI model, [Firefly](https://zapier.com/blog/adobe-firefly/), out on the web for free or through [Adobe Express](https://zapier.com/blog/adobe-express-ai), but it's at its best in the latest version of Photoshop.

Firefly has a few tricks up its sleeve. In addition to being capable of generating new images from a detailed text description, it can create text effects from a written prompt (think, the word "TOAST" written with letters that look like they're made from toast), recolor vector artwork, or add AI-generated elements to your images. You can test all these out through the web app, but it's that last feature where Firefly stands out.

Taken purely as a text-to-image generator, Firefly's results can be pretty hit and miss. It can match the best image generators like Midjourney for some prompts, but for others, I question what it was aiming to do. On the other hand, its integration with Photoshop, the industry standard image editor, is next level.

The two best features are Generative Fill and Generative Expand. With Generative Fill, you use Photoshop's regular tools to select an area of your image, and then, just by clicking a button and typing a prompt, you can replace it with something else. With Generative Expand, you can add to the outside of your image. Crucially, both tools understand the context of your image. In the screenshot above, you can see that Photoshop has matched the depth-of-field blur for the forest I added using Generative Fill. It looks cohesive.

As much as DALL·E and Stable Diffusion have started the conversation about image-generating AIs, Adobe's Firefly is the first implementation of an AI photo generator that really hints at what's to come. It isn't a party trick but a tool that's available to the millions of professionals who use Adobe apps every day.

**Firefly pricing:** Limited free credits; from $9.99 for Firefly Standard with 2,000 credits/month; Photoshop is available from $19.99/month as part of the Creative Cloud Photography Plan, which comes with 500 generative credits.

## The best AI image generator for graphic design

### Recraft

https://images.ctfassets.net/lzny33ho1g45/3ZU5phnoABT9vgevnLFlcG/6eb9b110464e7ed161fe54816ef0f78c/image2.png

**Recraft pros:**

- One of the most powerful and usable AI image generators
- Graphic design features are second to none

**Recraft cons:**

- More complicated to use than some of the other apps

Recraft is probably the most impressive app on this list. Its model is excellent and able to generate whatever you want, from photorealistic images to interesting logo designs. But it's the tools that Recraft has built around its model that really make it stand out.

Here's one example. Recraft allows you to create image sets that all fit the same style and color panel from a single set of prompts. You have all the style, color, and controls you need to dial things in, and it does an exceptional job right off the bat. Once you're happy with your images, you can export them as JPGs (fine), PNGs (better), or SVGs (amazing). Instead of being limited to small individual images, right from Recraft, you can create matching scalable design elements.

On top of that, you can use Recraft to create product mockups that combine multiple AI elements, in-paint and out-paint to add elements and combine images, adjust images and AI-generated work, remove backgrounds, and so much more. It's got collaboration tools, a great workspace, and you can export your work to other apps like Photoshop or Illustrator. It's a real continuation of what Adobe has done integrating Firefly into Photoshop.

**Recraft Pricing:** Free for 50 credits/day and limited features. From $12/month for Basic with 1,000 credits/month, commercial rights, and more artistic controls.

## Other AI image generators worth trying out

Over the past year, the overall standard of image generators has really improved. There are now a dozen different models that are almost equivalent in quality. I feel the seven above are the best choices for most people, but there are a handful of other apps that warrant mentioning:

- [Google Imagen 3](https://imagen.research.google/). Google's Imagen model is really great, and [if you already pay for Google Gemini](https://zapier.com/blog/gemini-for-workspace/), it's the first model you should look at.
- [Generative AI by Getty](https://www.gettyimages.ie/ai). Designed to generate commercially safe images, Generative AI by Getty is...fine. If you need images with zero commercial risk, it's worth a look—but the legal system doesn't seem to care about companies using images from Midjourney, Ideogram, or DALL·E.
- [Leonardo.Ai](http://leonardo.ai/). In addition to offering FLUX, image creation tool Leonardo.Ai has developed its own Phoenix model. It's a solid platform that just lacks a few features.
- [DALL·E 3](https://openai.com/index/dall-e-3/). DALL·E 3 is still available as a [GPT](https://zapier.com/blog/custom-chatgpt/). If you've got a soft spot for it, you can keep using it, but it's actually considered a legacy model now.
- [Luma Photon](https://lumalabs.ai/photon). Luma Photon is another great model, though I found the [Dream Machine](https://lumalabs.ai/dream-machine) app that uses it a bit too offbeat.
- [Playground](https://playground.com/design). Playground is great for creating designs, but its reliance on a template system meant I felt it was a little out of scope for the list.
- [MiniMax Image-01](https://www.minimax.io/news/image-01). Image-01 is doing really well in Artificial Analysis's leaderboards, though it's only available as an API. If you're a developer, it's worth a look.

If you want a laundry list of every AI image generator out there, including those that are built on top of all the models I've talked about, I made that too. It includes more than two dozen image generators: some are built into other tools, like [AI writing apps](https://zapier.com/blog/best-ai-writing-generator/), [photo editing apps](https://zapier.com/blog/best-ai-photo-editor/), or [stock photo sites](https://zapier.com/blog/best-free-stock-photos/); some let you [select from multiple models](https://zapier.com/blog/hugging-face/); and each one differs on how it approaches AI image generation. So if none of the apps on this list feel natural to you, check out my list of the [top AI art generators](https://zapier.com/blog/ai-art-generator/), and see if anything stands out.

## How to use an AI image generator

Ok, so you know what the best options are, but...now what? The team at Zapier has put together a bunch of resources to help you understand how to use these tools—and put them to work.

First, tutorials and walkthroughs for some of the best AI image generators:

- [How to use the ChatGPT image generator](https://zapier.com/blog/chatgpt-image-generation/)
- [How to use Midjourney](https://zapier.com/blog/how-to-use-midjourney/)
- [How to use Stable Diffusion](https://zapier.com/blog/how-to-use-stable-diffusion/)
- [How to use FLUX.1](https://zapier.com/blog/flux-ai-image/)
- [How to use Adobe Firefly](https://zapier.com/blog/adobe-firefly/)

Plus, a guide for [how to write effective AI art prompts](https://zapier.com/blog/ai-art-prompts/), so you can get what you're looking for faster (and better) when generating images.

Once you've got the basics down, it's time to use these tools for more than just creating wacky pictures. Here are some [tips for how to use AI image generators at work](https://zapier.com/blog/ai-image-examples-for-business/).

And finally, you can [automate your AI image generators](https://zapier.com/blog/automate-ai-images/), so they do their magic behind the scenes and connect to all the other apps you use.

## The legal and ethical implications of AI-generated images

AI-generated images are everywhere now, but that doesn't mean we shouldn't be asking questions about [how they should (or shouldn't) be used](https://zapier.com/blog/how-to-use-ai-badly/).

There aren't clear laws in place surrounding AI-generated images. And that goes for both sides of the coin: the U.S. Copyright Office [suggests that](https://fortune.com/2023/02/23/no-copyright-images-made-ai-artificial-intelligence/) AI-generated content isn't copyright-protected [without some kind of significant human input to the process](https://petapixel.com/2025/01/30/us-copyright-office-softens-its-stance-toward-registering-ai-generated-artworks/), and there aren't rules to protect artists whose work was scraped for AI training. (That's why Firefly was trained on licensed images and public domain content only.) They've [reaffirmed this stance](https://www.reuters.com/legal/litigation/us-copyright-office-denies-protection-another-ai-created-image-2023-09-06/), and the courts [have sided with their interpretation](https://www.reuters.com/world/us/us-appeals-court-rejects-copyrights-ai-generated-art-lacking-human-creator-2025-03-18/).

You're not likely to get into trouble for using AI-generated images for a few social media posts or blog hero images, but because there's no line drawn in the sand yet, it can be risky to develop an entire strategy around AI-generated art. (For what it's worth, [Hollywood seems to be quietly using it](https://nofilmschool.com/the-brutalist-ai).)

Then there's the [issue of bias](https://zapier.com/blog/ai-ethics/). As of now, AI has a lot of the same biases as humans, and that can lead to everything from the portrayal of stereotypes to harmful content. I experienced this myself with the outputs I got from some of the apps while testing them, though other tools take deliberate steps to add diversity to the images they generate. It's up to us as humans to avoid it by reviewing AI-generated content for bias and refining our prompts to eliminate that bias as much as possible.

## What's next for AI image generators?

AI image generating is a rapidly evolving space—and more powerful models are available each time I update this article. (I literally updated the article six weeks ago, and three new or upgraded image models dropped, so I had to update it again.) It's wild how good text-to-image models like GPT-4o, Reve, Midjourney, Ideogram, and FLUX.1 are getting at rendering tricky concepts repeatedly. While they're still a somewhat niche tool now, if they continue getting better at this pace, they could really shake things up.

**Related reading:**

- [The best AI productivity tools](https://zapier.com/blog/best-ai-productivity-tools/)
- [The best AI-powered social media management platforms](https://zapier.com/blog/best-ai-social-media-management/)
- [The best AI photo editors](https://zapier.com/blog/best-ai-photo-editor)
- [The best photo editors for iPhone and Android](https://zapier.com/blog/best-photo-editing-apps-iphone-android/)
- [The best free AI tools](https://zapier.com/blog/free-ai-tools/)
- [What is nano banana? Google's AI image generation model](https://zapier.com/blog/gemini-nano-banana)

_This article was originally published in March 2023. The most recent update was in May 2025._

[https://www.gravatar.com/avatar/68034d9772a3b2d120e7e7b08b251ea3?d=https://res.cloudinary.com/zapier-media/image/upload/f_auto/q_auto/Zapier%20logos/120px_2x_dt6fwm.png  
Harry Guinness

Harry Guinness is a writer and photographer from Dublin, Ireland. His writing has appeared in the New York Times, Lifehacker, the Irish Examiner, and How-To Geek. His photos have been published on hundreds of sites—mostly without his permission.](https://zapier.com/blog/author/harry-guinness)

**tags**

[Artificial intelligence (AI)](https://zapier.com/blog/tags/artificial-intelligence)  
[Images](https://zapier.com/blog/tags/images)

**mentioned apps**

[OpenAI](https://zapier.com/blog/apps/openai)

</details>

<details>
<summary>understanding-multimodal-llms-by-sebastian-raschka-phd</summary>

[![Ahead of AI](https://substackcdn.com/image/fetch/$s_!96vs!,w_80,h_80,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f25d0a-212b-4853-8bcb-128d0a3edbbf_1196x1196.png)](https://magazine.sebastianraschka.com/)

# [![Ahead of AI](https://substackcdn.com/image/fetch/$s_!xQ0c!,e_trim:10:white/e_trim:10:transparent/h_72,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5083e6d3-fbc9-4870-95b9-6e85d02f62a6_9366x2023.png)](https://magazine.sebastianraschka.com/)

SubscribeSign in

![User's avatar](https://substackcdn.com/image/fetch/$s_!CfW_!,w_64,h_64,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F61f4c017-506f-4e9b-a24f-76340dad0309_800x800.jpeg)

Discover more from Ahead of AI

Ahead of AI specializes in Machine Learning & AI research and is read by tens of thousands of researchers and practitioners who want to stay ahead in the ever-evolving field.

Over 129,000 subscribers

Subscribe

By subscribing, I agree to Substack's [Terms of Use](https://substack.com/tos), and acknowledge its [Information Collection Notice](https://substack.com/ccpa#personal-data-collected) and [Privacy Policy](https://substack.com/privacy).

Already have an account? Sign in

# Understanding Multimodal LLMs

### An introduction to the main techniques and latest models

[![Sebastian Raschka, PhD's avatar](https://substackcdn.com/image/fetch/$s_!CfW_!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F61f4c017-506f-4e9b-a24f-76340dad0309_800x800.jpeg)](https://substack.com/@rasbt)

[Sebastian Raschka, PhD](https://substack.com/@rasbt)

Nov 03, 2024

534

[57](https://magazine.sebastianraschka.com/p/understanding-multimodal-llms/comments)
35

Share

It was a wild two months. There have once again been many developments in AI research, with two Nobel Prizes awarded to AI and several interesting research papers published.

Among others, Meta AI released their latest Llama 3.2 models, which include open-weight versions for the 1B and 3B large language models and two multimodal models.

In this article, I aim to explain how multimodal LLMs function. Additionally, I will review and summarize roughly a dozen other recent multimodal papers and models published in recent weeks (including Llama 3.2) to compare their approaches.

(To see a table of contents menu, click on the stack of lines on the left-hand side.)

[![](https://substackcdn.com/image/fetch/$s_!Pq2z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d76dab1-362f-45b6-9b12-a12ac131edc5_1600x944.png)](https://substackcdn.com/image/fetch/$s_!Pq2z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d76dab1-362f-45b6-9b12-a12ac131edc5_1600x944.png) _An illustration of a multimodal LLM that can accept different input modalities (audio, text, images, and videos) and returns text as the output modality._

* * *

**But before we begin, I also have some exciting news to share on the personal front! My book,** _**"Build A Large Language Model (From Scratch)"**_ **, is now finally [available on Amazon](https://amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167)!**

[![](https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png)](https://substackcdn.com/image/fetch/$s_!woQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png) _[Build a Large Language Model (From Scratch)](https://amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167) now available on Amazon_

Writing this book was a tremendous effort, and I’m incredibly grateful for all the support and motivating feedback over the past two years—especially in these last couple of months, as so many kind readers have shared their feedback. Thank you all, and as an author, there is nothing more motivating than to hear that the book makes a difference in your careers!

For those who have finished the book and are eager for more, stay tuned! I’ll be adding some bonus content to the GitHub repository in the coming months.

**P.S. If you have read the book, I'd really appreciate it if you could leave a [brief review](https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167/); it truly helps us authors!**

* * *

# 1\. Use cases of multimodal LLMs

What are multimodal LLMs? As hinted at in the introduction, multimodal LLMs are large language models capable of processing multiple types of inputs, where each "modality" refers to a specific type of data—such as text (like in traditional LLMs), sound, images, videos, and more. For simplicity, we will primarily focus on the image modality alongside text inputs.

A classic and intuitive application of multimodal LLMs is image captioning: you provide an input image, and the model generates a description of the image, as shown in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!8kaL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93884822-79f1-498d-a33a-8a367ba57134_1500x1222.png)](https://substackcdn.com/image/fetch/$s_!8kaL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93884822-79f1-498d-a33a-8a367ba57134_1500x1222.png) _Example use of a multimodal LLM explaining [a meme](https://x.com/PainSci/status/1309570607458086914)._

Of course, there are many other use cases. For example, one of my favorites is extracting information from a PDF table and converting it into LaTeX or Markdown.

# 2\. Common approaches to building multimodal LLMs

There are two main approaches to building multimodal LLMs:

- Method A: Unified Embedding Decoder Architecture approach;

- Method B: Cross-modality Attention Architecture approach.


(By the way, I don’t believe official terms for these techniques exist yet, but let me know if you’ve come across any. For instance, briefer descriptions may be "decoder-only" and "cross-attention-based" approaches.)

[![](https://substackcdn.com/image/fetch/$s_!8miE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53956ae8-9cd8-474e-8c10-ef6bddb88164_1600x938.png)](https://substackcdn.com/image/fetch/$s_!8miE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53956ae8-9cd8-474e-8c10-ef6bddb88164_1600x938.png) _The two main approaches to developing multimodal LLM architectures._

As shown in the figure above, the _**Unified Embedding-Decoder Architecture**_ utilizes a single decoder model, much like an unmodified LLM architecture such as GPT-2 or Llama 3.2. In this approach, images are converted into tokens with the same embedding size as the original text tokens, allowing the LLM to process both text and image input tokens together after concatenation.

The _**Cross-Modality Attention Architecture**_ employs a cross-attention mechanism to integrate image and text embeddings directly within the attention layer.

In the following sections, we will explore how these methods work on a conceptual level. Then, we will look at recent research papers on multimodal LLMs to see how they are applied in practice.

## **2.1 Method A: Unified Embedding Decoder Architecture**

Let’s begin with the unified embedding decoder architecture, illustrated again in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!Ws6n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91955021-7da5-4bc4-840e-87d080152b18_1166x1400.png)](https://substackcdn.com/image/fetch/$s_!Ws6n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91955021-7da5-4bc4-840e-87d080152b18_1166x1400.png) _Illustration of the unified embedding decoder architecture, which is an unmodified decoder-style LLM (like GPT-2, Phi-3, Gemma, or Llama 3.2) that receives inputs consisting of image token and text token embeddings._

In the unified embedding-decoder architecture, an image is converted into embedding vectors, similar to how input text is converted into embeddings in a standard text-only LLM.

For a typical text-only LLM that processes text, the text input is usually tokenized (e.g., using Byte-Pair Encoding) and then passed through an embedding layer, as shown in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!dOba!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc97009dd-cee6-455f-87fe-64c33a868e9f_986x858.png)](https://substackcdn.com/image/fetch/$s_!dOba!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc97009dd-cee6-455f-87fe-64c33a868e9f_986x858.png) _Illustration of the standard process for tokenizing text and converting it into token embedding vectors, which are subsequently passed to an LLM during training and inference._

### **2.1.1 Understanding Image encoders**

Analogous to the tokenization and embedding of text, image embeddings are generated using an image encoder module (instead of a tokenizer), as shown in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!PlBh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15e9cc2f-95de-4723-9de5-9f2af7573aaa_790x750.png)](https://substackcdn.com/image/fetch/$s_!PlBh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F15e9cc2f-95de-4723-9de5-9f2af7573aaa_790x750.png) _Illustration of the process for encoding an image into image patch embeddings._

What happens inside the image encoder shown above? To process an image, we first divide it into smaller patches, much like breaking words into subwords during tokenization. These patches are then encoded by a pretrained vision transformer (ViT), as shown in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!_DNf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffef5f8cb-c76c-4c97-9771-7fdb87d7d8cd_1600x1135.png)](https://substackcdn.com/image/fetch/$s_!_DNf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffef5f8cb-c76c-4c97-9771-7fdb87d7d8cd_1600x1135.png) _Illustration of a classic vision transformer (ViT) setup, similar to the model proposed in [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929) (2020)._

Note that ViTs are often used for classification tasks, so I included the classification head in the figure above. However, in this case, we only need the image encoder part.

### **2.1.2 The role of the linear projection module**

The "linear projection" shown in the previous figure consists of a single linear layer (i.e., a fully connected layer). The purpose of this layer is to project the image patches, which are flattened into a vector, into an embedding size compatible with the transformer encoder. This linear projection is illustrated in the figure below. An image patch, flattened into a 256-dimensional vector, is up-projected to a 768-dimensional vector.

[![](https://substackcdn.com/image/fetch/$s_!i9i4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee32d720-92d7-48c2-b39d-adf61a870075_1600x681.png)](https://substackcdn.com/image/fetch/$s_!i9i4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee32d720-92d7-48c2-b39d-adf61a870075_1600x681.png) _Illustration of a linear projection layer that projects flattened image patches from a 256-dimensional into a 768-dimensional embedding space._

For those who prefer seeing a code example, In PyTorch code, we could implement the linear projection for the image patches as follows:

```
import torch

class PatchProjectionLayer(torch.nn.Module):

    def __init__(self, patch_size, num_channels, embedding_dim):
        super().__init__()
        self.patch_size = patch_size
        self.num_channels = num_channels
        self.embedding_dim = embedding_dim
        self.projection = torch.nn.Linear(
            patch_size * patch_size * num_channels, embedding_dim
        )

    def forward(self, x):

        batch_size, num_patches, channels, height, width = x.size()
        x = x.view(batch_size, num_patches, -1)  # Flatten each patch
        x = self.projection(x)  # Project each flattened patch
        return x

# Example Usage:
batch_size = 1
num_patches = 9  # Total patches per image
patch_size = 16  # 16x16 pixels per patch
num_channels = 3  # RGB image
embedding_dim = 768  # Size of the embedding vector

projection_layer = PatchProjectionLayer(patch_size, num_channels, embedding_dim)

patches = torch.rand(
    batch_size, num_patches, num_channels, patch_size, patch_size
)

projected_embeddings = projection_layer(patches)
print(projected_embeddings.shape)

# This prints
# torch.Size([1, 9, 768])
```

If you have read my [Machine Learning Q and AI](https://www.amazon.com/Machine-Learning-AI-Essential-Questions/dp/1718503768/) book by chance, you may know there are ways to replace linear layers with convolution operations that can be implemented to be mathematically equivalent. Here, this can be especially handy as we can combine the creation of patches and projection into two lines of code:

```
layer = torch.nn.Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))

image = torch.rand(batch_size, 3, 48, 48)
projected_patches = layer(image)

print(projected_patches.flatten(-2).transpose(-1, -2).shape)
# This prints
# torch.Size([1, 9, 768])
```

### **2.1.3 Image vs text tokenization**

Now that we briefly discussed the purpose of the image encoder (and the linear projection that is part of the encoder), let's return to the text tokenization analogy from earlier and look at text and image tokenization and embedding side by side, as depicted in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!zjmg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d56ea06-d202-4eb7-9e01-9aac492ee309_1522x1206.png)](https://substackcdn.com/image/fetch/$s_!zjmg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d56ea06-d202-4eb7-9e01-9aac492ee309_1522x1206.png) _Image tokenization and embedding (left) and text tokenization and embedding (right) side by side._

As you can see in the figure above, I included an additional _**projector**_ module that follows the image encoder. This _projector_ is usually just another _**linear projection**_ layer that is similar to the one explained earlier. The purpose is to project the image encoder outputs into a dimension that matches the dimensions of the embedded text tokens, as illustrated in the figure below. (As we will see later, the projector is sometimes also called adapter, adaptor, or connector.)

[![](https://substackcdn.com/image/fetch/$s_!TaTW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d0be64c-da90-4193-86db-804f6a8a0abb_1542x1242.png)](https://substackcdn.com/image/fetch/$s_!TaTW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d0be64c-da90-4193-86db-804f6a8a0abb_1542x1242.png) _Another side-by-side comparison between image tokenization and text tokenization, where the role of the projector is to match the text token embedding dimensions._

Now that the image patch embeddings have the same embedding dimension as the text token embeddings, we can simply concatenate them as input to the LLM, as shown in the figure at the beginning of this section. Below is the same figure again for easier reference.

[![](https://substackcdn.com/image/fetch/$s_!FTft!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa219f185-211b-4569-9398-2e080e2c5619_1166x1400.png)](https://substackcdn.com/image/fetch/$s_!FTft!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa219f185-211b-4569-9398-2e080e2c5619_1166x1400.png) _After projecting the image patch tokens into the same dimension as the text token embeddings, we can simply concatenate them as input to a standard LLM._

By the way, the image encoder we discussed in this section is usually a pretrained vision transformer. A popular choice is [CLIP](https://github.com/openai/CLIP) or [OpenCLIP](https://github.com/mlfoundations/open_clip).

However, there are also versions of Method A that operate directly on patches, such as [Fuyu](https://www.adept.ai/blog/fuyu-8b), which is shown in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!LB1L!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28269d0d-b806-4ae7-bf96-b282affd7e93_1600x645.png)](https://substackcdn.com/image/fetch/$s_!LB1L!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28269d0d-b806-4ae7-bf96-b282affd7e93_1600x645.png) _Annotated figure of the Fuyu multimodal LLM that operates directly on the image patches without image encoder. (Annotated figure from [https://www.adept.ai/blog/fuyu-8b](https://www.adept.ai/blog/fuyu-8b).)_

As illustrated in the figure above, Fuyu passes the input patches directly into a linear projection (or embedding layer) to learn its own image patch embeddings rather than relying on an additional pretrained image encoder like other models and methods do. This greatly simplifies the architecture and training setup.

## **2.2 Method B: Cross-Modality Attention Architecture**

Now that we have discussed the unified embedding decoder architecture approach to building multimodal LLMs and understand the basic concept behind image encoding, let's talk about an alternative way of implementing multimodal LLMs via cross-attention, as summarized in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!7Xvv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9c06055-b959-45d1-87b2-1f4e90ceaf2d_1296x1338.png)](https://substackcdn.com/image/fetch/$s_!7Xvv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9c06055-b959-45d1-87b2-1f4e90ceaf2d_1296x1338.png) _An illustration of the Cross-Modality Attention Architecture approach to building multimodal LLMs._

In the Cross-Modality Attention Architecture method depicted in the figure above, we still use the same image encoder setup we discussed previously. However, instead of encoding the patches as input to the LLM, we connect the input patches in the multi-head attention layer via a cross-attention mechanism.

The idea is related and goes back to the original transformer architecture from the 2017 [Attention Is All You Need](https://arxiv.org/abs/1706.03762) paper, highlighted in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!JYyE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d028b95-7965-43e0-b8fc-350609a69377_1370x1582.png)](https://substackcdn.com/image/fetch/$s_!JYyE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d028b95-7965-43e0-b8fc-350609a69377_1370x1582.png) _High-level illustration of the cross-attention mechanism used in the original transformer architecture. (Annotated figure from the "Attention Is All You Need" paper: https://arxiv.org/abs/1706.03762.)_

Note that the original "Attention Is All You Need" transformer depicted in the figure above was originally developed for language translation. So, it consists of a text **en** coder (left part of the figure) that takes the sentence to be translated and generates the translation via a text **de** coder (right part of the figure). In the context of multimodal LLM, the encoder is an image encoder instead of a text encoder, but the same idea applies.

How does cross-attention work? Let's have a look at a conceptual drawing of what happens inside the regular self-attention mechanism.

[![](https://substackcdn.com/image/fetch/$s_!HqoQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff763532b-1eed-4f7d-ae2c-7783d4f4fc46_1440x1194.png)](https://substackcdn.com/image/fetch/$s_!HqoQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff763532b-1eed-4f7d-ae2c-7783d4f4fc46_1440x1194.png) _Outline of the regular self-attention mechanism. (This flow depicts one of the heads in a regular multi-head attention module.)_

In the figure above, x is the input, and _Wq_ is a weight matrix used to generate the queries ( _Q_). Similarly, _K_ stands for keys, and _V_ stands for values. A represents the attention scores matrix, and _Z_ are the inputs (x) transformed into the output context vectors. (If this seems confusing, you may find a comprehensive introduction in Chapter 3 of my [Build a Large Language Model from Scratch book](https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167/) helpful; alternatively, you may also find my article, [Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs](https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention) helpful here.)

In cross-attention, in contrast to self-attention, we have two different input sources, as illustrated in the following figure.

[![](https://substackcdn.com/image/fetch/$s_!3PZD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4cc6f4-ca9a-431b-b572-95a1fda373a7_1508x1120.png)](https://substackcdn.com/image/fetch/$s_!3PZD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4cc6f4-ca9a-431b-b572-95a1fda373a7_1508x1120.png) _Illustration of cross attention, where there can be two different inputs x1 and x2_

As illustrated in the previous two figures, in self-attention, we work with the same input sequence. In cross-attention, we mix or combine two different input sequences.

In the case of the original transformer architecture in the _Attention Is All You Need_ paper, the two inputs _x1_ and _x2_ correspond to the sequence returned by the encoder module on the left ( _x2_) and the input sequence being processed by the decoder part on the right ( _x1_). In the context of a multimodal LLM, _x2_ is the output of an image encoder. (Note that the queries usually come from the decoder, and the keys and values typically come from the encoder.)

Note that in cross-attention, the two input sequences _x1_ and _x2_ can have different numbers of elements. However, their embedding dimensions must match. If we set _x1 = x2_, this is equivalent to self-attention.

# 3\. Unified decoder and cross-attention model training

Now that we have talked a bit about the two major multimodal design choices, let's briefly talk about how we deal with the three major components during model training, which are summarized in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!e2P-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24a12032-d32e-41f6-b390-4e321e1ea29f_1600x770.png)](https://substackcdn.com/image/fetch/$s_!e2P-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24a12032-d32e-41f6-b390-4e321e1ea29f_1600x770.png) _An overview of the different components in a multimodal LLM. The components numbered 1-3 can be frozen or unfrozen during the multimodal training process._

Similar to the development of traditional text-only LLMs, the training of multimodal LLMs also involves two phases: pretraining and instruction finetuning. However, unlike starting from scratch, multimodal LLM training typically begins with a pretrained, instruction-finetuned text-only LLM as the base model.

For the image encoder, CLIP is commonly used and often remains unchanged during the entire training process, though there are exceptions, as we will explore later. Keeping the LLM part frozen during the pretraining phase is also usual, focusing only on training the projector—a linear layer or a small multi-layer perceptron. Given the projector's limited learning capacity, usually comprising just one or two layers, the LLM is often unfrozen during multimodal instruction finetuning (stage 2) to allow for more comprehensive updates. However, note that in the cross-attention-based models (Method B), the cross-attention layers are unfrozen throughout the entire training process.

After introducing the two primary approaches (Method A: Unified Embedding Decoder Architecture and Method B: Cross-modality Attention Architecture), you might be wondering which is more effective. The answer depends on specific trade-offs.

The Unified Embedding Decoder Architecture (Method A) is typically easier to implement since it doesn't require any modifications to the LLM architecture itself.

The Cross-modality Attention Architecture (Method B) is often considered more computationally efficient because it doesn't overload the input context with additional image tokens, introducing them later in the cross-attention layers instead. Additionally, this approach maintains the text-only performance of the original LLM if the LLM parameters are kept frozen during training.

We will revisit the discussion on modeling performance and response quality in a later section, where we will discuss NVIDIA's NVLM paper.

This marks the end of what turned out to be a rather extensive introduction to multimodal LLMs. As I write this, I realize that the discussion has become lengthier than initially planned, which probably makes this a good place to conclude the article.

However, to provide a practical perspective, it would be nice to examine a few recent research papers that implement these approaches. So, we will explore these papers in the remaining sections of this article.

# 4\. Recent multimodal models and methods

For the remainder of this article, I will review recent literature concerning multimodal LLMs, focusing specifically on works published in the last few weeks to maintain a reasonable scope.

Thus, this is not a historical overview or comprehensive review of multimodal LLMs but rather a brief look at the latest developments. I will also try to keep these summaries short and without too much fluff as there are 10 of them.

The conclusion section at the end of this has an overview that compares the methods used in these papers.

## **4.1 The Llama 3 Herd of Models**

_[The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)_ paper (July 31, 2024) by Meta AI came out earlier this summer, which feels like ages ago in LLM terms. However, given that they only described but did not release their multimodal models until much later, I think it's fair to include Llama 3 in this list. (Llama 3.2 models were officially announced and made available on September 25.)

The multimodal Llama 3.2 models, which come in an 11-billion and 90-billion parameter version, are image-text models that use the previously described cross-attention-based approach, which is illustrated in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!fTYU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c8578fa-70f2-474f-9e98-87621f2dce96_1600x833.png)](https://substackcdn.com/image/fetch/$s_!fTYU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c8578fa-70f2-474f-9e98-87621f2dce96_1600x833.png) _Illustration of the multimodal LLM approach used by Llama 3.2. (Annotated figure from the Llama 3 paper: https://arxiv.org/abs/2407.21783.The video and speech parts are visually occluded to focus the attention on the image part.)_

Note that while the figure also depicts video and speech as possible modalities, the models that were released as of this writing focus only on image and text.

Llama 3.2 uses the cross-attention-based approach. However, it differs a bit from what I wrote about earlier, namely that in multimodal LLM development, we usually freeze the image encoder and only update the LLM parameters during pretraining.

Here, the researchers almost take the opposite approach: they update the image encoder but do not update the language model's parameters. They write that this is intentional and done to preserve the text-only capabilities so that the 11B and 90B multimodal models can be used as drop-in replacements for the Llama 3.1 8B and 70B text-only model on text tasks.

The training itself is done in multiple iterations, starting with the Llama 3.1 text models. After adding the image encoder and projection (here called "adapter") layers, they pretrain the model on image-text data. Then, similar to the Llama 3 model text-only training (I wrote about it in [an earlier article](https://magazine.sebastianraschka.com/i/147749119/llama-overview)), they follow up with instruction and preference finetuning.

Instead of adopting a pretrained model such as CLIP as an image encoder, the researchers used a vision transformer that they pretrained from scratch. Specifically, they adopted the  ViT-H/14 variant (630 million parameters) of the classic vision transformer architecture ( [Dosovitskiy et al., 2020](https://arxiv.org/abs/2010.11929)). They then pretrained the ViT on a dataset of 2.5 billion image-text pairs over five epochs; this was done before connecting the image encoder to the LLM. (The image encoder takes 224×224 resolution images and divides them into a 14×14 grid of patches, with each patch sized at 16×16 pixels.)

As the cross-attention layers add a substantial amount of parameters, they are only added in every fourth transformer block. (For the 8B model, this adds 3B parameters, and for the 70B model, this adds 20 billion parameters.)

## **4.2 Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models**

_[The Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models](https://www.arxiv.org/abs/2409.17146)_ paper (September 25, 2024) is notable because it promises to open source not only the model weights but also the dataset and source code similar to the language-only OLMo LLM. (This is great for LLM research as it allows us to take a look at the exact training procedure and code and also lets us run ablation studies and reproduce results on the same dataset.)

If you are wondering why there are two names in the paper title, Molmo refers to the model (Multimodal Open Language Model), and PixMo (Pixels for Molmo) is the dataset.

[![](https://substackcdn.com/image/fetch/$s_!9P0w!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73337002-8feb-4f1b-a109-1407096e32c5_1104x704.png)](https://substackcdn.com/image/fetch/$s_!9P0w!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73337002-8feb-4f1b-a109-1407096e32c5_1104x704.png) _Illustration of the Molmo decoder-only approach (Method A). Annotated figure adapted from the Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models paper: https://www.arxiv.org/abs/2409.17146._

As illustrated in the figure above, the image encoder employs an off-the-shelf vision transformer, specifically CLIP. The term "connector" here refers to a "projector" that aligns image features with the language model.

Molmo streamlines the training process by avoiding multiple pretraining stages, choosing instead a simple pipeline that updates all parameters in a unified approach—including those of the base LLM, the connector, and the image encoder.

The Molmo team offers several options for the base LLM:

- OLMo-7B-1024 (a fully open model backbone),

- OLMoE-1B-7B (a mixture-of-experts architecture; the most efficient model),

- Qwen2 7B (an open-weight model that performs better than OLMo-7B-1024),

- Qwen2 72B (an open-weight model and the best-performing model)


## **4.3 NVLM: Open Frontier-Class Multimodal LLMs**

NVIDIA's _[NVLM: Open Frontier-Class Multimodal LLMs](https://arxiv.org/abs/2409.11402)_ paper (September 17, 2024) is particularly interesting because, rather than focusing on a single approach, it explores both methods:

- Method A, the Unified Embedding Decoder Architecture ("decoder-only architecture," NVLM-D), and

- Method B, the Cross-Modality Attention Architecture ("cross-attention-based architecture," NVLM-X).


Additionally, they develop a hybrid approach (NVLM-H) and provide an apples-to-apples comparison of all three methods.

[![](https://substackcdn.com/image/fetch/$s_!6n6Y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45916952-b1ee-4972-a956-e45703e3fe36_1600x927.png)](https://substackcdn.com/image/fetch/$s_!6n6Y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45916952-b1ee-4972-a956-e45703e3fe36_1600x927.png) _Overview of the three multimodal approaches. (Annotated figure from the NVLM: Open Frontier-Class Multimodal LLMs paper: https://arxiv.org/abs/2409.11402)_

As summarized in the figure below, NVLM-D corresponds to Method A, and NVLM-X corresponds to Method B, as discussed earlier. The concept behind the hybrid model (NVLM-H) is to combine the strengths of both methods: an image thumbnail is provided as input, followed by a dynamic number of patches passed through cross-attention to capture finer high-resolution details.

In short, the research team find that:

- NVLM-X demonstrates superior computational efficiency for high-resolution images.

- NVLM-D achieves higher accuracy in OCR-related tasks.

- NVLM-H combines the advantages of both methods.


Similar to Molmo and other approaches, they begin with a text-only LLM rather than pretraining a multimodal model from scratch (as this generally performs better). Additionally, they use an instruction-tuned LLM instead of a base LLM. Specifically, the backbone LLM is Qwen2-72B-Instruct (to my knowledge, Molmo used the Qwen2-72B base model).

While training all LLM parameters in the NVLM-D approach, they found that for NVLM-X, it works well to freeze the original LLM parameters and train only the cross-attention layers during both pretraining and instruction finetuning.

For the image encoder, instead of using a typical CLIP model, they use [InternViT-6B](https://arxiv.org/abs/2312.14238), which remains frozen throughout all stages.

The projector is a multilayer perceptron rather than a single linear layer.

## **4.4 Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution**

The previous two papers and models, Molmo and NVLM, were based on Qwen2-72B LLM. In this paper, the Qwen research team itself announces a multimodal LLM, _[Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://arxiv.org/abs/2409.12191)_ (October 3rd, 2024).

At the core of this work is their so-called "Naive Dynamic Resolution" mechanism (the term "naive" is intentional and not a typo for "native," though "native" could also be fitting). This mechanism allows the model to handle images of varying resolutions without simple downsampling, enabling the input of images in their original resolution.

[![](https://substackcdn.com/image/fetch/$s_!Zrt8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2247e684-253a-462e-afb4-549411d5741a_1490x1068.png)](https://substackcdn.com/image/fetch/$s_!Zrt8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2247e684-253a-462e-afb4-549411d5741a_1490x1068.png) _An overview of the multimodal Qwen model, which can process input images with various different resolutions natively. (Annotated figure from the Qwen2-VL paper: https://arxiv.org/abs/2409.12191)_

The native resolution input is implemented via a modified ViT by removing the original absolute position embeddings and introducing 2D-RoPE.

They used a classic vision encoder with 675M parameters and LLM backbones of varying sizes, as shown in the table below.

[![](https://substackcdn.com/image/fetch/$s_!NdAJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ce9ce4a-d7ec-476d-91cb-29b6f5440b3b_1396x482.png)](https://substackcdn.com/image/fetch/$s_!NdAJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ce9ce4a-d7ec-476d-91cb-29b6f5440b3b_1396x482.png) The components of the different Qwen2-VL models. (Annotated figure from the Qwen2-VL paper: https://arxiv.org/abs/2409.12191)

The training itself consists of 3 stages: (1) pretraining only the image encoder, (2) unfreezing all parameters (including LLM), and (3) freezing the image encoder and instruction-finetuning only the LLM.

## **4.5 Pixtral 12B**

_[Pixtral 12B](https://mistral.ai/news/pixtral-12b/)_ (September 17, 2024), which uses the Method A: Unified Embedding Decoder Architecture approach, is the first multimodal model from Mistral AI. Unfortunately, there is no technical paper or report available, but the Mistral team shared a few interesting tidbits in their [blog post](https://mistral.ai/news/pixtral-12b/).

Interestingly, they chose not to use a pretrained image encoder, instead training one with 400 million parameters from scratch. For the LLM backbone, they used the 12-billion-parameter [Mistral NeMo](https://mistral.ai/news/mistral-nemo/) model.

Similar to Qwen2-VL, Pixtral also supports variable image sizes natively, as illustrated in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!eW3C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37bb0f12-4533-4f44-8907-1da868006ff3_1144x726.png)](https://substackcdn.com/image/fetch/$s_!eW3C!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37bb0f12-4533-4f44-8907-1da868006ff3_1144x726.png) _Illustration of how Pixtral processes images of different sizes. (Annotated figure from the Pixtral blog  post: https://mistral.ai/news/pixtral-12b/)_

## **4.6 MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning**

The _[MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning](https://arxiv.org/abs/2409.20566)_ paper (September 30, 2024) provides practical tips and introduces a mixture-of-experts multimodal model alongside a dense model similar to Molmo. The models span a wide size range, from 1 billion to 30 billion parameters.

The models described in this paper focuse on Method A, a Unified Embedding Transformer Architecture, which structures inputs effectively for multimodal learning.

In addition, the paper has a series of interesting ablation studies looking into data mixtures and the effects of using coordinate tokens.

[![](https://substackcdn.com/image/fetch/$s_!fMsE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71b22b97-e901-4c5f-a9c2-67e32c867823_1402x1178.png)](https://substackcdn.com/image/fetch/$s_!fMsE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71b22b97-e901-4c5f-a9c2-67e32c867823_1402x1178.png) _Illustration of the MM1.5 approach, which includes additional coordinate tokens to denote bounding boxes. (Annotated figure from the MM1.5 paper: https://arxiv.org/abs/2409.20566.)_

## **4.7 Aria: An Open Multimodal Native Mixture-of-Experts Model**

The _[Aria: An Open Multimodal Native Mixture-of-Experts Model](https://arxiv.org/abs/2410.05993)_ paper (October 8, 2024) introduces another mixture-of-experts model approach, similar to one of the variants in the Molmo and MM1.5 lineups.

The Aria model has 24.9 billion parameters, with 3.5 billion parameters allocated per text token. The image encoder ( [SigLIP](https://arxiv.org/abs/2303.15343)) has 438-million-parameters.

This model is based on a cross-attention approach with the following overall training procedure:

1. Training the LLM backbone entirely from scratch.

2. Pretraining both the LLM backbone and the vision encoder.


## **4.8 Baichuan-Omni**

The _[Baichuan-Omni Technical Report](https://arxiv.org/abs/2410.08565)_ (October 11, 2024) introduces Baichuan-Omni, a 7-billion-parameter multimodal LLM based on Method A: the Unified Embedding Decoder Architecture approach, as shown in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!-IYi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F142c39bd-2d3f-4813-9363-5ecf616cb784_2102x1326.png)](https://substackcdn.com/image/fetch/$s_!-IYi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F142c39bd-2d3f-4813-9363-5ecf616cb784_2102x1326.png) _An overview of the Baichuan-Omni model, which can handle various input modalities. (Annotated figure from the Baichuan-Omni paper: https://arxiv.org/abs/2410.08565)_

The training process for Baichuan-Omni involves a three-stage approach:

1. **Projector training**: Initially, only the projector is trained, while both the vision encoder and the language model (LLM) remain frozen.

2. **Vision encoder training**: Next, the vision encoder is unfrozen and trained, with the LLM still frozen.

3. **Full model training**: Finally, the LLM is unfrozen, allowing the entire model to be trained end-to-end.


The model utilizes the SigLIP vision encoder and incorporates the [AnyRes](https://arxiv.org/abs/2204.07156) module to handle high-resolution images through down-sampling techniques.

While the report does not explicitly specify the LLM backbone, it is likely based on the Baichuan 7B LLM, given the model's parameter size and the naming convention.

## **4.9 Emu3: Next-Token Prediction is All You Need**

The _Emu3: Next-Token Prediction is All You Need_ paper (September 27, 2024) presents a compelling alternative to diffusion models for image generation, which is solely based on a transformer-based decoder architecture. Although it's not a multimodal LLM in the classic sense (i.e., models focused on image understanding rather than generation), Emu3 is super interesting as it demonstrates that it's possible to use transformer decoders for image generation, which is a task typically dominated by diffusion methods. (However, note that there have been other similar approaches before, such as [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://arxiv.org/abs/2406.06525).)

[![](https://substackcdn.com/image/fetch/$s_!IWU7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F775db9c7-662f-4314-a5c4-c3f5efe0238d_1056x904.png)](https://substackcdn.com/image/fetch/$s_!IWU7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F775db9c7-662f-4314-a5c4-c3f5efe0238d_1056x904.png) _Emu3 is primarily an LLM for image generation as an alternative to diffusion models. (Annotated figure from the Emu3 paper: https://arxiv.org/abs/2409.18869)_

The researchers trained Emu3 from scratch and then used [Direct Preference Optimization](https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb) (DPO) to align the model with human preferences.

The architecture includes a vision tokenizer inspired by [SBER-MoVQGAN](https://arxiv.org/abs/2209.09002). The core LLM architecture is based on Llama 2, yet it is trained entirely from scratch.

## **4.10 Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation**

We previously focused on multimodal LLMs for image understanding and just saw one example for image generation with Emu 3 above. Now, the _[Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://arxiv.org/abs/2410.13848)_ paper (October 17, 2024) introduces a framework that unifies multimodal understanding and generation tasks within a single LLM backbone.

A key feature of Janus is the decoupling of visual encoding pathways to address the distinct requirements of understanding and generation tasks. The researchers argue that image understanding tasks require high-dimensional semantic representations, while generation tasks require detailed local information and global consistency in images. By separating these pathways, Janus effectively manages these differing needs.

The model employs the SigLIP vision encoder, similar to that used in Baichuan-Omni, for processing visual inputs. For image generation, it utilizes a [Vector Quantized (VQ)](https://arxiv.org/abs/2406.06525) tokenizer to handle the generation process. The base LLM in Janus is the [DeepSeek-LLM](https://arxiv.org/abs/2401.02954) with 1.3 billion parameters.

[![](https://substackcdn.com/image/fetch/$s_!9UFg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89d62626-4386-4e73-8992-158550752ce2_1434x692.png)](https://substackcdn.com/image/fetch/$s_!9UFg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89d62626-4386-4e73-8992-158550752ce2_1434x692.png) _An overview of the unified decoder-only framework used in Janus. (Annotated figure from the Janus paper: https://arxiv.org/abs/2410.13848.)_

The training process for the model in this image follows three stages, as shown in the figure below.

[![](https://substackcdn.com/image/fetch/$s_!Da5n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fb4f079-0771-4d21-8805-fded73134983_1536x648.png)](https://substackcdn.com/image/fetch/$s_!Da5n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2fb4f079-0771-4d21-8805-fded73134983_1536x648.png) Illustration of the 3-stage training process of the Janus model. (Annotated figure from the Janus paper: https://arxiv.org/abs/2410.13848)

In Stage I, only the projector layers and image output layer are trained while the LLM, understanding, and generation encoders remain frozen. In Stage II, the LLM backbone and text output layer are unfrozen, allowing for unified pretraining across understanding and generation tasks. Finally, in Stage III, the entire model, including the SigLIP image encoder, is unfrozen for supervised fine-tuning, enabling the model to fully integrate and refine its multimodal capabilities.

# Conclusion

As you may have noticed, I almost entirely skipped both the modeling and the computational performance comparisons. First, comparing the performance of LLMs and multimodal LLMs on public benchmarks is challenging due to prevalent data contamination, meaning that the test data may have been included in the training data.

Additionally, the architectural components vary so much that making an apples-to-apples comparison is difficult. So, big kudos to the NVIDIA team for developing NVLM in different flavors, which allowed for a comparison between the decoder-only and cross-attention approaches at least.

In any case, the main takeaway from this article is that multimodal LLMs can be built successfully in many different ways. Below is a figure that summarizes the different components of the models covered in this article.

[![](https://substackcdn.com/image/fetch/$s_!R_9Y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb043e6d7-78e5-4628-987a-b333d3a58829_2224x1180.png)](https://substackcdn.com/image/fetch/$s_!R_9Y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb043e6d7-78e5-4628-987a-b333d3a58829_2224x1180.png) An overview of the different models covered in this article along with their subcomponents and training approaches.

I hope you found reading this article educational and now have a better understanding of how multimodal LLMs work!

* * *

_This magazine is a personal passion project. For those who wish to support me, please consider purchasing a copy of my [Build a Large Language Model (From Scratch) book](https://amzn.to/4fqvn0D). (I am confident that you'll get lots out of this book as it explains how LLMs work in a level of detail that is not found anywhere else.)_

[![](https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png)](https://substackcdn.com/image/fetch/$s_!woQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png) _[Build a Large Language Model (From Scratch)](https://amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167) now available on Amazon_

_If you read the book and have a few minutes to spare, I'd really appreciate a [brief review](https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167). It helps us authors a lot!_

Alternatively, I also recently enabled the paid subscription option on Substack to support this magazine directly.

**Your support means a great deal! Thank you!**

Subscribe

[![Mayank Bhaskar's avatar](https://substackcdn.com/image/fetch/$s_!bV82!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1adbac6a-06f7-49dc-a843-4e5dca5359b2_400x400.jpeg)](https://substack.com/profile/5890941-mayank-bhaskar)

[![Ricky's avatar](https://substackcdn.com/image/fetch/$s_!O52Y!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a899ecb-195a-4309-ba73-1b1014e73a9d_144x144.png)](https://substack.com/profile/130044073-ricky)

[![Ryan Callihan's avatar](https://substackcdn.com/image/fetch/$s_!Q83H!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28f8fd01-1bad-4d39-b8b8-ebc29a6c1b45_758x892.jpeg)](https://substack.com/profile/107478147-ryan-callihan)

[![Peter's avatar](https://substackcdn.com/image/fetch/$s_!xWtE!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6f12c7-a822-4f07-a344-da961d2de9fa_144x144.png)](https://substack.com/profile/14044462-peter)

[![Anthony's avatar](https://substackcdn.com/image/fetch/$s_!sSwk!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3414c88-720c-4011-9733-16d27e7b1d8d_848x848.jpeg)](https://substack.com/profile/36692218-anthony)

534 Likes∙

[35 Restacks](https://substack.com/note/p-151078631/restacks?utm_source=substack&utm_content=facepile-restacks)

534

[57](https://magazine.sebastianraschka.com/p/understanding-multimodal-llms/comments)
35

Share

#### Discussion about this post

CommentsRestacks

![User's avatar](https://substackcdn.com/image/fetch/$s_!TnFC!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Favatars%2Fdefault-light.png)

[![Xiaolong's avatar](https://substackcdn.com/image/fetch/$s_!h0SC!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec0f724f-c21e-4113-a45f-a27c74be5613_150x150.jpeg)](https://substack.com/profile/285603423-xiaolong?utm_source=comment)

[Xiaolong](https://substack.com/profile/285603423-xiaolong?utm_source=substack-feed-item)

[Nov 11, 2024](https://magazine.sebastianraschka.com/p/understanding-multimodal-llms/comment/76611658 "Nov 11, 2024, 2:17 AM")

Liked by Sebastian Raschka, PhD

Thank you for sharing it!

However, in the last "overview" diagram, the "Method" of Molmo and NVLM seems to be filled in incorrectly. That is, "Both + Hybrid" should correspond to NVLM instead of Molmo.

Expand full comment

Like (3)

Reply

Share

[1 reply by Sebastian Raschka, PhD](https://magazine.sebastianraschka.com/p/understanding-multimodal-llms/comment/76611658)

[![chamidou2k's avatar](https://substackcdn.com/image/fetch/$s_!Sl6y!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bd072b3-5b00-4d8d-838e-5914d7c1d1df_144x144.png)](https://substack.com/profile/186561238-chamidou2k?utm_source=comment)

[chamidou2k](https://substack.com/profile/186561238-chamidou2k?utm_source=substack-feed-item)

[Aug 24](https://magazine.sebastianraschka.com/p/understanding-multimodal-llms/comment/148810012 "Aug 24, 2025, 10:19 PM")

Liked by Sebastian Raschka, PhD

An excellent article to help me gain a full picture of multi modality

Expand full comment

Like (2)

Reply

Share

[55 more comments...](https://magazine.sebastianraschka.com/p/understanding-multimodal-llms/comments)

TopLatestDiscussions

[The Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison)

[From DeepSeek-V3 to Kimi K2: A Look At Modern LLM Architecture Design](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison)

Jul 19•
[Sebastian Raschka, PhD](https://substack.com/@rasbt)

1,104

[43](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison/comments)

![](https://substackcdn.com/image/fetch/$s_!LmVE!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45c50202-0e8b-4e64-8296-4e2ccf4cb287_1756x1227.png)

[Understanding Reasoning LLMs](https://magazine.sebastianraschka.com/p/understanding-reasoning-llms)

[Methods and Strategies for Building and Refining Reasoning Models](https://magazine.sebastianraschka.com/p/understanding-reasoning-llms)

Feb 5•
[Sebastian Raschka, PhD](https://substack.com/@rasbt)

1,089

[40](https://magazine.sebastianraschka.com/p/understanding-reasoning-llms/comments)

![](https://substackcdn.com/image/fetch/$s_!QwUc!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6ebc5c9-461f-4d3a-889b-b8ea4e14e5ba_1600x830.png)

[Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs](https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention)

[This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama.](https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention)

Jan 14, 2024

380

[41](https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention/comments)

![](https://substackcdn.com/image/fetch/$s_!3NS4!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69bfee26-ea3b-42a6-8a1a-6b8187852082_738x564.png)

See all

Ready for more?

Subscribe

</details>

<details>
<summary>what-are-some-real-world-applications-of-multimodal-ai</summary>

# What are some real-world applications of multimodal AI?

Multimodal AI, which processes and combines different data types like text, images, audio, and sensor inputs, has practical applications across industries. By integrating multiple data sources, these systems improve accuracy and functionality in tasks that require contextual understanding. Below are three key areas where multimodal AI is being applied effectively today.

In healthcare, multimodal AI enhances diagnostics and patient care by merging medical imaging, electronic health records (EHRs), and sensor data. For example, a system might analyze a chest X-ray (image), a patient’s symptom descriptions (text), and vital signs from wearables (sensor data) to detect pneumonia. Models like Google’s Med-PaLM 2 combine vision and language processing to interpret radiology images alongside clinical notes, reducing misdiagnosis risks. Another use case is monitoring postoperative recovery: wearable devices track movement and heart rate, while speech analysis detects pain or fatigue in a patient’s voice, enabling proactive interventions.

Autonomous vehicles rely heavily on multimodal AI to fuse data from cameras, LiDAR, radar, and GPS. A self-driving car processes road signs (visual data), pedestrian movements (video), and proximity sensor readings to navigate safely. Tesla’s Autopilot, for instance, uses neural networks to combine camera feeds with ultrasonic sensors, improving object detection in varied lighting or weather. Similarly, companies like Waymo train models to correlate map data with real-time sensor inputs, ensuring precise localization and path planning. This redundancy across modalities helps address limitations of single-sensor systems, such as camera failures in low light.

Customer service and content moderation also benefit from multimodal approaches. Virtual assistants like Amazon’s Alexa process voice commands while analyzing user history (text) to personalize responses. In moderation, platforms like YouTube use AI to flag harmful content by scanning video frames (images), audio for hate speech, and user comments (text) simultaneously. For example, a post containing violent imagery and threatening text would be detected faster than if each modality were analyzed separately. Tools like OpenAI’s CLIP enable cross-modal matching, such as linking inappropriate images to their descriptive captions, improving accuracy in filtering violations. These systems reduce reliance on manual review while scaling to handle large data volumes.

</details>

<details>
<summary>what-are-vision-language-models-nvidia-glossary</summary>

# Vision Language Models

Vision language models (VLMs) are multimodal, generative AI models capable of understanding and processing video, image, and text.

## What Are Vision Language Models?

Vision language models are multimodal AI systems built by combining a large language model (LLM) with a vision encoder, giving the LLM the ability to “see.”

With this ability, VLMs can process and provide advanced understanding of video, image, and text inputs supplied in the prompt to generate text responses.

https://www.nvidia.com/content/nvidiaGDC/us/en_US/glossary/vision-language-models/_jcr_content/root/responsivegrid/nv_container_copy/nv_image.coreimg.100.1290.png/1736201901571/metropolis-iva-diagram-vlm-glossary-ces25-3576177-r1--1-.png

Figure 1: Use cases for vision language models

Unlike traditional [computer vision](https://www.nvidia.com/en-us/glossary/computer-vision/) models, VLMs are not bound by a fixed set of classes or a specific task like classification or detection. Retrained on a vast corpus of text and image/video-caption pairs, VLMs can be instructed in natural language and used to handle many classic vision tasks plus new generative AI-powered tasks such as summarization and visual Q&A.

## Why Are Vision Language Models Important?

To understand the importance of VLMs, it’s helpful to know how past computer vision (CV) models work. Traditional convolutional neural network ( [CNN](https://www.nvidia.com/en-us/glossary/convolutional-neural-network/))-based CV models are trained for a specific task on a bounded set of classes. For example:

- A classification model that identifies whether an image contains a cat or a dog
- An optical character detection and recognition CV model that reads text in an image but doesn’t interpret the format or any visual data within a document

Previous CV models were trained for a specific purpose and did not have the ability to go beyond the task or set of classes they were developed for and trained on. If the use case changed at all or required a new class to be added to the model, a developer would have to collect and label a large number of images and retrain the model. This is an expensive, time-consuming process. Additionally, CV models don't have any natural language understanding.

VLMs bring a new class of capabilities by combining the power of foundation models, like [CLIP](https://github.com/openai/CLIP), and LLMs to have both vision and language capabilities. Out of the box, VLMs have strong zero-shot performance on a variety of vision tasks, like visual question-answering, classification, and optical character recognition. They are also extremely flexible and can be used not just on a fixed set of classes but for nearly any use case by simply changing a text prompt.

Using a VLM is very similar to interacting with an LLM. The user supplies text prompts that can be interleaved with images. The inputs are then used to generate text output. The input prompts are open-ended, allowing the user to instruct the VLM to answer questions, summarize, explain the content, or reason with the image. Users can chat back and forth with the VLM, with the ability to add images into the context of the conversation. VLMs can also be integrated into visual agents to autonomously perform vision tasks.

## How Do Vision Language Models Work?

Most VLMs follow an architecture with three parts:

- A vision encoder
- A projector
- An LLM

The vision encoder is typically a CLIP-based model with a transformer architecture that has been trained on millions of image-text pairs, giving it the ability to associate images and text. The projector is a set of layers that translates the output of the vision encoder into a form the LLM can understand, often interpreted as image tokens. This projector can be a simple line layer like LLaVA and VILA, or something more complex like the cross-attention layers used in Llama 3.2 Vision.

Any off-the-shelf LLM can be used to build a VLM. There are hundreds of VLM variants that combine various LLMs with vision encoders.

https://www.nvidia.com/content/nvidiaGDC/us/en_US/glossary/vision-language-models/_jcr_content/root/responsivegrid/nv_container_copy_co_300503066/nv_image.coreimg.svg/1736168815674/vlm-architecture-diagram.svg

Figure 2: A common three-part architecture for vision language models

## How Are Vision Language Models Trained?

VLMs are trained in several stages that include pretraining, followed by supervised fine-tuning. Optionally, parameter efficient fine-tuning (PEFT) can be applied as a final stage to create a domain-specific VLM on custom data.

The pretraining stage aligns the vision encoder, projector, and LLM to essentially speak the same language when interpreting the text and image input. This is done using large corpora of text and images with image-caption pairs and interleaved image-text data. Once the three components have been aligned through pretraining, the VLM goes through a supervised fine-tuning stage to help it understand how to respond to user prompts.

The data used in this stage are a blend of example prompts with text and/or image input and the expected response of the model. For example, this data could be prompts telling the model to describe the image or to count all the objects in the frame with the expected correct response. After this round of training, the VLM will understand how to best interpret images and respond to user prompts.

https://www.nvidia.com/content/nvidiaGDC/us/en_US/glossary/vision-language-models/_jcr_content/root/responsivegrid/nv_container_copy_co_1755415045/nv_image.coreimg.svg/1736168816034/vlm-training-process-diagram.svg

Figure 3: Training for VLMs is often done in several stages to target certain parts of the model

Once the VLM is trained, it can be used in the same way as an LLM by providing prompts that can also include images interleaved in text. The VLM will then generate a text response based on the inputs. VLMs are typically deployed with an OpenAI style REST API interface to make it easy to interact with the model.

More advanced techniques are currently being researched to enhance vision capabilities:

- Ensembling vision encoders to process image inputs
- Breaking apart high-resolution image inputs into smaller tiles for processing
- Increasing context length to improve long video understanding

All of these advancements are progressing the capabilities of VLMs from only understanding single-image input to being highly capable models that can compare and contrast images, accurately read text, understand long videos, and have strong spatial understanding.

## How Are Vision Language Models Benchmarked?

Several common benchmarks, such [MMMU](https://mmmu-benchmark.github.io/), [Video-MME](https://video-mme.github.io/home_page.html), [MathVista](https://mathvista.github.io/), [ChartQA](https://github.com/vis-nlp/ChartQA) , and [DocVQA](https://www.docvqa.org/), exist to determine how well vision-language models perform on a variety of tasks, such as:

- Visual question-answering
- Logic and reasoning
- Document understanding
- Multi-image comparisons
- Video understanding

Most benchmarks consist of a set of images with several associated questions, often posed as multiple-choice questions. The multiple-choice format is the easiest way to consistently benchmark and compare VLMs. These questions test the VLMs perception, knowledge, and reasoning capabilities. When running these benchmarks, the VLM is provided with the image, question, and several multiple-choice answers it must choose from.

https://www.nvidia.com/content/nvidiaGDC/us/en_US/glossary/vision-language-models/_jcr_content/root/responsivegrid/nv_container_copy_co_42410027/nv_image.coreimg.100.1290.jpeg/1736168816436/vlm-mmmu-ari.jpeg

Figure 4: Example multiple-choice questions for VLMs used in the MMMU benchmark

Source ( [MMMU](https://mmmu-benchmark.github.io/))

The accuracy of the VLM is the number of correct choices over the set of multiple-choice questions. Some benchmarks also include numerical questions where the VLM must perform a specific calculation and be within a certain percentage of the answer to be considered correct. Often these questions and images come from academic sources, such as college-level textbooks.

## How Are Vision Language Models Used?

VLMs are quickly becoming the go-to tool for all types of vision-related tasks due to their flexibility and natural language understanding. VLMs can be easily instructed to perform a wide variety of tasks through natural language:

1. Visual questions-answering
2. Image and video summarization
3. Parsing text and handwritten documents

Previous applications that would have required a large ensemble of specially trained models can now be accomplished with just a single VLM.

VLMs are especially good at summarizing the contents of images and can be prompted to perform specific tasks based on the contents. Take for example, an education use case—a VLM could be given an image of a handwritten math problem, and it could use its optical character recognition and reasoning capabilities to interpret the problem and produce a step-by-step guide on how to solve it. VLMs can not only understand the content of the image but also reason and perform specific tasks.

https://www.nvidia.com/content/nvidiaGDC/us/en_US/glossary/vision-language-models/_jcr_content/root/responsivegrid/nv_container_copy_co_531349501/nv_image.coreimg.svg/1736168816834/vlm-real-world-diagram.svg

Figure 5: video analytics AI agents transform video and image data into real-world insights

With vast amounts of video being produced every day, it's infeasible to review and extract insights from this volume of video that is produced by all industries. VLMs can be integrated into a larger system to build video analytics AI agents capable of detecting specific events when prompted. These systems could be used to detect malfunctioning robots in a warehouse or generate out-of-stock alerts when shelves are empty. Their general understanding goes beyond simple detection and could be used to generate automated reports. For example, an intelligent traffic system could detect, analyze, and produce reports of traffic hazards, such as fallen trees, stalled vehicles, or collisions.

VLMs can be used with technologies like graph databases to understand long videos. This helps them capture the complexity of objects and events in a video. Such systems could be used to summarize operations in a warehouse to find bottlenecks and inefficiencies or produce sports commentary for football, basketball, or soccer games.

## What Are the Challenges of Vision Language Models?

Vision language models are maturing quickly, but they still have some limitations, particularly around spatial understanding and long-context video understanding.

Most VLMs use CLIP-based models as the vision encoder, which are limited to 224x224 or 336x336 image input size. This relatively small input image makes it difficult for small objects and details to be detected. For example, an HD 1080x1920 frame from a video must be downsized or cropped to a much smaller input resolution, making it difficult to retain details for small objects or fine details. To fix this, VLMs are starting to use tiling methods that allow a big image to be broken into smaller pieces and then fed into the model. There's also ongoing research to explore the use of higher-resolution image encoders.

VLMs also have difficulty providing precise locations for objects. The training data for CLIP-based vision encoders consists mostly of short text descriptions of images, like captions. These descriptions don't include detailed, fine-grained object locations, and this limitation impacts CLIP’s spatial understanding. This is inherited by VLMs that use it as a vision encoder. New approaches are exploring the use of ensembling several vision encoders to address these limitations [2408.15998 (arxiv.org)](https://arxiv.org/pdf/2408.15998).

Long video understanding is a challenge due to the need to take into account visual information across potential hours of video to properly analyze or answer questions. Like LLMs, VLMs have limited context length meaning—only a certain number of frames from a video can be included to answer questions. Approaches to increase context length and train VLMs on more video-based data are being researched, such as LongVILA [2408.10188 (arxiv.org)](https://www.arxiv.org/pdf/2408.10188).

VLMs may not have seen enough data for very specific use cases, such as finding manufacturing defects in a specific product line. This limitation can be overcome by fine-tuning the VLM on domain-specific data or using multi-image VLMs with in-context learning to provide examples that can teach the model new information without explicitly training the model. Training the model on domain-specific data with PEFT is another technique that can be used to improve a VLM’s accuracy on custom data.

</details>

<details>
<summary>what-is-optical-character-recognition-ocr-explained</summary>

# What Is Optical Character Recognition (OCR)?

https://blog.roboflow.com/content/images/size/w1200/2024/04/image-730.webp

[Petru P.](https://blog.roboflow.com/author/potrimba/)

Published  
Nov 21, 2023  
• 7 min read

Have you ever wondered how a computer can understand the words on a photo, just like you do?  That's where Optical Character Recognition, or [OCR](https://roboflow.com/ocr?ref=blog.roboflow.com), steps in. OCR takes the text you see in images – be it from a book, a receipt, or an old letter – and turns it into something your computer can read, edit, and search.

OCR finds widespread applications in tasks such as automated data entry, document digitization, text extraction from images, invoice processing, form recognition, and enhancing accessibility for visually impaired individuals.

Let's explore the fundamentals of OCR, understanding its workings, the challenges it addresses, and why it remains a crucial component of present and future technology.

## What Is Optical Character Recognition?

Optical Character Recognition (OCR) involves converting both handwritten and typed text from various sources, including images, videos, and scanned documents like PDFs, into a digitally editable format.

The output from OCR can be used by a computer to make decisions. Common use cases of OCR include:

Using OCR to read product identifiers on an assembly line. When each identifier is read, a piece of software can update an inventory tracking system to note the package with the given identifier has arrived.

Using OCR for scanned document recognition. This involves scanning printed documents, after which OCR software converts them into searchable and editable text. This method is widely employed to automate the handling of legal documents, extract data from bank statements and invoices, and streamline tasks like invoice processing and financial record-keeping.

Using OCR for “scene text recognition”, wherein an OCR system recognizes text from natural scenes, such as street signs, storefronts, or license plates.

Using OCR for alphanumeric, printed text, such as text that was written on a typewriter, or text that was printed out. But, you can also use OCR on handwriting. This usually involves using a separate system due to the differences in handwriting compared to printed text.

https://blog.roboflow.com/content/images/2024/04/image-733.webp_Application of OCR on the text of a book._ [_Source_](https://www.edenai.co/post/optical-character-recognition-ocr-which-solution-to-choose?ref=blog.roboflow.com).

## How Optical Character Recognition Works

Let's discuss the typical steps modern OCR software uses to read text:

1. **Image pre-processing**: After an image has been collected, the image undergoes pre-processing to enhance image quality, improving recognition. Pre-processing may involve resizing, contrast enhancement, binarization, noise reduction, and other techniques.
2. **Text Detection**: Using a specialized deep-learning model trained on large datasets of images and text, the computer vision model detects regions in the input image that likely contain text. This process is usually a crucial step.
3. **Layout Analysis**: After detecting text regions, the computer vision model conducts layout analysis to determine the structure and order of the text in the image. This step ensures the preservation of context and organizes the output for readability, but is not run by all OCR systems.
4. **Text Recognition**: Detected text regions pass through a deep learning-based text recognition model, utilizing a combination of convolutional neural networks (CNNs) and recurrent neural networks (RNNs). This model recognizes individual characters and words in the input image, converting them into machine-readable text.
5. **Language Model**: The final output undergoes post-processing to remove noise, correct spelling mistakes, and enhance overall accuracy. The predicted sequence of characters may contain errors, especially for long or uncommon words. Language models, acting as word processors, refine the output by predicting the probability of a sequence of words based on the input image. Statistical models and advanced methods, including deep learning, may be employed for this purpose.

https://blog.roboflow.com/content/images/2024/04/image-738.webp_An example OCR system pipeline._

Having acquired an understanding of how OCR operates, let's examine its algorithms and investigate their operational mechanisms, covering the old and the new.

## Traditional Approaches to OCR

The first OCR algorithms rooted in image processing were typically rule-based systems. One well-known OCR that uses this approach is [Tesseract](https://github.com/tesseract-ocr/tesseract?ref=blog.roboflow.com). These systems relied on manually crafted features and heuristic rules to identify characters within images. The approach involved segmenting characters into individual units and applying a set of rules for character classification.

However, the accuracy and performance of these algorithms were often constrained due to the intricate process of developing and fine-tuning the necessary handcrafted features and rules for effective recognition.

### Tesseract

Tesseract, an open-source optical character recognition engine, originated at Hewlett-Packard Laboratories in the 1980s and subsequently became open-source in 2005.

Initially designed to recognize English text exclusively, Tesseract has evolved into a versatile OCR engine. Working from traditional image processing principles, which involves manual logic unlike the deep learning processes in modern systems, Tesseract analyzes images to identify patterns for character recognition.

First, Tesseract preprocesses the image to enhance input quality, a step which encompasses tasks like contrast improvement and noise removal. Following this, Tesseract employs feature extraction techniques, including edge detection and pattern recognition, to identify and recognize characters.

https://blog.roboflow.com/content/images/2024/04/image-741.webp_Tesseract OCR engine pipeline._ [_Source_](https://www.researchgate.net/figure/Tesseract-OCR-engine-architecture_fig4_265087843?ref=blog.roboflow.com).

## Deep Learning Approaches to Optical Character Recognition

With the rise of deep learning, the integration of neural networks into OCR systems has gained substantial popularity. In particular, deep learning methodologies like [Convolutional Neural Networks](https://blog.roboflow.com/what-is-a-convolutional-neural-network/) and Long Short-Term Memory networks are leveraged, for precise text recognition. Neural networks regularly achieve better performance than traditional OCR techniques.

In recent years, there has also been a surge in novel approaches that leverage pre-trained image and text [Transformers](https://blog.roboflow.com/what-is-a-transformer/), a deep learning architecture. Transformers are ushering in a new era of end-to-end optical word recognition.

### PaddleOCR

[Paddle OCR](https://arxiv.org/abs/2009.09941?ref=blog.roboflow.com) is an open-source engine developed by Baidu's PaddlePaddle team. Leveraging deep learning techniques, including CNNs and recurrent neural networks, Paddle OCR excels in accurate text recognition. It comprises two key components: the detector and the extractor. The detector is tasked with pinpointing text within an image or document. It employs various algorithms, such as [EAST (Efficient and Accurate Scene Text)](https://paperswithcode.com/paper/east-an-efficient-and-accurate-scene-text?ref=blog.roboflow.com) or [DB (Differentiable Binarization)](https://arxiv.org/abs/1911.08947?ref=blog.roboflow.com) detectors, to identify text regions.

https://blog.roboflow.com/content/images/2024/04/image-745.webp_DB (Differentiable Binarization) architecture._ [_Source_](https://arxiv.org/pdf/2009.09941.pdf?ref=blog.roboflow.com).

After the detector locates the text, the extractor comes into play, retrieving the text from the image. It employs a blend of Convolutional Neural Networks and Recurrent Neural Networks for precise text recognition. CNNs are utilized to extract features from the text, while RNNs play a crucial role in recognizing the sequence of characters.

https://blog.roboflow.com/content/images/2024/04/image-748.webp_CRNN Extractor architecture._ [_Source_](https://arxiv.org/pdf/1507.05717.pdf?ref=blog.roboflow.com).

Paddle OCR stands out for its remarkable speed, making it among the swiftest OCR engines. Its efficiency is attributed to the utilization of parallel computing and GPU acceleration. This feature renders it particularly suitable for extensive OCR tasks, including document scanning and image recognition. Moreover, its adaptability shines through as it can be tailored and fine-tuned for specific tasks and datasets, enhancing its versatility and robustness in various OCR applications.

### TrOCR

[Transformer-based Optical Character Recognition (TrOCR)](https://arxiv.org/abs/2109.10282?ref=blog.roboflow.com) is one of many transformer-based [OCR models](https://blog.roboflow.com/best-ocr-models-text-recognition/). In contrast to traditional OCR systems, TrOCR adopts a methodology where both input image processing and the generation of corresponding text output occur within a single model.

The encoder segment of TrOCR employs a transformer-based architecture to handle the input image, segmenting it into a grid of patches and extracting visual features from each patch. Simultaneously, the decoder component utilizes a transformer-based model to produce the relevant text output, incorporating the visual features extracted from the image.

https://blog.roboflow.com/content/images/2024/04/image-752.webp_TrOCR Architecture._ [_Source_](https://arxiv.org/pdf/2109.10282.pdf?ref=blog.roboflow.com).

This comprehensive and transformer-based methodology empowers TrOCR to attain strong performance across diverse OCR benchmarks, establishing the model as a highly dependable and effective tool for text recognition tasks.

## Advantages of Modern OCR Techniques

One of the primary advantages of OCR is its ability to automate the data entry process. Traditional manual data entry is not only time-consuming but also prone to errors. OCR technology streamlines this process by automatically extracting text from images or scanned documents, eliminating the need for human input. This automation significantly reduces the time required for tasks such as transcribing printed or handwritten text into digital formats.

In addition, OCR facilitates the digitization of documents, leading to improved efficiency in document management. By converting physical documents into digital formats, OCR enables easy storage, retrieval, and organization of information.

Digital documents are more accessible and can be quickly searched, eliminating the need for manual sorting through paper files. This advantage is particularly crucial in business settings where quick access to relevant information is essential.

## Limitations of Modern OCR Techniques

OCR systems, while proficient in recognizing printed text, often face challenges when it comes to accurately interpreting handwritten text. Handwriting is inherently diverse, varying in styles, shapes, and legibility. Unlike printed text, which follows standardized fonts and structures, handwritten text can exhibit significant variability, making it difficult for OCR algorithms to consistently and accurately recognize every nuance.

This limitation is particularly pronounced in scenarios where the handwriting is cursive, unconventional, or poorly formed. Overcoming this challenge requires more advanced techniques, such as integrating machine learning [models](https://blog.roboflow.com/best-ocr-models-text-recognition/) specifically trained on diverse handwritten datasets.

Furthermore, OCR systems can be sensitive to the quality of the input image and may struggle with images that have poor resolution, low contrast, or significant noise. Additionally, documents with complex layouts, multiple columns, or irregular text arrangements pose challenges for traditional OCR methods.

The image preprocessing steps performed by OCR engines, such as Tesseract, are crucial for improving recognition accuracy, but they may not always suffice for images with inherent complexities. Complex layouts can disrupt the OCR's ability to accurately segment text regions and extract meaningful content, leading to errors in character recognition.

To mitigate these issues, additional preprocessing techniques or more advanced OCR methods may be necessary, adding complexity to the implementation process.

## Optical Character Recognition

Optical Character Recognition (OCR) is the extraction of text from scanned documents or images, converting it into machine-readable data to enhance information accessibility.

OCR can reduce the time and resources needed for managing non-searchable or elusive data, eliminating manual data input, reducing errors, and boosting productivity. However, challenges such as handwritten text recognition and sensitivity to image quality persist in OCR systems.

Despite these challenges, OCR remains pivotal in present and future technology, automating data entry, improving document management, and enhancing accessibility. Its adaptability and multilingual support position OCR as a fundamental component in shaping technological advancements.

### **Cite this Post**

Use the following entry to cite this post in your research:

_[Petru P.](https://blog.roboflow.com/author/potrimba/). (Nov 21, 2023)._
_What Is Optical Character Recognition (OCR)?. Roboflow Blog: https://blog.roboflow.com/what-is-optical-character-recognition-ocr/_

### Written by

Petru P.

Machine Learning Engineer @ Google

</details>
