One-sentence Summary

Google's EmbeddingGemma Team proposes EmbeddingGemma 2, an open multimodal embedding model that unifies text, image, audio, video, and mixed-modal representations through simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression, establishes a new Pareto frontier on comprehensive embedding benchmarks across all modalities by outperforming specialist models and models two to twelve times larger, and enables flexible deployment through selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights.

Key Contributions

• EmbeddingGemma 2 is an open multimodal embedding model that maps text, images, audio, video, and mixed modality inputs into a unified space using simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression.
• Checkpoint merging along a single finetuning trajectory and a separate post-merging quantization-aware training tail-patch stage preserve finetuned quality while producing int4 Q4_0 and Q4_C checkpoints close to half-precision performance on evaluations such as MTEB(English, v2).
• Experiments across text, vision, and audio benchmarks show that EmbeddingGemma 2 sets a new Pareto frontier over specialist models and models two to twelve times larger, while selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights support resource-constrained deployment, including 191 MB peak on-device text-only memory on a Google Pixel 11 Pro.

Introduction

Embedding models are central to modern retrieval, semantic similarity, clustering, and retrieval-augmented generation because they map data into fixed-length vectors that place semantically related items close together. For text-only use, this is well established, but working with images, video, and audio traditionally requires extra tools such as automatic speech recognition or image captioning, which add cost and can miss fine-grained visual detail or nonverbal and environmental sounds.

Prior multimodal embedding models either focus on vision-text alignment without natively handling mixed-modal interactions, or they adapt multimodal large language models that are often large, slow, and poorly suited to latency-sensitive, high-throughput, or on-device applications. The authors address this gap with EmbeddingGemma 2, a 744M-parameter general-purpose multimodal embedding model that maps text, images, video, audio, and arbitrary interleaved combinations into a single embedding space. Its modular design, efficient architecture, multistage training, and quantization support make high-quality multimodal embeddings practical even in resource-constrained settings.

Dataset

• Text data. The authors use the MTEB evaluation suite, which contains 162 quality-controlled tasks. These tasks cover 250+ natural languages and 14+ programming languages, span 20 domains such as legal, medical, news, programming, and web, and include 10 task types: bitext mining, classification, clustering, instruction retrieval, multilabel classification, pair classification, reranking, retrieval, semantic text similarity, and summarization. The paper uses this data to evaluate text and code embeddings in multilingual and English settings. The excerpt does not describe a training split, mixture ratio, filtering pipeline, cropping strategy, or metadata schema for this benchmark.

• Vision data. The authors use MMEB-v2 and MIEB(lite). MMEB-v2 includes 78 tasks across image, video, and visual document modalities, including PDFs, slides, and charts, with tasks such as retrieval, classification, question answering, visual grounding, and moment retrieval. MIEB(lite) complements this with linear-probe classification, clustering, multiple choice, compositionality, and visual semantic text similarity. These datasets are used to evaluate cross-modal and mixed-modal visual embeddings, including zero-shot behavior and linear-probe measurements. The excerpt does not specify per-subset filtering rules, cropping strategies, metadata construction, or training split and mixture ratios.

• Audio data. The authors use MAEB and MSEB(Retrieval). MAEB contains 30 tasks across five acoustic domains: speech, music, environmental sound, bioacoustics, and emotion. It covers unimodal and cross-modal retrieval, reranking, clustering, and classification. MSEB(Retrieval) focuses on multilingual speech understanding and evaluates retrieval of a text passage from a spoken query in same-language and cross-language settings across 26 languages and regional locales. The paper uses these datasets to compare native audio embeddings with ASR-derived text embeddings and to evaluate truncated embeddings at reduced dimensions, including 256-dimensional and 128-dimensional settings where discussed. The excerpt does not provide training split, mixture ratio, filtering rules, cropping strategy, or metadata schema.

• Overall usage. The described datasets are evaluation benchmark suites rather than a newly constructed training corpus. The authors use them to assess modality-specific and cross-modal embedding quality, task-type performance, language coverage, and the effect of lower-dimensional embeddings.

Experiment

The experiments evaluate EmbeddingGemma 2 on broad embedding benchmarks covering text, code, images, video, visual documents, and audio, while ablations isolate the impact of key design choices. They show that shared per-layer inputs, a contrastive finetuning stage, and trainable vision encoders improve multimodal representation quality, whereas vision token budgets trade efficiency against modality specific accuracy. Across the main evaluation suite, the model provides strong general purpose embeddings relative to larger or specialist baselines, especially for multilingual text, code, vision, and native audio, with remaining weaknesses in some zero shot paralinguistic audio tasks.

Quantized int4 EmbeddingGemma 2 checkpoints remain close to the bfloat16 baseline on multilingual, English, and code text benchmarks, with per block and per channel quantization performing similarly. The clearest degradation appears in audio retrieval, where scores fall more noticeably than in text tasks, especially for cross language retrieval. EmbeddingGemma 1 shows a similar quantization trend on the text benchmark metrics it reports. Text and code embeddings are largely robust to int4 quantization. Audio retrieval is more sensitive to quantization than text retrieval, with cross language performance showing a clearer drop.

Replacing the dedicated Per-Layer Embeddings lookup table with projection-only per-layer inputs improves overall performance and helps Image, Video, and VisDoc tasks. This design also reduces parameter overhead while deriving per-layer signals from the shared embedding space. The gain is especially notable on Video and VisDoc, where projection-only scores are higher than the lookup-table variant. Projection-only achieves higher overall performance than using a dedicated PLE lookup table Projection-only outperforms the lookup variant on Image, Video, and VisDoc metrics Removing the lookup table reduces parameter overhead while improving representation quality

Extending the pre-finetuning phase for a similar training length does not improve performance relative to the initial multimodal checkpoint. Moving to the finetuning stage with contrastive and spread-out objectives raises overall, image, and video performance, while visual document scores remain nearly unchanged. This suggests that the change in recipe, rather than extra training alone, enables better representation learning. Additional distillation yields little gain from the pre-finetuning checkpoint. The finetuning stage gives the clearest improvements on image and video tasks. Visual document performance stays broadly stable across the tested stages.

Training the vision encoder, rather than keeping it frozen, improves multimodal embedding performance across image, video, and document tasks. The improvement is most noticeable for image and video tasks, while the gain on VisDoc is smaller. Trainable vision encoder settings outperform frozen vision encoder settings on overall multimodal performance. Image and video tasks benefit most from updating the vision encoder during training. The frozen vision encoder remains relatively strong on VisDoc, but trainable tuning still provides an advantage. The comparison starts from the same text pre-finetuning checkpoint, indicating that vision encoder training is the key difference.

Increasing the vision token budget at inference generally improves overall visual embedding performance, but the effect varies by modality. Visual document tasks benefit strongly from budgets larger than the training budget, while image quality tends to saturate and video performance can decline at very large budgets. Training with a smaller per-frame video budget improves video tasks when evaluated at a matching small budget, but weakens image and visual document performance. Larger inference vision token budgets improve overall performance, especially for visual document tasks. Small-budget video training helps video retrieval at small inference budgets but reduces image and visual document quality.

Experiments evaluate int4 quantization, per-layer input design, training-stage changes, vision-encoder tuning, and inference vision-token budgets across text, code, audio, image, video, and visual-document tasks. The results show that int4 quantization preserves text and code embeddings well but weakens audio retrieval, especially cross-language, while projection-only per-layer inputs improve overall performance and reduce parameter overhead, with especially clear benefits for video and visual documents. Extending pre-finetuning alone gives little gain, but moving to contrastive finetuning and training the vision encoder improves representation quality, particularly for image and video tasks. Larger inference token budgets help overall visual embeddings and visual documents, although very large budgets can saturate or hurt video performance.