LLaVA-OneVision: Easy Visual Task Transfer
Bo LiYuanhan ZhangDong GuoRenrui ZhangFeng LiHao ZhangKaichen ZhangPei-Yuan ZhangYanwei LiZiwei Liu
Presents LLaVA-OneVision, an open multimodal model that advances performance across single-image, multi-image, and video scenarios simultaneously through visual task transfer from images to videos.
Building general-purpose artificial intelligence assistants requires models that can seamlessly understand multiple visual formats, including individual photographs, sequences of related images, and continuous video streams. Historically, open-source large multimodal models have specialized in a single visual domain, often suffering significant performance degradation when applied across different formats. Concurrently, top-tier proprietary models dominate cross-scenario capabilities, leaving an open-source gap for an efficient, unified architecture capable of handling complex visual tasks across various operational contexts.
The article introduces and evaluates LLaVA-OneVision, an open-source family of large multimodal models designed to push the state-of-the-art across single-image, multi-image, and video understanding within a single, unified framework. Specifically, the authors aimed to demonstrate that careful data curation and a unified visual representation scheme enable strong transfer learning across distinct modalities without degrading core image reasoning capabilities.
To achieve this, the authors implemented a streamlined architecture connecting a pre-trained visual encoder (SigLIP) to a language model (Qwen-2) via a two-layer projection module. They introduced an adaptive representation strategy that caps the maximum visual token budget across single images (up to 7,290 tokens), multiple images (up to 8,748 tokens across up to 12 images), and video sequences (up to 6,272 tokens across up to 32 frames). The training pipeline followed a staged curriculum learning approach: initial language-image alignment, followed by high-quality synthetic knowledge injection (utilizing 4 million samples, of which 99.8% were synthetic), single-image instruction tuning on 3.2 million samples, and finally multi-modal fine-tuning across 1.6 million balanced mixed-modality samples. Models were evaluated across three parameter scales (0.5B, 7B, and 72B) across dozens of public academic and conversational benchmarks using standardized zero-shot evaluation protocols.
The findings confirm that LLaVA-OneVision establishes a new baseline for open multimodal systems, outperforming prior open models across all three vision scenarios. First, the 72B model achieves performance situated between commercial benchmarks GPT-4V and GPT-4o, notably surpassing GPT-4V on specialized single-image tasks such as document analysis (91.3% on DocVQA) and mathematical reasoning (67.5% on MathVista). Second, cross-modality tuning yielded massive improvements on multi-image tasks, where the 72B model reached 91.6% on MathVerse and 94.9% on SciVerse out-of-domain benchmarks. Third, video understanding transferred effectively from static image training, enabling competitive performance on benchmarks like VideoMME (66.2%) without requiring dedicated video architectures. Finally, the framework exhibited strong emergent cross-scenario capabilities not explicitly seen during training, such as interpreting multi-camera self-driving video, tracking visual markers in dynamic videos, and translating static images into detailed video generation instructions.
These results carry significant strategic implications for enterprise AI deployment. Organizations can reduce computational and operational complexity by deploying a single, modular open-source model capable of addressing diverse document, video, and multi-view inspection workflows rather than managing fragmented, task-specific pipelines. The finding that 99.8% synthetic pre-training data drove major knowledge gains suggests a cost-effective path around expensive, legally risky public web data scraping. Furthermore, the availability of lightweight variants (0.5B and 7B) enables deployment on resource-constrained edge devices while preserving high OCR and perceptual reasoning capabilities.
Decision-makers and engineering teams should consider adopting the LLaVA-OneVision framework as an open baseline for applications requiring multi-image reasoning, graphical user interface automation, and video QA. When designing internal pipelines, teams should adopt staged curriculum training and balance visual token allocation across modalities. For critical production workflows requiring highly open-ended, multi-turn conversational chat in the wild, organizations should conduct targeted pilot validations and implement additional preference alignment, as a noticeable performance gap relative to frontier proprietary models like GPT-4o remains in complex visual dialogue.
Confidence in these findings is supported by standardized zero-shot evaluation across standardized benchmarks using the LMMs-Eval framework. However, several constraints exist: training was executed under fixed compute budgets without exhaustive hyperparameter tuning across all sub-components, proprietary synthetic datasets created via commercial APIs cannot be fully redistributed, and models rely heavily on language backbone scaling to solve reasoning-intensive visual tasks.
- Paper: Improved Baselines with Visual Instruction Tuning, Haotian Liu et al. (2024). Reading LLaVA-1.5 provides the foundational design principles of visual instruction tuning and data efficiency that LLaVA-OneVision directly builds upon and extends.
- Paper: Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces, Jihan Yang et al. (2025). This benchmark work evaluates how advanced multimodal models process, remember, and reason about three-dimensional spaces, directly extending LLaVA-OneVision's cross-scenario capabilities.
