MMBench: Is Your Multi-modal Model an All-around Player?

Yuanzhan LiuHaodong DuanYuanhan ZhangBo LiSongyang ZhangWangbo ZhaoYike YuanJiaqi WangConghui HeZiwei Liu

article2023ECCV2,467 citations

Introduces MMBench, a bilingual benchmark that reliably assesses vision-language models across fine-grained perception and reasoning tasks using a circular evaluation strategy to eliminate option bias.

Listen

Rapid advancements in multimodal artificial intelligence have led to the widespread emergence of vision-language models capable of processing both image and text inputs. However, evaluating these models effectively remains a major challenge. Traditional objective benchmarks rely on rigid exact-matching metrics that penalize valid paraphrased answers, while human subjective reviews are expensive, difficult to scale, and susceptible to annotator bias. In addition, existing benchmarks rarely provide granular assessments across diverse cognitive skills.

The article introduces and evaluates MMBench, a bilingual objective benchmark designed to provide a comprehensive, fine-grained, and robust assessment of vision-language models across perception and reasoning capabilities.

To establish a dependable evaluation, the creators constructed a curated dataset of 3,217 multiple-choice questions organized into a hierarchical taxonomy covering 20 fine-grained skills across perception and reasoning. Over 80% of the questions were newly sourced from the internet, and quality control pipelines filtered out text-only inferable prompts and flawed questions. The evaluation framework introduced two key innovations: a choice extraction mechanism powered by large language models to accurately parse free-form responses from models with poor instruction-following ability, and a circular evaluation strategy (CircularEval) that prompts a model multiple times per question with rotated answer choices, scoring a success only if the model is correct across all passes.

The benchmark revealed several critical findings across 21 evaluated models. First, circular evaluation substantially reduced random guessing artifacts and model option bias, causing typical performance drops of 7 to 27 percentage points compared to single-pass evaluations and widening discernible gaps between models. Second, InternLM-XComposer2 achieved the top overall test score among open-source models at 78.1%, outperforming proprietary solutions such as GPT-4v (74.3%) and Gemini-Pro-V (70.2%), while leading proprietary models like Qwen-VL-Max reached 75.4%. Third, the underlying language model backbone was shown to be decisive; upgrading language backbones yielded major accuracy gains across identical visual architectures. Fourth, proprietary models demonstrated clear advantages over open-source counterparts in tasks requiring external knowledge retrieval (such as celebrity identification) and structured visual parsing (such as diagrams and tables). Finally, nearly all evaluated models struggled with low-level visual perception (e.g., comparing image brightness or sharpness) and spatial relationship reasoning.

These findings indicate that single-pass benchmarks have historically overestimated model capabilities due to selection bias and guesswork. For organizations developing or procuring visual AI solutions, performance heavily depends on the underlying language reasoning core and balanced bilingual training corpora rather than visual encoders alone. Furthermore, safety and content moderation mechanisms in commercial models introduce slight performance penalties (up to 2.4% test score variations) when models refuse to answer questions involving real individuals or sensitive scenes.

Stakeholders and researchers should adopt multi-pass consistency evaluations like CircularEval to prevent inflated metrics when benchmarking AI assets. Developers should focus research efforts on addressing systemic weaknesses across models, specifically structured document understanding, spatial reasoning, and low-level visual attribute analysis. Development teams building cross-lingual tools should also ensure training data balance, as most models showed performance drops of several percentage points on the Chinese benchmark split relative to English.

While MMBench offers high evaluation reliability—evidenced by a 91.5% alignment between human annotators and the language model choice extractor—the framework remains bounded by its multiple-choice structure. It does not directly assess free-form long-form generation or multi-turn conversational nuances. Readers can have high confidence in the relative capability rankings, but should exercise caution when extrapolating multiple-choice accuracy directly to open-ended visual generation workflows.

Nanyang Technological University National University of Singapore Shanghai Artificial Intelligence Laboratory The Chinese University of Hong Kong Zhejiang University bilingual language model large language models MMBench Vision‑Language Model CircularEval

arXiv: 2307.06281 open-compass/MMBench

Paper: VQA: Visual Question Answering, Stanislaw Antol et al. (2015). Establishes the foundational visual question answering paradigm and multiple-choice formulation that MMBench builds upon to evaluate multimodal perception and reasoning.
Paper: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, Yash Goyal et al. (2016). Identifies critical language priors and dataset shortcuts in VQA evaluation that motivate MMBench's design of circular evaluation and debiased question curation.
Paper: GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering, Drew A. Hudson et al. (2019). Pioneers fine-grained compositional reasoning metrics for vision-language models, providing the conceptual groundwork for MMBench's hierarchical skill taxonomy.
Paper: Measuring Massive Multitask Language Understanding, Dan Hendrycks et al. (2020). Defines the standardized multi-task multiple-choice evaluation framework for large language models that MMBench extends into the multimodal domain.
Paper: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, Pan Lu et al. (2022). Introduces multi-modal multi-choice reasoning across structured academic domains, directly preceding MMBench's evaluation of visual-reasoning capabilities.
Paper: Towards VQA Models That Can Read, Amanpreet Singh et al. (2019). Establishes OCR and text-dependent visual reasoning as core multimodal competencies evaluated within MMBench's fine-grained perception categories.
Paper: CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning, Justin Johnson et al. (2016). Provides a diagnostic benchmark framework for isolating spatial relationships, attribute recognition, and multi-step reasoning evaluated in MMBench.
Paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng et al. (2023). Validates the use of large language models as automated judges to interpret open-ended model outputs, foundational to MMBench's LLM-based choice extractor.

Paper: MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, Xiang Yue et al. (2023). Extends general vision-language benchmarking to college-level and professional domain expertise across heterogeneous image modalities.
Paper: Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis, Chaoyou Fu et al. (2025). Broadens static image-based multimodal evaluation benchmarks like MMBench into comprehensive temporal and dynamic video analysis.
Paper: Improved Baselines with Visual Instruction Tuning, Haotian Liu et al. (2024). Demonstrates architectural and data-scaling baselines for visual instruction tuning using benchmarks like MMBench to quantify perceptual and reasoning gains.
Paper: Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, Peng Wang et al. (2024). Applies multimodal architectural improvements across arbitrary image resolutions and documents, addressing specific low-level visual perception gaps exposed by MMBench.
Paper: LLaVA-OneVision: Easy Visual Task Transfer, Bo Li et al. (2024). Builds unified open-source multimodal models handling single-image, multi-image, and video reasoning evaluated across standard benchmarks including MMBench.
Paper: Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks, Zhe Chen et al. (2024). Scales vision foundation models and aligns them with language backbones, evaluating broad generic visual-linguistic capabilities against MMBench metrics.
Paper: Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces, Jihan Yang et al. (2025). Deepens the investigation of 3D spatial reasoning in multimodal models, targeting the critical spatial perception deficiencies identified in MMBench.
Paper: From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge, Dawei Li et al. (2025). Surveys the broader methodology, biases, and prompt design strategies of LLM-as-a-judge systems, generalizing the choice-extraction paradigm used in MMBench.

Cover for MMBench: Is Your Multi-modal Model an All-around Player?

Abstract

Large vision-language models (VLMs) have recently achieved remarkable progress, exhibiting impressive multimodal perception and reasoning abilities. However, effectively evaluating these large VLMs remains a major challenge, hindering future development in this domain. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but lack fine-grained ability assessment and robust evaluation metrics. Meanwhile, subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, which is not scalable and may display significant bias. In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of the following key features: 1. MMBench is meticulously curated with well-designed quality control schemes, surpassing existing similar benchmarks in terms of the number and variety of evaluation questions and abilities; 2. MMBench introduces a rigorous CircularEval strategy and incorporates large language models to convert free-form predictions into pre-defined choices, which helps to yield accurate evaluation results for models with limited instruction-following capabilities. 3. MMBench incorporates multiple-choice questions in both English and Chinese versions, enabling an apples-to-apples comparison of VLMs' performance under a bilingual context. To summarize, MMBench is a systematically designed objective benchmark for a robust and holistic evaluation of vision-language models. We hope MMBench will assist the research community in better evaluating their models and facilitate future progress in this area. The evalutation code of MMBench has been integrated into VLMEvalKit: this https URL.

Topics

bilingual language model large language models reasoning

Keywords

MMBench Vision‑Language Model CircularEval Bilingual Benchmark Multiple‑Choice Question Quality Control Large Language Model Ability Dimension Instruction‑Following Evaluation Strategy

1 Introduction
2 Related Work
2.1 Multimodal Datasets
2.2 Multimodal Models
3 The construction of MMBench
3.1 The Hierachical Ability Taxonomy of MMBench
3.2 Data Collection and Quality Control
3.3 MMBench Statistics
4 Evaluation Strategy
4.1 LLM-involved Choice Extraction
4.2 LLM as the Choice Extractor: A Feasibility Analysis
4.3 CircularEval Strategy
5 Evaluation Results
5.1 Experimental Setup
5.2 Main Results
5.3 Fine-grained Analysis
6 Conclusion
A More Details about the Data
A.1 Definition about Each Leaf Ability
A.2 Data Sources of MMBench
B More Details on MMBench Construction
C More Details on LLM-based Choice Extraction
D Evaluation Settings and Results
D.1 Evaluation Settings
D.2 Model Settings
D.3 More Results
References

Knowls

Knowl 1 — CircularEval Evaluation Protocol for Multiple-Choice VLMs
algorithm
CircularEval is an evaluation protocol designed to evaluate vision-language models (VLMs) on multiple-choice questions while eliminating choice-position bias and minimizing the impact of random guessing.
Input: Problem quadruple $P = (Q, C, I, A)$ , where $Q$ is the question text, $C = (c_1, c_2, \dots, c_N)$ is an ordered list of $N$ candidate choices with $2 \le N \le 4$ , $I$ is the image, and $A \in \{1, \dots, N\}$ is the index of the correct answer; Vision-Language Model $\mathcal{M}$ ; Choice Extractor $\mathcal{E}$ .
Output: Boolean evaluation outcome $S \in \{\text{Correct}, \text{Incorrect}\}$ .
for $k = 0$ to $N - 1$ do
    Construct shifted choice list $C^{(k)} = (c_{1 + k \pmod N}, c_{2 + k \pmod N}, \dots, c_{N + k \pmod N})$
    Determine updated ground-truth choice index $A^{(k)} = ((A - 1 - k) \pmod N) + 1$
    Format prompt $T^{(k)}$ from $Q$ and $C^{(k)}$
    Generate free-form response $R^{(k)} = \mathcal{M}(I, T^{(k)})$
    Extract predicted choice index $\hat{A}^{(k)} = \mathcal{E}(Q, C^{(k)}, R^{(k)})$
    if $\hat{A}^{(k)} \ne A^{(k)}$ then
        return Incorrect (early termination)
    end if
end for
return Correct
A problem is scored as correct under CircularEval if and only if the model predicts the correct choice across all $N$ circularly permuted passes. If the model makes an incorrect prediction on any pass $k$ , evaluation for that problem is terminated immediately without running the remaining passes, keeping the overall evaluation computational overhead strictly below $N\times$ single-pass inference.
Knowl 2 — Two-Step Choice Extraction for Free-Form VLM Predictions
algorithm
Because many vision-language models produce open-ended textual answers rather than discrete option letters, a two-step parsing mechanism converts free-form outputs into standardized choice labels.
Input: Question text $Q$ , candidate choice list $C = (c_1, \dots, c_N)$ , free-form VLM generation $R$ , Large Language Model $\mathcal{LLM}$ (e.g., GPT-4).
Output: Predicted choice label $L \in \{\text{'A'}, \text{'B'}, \text{'C'}, \text{'D'}, \text{'Z'}\}$ .
Attempt heuristic extraction $L_{\text{heur}} = \text{ExtractChoiceRegex}(R)$
if $L_{\text{heur}} \in \{\text{'A'}, \dots, \text{'D'}\}$ then
    return $L_{\text{heur}}$
end if
Construct extraction prompt $\Pi(Q, C, R)$ instructing $\mathcal{LLM}$ to match $R$ strictly to the literal meaning of one candidate in $C$ , or output 'Z' if no option matches
Generate extractor response $L_{\text{llm}} = \mathcal{LLM}(\Pi(Q, C, R))$
if $L_{\text{llm}} \in \{\text{'A'}, \dots, \text{'D'}, \text{'Z'}\}$ then
    return $L_{\text{llm}}$
else
    return 'Z'
end if
On a validation subset of hard samples where heuristic extraction fails, GPT-4 achieves a 91.5% choice extraction alignment rate with human annotators, compared to approximately 85% for GPT-3.5-Turbo and Qwen-Max, and 87.0% for InternLM2-7B.
Knowl 3 — MMBench Hierarchical Ability Taxonomy and Dataset Structure
definition
MMBench is a bilingual multimodal benchmark comprising 3,217 multiple-choice question-answer quadruples $(Q_i, C_i, I_i, A_i)$ organized into a three-level hierarchical taxonomy of capabilities:
- Level 1 (2 primary abilities):
  1. Perception (2,068 problems): Gathering and parsing visual information from sensory inputs.
  2. Reasoning (1,149 problems): Drawing deductive, inductive, or domain-specific conclusions from perceived visual and textual evidence.
- Level 2 (6 sub-abilities):
  1. Coarse Perception (902 problems)
  2. Fine-grained Perception (Single-instance) (720 problems)
  3. Fine-grained Perception (Cross-instance) (446 problems)
  4. Attribute Reasoning (409 problems)
  5. Relation Reasoning (432 problems)
  6. Logic Reasoning (308 problems)
- Level 3 (20 leaf abilities):
  - Under Coarse Perception: Image Style, Image Scene, Image Emotion, Image Quality, Image Topic.
  - Under Fine-grained Perception (Single-instance): Object Localization, Attribute Recognition, Celebrity Recognition, OCR.
  - Under Fine-grained Perception (Cross-instance): Spatial Relationship, Attribute Comparison, Action Recognition.
  - Under Attribute Reasoning: Physical Property Reasoning, Function Reasoning, Identity Reasoning.
  - Under Relation Reasoning: Social Relation, Physical Relation, Nature Relation.
  - Under Logic Reasoning: Structuralized Image-Text Understanding, Future Prediction.
Each leaf ability contains at least 125 questions. The benchmark is split into a public development set (dev, 40% of samples, answers released) and a hidden test set (test, 60% of samples, evaluated via server submission).
Knowl 4 — Dual-Stage Quality Control Pipeline for Benchmark Sample Filtering
algorithm
To ensure questions genuinely evaluate visual reasoning and contain no ambiguous or corrupted ground truths, MMBench uses a dual-stage filtering pipeline combining LLM text-only verification and VLM consensus verification.
Input: Candidate question-answer dataset $\mathcal{D} = \{(Q_i, C_i, I_i, A_i)\}_{i=1}^M$ , text-only LLM ensemble $\mathcal{E}_{\text{LLM}}$ , VLM ensemble $\mathcal{E}_{\text{VLM}}$ .
Output: Curated high-quality dataset $\mathcal{D}^*$ .
$\mathcal{D}^* \leftarrow \emptyset$
for each sample $(Q, C, I, A) \in \mathcal{D}$ do
    // Stage 1: Text-Only Question Detection
     $n_{\text{text\_correct}} \leftarrow 0$
    for each model $\mathcal{M} \in \mathcal{E}_{\text{LLM}}$ do
        Infer $\hat{A} = \mathcal{M}(Q, C)$ without image $I$ under CircularEval
        if $\hat{A} == A$ then
             $n_{\text{text\_correct}} \leftarrow n_{\text{text\_correct}} + 1$
        end if
    end for
    if $n_{\text{text\_correct}} > \frac{1}{2} |\mathcal{E}_{\text{LLM}}|$ then
        Perform manual inspection; discard sample if answerable without image
        continue
    end if
    // Stage 2: Flawed Sample Detection
     $n_{\text{vlm\_correct}} \leftarrow 0$
    for each model $\mathcal{V} \in \mathcal{E}_{\text{VLM}}$ do
        Infer $\hat{A}_v = \mathcal{V}(I, Q, C)$ under CircularEval
        if $\hat{A}_v == A$ then
             $n_{\text{vlm\_correct}} \leftarrow n_{\text{vlm\_correct}} + 1$
        end if
    end for
    if $n_{\text{vlm\_correct}} == 0$ then
        Perform manual inspection; discard sample if question, options, or answer are flawed
        continue
    end if
     $\mathcal{D}^* \leftarrow \mathcal{D}^* \cup \{(Q, C, I, A)\}$
end for
return $\mathcal{D}^*$
Knowl 5 — Performance Impact of CircularEval Compared to Vanilla Single-Pass Evaluation
empirical result
Evaluating models under CircularEval imposes a strict multi-pass consistency constraint that significantly reduces scores compared to standard single-pass evaluation (VanillaEval), highlighting position bias and random guessing vulnerability across vision-language models on the MMBench-dev split:
- Non-instruction-tuned or weak models: OpenFlamingo v2 drops from 36.7% Top-1 accuracy under VanillaEval to 2.6% under CircularEval (a drop of $-34.1\%$ ). VisualGLM-6B drops from 63.1% to 36.1% ( $-27.0\%$ ). MiniGPT4-7B drops from 56.8% to 32.7% ( $-24.1\%$ ). InstructBLIP-7B drops from 61.4% to 37.4% ( $-24.0\%$ ).
- High-performing open-source and proprietary models: InternLM-XComposer2 drops from 83.8% to 79.1% ( $-4.7\%$ ), Qwen-VL-Max drops from 85.1% to 76.4% ( $-8.7\%$ ), GPT-4v drops from 85.1% to 74.3% ( $-10.8\%$ ), and Gemini-Pro-V drops from 82.6% to 70.9% ( $-11.7\%$ ).
- Separation of model tiers: The performance gap between LLaVA-v1.5-13B and LLaVA-v1.5-7B is 2.1% under VanillaEval, but widens to 4.7% under CircularEval (67.2% vs. 62.5%), demonstrating that CircularEval more reliably distinguishes fine-grained model capabilities.

Knowl 6 — Vision-Language Model Accuracy Across Level-2 Abilities on MMBench-Test

data/table

The following table reports Top-1 accuracy (%) under CircularEval on the MMBench test split across the 6 Level-2 ability dimensions: Coarse Perception (CP), Fine-grained Perception Single-instance (FP-S), Fine-grained Perception Cross-instance (FP-C), Attribute Reasoning (AR), Logic Reasoning (LR), and Relation Reasoning (RR).

Model	Overall	CP	FP-S	FP-C	AR	LR	RR
Text-Only Baseline
GPT-4-Turbo (0125)	2.9	0.6	1.2	4.1	3.7	4.9	7.4
Open-Source VLMs
OpenFlamingo v2	2.3	1.1	3.5	1.5	5.3	0.0	2.7
MiniGPT4-7B	30.5	37.0	31.8	17.2	49.8	9.2	25.6
IDEFICS-9B-Instruct	35.2	48.3	31.3	29.6	47.8	11.4	25.2
VisualGLM-6B	35.4	40.2	38.5	26.2	47.8	19.6	29.5
InstructBLIP-7B	38.3	46.7	39.0	31.8	55.5	8.7	31.0
MiniGPT4-13B	38.8	44.6	42.9	23.2	64.9	8.2	32.9
InstructBLIP-13B	39.8	47.2	42.9	21.0	60.4	12.5	38.8
Qwen-VL-Chat	60.9	68.5	67.7	50.2	78.0	37.0	45.7
MiniCPM-V	61.4	65.6	69.4	51.3	70.6	35.3	59.7
LLaVA-v1.5-7B	63.4	70.0	68.0	57.7	77.6	33.2	56.2
mPLUG-Owl2	63.5	68.1	69.1	55.8	78.4	37.0	57.0
CogVLM-Chat-17B	63.6	72.8	66.6	55.4	71.4	33.7	62.0
Yi-VL-6B	65.5	72.8	72.9	56.2	75.5	41.3	55.4
LLaVA-v1.5-13B	66.9	73.1	72.4	60.3	75.5	35.9	65.5
Yi-VL-34B	68.4	72.0	78.0	54.7	81.2	38.6	68.2
LLaVA-InternLM2-20B	72.3	78.3	76.6	68.2	78.4	46.2	69.4
InternLM-XComposer2	78.1	80.4	83.5	73.0	83.7	63.6	74.4
Proprietary VLMs
Qwen-VL-Plus	64.6	66.5	79.1	50.2	73.9	42.9	57.8
Gemini-Pro-V	70.2	70.0	78.9	65.9	82.9	46.2	65.9
GPT-4v	74.3	77.6	73.8	71.5	85.3	63.6	68.6
Qwen-VL-Max	75.4	74.8	87.2	67.0	85.3	54.9	70.5

The text-only GPT-4 baseline achieves near-random overall accuracy (2.9%), validating that the benchmark questions require image understanding. InternLM-XComposer2 achieves the highest overall accuracy (78.1%) among all evaluated models. Upgrading the underlying LLM backbone within the same vision framework consistently yields large reasoning gains (e.g., LLaVA-InternLM2-20B achieves 72.3% vs. 63.4% for LLaVA-v1.5-7B).

Knowl 7 — Bilingual Performance Disparity on MMBench-CN
empirical result
Evaluating models on MMBench-CN (the human-verified Chinese translation of MMBench) reveals language-dependent performance shifts across vision-language models:
1. General performance reduction: The majority of models exhibit lower overall accuracy on MMBench-CN compared to English MMBench. For example, LLaVA-v1.5-7B drops from 63.4% to 56.9% ( $-6.5\%$ on test), and CogVLM-Chat-17B drops from 63.6% to 54.0% ( $-9.6\%$ on test). This gap stems from predominantly English-skewed multimodal pretraining and instruction tuning.
2. Resilience of high-capacity models: Models achieving $>70\%$ accuracy on the English test split exhibit a maximum English-to-Chinese performance gap of 2.0%.
3. Balanced multimodal alignment: InternLM-XComposer2 achieves 78.1% on English MMBench-test and 77.1% on MMBench-CN-test (a drop of 1.0%), demonstrating that training on balanced bilingual multimodal corpora and employing strong bilingual language backbones mitigates language-transfer performance decay.
Knowl 8 — Core Ability Bottlenecks for Vision-Language Models
limitation
Across all evaluated open-source and proprietary VLMs, the lowest maximum achieved accuracies ( $A_{\max}$ ) among all 20 Level-3 leaf abilities identify three fundamental bottlenecks in current vision-language architectures:
1. Low-level visual quality assessment ( $A_{\max} = 61.3\%$ on Image Quality): Models struggle to recognize and compare basic image-level physical attributes, such as relative brightness, contrast ratio, sharpness, and compression artifacts.
2. Structuralized visual parsing ( $A_{\max} = 61.5\%$ on Structuralized Image-Text Understanding): Models consistently fail on reading and interpreting structured visual artifacts such as charts, coordinate graphs, code diagrams, and table layouts.
3. Inter-object and 3D spatial reasoning ( $A_{\max} = 64.0\%$ on Physical Relation Reasoning, $A_{\max} = 68.0\%$ on Spatial Relationship): Models perform poorly on determining directional coordinates, 3D relative positioning, and point-of-view spatial transformations between visual entities.
Knowl 9 — Distinct Competency Gaps Between Proprietary and Open-Source Models
empirical result
Fine-grained comparison between proprietary models (GPT-4v, Gemini-Pro-V, Qwen-VL-Max) and top open-source models trained purely on open datasets (such as LLaVA-InternLM2-20B) reveals that proprietary models hold significant advantages in two specific categories:
1. Structuralized visual comprehension: Understanding and extracting quantitative data from complex diagrams, layouts, tables, and embedded code blocks.
2. Knowledge-intensive reasoning: Answering queries requiring retrieval of external world knowledge paired with visual evidence, including Celebrity Recognition, Physical Property Reasoning, and Natural Relation Reasoning.
In contrast, on standard coarse visual perception, single-object localization, and general attribute perception, top open-source models perform comparably to leading proprietary models.
Knowl 10 — Content Moderation Rejection Rates and Upper-Bound Accuracy Impact
empirical result
Proprietary VLMs implement safety refusal guardrails that trigger rejections during automated benchmark evaluation:
- Rejection rates: GPT-4v rejects answering in 1.8% of all CircularEval passes, Gemini-Pro-V in 1.6%, and Qwen-VL-Max in 0.1%.
- Dominant refusal categories: 74% of GPT-4v rejections occur in the Celebrity Recognition category (due to policies restricting the identification of real individuals), whereas Gemini-Pro-V shows no single dominant trigger category.
- Upper-bound estimation: Because CircularEval counts any refusal in a cycle as a failure for that problem, recalculating model accuracy under the assumption that all refused questions are answered correctly increases MMBench-test accuracy by $+1.9\%$ for GPT-4v (74.3% to 76.2%), $+2.4\%$ for Gemini-Pro-V (70.2% to 72.6%), and $+0.1\%$ for Qwen-VL-Max (75.4% to 75.5%). These adjustments confirm that content moderation does not significantly alter relative benchmark rankings.

Coverage note — None was omitted; all contributed aspects including benchmark taxonomy, CircularEval, LLM-based choice extraction, quality control mechanisms, bilingual splits, empirical results, and failure analyses are fully covered.

References

1.W3c school. In https://www.w3schools.com/, 2023. 17
2.01-ai. Yi-vl. https://huggingface.co/01-ai/Yi-VL-34B, 2023. 4, 8, 10, 22, 23, 24, 25, 26
3.Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957, 2019. 3
4.Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022. 4, 7, 8, 10, 22, 23, 24, 25, 26
5.Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 17
6.Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 4, 8, 10, 20, 22, 23, 24, 25, 26
7.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 4
8.Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 4, 22, 23, 24, 25, 26
9.Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 2, 3, 17
10.XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/InternLM/xtuner, 2023. 4, 10, 22, 23, 24, 25, 26
11.Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 4, 8, 10, 20, 22, 23, 24, 25, 26
12.Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024. 4, 8, 9, 10, 22, 23, 24, 25, 26
13.Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, 2022. 10, 20, 22, 23, 24, 25, 26
14.Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. 1, 8
15.Haodong Duan, Yue Zhao, Kai Chen, Yuanjun Xiong, and Dahua Lin. Mitigating representation bias in action recognition: Algorithms and benchmarks, 2022. 17
16.Jerry A Fodor. The modularity of mind. MIT press, 1983. 4
17.Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306.13394, 2023. 3, 18
18.Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans, 2023. 2
19.Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 2, 3
20.Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018. 3
21.V. Hosu, H. Lin, T. Sziranyi, and D. Saupe. Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, 29:4041–4056, 2020. 17
22.Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 4
23.Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019. 2, 3, 19, 21
24.Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017. 17
25.Alexander Kuhnle and Ann Copestake. Shapeworld-a new test methodology for multimodal language understanding. arXiv preprint arXiv:1704.04517, 2017. 17
26.Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023. 8, 10, 20, 22, 23, 24, 25, 26
27.Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 18
28.Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 4
29.Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Dual-glance model for deciphering social relationships. In Proceedings of the IEEE international conference on computer vision, pages 2650–2659, 2017. 17
30.Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023. 22, 23, 24, 25, 26
31.Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023. 17
32.Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 4, 8, 10, 22, 23, 24, 25, 26
33.Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023. 2, 4, 9, 17
34.Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. 3, 17
35.Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 2, 3, 6, 19, 21
36.Mike Oaksford and Nick Chater. Bayesian rationality: The probabilistic approach to human reasoning. Oxford University Press, 2007. 4
37.OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. 1, 2, 4, 5, 8, 10, 17, 20, 23, 24, 25, 26
38.OpenBMB. Omnilmm: Large multi-modal models for strong performance and efficient deployment. https://github.com/OpenBMB/OmniLMM, 2023. 22, 23, 24, 25, 26
39.OpenBMB. Minicpm: Unveiling the potential of end-side large language models, 2024. 8, 10, 20, 22, 23, 24, 25, 26
40.Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. 4
41.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 4
42.Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 3, 17, 19, 21
43.Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023. 22, 23, 24, 25, 26
44.Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2, 4, 5, 8, 10, 17, 20, 23, 24, 25, 26
45.InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM-techreport, 2023. 8, 9, 18
46.Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 4
47.Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. ArXiv, abs/2311.03079, 2023. 8, 10, 22, 23, 24, 25, 26
48.Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Jiao Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. 2023. 2
49.Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2, 3, 4, 22
50.Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. ArXiv, abs/2311.04257, 2023. 8, 10, 23, 24, 25, 26
51.Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 3
52.Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112, 2023. 22, 23, 24, 25, 26
53.Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 2, 4, 9
54.Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. 17
55.Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 3
56.Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2, 4, 8, 10, 20, 22, 23, 24, 25, 26

Citation

MLA

Liu, Y., et al. “MMBench: Is Your Multi-modal Model an All-around Player?”. arXiv, 2023, http://arxiv.org/abs/2307.06281v5.

APA

Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., & Lin, D. (2023). MMBench: Is Your Multi-modal Model an All-around Player?. arXiv. http://arxiv.org/abs/2307.06281v5

Chicago

Liu, Y., H. Duan, Y. Zhang, et al. 2023. “MMBench: Is Your Multi-modal Model an All-around Player?”. arXiv. http://arxiv.org/abs/2307.06281v5.

Harvard

Liu, Y. et al. (2023) “MMBench: Is Your Multi-modal Model an All-around Player?”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2307.06281v5.

Vancouver

1. Liu Y, Duan H, Zhang Y, et al (2023) MMBench: Is Your Multi-modal Model an All-around Player?. arXiv

BibTeX

@article{liu2023mmbench,
  title = {MMBench: Is Your Multi-modal Model an All-around Player?},
  author = {Liu, Yuan and Duan, Haodong and Zhang, Yuanhan and Li, Bo and Zhang, Songyang and Zhao, Wangbo and Yuan, Yike and Wang, Jiaqi and He, Conghui and Liu, Ziwei and Chen, Kai and Lin, Dahua},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2307.06281v5},
  eprint = {2307.06281}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

MMBench: Is Your Multi-modal Model an All-around Player?

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — CircularEval Evaluation Protocol for Multiple-Choice VLMs

Knowl 2 — Two-Step Choice Extraction for Free-Form VLM Predictions

Knowl 3 — MMBench Hierarchical Ability Taxonomy and Dataset Structure

Knowl 4 — Dual-Stage Quality Control Pipeline for Benchmark Sample Filtering

Knowl 5 — Performance Impact of CircularEval Compared to Vanilla Single-Pass Evaluation

Knowl 6 — Vision-Language Model Accuracy Across Level-2 Abilities on MMBench-Test

Knowl 7 — Bilingual Performance Disparity on MMBench-CN

Knowl 8 — Core Ability Bottlenecks for Vision-Language Models

Knowl 9 — Distinct Competency Gaps Between Proprietary and Open-Source Models

Knowl 10 — Content Moderation Rejection Rates and Upper-Bound Accuracy Impact

References

Citation

BibTeX

Source Code

Access the Paper