SPICE: Semantic Propositional Image Caption Evaluation
Peter AndersonBasura FernandoMark JohnsonStephen Gould
Proposes SPICE, a scene-graph-based image caption evaluation metric that correlates significantly better with human judgment than traditional n-gram methods while enabling targeted diagnostic evaluation of semantic capabilities like counting and color comprehension.
Automated image captioning has advanced rapidly, yet evaluating generated captions remains a critical bottleneck. Standard automated metrics such as Bleu, ROUGE, CIDEr, and METEOR rely heavily on surface-level n-gram overlap. Because matching phrases is neither necessary nor sufficient to convey meaning, these conventional metrics frequently misjudge caption quality, penalize descriptive detail, and fail to reward captions that humans prefer.
The article introduces and evaluates SPICE (Semantic Propositional Image Caption Evaluation), a novel automated evaluation metric designed to measure caption quality based on semantic propositional content rather than phrase overlap. The authors set out to demonstrate that evaluating the semantic content of captions—specifically the objects, attributes, and relationships they describe—better correlates with human judgment than existing n-gram metrics.
To achieve this, the approach converts both candidate and human reference captions into semantic scene graphs using dependency parsing and linguistic transformation rules. These scene graphs represent sentences as structured logical tuples denoting objects, their attributes, and their relationships. The metric then computes an F-score (a balanced measure of precision and recall) over matching tuples between the candidate and reference graphs, utilizing synonym matching to recognize related terms. The authors tested SPICE against standard metrics across four established benchmark datasets, including evaluation data from the 2015 MS COCO Captioning Challenge covering 15 competition entries, human captions, and 255,000 human quality judgments.
The analysis yielded several key findings. First, SPICE achieved a system-level correlation of 0.88 with human quality judgments on the MS COCO dataset, dramatically outperforming METEOR (0.53) and CIDEr (0.43). Second, SPICE was the only automated metric that correctly ranked human-written descriptions above all machine-generated entries and rewarded appropriate descriptive detail. Third, on pairwise classification tasks between competing models (PASCAL-50S dataset), SPICE showed the highest accuracy (68.2%) in identifying which model humans preferred. Finally, because the metric decomposes into explicit semantic categories, it enables detailed diagnostic evaluation; for example, subcategory analysis revealed that while top models effectively identified colors (sometimes surpassing human baselines), none demonstrated a meaningful capability to count objects accurately.
These findings indicate that evaluating semantic meaning aligns automated benchmarks much closer to genuine human perception. For research and technical leaders, adopting semantic evaluation mitigates the risk of selecting models that merely mimic superficial phrasing without understanding scene composition. It also provides actionable diagnostic insights into specific model weaknesses, such as counting or relational reasoning, which traditional metrics obscure.
Organizations developing or deploying vision-language systems should integrate SPICE alongside existing tools to benchmark model performance and guide targeted system improvements. Where caption fluency and grammatical correctness are also critical risks, practitioners should combine SPICE with fluency measures (such as surprisal metrics) in an ensemble evaluation. Future work should focus on integrating more advanced semantic parsers to further enhance metric precision and establishing upper-bound performance benchmarks against expert human semantic parsing.
Confidence in these findings is high for system-level ranking and model comparison on natural image datasets. However, stakeholders should note two primary limitations: SPICE relies on automated dependency parsing, which can introduce errors, and it does not explicitly penalize grammatical fluency issues on its own.
- Paper: CIDEr: Consensus-based image description evaluation, Ramakrishna Vedantam et al. (2014). Reading CIDEr is essential because SPICE directly builds upon and compares against its TF-IDF-weighted n-gram approach for image caption evaluation.
- Paper: Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering, Peter Anderson et al. (2018). Understanding bottom-up object detection and top-down attention mechanisms provides the visual feature grounding necessary to comprehend how SPICE constructs scene graphs.
- Paper: VQA: Visual Question Answering, Stanislaw Antol et al. (2015). Familiarity with the foundational Visual Question Answering benchmark helps clarify how semantic evaluation metrics like SPICE are used to probe fine-grained visual reasoning.
- Paper: CLIPScore: A Reference-free Evaluation Metric for Image Captioning, Jack Hessel et al. (2021). CLIPScore naturally extends this work by demonstrating how pretrained vision-language models can replace both reference-based metrics like SPICE and human evaluations for caption scoring.
- Paper: BERTScore: Evaluating Text Generation with BERT, Tianyi Zhang et al. (2019). BERTScore builds upon SPICE's goal of semantic evaluation by applying contextual token embeddings to measure generation quality without relying strictly on structured scene graphs.
- Paper: G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, Yang Liu et al. (2023). G-Eval continues the exploration of advanced evaluation metrics by showing how large language models can be prompted to judge natural language generation with strong human alignment.
