keyword

Denotation Graphs

A denotation graph is a semantic knowledge structure that organizes linguistic expressions into a subsumption hierarchy based on their visual denotations, defined as the set of images or scenes each expression accurately describes. In this graph, nodes represent text elements, phrases, or full descriptions alongside the specific visual instances to which they refer, while directed edges establish entailment or subset relations between broader and more specific concepts. By grounding abstract language in concrete visual references and structuring their logical inclusions, a denotation graph enables automated systems to evaluate semantic similarity, capture textual entailments, and perform complex semantic inferences over event descriptions.

1 item

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Peter Young, Alice Lai, Micah Hodosh, J. Hockenmaier

OrganizationsUniversity of Illinois Urbana-Champaign

Why you should read this

Proposes a method for grounding semantic inference in visual denotation graphs constructed from paired image captions, yielding novel denotational similarity metrics that outperform standard distributional models in recognizing textual entailment and semantic textual similarity.

We propose to use the visual denotations of linguistic expressions (i.e. the set of images they describe) to define novel denotational similarity metrics, which we show to be at least as beneficial as distributional similarities for two tasks that require semantic inference. To compute these denotational similarities, we construct a denotation graph, i.e. a subsumption hierarchy over constituents and their denotations, based on a large corpus of 30K images and 150K descriptive captions.

Source

http://www.mitpressjournals.org/doi/pdf/10.1162/tacl_a_00166

Added

2026-09-11