ReferItGame: Referring to Objects in Photographs of Natural Scenes

Sahar KazemzadehVicente OrdonezMark MattenTamara L. Berg

article2014EMNLP1,834 citations

Presents a crowdsourced two-player game that simultaneously collects and verifies a large-scale real-world dataset of natural language referring expressions for objects in photographs, accompanied by an optimization-based generation model.

Listen

Understanding how humans refer to objects in the physical world is a central requirement for developing advanced artificial intelligence in robotics, image search, and natural language interfaces. While prior research on referring expression generation focused on simple, artificial setups or isolated objects against plain backgrounds, real-world applications require systems that can describe items within complex, cluttered visual scenes. Progress in this domain has been hindered by the lack of large-scale, verified datasets capturing how people naturally identify specific objects in complex photographs.

To address this gap, the article set out to construct an efficient data collection framework, release a comprehensive real-world dataset of natural language referring expressions, and evaluate a new optimization model designed to automatically generate appropriate descriptions for objects in natural photographs.

The authors developed an interactive two-player crowdsourced game called ReferItGame, in which one player describes an outlined object and the other locates it by clicking on the scene. Successful localizations automatically verified description accuracy. The resulting dataset gathered 130,525 verified expressions across 96,654 distinct objects in 19,894 natural photographs. Using this corpus, the authors analyzed linguistic patterns and built a mathematical optimization model that jointly determines which visual attributes to mention and predicts their specific values using visual classifiers and category-specific statistical priors. The system was experimentally tested across three distinct test sets of 500 objects each, evaluating general cases, prominent categories, and scenes with multiple identical object types.

The investigation produced several key findings. First, human descriptions are remarkably concise: 50% use no secondary modifiers beyond the object name, 41% use only one attribute, 9% use two, and fewer than 1% use more. Second, attribute usage depends heavily on the object category and visual context; for example, color is frequently used for cars but rarely for mountains, and the presence of multiple identical objects increases the use of relative spatial relationships from 12% to 26%. Third, people rely consistently on prominent reference anchors, notably people, walls, and trees, when describing spatial context (such as "to the left of the man"). Fourth, the proposed generation model significantly outperformed a statistics-only baseline across all benchmarks, achieving precision rates of roughly 36% to 38% (a 24% to 31% relative improvement) and recall rates of 48% to 60% (a 23% to 30% relative improvement).

These findings demonstrate that automated generation of referring expressions in real-world environments requires modeling category-specific language habits and scene context rather than applying uniform attribute rules. Incorporating visual models alongside contextual priors reduces the risk of generating unnatural, redundant, or confusing instructions, which is critical for human-robot collaboration and automated visual search.

Organizations developing grounded vision and language technologies should adopt joint attribute-selection models that account for entry-level naming conventions and contextual clutter. Future technical development should focus on fully automated visual recognition systems to remove reliance on pre-segmented image regions and incorporate linguistic models for entry-level object naming.

The primary limitations include reliance on pre-segmented image regions rather than end-to-end visual detection, as well as occasional visual classifier errors that propagate into the final text descriptions. Additionally, human agreement on attribute matching across identical objects was measured at 72.31%, indicating natural speaker variation that caps theoretical model accuracy. Nevertheless, the scale of the dataset and the consistency of the experimental gains provide strong confidence in the core methodology and findings.

The Bishop's School University of North Carolina at Chapel Hill gameplay natural languages optimization Referring Objects Object

Kazemzadeh et al (2014).pdf

Paper: Selective Search for Object Recognition, Jasper R. R. Uijlings et al. (2013). Selective Search provides the foundational object and region proposal generation techniques that visual referring and grounding models rely on to extract candidate regions in natural scenes.
Paper: Every Picture Tells a Story: Generating Sentences from Images, Ali Farhadi et al. (2010). This work establishes the early paradigms for connecting natural language descriptions to semantic visual entities in photographs.
Paper: Indoor Segmentation and Support Inference from RGBD Images, N. Silberman et al. (2012). It provides crucial groundwork for segmenting objects and understanding spatial relationships across cluttered, real-world scenes.

Paper: Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models, Bryan A. Plummer et al. (2015). This paper builds directly upon the idea of grounding phrases to specific image regions by introducing large-scale phrase-to-bounding-box correspondences in Flickr30k Entities.
Paper: Deep visual-semantic alignments for generating image descriptions, Andrej Karpathy et al. (2015). It extends region-level visual-linguistic grounding into deep neural network architectures that jointly align image regions with sentence fragments.
Paper: Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations, Ranjay Krishna et al. (2016). It expands referring and grounded vision-language representations to an extensive scale with dense scene graphs, attributes, and relationships.
Paper: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, Shilong Liu et al. (2023). It advances referring expression comprehension into modern open-set vision-language object detection architectures, evaluating directly on datasets like RefCOCO.
Paper: Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering, Peter Anderson et al. (2018). It utilizes object-centric region features to implement bottom-up attention mechanisms for grounding visual language tasks.
Paper: A simple neural network module for relational reasoning, Adam Santoro et al. (2017). It designs neural modules explicitly targeting the relational and spatial reasoning between entities required by referring expressions.

Cover for ReferItGame: Referring to Objects in Photographs of Natural Scenes

Abstract

In this paper we introduce a new game to crowd-source natural language referring expressions. By designing a two player game, we can both collect and verify referring expressions directly within the game. To date, the game has produced a dataset containing 130,525 expressions, referring to 96,654 distinct objects, in 19,894 photographs of natural scenes. This dataset is larger and more varied than previous REG datasets and allows us to study referring expressions in real-world scenes. We provide an in depth analysis of the resulting dataset. Based on our findings, we design a new optimization based model for generating referring expressions and perform experimental evaluations on 3 test sets.

Topics

gameplay natural languages optimization optimization problems computer games real scenes

Keywords

Referring Objects Object Expressions Word Natural Referring Expressions Expression Attribute Dataset

1 Introduction
2 Related Work
3 Referring Expression Game (ReferItGame)
3.1 Game Play
3.2 Playing Against the Computer
4 ReferItGame Dataset
4.1 Images and Labels
4.2 Collecting the Dataset
4.3 Processing the Dataset
4.4 Dataset Analysis
5 Generating Referring Expressions
5.1 Generation Model
5.1.1 Content-based potentials
5.1.2 Prior statistics-based potentials
5.2 Experiments
6 Conclusions & Future Work
Acknowledgments
References

Knowls

Knowl 1 — ReferItGame Two-Player Data Collection and Verification Game
model/method
ReferItGame is a two-player online human computation game designed to crowd-source and simultaneously verify natural language referring expressions for objects in natural scene photographs.

In each game round:
- Player 1 (the describer) is presented with an image where a target object is highlighted by a segmentation outline, along with a text box to write a natural language referring expression identifying that object.
- Player 2 (the localizer) receives the same image without the object outline, along with Player 1's referring expression, and is instructed to click on the location of the described object.
- Verification and Reward: If Player 2 clicks within the ground-truth segmentation mask of the target object, both players earn game points, the referring expression is verified as valid and stored in the dataset, and the player roles swap for the next image. If Player 2 clicks outside the target object, no points are awarded and player roles remain unchanged.
A round timer encourages fast, spontaneous language generation. When only one human player is connected, single-player "canned" games pair the user against pre-recorded human referring expressions and click locations using lemmatized bag-of-words cosine similarity matching.
Knowl 2 — ReferItGame Dataset
definition
The ReferItGame dataset is a large-scale collection of natural language referring expressions grounded in real-world photographs, constructed on top of the ImageCLEF IAPR TC-12 image retrieval benchmark and the SAIAPR TC-12 segmentation annotations.

Key specifications of the dataset include:
- Total verified expressions: 130,525 successfully completed games (comprising 120,094 two-player live games and 10,431 canned single-player games against the computer).
- Distinct referred objects: 96,654 segmented objects spanning 238 annotated categories (including people, animals, vehicles, indoor objects, and natural background elements such as grass, sky, rock, and water).
- Total images: 19,894 complex natural scene photographs depicting everyday activities, landscapes, sports, and urban environments.
- Multi-expression coverage: 10,304 objects have 2 or more distinct referring expressions collected to capture inter-annotator variation, while the remaining objects have 1 collected expression.
- Quality filtering: Expressions resulting in failed localizations during gameplay are excluded from the primary dataset analyses and generation benchmarks.
Knowl 3 — 7-Tuple Representation and Template-Based Parsing for Referring Expressions
model/method
Referring expressions in natural scenes are parameterized as a 7-tuple of attribute slots $R = \{r_1, r_2, r_3, r_4, r_5, r_6, r_7\}$ :
- $r_1$ : Entry-level category attribute (the head noun naming the target object, e.g., "car", "man", "road").
- $r_2$ : Color attribute (e.g., "red", "white", "black").
- $r_3$ : Size attribute (e.g., "big", "tiny", "tall").
- $r_4$ : Absolute location attribute (e.g., "top", "bottom right", "middle").
- $r_5$ : Relative location relation attribute (e.g., "to the left of", "on top of", "behind").
- $r_6$ : Relative location reference object attribute (the landmark entity, e.g., "tree", "man", "table").
- $r_7$ : Generic attribute (infrequent modifier descriptors such as material or shape, e.g., "wooden", "round").
To parse raw text into this tuple without manual annotation, a dependency parser pipeline operates as follows:
1. Sentences are parsed using Stanford CoreNLP dependency trees. A breadth-first search identifies the head noun of the sentence as the subject word $S$ .
2. Predefined lexicons categorize observed vocabulary words into candidate attribute types using part-of-speech tagging and seed word lists.
3. Collapsed grammatical dependency relations are mapped to attributes via template rules:
  - $\text{rel}(S, \text{color\_word}) \lor \text{prep\_in}(S, \text{color\_word}) \implies r_2 = \text{color\_word}$
  - $\text{rel}(S, \text{size\_word}) \implies r_3 = \text{size\_word}$
  - $\text{rel}(S, \text{abs\_loc\_word}) \lor (\text{prep\_on}(S, \text{orientation\_word}) \land \neg\text{prep\_of}(S, \cdot)) \implies r_4 = \text{abs\_loc\_word}$
  - $\text{prep\_rel\_loc\_word}(S, \text{object\_word}) \lor (\text{prep\_(on|to|at)}(S, \text{orientation\_word}) \land \text{prep\_of}(S, \text{object\_word})) \implies r_5 = \text{orientation\_word}, r_6 = \text{object\_word}$
  - $\text{amod}(S, \text{generic\_word}) \implies r_7 = \text{generic\_word}$
This rule-based dependency parser achieves 91% accuracy when evaluated against 4,500 manually parsed human expressions.
Knowl 4 — Empirical Distributional Properties of Natural Scene Referring Expressions
empirical result
Statistical analysis of 130,525 verified natural scene referring expressions reveals key linguistic characteristics:
- Attribute sparsity: 50% of referring expressions contain 0 modifier attributes beyond the entry-level object category noun ( $r_1$ ), 41% contain 1 modifier attribute, and 9% contain 2 modifier attributes. Expressions containing more than 2 modifier attributes are exceedingly rare.
- Category dependency: The likelihood and type of modifier attributes used are highly category-specific. People and plant categories elicit significantly higher modifier frequencies than background categories such as sky or rock. Color attributes are heavily used for cars and people/clothing, but almost never for mountains or sky.
- Multi-instance effect: The presence of multiple instances of the same object class in a scene increases the usage frequency of all attribute types compared to scenes containing only a single instance of that category.
- Salience of relative reference objects: Relative location reference objects ( $r_6$ ) are strongly biased toward people ("people", "man", "guy", "woman") and salient structural background objects ("tree", "wall", "building").
- Entry-level category naming: Human expressions systematically favor psychological entry-level category nouns over formal ontology labels; for example, objects annotated as "street" are predominantly called "road" or "ground", and objects annotated as "woman" are frequently referred to as "person".
Knowl 5 — Integer Linear Programming Optimization Framework for Referring Expression Generation
model/method
Referring expression generation (REG) in natural scenes is formulated as an optimization problem that jointly selects which attribute types to include and which attribute values to assign. Given an input tuple $I = \{P, S\}$ , where $P$ is the target segmented object and $S$ is the surrounding scene containing multiple objects, the system selects an optimal attribute-value tuple $R^* = \{r_1^*, \dots, r_7^*\}$ from attribute vocabularies $V_{r_i} \cup \{\varepsilon\}$ , where $\varepsilon$ denotes omitting attribute $r_i$ .

The optimal referring expression is computed as: $R^* = \arg\max_{R} E(R, P, S) \quad \text{subject to} \quad f_k(R) \le b_k$

The objective energy function decomposes into content-based visual potentials, category-specific unary prior potentials, and pairwise attribute prior potentials: $E(R, P, S) = \alpha \sum_{i=2}^{6} \phi_i(r_i, P, S) + \beta \sum_{i=1}^{7} \psi_i(r_i, \text{type}(P)) + \sum_{i > j} \psi_{i,j}(r_i, r_j)$ where:
- $\phi_i(r_i, P, S)$ is the visual compatibility potential between attribute value $r_i$ and scene/object properties $(P, S)$ .
- $\psi_i(r_i, \text{type}(P))$ is the unary prior potential based on co-occurrence between object category $\text{type}(P)$ and attribute $r_i$ .
- $\psi_{i,j}(r_i, r_j)$ is the pairwise prior potential scoring compatibility between pairs of attributes.
- $\alpha, \beta > 0$ are weighting hyperparameters tuned on validation data.
The linear constraints $f_k(R) \le b_k$ enforce:
1. Length and verbosity control: $\sum_{i=1}^{7} \mathbf{1}[r_i = \varepsilon] \le 7 - \gamma(P, S)$ where $\gamma(P, S)$ specifies the minimum number of non- $\varepsilon$ attributes required, increasing when $S$ contains multiple objects of class $\text{type}(P)$ .
2. Spatial relation coupling: $r_5 = \varepsilon \iff r_6 = \varepsilon$ , ensuring relative spatial relations and landmark reference objects are included or excluded together.
3. Mandatory category naming: $r_1 \ne \varepsilon$ , guaranteeing an entry-level object name is always generated.
The model is solved using commercial binary integer linear programming software (IBM ILOG CPLEX) with binary indicator variables over candidate attribute values and pairwise variable activations.
Knowl 6 — Content-Based Visual Potential Functions for REG
equation
In the referring expression generation model, content-based potential functions $\phi_i(r_i, P, S)$ evaluate visual and spatial compatibility for attributes $r_2$ through $r_6$ ( $r_1$ and $r_7$ are unmodeled visually and rely solely on prior potentials):
1. Color Potential ( $r_2 = c_k \in V_{r_2}$ ): $\phi_2(r_2 = c_k, P, S) = \text{sim}(\text{hist}_{c_k}, \text{hist}(P))$ where $\text{hist}(P)$ is the normalized HSV color histogram of target object region $P$ , $\text{hist}_{c_k}$ is the mean HSV histogram of all training objects referred to with color term $c_k$ , and $\text{sim}(\cdot, \cdot)$ is cosine similarity.
2. Size Potential ( $r_3 = s_k \in V_{r_3}$ ): $\phi_3(r_3 = s_k, P, S) = \frac{1}{\sigma_{s_k} \sqrt{2\pi}} \exp\left( -\frac{(\text{size}(P) - \mu_{s_k})^2}{2\sigma_{s_k}^2} \right)$ where $\text{size}(P) \in [0, 1]$ is the pixel area of object $P$ normalized by total image area, and $(\mu_{s_k}, \sigma_{s_k}^2)$ are Gaussian distribution parameters fitted to training objects described with size word $s_k$ .
3. Absolute Location Potential ( $r_4 = a_k \in V_{r_4}$ ): $\phi_4(r_4 = a_k, P, S) = \frac{1}{\sqrt{(2\pi)^2 |\Sigma_{a_k}|}} \exp\left( -\frac{1}{2} (\text{loc}(P) - \mu_{a_k})^T \Sigma_{a_k}^{-1} (\text{loc}(P) - \mu_{a_k}) \right)$ where $\text{loc}(P) \in [0, 1]^2$ denotes the 2D centroid coordinates of object $P$ in normalized image coordinates, and $(\mu_{a_k}, \Sigma_{a_k})$ are the bivariate Gaussian mean vector and covariance matrix estimated for absolute location term $a_k$ .
4. Relative Location Relation Potential ( $r_5 = l_k \in V_{r_5}$ ): $\phi_5(r_5 = l_k, P, S) = \mathbf{1}[l_k \ne \varepsilon] \cdot g(\text{count}(\text{type}(P), S))$ where $\text{count}(\text{type}(P), S)$ is the number of objects sharing category $\text{type}(P)$ in scene $S$ , and $g$ assigns empirical probabilities: $0.12$ for single-instance objects, $0.22$ for two instances, and $0.26$ for three or more instances.
5. Relative Object Landmark Potential ( $r_6 = o_k \in V_{r_6}$ ): $\phi_6(r_6 = o_k, P, S) = \mathbf{1}[o_k \in \text{objectsnear}(\text{location}(P), S)]$ where $\text{objectsnear}(\text{location}(P), S)$ is the set of object classes present in scene $S$ within a spatial proximity neighborhood of target object $P$ .
Knowl 7 — Prior Statistics-Based Potential Functions for REG
equation
In the referring expression generation model, prior potential functions $\psi_i$ and $\psi_{i,j}$ encode global and category-specific attribute occurrence statistics estimated from training corpus $D = \{(P^{(j)}, S^{(j)}, R^{(j)})\}_{j=1}^{|D|}$ :
1. Unary Category Prior Potential: $\psi_i(r_i, \text{type}(P)) = \frac{\sum_{j=1}^{|D|} \mathbf{1}\left[\left(r_i^{(j)} \ne \varepsilon\right) \land \left(\text{type}(P^{(j)}) = \text{type}(P)\right)\right]}{\sum_{j=1}^{|D|} \mathbf{1}\left[\text{type}(P^{(j)}) = \text{type}(P)\right]} + \frac{\sum_{j=1}^{|D|} \mathbf{1}\left[r_i^{(j)} \ne \varepsilon\right]}{|D|} + \lambda$ where $\text{type}(P)$ denotes the object class of $P$ , and $\lambda$ is an additive Laplace smoothing term.
2. Pairwise Prior Potentials: $\sum_{i < j} \psi_{i,j}(r_i, r_j) = \sum_{i < j} \psi_{i,j}^{(1)}(r_i, r_j) + \psi_{5,6}^{(2)}(r_5, r_6)$ where $\psi_{i,j}^{(1)}(r_i, r_j)$ captures attribute co-occurrence statistics across attribute slots: $\psi_{i,j}^{(1)}(r_i, r_j) = \begin{cases} 1 & \text{if } r_i = r_j = \varepsilon \\ C + \lambda & \text{otherwise} \end{cases} \quad \text{with } C = \frac{1}{|D|}\sum_{t=1}^{|D|} \mathbf{1}\left[r_i^{(t)} \ne \varepsilon \land r_j^{(t)} \ne \varepsilon\right]$ and $\psi_{5,6}^{(2)}(r_5, r_6)$ measures lexical cohesion between specific relative spatial prepositions ( $r_5 = a$ ) and landmark object nouns ( $r_6 = b$ ): $\psi_{5,6}^{(2)}(r_5 = a, r_6 = b) = \frac{1}{|D|} \sum_{t=1}^{|D|} \mathbf{1}\left[r_5^{(t)} = a \land r_6^{(t)} = b\right]$
Knowl 8 — Experimental Evaluation Protocol and Test Set Benchmarks for Natural Scene REG
experimental setup
The referring expression generation model is evaluated across three distinct 500-object test sets designed to evaluate different scene conditions:
- Test Set A (Random Sample): 500 objects sampled uniformly at random across the entire dataset to evaluate overall distribution fidelity.
- Test Set B (Balanced Frequent Foreground Categories): 500 objects sampled with balanced category representation from the most frequent foreground classes, excluding background elements such as sky, ground, floor, sand, and sidewalk.
- Test Set C (Cluttered Multi-Instance Scenes): 500 objects sampled from images containing two or more distinct instances of the same object class, excluding background elements.
For each test object, 3 independent referring expressions were gathered via ReferItGame and manually annotated with ground-truth attribute labels. The human agreement baseline—measured as the mean matching accuracy of attribute values across pairs of human users on the same object—is 72.31%.

Knowl 9 — Referring Expression Generation Precision and Recall Across Scene Variations

data/table

Performance of the Integer Linear Programming generation model (Full Model) and a baseline model incorporating solely prior potentials ( $\psi_i, \psi_{i,j}$ ) without visual content potentials ( $\phi_i$ ) across Test Sets A, B, and C:

Model / Test Set	Precision (%)	Recall (%)
Baseline – Test Set A	27.92	43.27
Full Model – Test Set A	36.28	53.44
Baseline – Test Set B	29.87	50.57
Full Model – Test Set B	36.68	59.80
Baseline – Test Set C	28.85	37.41
Full Model – Test Set C	37.73	48.54

The Full Model outperforms the prior-only baseline by large margins across all test sets:

On Test Set A (random distribution), the full model improves precision by +8.36% and recall by +10.17% over baseline.
On Test Set B (balanced foreground categories), the full model achieves its highest recall of 59.80% (+9.23% over baseline).
On Test Set C (multi-instance cluttered scenes), the full model achieves its highest precision of 37.73% (+8.88% over baseline) and increases recall by +11.13%.

Knowl 10 — Limitations of Low-Level Content Models in Natural Scene REG
limitation
The referring expression generation optimization system demonstrates three primary limitations:
- Visual classifier errors: Global color representations (HSV histograms) and bounding-box spatial Gaussian approximations frequently mispredict attribute values when objects contain mixed colors, non-uniform textures, or complex geometries (e.g., misclassifying a man wearing a red shirt as "yellow").
- Unmodeled visual attributes: Fine-grained generic attributes ( $r_7$ , including shape and material) and entry-level object category names ( $r_1$ ) are driven solely by dataset co-occurrence priors rather than visual recognition models due to data sparsity in infrequent attribute classes.
- Attribute selection failures: Objective energy trade-offs between prior potentials and weak visual potentials can lead the integer program to generate incorrect attribute subsets or omit critical discriminative attributes.

Coverage note — None was omitted; all primary contributions, including the collection game, dataset characteristics, parsing pipeline, optimization framework, potential formulations, test sets, evaluation results, and failure modes, are fully represented.

References

1.Ahmet Aker and Robert Gaizauskas. 2010. Generating image descriptions using dependency relational patterns. In Association for Computational Linguistics (ACL).
2.Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven J. Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, Lara Schmidt, Jiangnan Shangguan, Jeffrey Mark Siskind, Jarrell W. Waggoner, Song Wang, Jinlian Wei, Yifan Yin, and Zhiqi Zhang. 2012. Video in sentences out. In Uncertainty in Artificial Intelligence (UAI).
3.Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science (CogSci), 19:233264.
4.Robert Dale and Ehud Reiter. 2000. Building natural language generation systems. In Cambridge University Press.
5.Jia Deng, Alexander C. Berg, Kai Li, and Fei-Fei Li. 2010. What does classifying more than 10,000 image categories tell us? In European Conference on Computer Vision (ECCV).
6.Jia Deng, Alex Berg, Sanjeev Satheesh, Hao Su, Aditya Khosla, and Fei-Fei Li. 2012. Large scale visual recognition challenge. In http://www.image-net.org/challenges/LSVRC/2012/index.
7.Jia Deng, Jonathan Krause, and Li Fei-Fei. 2013. Fine-grained crowdsourcing for fine-grained recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
8.Hugo Jair Escalante, Carlos A. Hernandez, Jesus A. Gonzalez, A. Lopez-Lopez, Manuel Montes, Eduardo F. Morales, L. Enrique Sucar, Luis Villasenor, and Michael Grubinger. 2010. The segmented and annotated iapr tc-12 benchmark. Computer Vision and Image Understanding (CVIU).
9.Rui Fang, Changsong Liu, Lanbo She, and Joyce Chai. 2013. Towards situated dialogue: Revisiting referring expression generation. In Empirical Methods on Natural Language Processing (EMNLP).
10.Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: generating sentences for images. In European Conference on Computer Vision (ECCV).
11.Yansong Feng and Mirella Lapata. 2010. How many words is a picture worth? automatic caption generation for news images. In Association for Computational Linguistics (ACL).
12.Yansong Feng and Mirella Lapata. 2013. Automatic caption generation for news images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4):797–812.
13.Nicholas FitzGerald, Yoav Artzi, and Luke Zettlemoyer. 2013. Learning distributions over logical forms for referring expression generation. In Empirical Methods on Natural Language Processing (EMNLP).
14.H. Paul Grice. 1975. Logic and conversation. page 4158.
15.Michael Grubinger, Paul D. Clough, Henning Muller, and Thomas Deselaers. 2006. The iapr benchmark: A new evaluation resource for visual information systems. In Proceedings of the International Workshop OntoImage (LREC).
16.Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2013. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In International Conference on Computer Vision (ICCV).
17.Emiel Krahmer and Kees van Deemter. 2012. Computational generation of referring expressions: A survey. In Computational Linguistics, volume 38, page 173218.
18.Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Neural Information Processing Systems (NIPS).
19.Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. 2011. Babytalk: Understanding and generating simple image descriptions. In IEEE Computer Vision and Pattern Recognition (CVPR).
20.Polina Kuznetsova, Vicente Ordonez, Alex Berg, Tamara L Berg, and Yejin Choi. 2012. Collective generation of natural image descriptions. In Association for Computational Linguistics (ACL).
21.Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2010. Natural reference to objects in a visual domain. In International Natural Language Generation Conference (INLG).
22.Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2011. Two approaches for generating size modifiers. In European Workshop on Natural Language Generation.
23.Margaret Mitchell, Ehud Reiter, and Kees van Deemter. 2013a. Typicality and object reference. In Cognitive Science (CogSci).
24.Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2013b. Generating expressions that refer to visible objects. In North American Chapter of the Association for Computational Linguistics (NAACL).
25.Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS).
26.Vicente Ordonez, Jia Deng, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2013. From large scale image categorization to entry-level categories. In International Conference on Computer Vision (ICCV).
27.Florent Perronnin, Zeynep Akata, Zaid Harchaoui, and Cordelia Schmid. 2012. Towards good practice in large-scale learning for image classification. In Computer Vision and Pattern Recognition (CVPR).
28.Yuan Ren, Kees Van Deemter, and Jeff Z Pan. 2010. Charting the potential of description logic for the generation of referring expressions. In International Natural Language Generation Conference (INLG).
29.Eleanor Rosch. 1978. Principles of categorization. Cognition and Categorization, page 2748.
30.Nitin Seemakurty, Jonathan Chu, Luis von Ahn, and Anthony Tomasic. 2010. Word sense disambiguation via human computation. In Human Computation Workshop.
31.Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing With Compositional Vector Grammars. In Association for Computational Linguistics (ACL).
32.Kees van Deemter, Ielka van der Sluis, and Albert Gatt. 2006. Building a semantically transparent corpus for the generation of referring expressions. In International Conference on Natural Language Generation (INLG).
33.Kees Van Deemter, Albert Gatt, Roger PG van Gompel, and Emiel Krahmer. 2012. Toward a computational psycholinguistics of reference production. In Topics in Cognitive Science, volume 4(2), page 166183.
34.Jette Viethen and Robert Dale. 2008. The use of spatial relations in referring expression generation. In International Natural Language Generation Conference (INLG).
35.Jette Viethen and Robert Dale. 2010. Speaker-dependent variation in content selection for referring expression generation. In Australasian Language Technology Workshop.
36.Jette Viethen, Margaret Mitchell, and Emiel Krahmer. 2013. Graphs and spatial relations in the generation of referring expressions. In European Workshop on Natural Language Generation.
37.Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In ACM Conf. on Human Factors in Computing Systems (CHI).
38.Luis von Ahn, Mihir Kedia, and Manuel Blum. 2006a. Verbosity: A game for collecting common-sense knowledge. In ACM Conference on Human Factors in Computing Systems (CHI).
39.Luis von Ahn, Ruoran Liu, and Manuel Blum. 2006b. Peekaboom: A game for locating objects in images. In ACM Conference on Human Factors in Computing Systems (CHI).
40.Terry Winograd. 1972. Understanding natural language. Cognitive Psychology, 3(1):1191.
41.Yezhou Yang, Ching Lik Teo, Hal Daume III, and Yiannis Aloimonos. 2011. Corpus-guided sentence generation of natural images. In Empirical Methods on Natural Language Processing (EMNLP).
42.Benjamin Z. Yao, Xiong Yang, Liang Lin, Mun Wai Lee, and Song-Chun Zhu. 2010. I2t: Image parsing to text description. Proc. IEEE, 98(8).

Citation

MLA

Kazemzadeh, S., et al. “ReferItGame: Referring to Objects in Photographs of Natural Scenes”. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 787–98, https://doi.org/10.3115/v1/D14-1086.

APA

Kazemzadeh, S., Ordonez, V., Matten, M., & Berg, T. (2014). ReferItGame: Referring to Objects in Photographs of Natural Scenes. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 787–798. https://doi.org/10.3115/v1/D14-1086

Chicago

Kazemzadeh, S., V. Ordonez, M. Matten, and T. Berg. 2014. “ReferItGame: Referring to Objects in Photographs of Natural Scenes”. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 787–98. https://doi.org/10.3115/v1/D14-1086.

Harvard

Kazemzadeh, S. et al. (2014) “ReferItGame: Referring to Objects in Photographs of Natural Scenes”, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 787–798. Available at: https://doi.org/10.3115/v1/D14-1086.

Vancouver

1. Kazemzadeh S, Ordonez V, Matten M, Berg T (2014) ReferItGame: Referring to Objects in Photographs of Natural Scenes. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp 787–798

BibTeX

@inproceedings{kazemzadeh-etal-2014-referitgame,
    title = "{R}efer{I}t{G}ame: Referring to Objects in Photographs of Natural Scenes",
    author = "Kazemzadeh, Sahar  and
      Ordonez, Vicente  and
      Matten, Mark  and
      Berg, Tamara",
    editor = "Moschitti, Alessandro  and
      Pang, Bo  and
      Daelemans, Walter",
    booktitle = "Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ({EMNLP})",
    month = oct,
    year = "2014",
    address = "Doha, Qatar",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D14-1086/",
    doi = "10.3115/v1/D14-1086",
    pages = "787--798"
}

Metadata:ACL Anthology

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by-nc-sa/4.0/

ReferItGame: Referring to Objects in Photographs of Natural Scenes

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — ReferItGame Two-Player Data Collection and Verification Game

Knowl 2 — ReferItGame Dataset

Knowl 3 — 7-Tuple Representation and Template-Based Parsing for Referring Expressions

Knowl 4 — Empirical Distributional Properties of Natural Scene Referring Expressions

Knowl 5 — Integer Linear Programming Optimization Framework for Referring Expression Generation

Knowl 6 — Content-Based Visual Potential Functions for REG

Knowl 7 — Prior Statistics-Based Potential Functions for REG

Knowl 8 — Experimental Evaluation Protocol and Test Set Benchmarks for Natural Scene REG

Knowl 9 — Referring Expression Generation Precision and Recall Across Scene Variations

Knowl 10 — Limitations of Low-Level Content Models in Natural Scene REG

References

Citation

BibTeX

Access the Paper