Matching Words and Pictures

Kobus BarnardPinar DuyguluDavid ForsythNando de FreitasDavid BleiMichael I. Jordan

article2003JMLR1,821 citations

Proposes probabilistic and statistical translation models to learn the joint distribution of segmented image regions and words, establishing a foundational framework for automatic image annotation, object recognition, and text-based image retrieval.

Listen

Managing and searching large collections of digital images presents a significant operational challenge across media archives, digital libraries, and commercial repositories. While users typically search for pictures using specific semantic terms and conceptual topics, conventional computer vision techniques primarily index images using low-level visual properties such as color histograms and textures, which fail to meet user needs. Captioning and labeling images manually is labor-intensive, inconsistent, and expensive. Combining textual information with visual features offers a powerful path toward automated indexing and retrieval.

The article evaluates whether probabilistic machine learning models can jointly learn the relationship between image regions and associated words from weakly labeled collections. It aims to demonstrate that such models can accurately predict keywords for whole images, assign names to specific visual regions, and treat broad-scale object recognition as a translation process.

To address this, the authors developed and compared several generative and latent-variable statistical models. The methods analyzed include multi-modal hierarchical aspect trees, discrete translation models adapted from statistical machine translation, continuous emission models, and mixture models based on latent Dirichlet allocation. The models were evaluated using a benchmark collection of several thousand annotated images across 160 distinct photographic themes. Images were segmented into major visual regions and described by forty visual features spanning size, color, texture, and shape. Performance was measured on held-out test data and novel image topics using word prediction accuracy, normalized error scores, information divergence, and human scoring on region-level correspondence.

The investigation produced several key findings regarding multi-modal learning. First, statistical models that jointly learn the links between image regions and words consistently outperform baseline frequency estimates. Across test sets, the best methods improved keyword prediction rates by roughly 50% over empirical baselines, correctly identifying approximately 0.9 out of 3 target keywords compared to 0.6 for the baseline. Second, continuous models that jointly learn visual features and word associations outperform discrete translation pipelines that quantize visual regions before learning text correspondences. Third, forcing entire images into strict global cluster trees frequently harms performance when applied to unfamiliar image themes; simpler continuous linear structures generalize significantly better across diverse topics. Finally, models based on latent Dirichlet allocation demonstrated superior resistance to overfitting when processing novel image categories.

These findings indicate that automated annotation and region labeling are viable strategies for streamlining media archive operations and building more intuitive visual search engines. In practice, organizations can automate initial metadata tagging and organize image collections for browsing without requiring expensive pixel-level manual labeling. However, the degradation in performance when handling completely novel topics indicates that deploying these systems on open-ended web collections introduces accuracy risks.

Organizations considering automated indexing should prioritize continuous joint models over discrete translation systems and apply probability thresholds to withhold low-confidence predictions. Future development should focus on incorporating explicit empty tokens so models can gracefully ignore background noise, integrating natural language processing to extract nouns from unstructured text captions, and exploring minimal human-in-the-loop validation to resolve visual ambiguities.

While the results strongly confirm that image regions can be linked to semantics, confidence is bounded by current technical constraints. The experiments relied on standardized vocabulary sets and pre-segmented image regions, meaning real-world performance will depend on segmentation reliability and the consistency of text annotations across diverse image collections.

Middle East Technical University University of Arizona University of British Columbia University of California correspondence models image annotation object recognition

Barnard et al (2003).pdf

Paper: Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Pinar Duygulu et al. (2002). This foundational paper first formulated object recognition and region naming as statistical machine translation between segmented image regions and words, which the source paper directly generalizes and compares against.
Paper: Co-clustering documents and words using bipartite spectral graph partitioning, Inderjit S. Dhillon (2001). It introduces bipartite spectral graph partitioning for co-clustering dual modalities, providing foundational principles for joint distribution modeling across visual features and text.
Paper: Text Classification from Labeled and Unlabeled Documents using EM, K. Nigam et al. (2000). It establishes the Expectation-Maximization framework for learning generative models from weakly labeled and unlabeled co-occurrence data that underpins statistical multimodal translation models.
Paper: Region Competition: Unifying Snakes, Region Growing, and Bayes/MDL for Multiband Image Segmentation, Song Chun Zhu et al. (1996). It details core statistical region segmentation techniques that produce the discrete visual regions required as inputs for the source paper's joint image-word models.

Paper: Connecting Modalities: Semi-supervised Segmentation and Annotation of Images Using Unaligned Text Corpora, Richard Socher et al. (2010). This work advances multimodal image-text modeling by learning joint visual-word representations from unaligned text corpora and weakly labeled images using canonical correlation analysis.
Paper: A Bayesian hierarchical model for learning natural scene categories, Li Fei-Fei et al. (2005). This paper builds directly on the topic-modeling and visual-word ideas by applying Latent Dirichlet Allocation to discover intermediate themes for visual scene categorization.
Paper: Every Picture Tells a Story: Generating Sentences from Images, Ali Farhadi et al. (2010). It extends region-word association models to full natural language sentence generation and retrieval by mapping images and text into a structured semantic meaning space.
Paper: Deep visual-semantic alignments for generating image descriptions, Andrej Karpathy et al. (2015). It modernizes the correspondence between image regions and text by using deep convolutional and recurrent neural networks to learn visual-semantic alignments and generate descriptions.
Paper: Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models, Bryan A. Plummer et al. (2015). It scales up the problem of region-phrase grounding by establishing a dedicated large-scale benchmark of explicit correspondences between image regions and entity mentions.
Paper: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, Kelvin Xu et al. (2015). It incorporates neural attention mechanisms to dynamically attend to specific image regions during word-by-word caption generation, directly realizing soft region-word alignment.
Paper: Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering, Peter Anderson et al. (2018). It combines bottom-up object proposals with top-down attention networks to align salient image regions with generated language for captioning and question answering.
Paper: Multimodal Machine Learning: A Survey and Taxonomy, Tadas Baltrušaitis et al. (2017). It provides a comprehensive taxonomy and survey of the multimodal machine learning landscape that evolved from early image-word translation and alignment models.
Paper: Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks, Xiujun Li et al. (2020). It utilizes detected object tags as linguistic anchor points to align image regions and text within modern transformer-based vision-language pre-training.
Paper: VL-BERT: Pre-training of Generic Visual-Linguistic Representations, Weijie Su et al. (2019). It generalizes visual-linguistic alignment into a unified pre-trained transformer architecture operating directly over paired image regions and text tokens.

Abstract

We present a new approach for modeling multi-modal data sets, focusing on the specific case of segmented images with associated text. Learning the joint distribution of image regions and words has many applications. We consider in detail predicting words associated with whole images (auto-annotation) and corresponding to particular image regions (region naming). Auto-annotation might help organize and access large collections of images. Region naming is a model of object recognition as a process of translating image regions to words, much as one might translate from one language to another. Learning the relationships between image regions and semantic correlates (words) is an interesting example of multi-modal data mining, particularly because it is typically hard to apply data mining techniques to collections of images. We develop a number of models for the joint distribution of image regions and words, including several which explicitly learn the correspondence between regions and words. We study multi-modal and correspondence extensions to Hofmann’s hierarchical clustering/aspect model, a translation model adapted from statistical machine translation (Brown et al.), and a multi-modal extension to mixture of latent Dirichlet allocation (MoM-LDA). All models are assessed using a large collection of annotated images of real

Keywords

correspondence models image annotation object recognition word prediction lexicon learning machine translation image segmentation image regions information retrieval

1. Introduction
1.1 Practical Applications
1.2 Annotation, Correspondence and Recognition
2. Input Representation and Preprocessing
3. Annotation Models
3.1 Multi-Modal Hierarchical Aspect Models
3.2 Mixture of Multi-Modal Latent Dirichlet Allocation
4. Simple Correspondence Models
4.1 Discrete Data Translation
4.2 Correspondence from a Hierarchical Clustering Model
5. Integrating Correspondence and Hierarchical Clustering
5.1 Linking Word Emission and Region Emission Probabilities with Mixture Weights
5.2 Paired Word and Region Emission at Nodes
5.3 Correspondence Models, NULL, Fertility and Refusal to Predict
6. Evaluation Methods
6.1 Measuring Annotation Performance
6.1.1 MEASURING THE QUALITY OF WORD POSTERIOR DISTRIBUTION
6.1.2 HOW WELL DO MODELS PREDICT WORDS?
6.2 Measuring Correspondence Performance
6.2.1 USING ANNOTATION AS A PROXY
6.2.2 MANUAL CORRESPONDENCE SCORING
7. Experiments
7.1 Annotation Results
7.1.1 THE NUMBER OF TRAINING ITERATIONS, AND OVER FITTING
7.1.2 SCORING ANNOTATIONS WITH THE NORMALIZED SCORE, AND THE EFFECT OF REFUSAL TO PREDICT
7.1.3 COMPARISON OF MODELS USING DIFFERENT SCORES
7.2 Correspondence Results
8. Discussion
Acknowledgments
References

Knowls

Knowl 1 — Multi-Modal Hierarchical Aspect Models for Image Annotation
model/method
Multi-modal hierarchical aspect models represent the joint distribution of image regions (blobs) and text annotations (words) by arranging latent topics in a tree hierarchy. Nodes in the tree generate both continuous visual features via Gaussian distributions and discrete text tokens via multinomial distributions. A cluster $c$ corresponds to a path from a leaf node to the root. Nodes near the root emit general visual and textual concepts shared across many clusters, while nodes near the leaves emit specific terms and features.

In the generative variant (denoted Model I-2), an annotated document $D = W \cup B$ containing a set of words $W$ and a set of segmented image blobs $B$ is generated according to:

$p(D) = \sum_c p(c) \prod_{w \in W} \left[ \sum_l p(w|l,c)p(l|c) \right]^{\frac{N_w}{N_{w,d}}} \prod_{b \in B} \left[ \sum_l p(b|l,c)p(l|c) \right]^{\frac{N_b}{N_{b,d}}}$

where $c$ indexes tree leaf paths (clusters), $l$ indexes tree levels from root to leaf, $w$ indexes the words in the document, and $b$ indexes the image blobs. The term $p(w|l,c)$ is a discrete multinomial word emission distribution, and $p(b|l,c)$ is a multivariate Gaussian distribution with diagonal covariance over the blob feature space. The term $p(l|c)$ represents the prior probability over levels given the cluster. Exponents $\frac{N_w}{N_{w,d}}$ and $\frac{N_b}{N_{b,d}}$ normalize for variations in word count $N_{w,d}$ and blob count $N_{b,d}$ relative to document maximums $N_w$ and $N_b$ .

In the document-specific non-generative variants, the vertical weights depend on the training instance: $p(l|d)$ in Model I-0 and $p(l|c,d)$ in Model I-1.

For image-based word prediction on uncaptioned images with blobs $B$ , the posterior distribution over vocabulary words is computed as:

$p(w|B) \propto \sum_c p(c) \left[ \sum_l p(w|l,c)p(l|c) \right] \prod_{b \in B} \left[ \sum_l p(b|l,c)p(l|c) \right]^{\frac{N_b}{N_{b,d}}}$
Knowl 2 — Dependent Multi-Modal Hierarchical Aspect Models
model/method
Dependent hierarchical aspect models (denoted D-models) eliminate the conditional independence assumption between image regions and text annotations within a cluster by conditioning word emissions directly on image blob emissions through the vertical mixture weights.

In Model D-0, the joint probability of the observations $D = W \cup B$ for document $d$ is given by:

$p(D|d) = \sum_c p(c) \prod_{w \in W} \left[ \sum_l p(w|l,c)p(l|B,c,d) \right]^{\frac{N_w}{N_{w,d}}} \prod_{b \in B} \left[ \sum_l p(b|l,c)p(l|d) \right]^{\frac{N_b}{N_{b,d}}}$

where the vertical mixing distribution inherited by the words, $p(l|B,c,d)$ , is determined by the posterior level probabilities of all observed blobs in the image:

$p(l|B,c,d) \propto \sum_{b \in B} p(l|b,c,d)$

Here, $c$ indexes clusters, $l$ indexes levels, $W$ is the set of words, $B$ is the set of blobs, $p(w|l,c)$ is the multinomial word probability, and $p(b|l,c)$ is the Gaussian blob emission probability. If a specific node in the tree path contributes minimally to explaining the image blobs, its contribution to emitting words is suppressed accordingly.

Model variants include D-1 (using cluster-dependent document level distributions $p(l|c,d)$ ) and D-2 (a generative formulation replacing $p(l|d)$ with cluster priors $p(l|c)$ ).
Knowl 3 — Mixture of Multi-Modal Latent Dirichlet Allocation (MoM-LDA)
model/method
The Mixture of Multi-Modal Latent Dirichlet Allocation (MoM-LDA) model is a hierarchical Bayesian framework that models images and accompanying annotations via shared continuous latent Dirichlet factors embedded under a discrete mixture component.

For an image with $M$ blobs and $N$ words, the generative process is defined as:
1. Choose a mixture component $c \sim \text{Multinomial}(\eta)$ , where $\eta$ is a $J$ -dimensional multinomial parameter.
2. Sample document-specific factor proportions $\theta \sim \text{Dir}(\alpha_c)$ , where $\alpha_c$ is a $K$ -dimensional Dirichlet parameter vector conditioned on mixture component $c$ .
3. For each word $w_n$ $w_{n}$ ( $n = 1, \dots, N$ $n = 1, \dots, N$ ):
  - Sample a latent factor $z_n \sim \text{Multinomial}(\theta)$ .
  - Sample word $w_n \sim p(w_n|z_n, c, \beta)$ , where $\beta_{cz}$ is the word multinomial conditioned on mixture component $c$ and factor $z_n$ .
4. For each blob $b_m$ $b_{m}$ ( $m = 1, \dots, M$ $m = 1, \dots, M$ ):
  - Sample a latent factor $s_m \sim \text{Multinomial}(\theta)$ .
  - Sample blob feature vector $b_m \sim \mathcal{N}(\mu_{c s_m}, \Sigma_{c s_m})$ , where $\mu_{cs}$ and diagonal covariance $\Sigma_{cs}$ parameterize the multivariate Gaussian for factor $s_m$ and mixture component $c$ .
Inference and parameter estimation are conducted via Expectation-Maximization with a variational E-step. Given an unannotated image with blobs, let $\phi$ represent the approximate posterior distribution over mixture components and $\gamma_c$ the posterior Dirichlet parameters for component $c$ . The predictive distribution for word $w$ is:

$p(w|b) = \sum_{c=1}^J p(c|\phi) \sum_{z=1}^K p(w|z,c) \frac{\gamma_{cz}}{\sum_{y=1}^K \gamma_{cy}}$
Knowl 4 — Paired Word-Region Correspondence Models with Graph-Matching EM
model/method
Paired correspondence hierarchical models (denoted C-models, including C-0, C-1, and C-2) explicitly pair image regions and words at the emission nodes such that observations are defined as pairs $(w,b) \in D$ .

The document likelihood assuming a known correspondence is:

$p(D|d) = \sum_c p(c) \prod_{(w,b) \in D} \left[ \sum_l p((w,b)|l,c)p(l|d) \right]$

where $p((w,b)|l,c) = p(w|l,c)p(b|l,c)$ , $c$ indexes clusters, $l$ indexes tree levels, and $p(l|d)$ are level weights.

Because training data contains only bags of words and bags of regions without explicit ground-truth alignments, correspondence is estimated during EM prior to the expectation step. The probability that word $w$ corresponds to blob $b$ (denoted $w \Leftrightarrow b$ ) is estimated across clusters:

$p(w \Leftrightarrow b) \approx \sum_c p(c) \sum_l p((w,b)|l,c)p(l|d)$

A maximum likelihood bipartite matching is solved in polynomial time using the Jonker-Volgenant algorithm, minimizing the edge costs defined as negative log probabilities $-\log p(w \Leftrightarrow b)$ . To handle unequal counts of words and regions, the smaller set is duplicated until cardinality matches before computing the bipartite match.
Knowl 5 — Discrete Translation Model for Image Region Lexicon Learning
model/method
The discrete translation approach models region-to-word association by analogy to statistical machine translation (IBM Model 1) on aligned bilingual corpora (bitexts).

Continuous visual feature vectors of image segments are first discretized into $K$ visual words (blob tokens) using K-means clustering (e.g., $K=500$ ). An image is treated as a parallel sentence pair consisting of a set of discrete blob tokens and a set of text annotation words.

Assuming unknown one-to-one correspondences between blob tokens and words within each image, the joint probability table $p(w|b)$ linking word tokens $w$ to blob tokens $b$ is treated as a missing-data problem and estimated across the entire corpus using the Expectation-Maximization (EM) algorithm.
Knowl 6 — Visual Feature Representation via Normalized Cuts Segmentation
definition
Images are represented as collections of segmented visual regions, termed blobs. Each image is segmented using the Normalized Cuts algorithm, and the 8 largest regions are retained. Each region is mapped to a 40-dimensional continuous feature vector capturing five visual properties:
1. Size (1 feature): The proportion of the total image area covered by the region.
2. Position (2 features): The spatial coordinates of the region center of mass normalized by image dimensions.
3. Color (12 features): Average and standard deviation of values in RGB, CIE-Lab, and normalized color spaces ( $r = R/(R+G+B)$ , $g = G/(R+G+B)$ ).
4. Texture (22 features): Mean and variance of responses from 16 filters: 4 Difference of Gaussian filters at different scales $\sigma$ , and 12 oriented filters aligned at $30^\circ$ increments.
5. Shape (3 features): The ratio of region area to perimeter squared (compactness), the moment of inertia about the region center of mass, and the ratio of region area to the area of its convex hull.
Continuous features are standardized to zero mean and unit variance across the training set, and emission covariances in Gaussian density estimations are regularized with a minimum variance threshold of $0.001$ .
Knowl 7 — Performance Metrics for Image Annotation and Region Word Prediction
definition
To evaluate multi-modal models against an empirical word frequency baseline, three primary evaluation measures are defined:
1. Relative Kullback-Leibler Divergence ( $E_{KL}$ ): Measures the quality of the predicted word posterior distribution $q(w|B)$ against a uniform distribution over the $K$ ground-truth words $p(w) = 1/K$ : $E_{KL} = \frac{1}{|D_{\text{test}}|} \sum_{d \in D_{\text{test}}} \left( E_{KL}^{(\text{empirical})} - E_{KL}^{(\text{model})} \right)$ where $E_{KL}^{(\text{model})} = \text{constant} - \frac{1}{K} \sum_{w \in \text{observed}} \log q(w|B)$ . Higher positive values indicate superior performance over the empirical prior.
2. Normalized Classification Score ( $E_{NS}$ ): Evaluates thresholded multi-label prediction accounting for vocabulary size $N$ and image ground-truth count $n$ : $E_{NS}^{(\text{model})} = \frac{r}{n} - \frac{w}{N - n}$ where $r$ is the number of correctly predicted words, and $w$ is the number of incorrectly predicted words. A score of 0 corresponds to predicting everything or nothing; 1 indicates perfect prediction; $-1$ is predicting the exact complement. Performance is reported as $E_{NS}^{(\text{model})} - E_{NS}^{(\text{empirical})}$ .
3. Precision-Recall Score on Top $n$ Words ( $E_{PR}$ ): $E_{PR}^{(\text{model})} = \frac{r}{n}$ where the model predicts exactly $n$ words corresponding to the number of true keywords for that image. Reported as $E_{PR}^{(\text{model})} - E_{PR}^{(\text{empirical})}$ .

Knowl 8 — Empirical Annotation Performance on the Corel Dataset

data/table

The table below summarizes image annotation performance (measured as the increase over the empirical training frequency baseline) on Corel test images divided into standard held-out data (drawn from the same 80 CD categories as training) and novel held-out data (drawn from 80 separate CD categories). Training used 10 EM iterations on roughly 155 vocabulary words.

Method	Held out PR	Novel PR	Held out NS	Novel NS
linear-I-0-doc-vert	0.095 (0.003)	0.057 (0.003)	0.174 (0.007)	0.081 (0.007)
binary-I-0-doc-vert	0.094 (0.004)	0.034 (0.005)	0.160 (0.007)	0.065 (0.008)
binary-I-2-doc-vert	0.090 (0.004)	0.036 (0.004)	0.170 (0.008)	0.074 (0.008)
linear-D-0-doc-vert	0.102 (0.002)	0.059 (0.004)	0.167 (0.006)	0.076 (0.008)
binary-D-0-doc-vert	0.094 (0.003)	0.037 (0.005)	0.157 (0.007)	0.064 (0.008)
binary-D-2-region-cluster	0.108 (0.003)	0.039 (0.005)	0.179 (0.003)	0.072 (0.005)
linear-C-0-region-only	0.067 (0.002)	0.035 (0.005)	0.124 (0.007)	0.046 (0.006)
binary-C-0-doc-vert	0.078 (0.002)	0.025 (0.004)	0.148 (0.006)	0.054 (0.007)
discrete-translation	0.073 (0.003)	0.029 (0.005)	0.111 (0.007)	0.016 (0.008)
MoM-LDA	0.050 (0.002)	0.038 (0.002)	0.107 (0.005)	0.041 (0.007)

Baseline empirical word frequency achieves a PR of approximately $0.19$ and an NS of $0.425$ . The results show that dependent clustering models (D-models) and continuous hierarchical aspect models consistently outperform the discrete translation model across all splits.

Knowl 9 — Manual Evaluation of Region-to-Word Correspondence Performance

data/table

Quantitative correspondence accuracy was measured directly on a benchmark of hand-labeled image regions across 10 sets of 50 images from the held-out collection. Human judges annotated each valid segment with plausible ground-truth vocabulary terms. Performance was measured via the increase in $E_{PR}$ relative to the empirical frequency baseline ( $0.094$ ).

Method	PR Measure Increase
linear-I-0-region-only	0.099 (0.02)
binary-I-0-region-cluster	0.101 (0.01)
binary-I-0-region-only	0.103 (0.01)
binary-I-2-region-cluster	0.101 (0.01)
binary-I-2-region-only	0.093 (0.01)
linear-D-0-region-only	0.132 (0.01)
binary-D-0-region-cluster	0.096 (0.01)
binary-D-0-region-only	0.104 (0.01)
binary-D-2-region-cluster	0.103 (0.01)
binary-D-2-region-only	0.092 (0.01)
linear-C-0-region-only	0.101 (0.01)
discrete-translation	0.066 (0.01)

The continuous linear dependent model (linear-D-0-region-only) attains the highest correspondence score ( $0.132$ ), significantly outperforming the discrete translation model ( $0.066$ ). Explicit correspondence models (C-models) improved correspondence relative to their whole-image annotation proxy scores, but fell short of the dependent continuous models due to lack of a NULL-token mechanism for unmodeled regions.

Knowl 10 — Generalization Effects of Clustering Topologies and Continuous vs. Discrete Representations
empirical result
Experimental comparisons across hierarchical aspect topologies (linear aspect chains vs. binary tree clustering) and feature representations reveal two fundamental behaviors in multi-modal learning:
1. Clustering vs. Aspect Topologies: Hierarchical tree clustering enforces global document grouping that aids training-set fitting but hurts generalization on novel held-out categories. On novel CD categories, linear topologies without horizontal cluster branching (e.g., linear-I-0 and linear-D-0) score substantially higher ( $E_{KL}$ reduction of $0.258$ -- $0.268$ ) than 9-level binary trees ( $E_{KL}$ reduction of $0.060$ -- $0.105$ ). Global image clustering binds words to specific scene contexts (e.g., associating 'sky' exclusively with jungle scenes), preventing correct transfer to novel scene contexts (e.g., 'sky' in jet images).
2. Continuous Distributions vs. Discrete Quantization: Jointly estimating continuous Gaussian parameters for image regions during multi-modal training outperforms discrete translation with pre-quantized K-means blob tokens. Early vector quantization introduces irreversible cluster assignment errors and artifacts that induce severe overfitting, causing the discrete translation model's $E_{KL}$ improvement on novel datasets to drop below zero ( $-0.072$ ).
Knowl 11 — Limitations of Absence of NULL Matching and Unsupervised Correspondence Symmetries
limitation
The correspondence and translation frameworks exhibit two structural limitations:
1. Lack of Explicit NULL Modeling: The correspondence formulation forces every segmented region to be linked to an annotation word and every word to a region. Because segmentation produces unnameable background fragments and annotators omit visually evident objects, models are corrupted by large outlier pools during training, compelling spurious word assignments.
2. Incomplete-Data Symmetries: Unsupervised co-occurrence estimation cannot disambiguate features of objects that always appear together in training bags (e.g., 'tiger' and 'grass'). Small amounts of supervisory region labeling are required to break likelihood symmetries.

Coverage note — None was omitted; all primary models (multi-modal hierarchical aspect models I/D/C, MoM-LDA, discrete translation), visual representations, inference formulations, evaluation metrics, and empirical findings are fully covered.

References

1.L. H. Armitage and P. G. B. Enser. Analysis of user need in image archives. Journal of Information Science, 23(4):287–299, 1997.
2.K. Barnard, P. Duygulu, and D. A. Forsyth. Clustering art. In IEEE Conference on Computer Vision and Pattern Recognition, pages II:434–441, Hawaii, 2001.
3.K. Barnard and D. A. Forsyth. Learning the semantics of words and pictures. In International Conference on Computer Vision, pages II:408–415, 2001.
4.D. Blei and M. Jordan. Modeling annotated data. Technical Report CSD-02-1202, U.C. Berkeley CS Division, 2002.
5.D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. In Advances in Neural Information Processing Systems 14, 2002.
6.P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer. The mathematics of machine translation: Parameter estimation. Computational Linguistics, 19(10):263–311, 1993.
7.C. Carson, S. Belongie, H. Greenspan, and J. Malik. Blobworld: Image segmentation using expectation-maximization and its application to image querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8):1026–1038, 2002.
8.G. Celeux, D. Chauveau, and J. Diebolt. On stochastic versions of the EM algorithm. Technical report 2514, INRIA, March 1995.
9.F. Chen, U. Gargi, L. Niles, and H. Schtze. Multi-modal browsing of images in web documents. In SPIE Document Recognition and Retrieval, 1999.
10.J. Chen, C. A. Bouman, and J. C. Dalton. Hierarchical browsing and search of large image databases. IEEE Transactions on Image Processing, 9(3):442–455, 2000.
11.A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39:1–38, 1977.
12.P. Duygulu, Kobus B., J. F. G de Freitas, and D. A. Forsyth. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In The Seventh European Conference on Computer Vision, pages IV:97–112, 2002.
13.P. G. B. Enser. Query analysis in a visual information retrieval context. Journal of Document and Text Management, 1(1):25–39, 1993.
14.P. G. B. Enser. Progress in documentation pictorial information retrieval. Journal of Documentation, 51(2):126–170, 1995.
15.M. M. Fleck, D. A. Forsyth, and C. Bregler. Finding naked people. In Bernard Buxton and Roberto Cipolla, editors, 4th European Conference on Computer Vision, pages II:591–602. Springer, 1996.
16.D. A. Forsyth. Computer vision tools for finding images and video sequences. Library Trends, 48 (2):326–355, 1999.
17.D. A. Forsyth and J. Ponce. Computer Vision - A Modern Approach. Prentice-Hall, 2002.
18.C. O. Frost, B. Taylor, A. Noakes, S. Markel, D. Torres, and K. M. Drabenstott. Browse and search patterns in a digital image database. Information retrieval, 1:287–313, 2000.
19.T. Hofmann. Learning and representing topic. A hierarchical mixture model for word occurrence in document databases. In Workshop on learning from text and the web, CMU, 1998.
20.T. Hofmann and J. Puzicha. Statistical models for co-occurrence data. A.I. Memo 1635, Massachusetts Institute of Technology, 1998.
21.R. Jonker and A. Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38:325–340, 1987.
22.D. Jurafsky and J.H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition. Prentice-Hall, 2000.
23.L.H. Keister. User types and queries: impact on image access systems. In Challenges in indexing electronic text and images. Learned Information, 1994.
24.M. La Cascia, S. Sethi, and S. Sclaroff. Combining textual and visual cues for content-based image retrieval on the world wide web. In IEEE Workshop on Content-Based Access of Image and Video Libraries, 1998.
25.C. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press. Cambridge, MA, 1999.
26.M. Markkula and E. Sormunen. End-user searching challenges indexing practices in the digital newspaper photo archive. Information retrieval, 1:259–285, 2000.
27.O. Maron. Learning from Ambiguity. Ph.D. dissertation, Massachusetts Institute of Technology, 1998.
28.O. Maron and A.L. Ratan. Multiple-instance learning for natural scene classification. In The Fifteenth International Conference on Machine Learning, 1998.
29.D. Melamed. Empirical methods for exploiting parallel texts. MIT Press, Cambridge, Massachusetts, 2001.
30.Y. Mori, H. Takahashi, and R. Oka. Image-to-word transformation based on dividing and vector quantizing images with words. In First International Workshop on Multimedia Intelligent Storage and Retrieval Management (in conjunction with ACM Multimedia Conference 1999), Orlando, Florida, 1999.
31.M. Oren, C. Papageorgiou, P. Sinha, and E. Osuna. Pedestrian detection using wavelet templates. In Computer vision and pattern recognition, pages 193–9, 1997.
32.S. Ornager. View a picture: Theoretical image analysis and empirical user studies on indexing and retrieval. Swedis Library Research, 2(3):31–41, 1996.
33.Shin’ichi Satoh and T. Kanade. Name-it: Association of face and name in video. In Proceedings of 1997 IEEE Computer Vision and Pattern Recognition (CVPR ’97), pages 368–373, June 1997.
34.H. Schneiderman and T. Kanade. A statistical approach to 3d object recognition applied to faces and cars. In IEEE Conference on Computer Vision and Pattern Recognition, page 100. IEEE, 2000.
35.J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(9):888–905, 2000.
36.R. K. Srihari. Extracting Visual Information from Text: Using Captions to Label Human Faces in Newspaper Photographs. Ph.d. thesis, SUNY at Buffalo, 1991.
37.R. K. Srihari and D. T. Burhans. Visual semantics: Extracting visual information from text accompanying pictures. In AAAI ’94, Seattle, WA, 1994.
38.R. K. Srihari, R. Chopra, D. Burhans, M. Venkataraman, and V. Govindaraju. Use of collateral text in image interpretation. In ARPA Image Understanding Workshop, Monterey, CA, 1994.
39.M. J. Swain, C. Frankel, and V. Athitsos. Webseer: An image search engine for the world wide web. Technical Report TR-96-14, Computer Science Department, University of Chicago, 1996.

Citation

MLA

Barnard, K., et al. “Matching Words and Pictures”. Journal of Machine Learning Research, vol. 3, no. Feb, 2003, pp. 1107–35, https://www.jmlr.org/papers/v3/barnard03a.html.

APA

Barnard, K., Duygulu, P., Forsyth, D., Freitas, N. de ., Blei, D. M., & Jordan, M. I. (2003). Matching Words and Pictures. Journal of Machine Learning Research, 3(Feb), 1107–1135. https://www.jmlr.org/papers/v3/barnard03a.html

Chicago

Barnard, K., P. Duygulu, D. Forsyth, N. de . Freitas, D. M. Blei, and M. I. Jordan. 2003. “Matching Words and Pictures”. Journal of Machine Learning Research 3 (Feb): 1107–35. https://www.jmlr.org/papers/v3/barnard03a.html.

Harvard

Barnard, K. et al. (2003) “Matching Words and Pictures”, Journal of Machine Learning Research, 3(Feb), pp. 1107–1135. Available at: https://www.jmlr.org/papers/v3/barnard03a.html.

Vancouver

1. Barnard K, Duygulu P, Forsyth D, Freitas N de, Blei DM, Jordan MI (2003) Matching Words and Pictures. Journal of Machine Learning Research 3:1107–1135

BibTeX

@article{barnard2003matching,
  title = {Matching Words and Pictures},
  author = {Barnard, Kobus and Duygulu, Pinar and Forsyth, David and Freitas, Nando de and Blei, David M. and Jordan, Michael I.},
  year = {2003},
  journal = {Journal of Machine Learning Research},
  volume = {3},
  number = {Feb},
  pages = {1107-1135},
  url = {https://www.jmlr.org/papers/v3/barnard03a.html}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Matching Words and Pictures

Abstract

Keywords

Table of Contents

Knowls

Knowl 1 — Multi-Modal Hierarchical Aspect Models for Image Annotation

Knowl 2 — Dependent Multi-Modal Hierarchical Aspect Models

Knowl 3 — Mixture of Multi-Modal Latent Dirichlet Allocation (MoM-LDA)

Knowl 4 — Paired Word-Region Correspondence Models with Graph-Matching EM

Knowl 5 — Discrete Translation Model for Image Region Lexicon Learning

Knowl 6 — Visual Feature Representation via Normalized Cuts Segmentation

Knowl 7 — Performance Metrics for Image Annotation and Region Word Prediction