Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary

Pinar DuyguluKobus BarnardNando de FreitasDavid Forsyth

article2002ECCV1,884 citationsBest paper in Cognitive Vision award

Proposes a statistical machine translation framework that uses the EM algorithm to automatically map segmented image regions to descriptive keywords from weakly labeled datasets.

Listen

Traditional computer vision techniques for object recognition often require extensive manual segmentations, rely heavily on rigid geometric models, or operate solely on isolated objects. Meanwhile, vast collections of digital images on the web are accompanied by loosely related text, such as captions, tags, and descriptive paragraphs. The article addresses the challenge of automatically identifying and localizing objects across diverse image collections without requiring fine-grained visual ground truth, treating object recognition as an automated cross-modal translation problem from image regions to descriptive words.

To demonstrate this capability, the article develops a statistical machine translation framework based on the Expectation-Maximization algorithm. The method models image understanding analogously to learning a bilingual lexicon from aligned sentences, establishing associations between discrete visual features and text vocabulary across weakly labeled datasets without supervised region annotations.

The approach begins by segmenting images into regions and extracting low-level feature vectors capturing color, texture, shape, position, and size. These region features are then clustered into discrete visual tokens, referred to as visual terms. Using a standard dataset of 4,500 annotated images, the algorithm iteratively estimates the conditional probability of textual words given specific visual terms, treating the true one-to-one correspondence between regions and keywords as hidden data. The models are evaluated on held-out test sets using standard precision and recall metrics.

The key findings reveal that statistical translation models can successfully learn visual-text correspondences from unaligned data. First, only a core subset of about 80 distinct words out of hundreds in the vocabulary can be predicted with non-zero probability, reflecting that many words never achieve the maximum likelihood given any visual feature. Second, introducing a null-token probability threshold filters out weakly supported associations, which systematically boosts precision across both training and test data at the expense of lower recall. Third, hierarchically clustering highly correlated and visually indistinguishable words (such as "horses" and "mare", or "sky" and "beach") significantly enhances overall prediction accuracy and recall, enabling reliable recognition of higher-level concepts.

These results imply that image indexing, search, and retrieval can scale significantly without the immense cost of detailed manual annotation. Viewing region-word correspondence through statistical machine learning decouples model development from rigid geometric priors and domain-specific feature engineering. However, the findings also highlight that visual-semantic ambiguity poses a fundamental bottleneck, as visually similar objects and co-occurring background terms cannot be separated without linguistic structure or richer contextual cues.

To build on this foundation, future work should integrate multi-word linguistic models and phrase structures rather than relying on an unstructured bag-of-words assumption. Further research is also recommended to incorporate contextual relationships between neighboring image regions and to explore joint feature learning, which would improve the granularity of visual token clustering before performing the translation step.

The conclusions are subject to certain limitations, including the reliance on a constrained vocabulary of 4,500 images and the potential for segmentation errors to propagate into inaccurate visual tokens. Co-occurring concepts that appear together consistently cannot be disentangled without negative examples or larger corpora. Confidence is high in the theoretical framing and broad statistical trends, but operational deployment for open-vocabulary retrieval requires caution until more expressive language and region representations are incorporated.

Cover for Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary

Abstract

We describe a model of object recognition as machine translation. In this model, recognition is a process of annotating image regions with words. Firstly, images are segmented into regions, which are classified into region types using a variety of features. A mapping between region types and keywords supplied with the images, is then learned, using a method based around EM. This process is analogous with learning a lexicon from an aligned bitext. For the implementation we describe, these words are nouns taken from a large vocabulary. On a large test set, the method can predict numerous words with high accuracy. Simple methods identify words that cannot be predicted well. We show how to cluster words that individually are difficult to predict into clusters that can be predicted wellfor example, we cannot predict the distinction between train and locomotive using the current set of features, but we can predict the underlying concept. The method is trained on a substantial collection of images. Extensive experimental results illustrate the strengths and weaknesses of the approach.

Table of Contents

  • 1 Introduction
  • 1.1 Annotated images and auto-annotation
  • 2 Using EM to learn a Lexicon
  • 2.1 EM algorithm for finding the correspondence between blobs and words
  • 2.2 Maximum Likelihood Estimation with EM
  • 3 Applying and Refining the Lexicon
  • 3.1 Controlling the Vocabulary by Refusing to Predict
  • 3.2 Clustering Indistinguishable Words
  • 4 Experimental Results
  • 4.1 Evaluating Annotation
  • 4.2 Correspondence
  • 5 Discussion
  • Acknowledgements
  • References

Knowls

  1. Knowl 1 — Object Recognition as Statistical Machine Translation

    model/method

    Object recognition and image annotation are formulated as a statistical machine translation problem between two modalities: a visual vocabulary of discrete image region descriptors ("blobs") and a textual vocabulary of discrete semantic labels ("words").

    In this framework, an annotated image collection is treated as an aligned bilingual corpus (bitext) where each document consists of an image paired with caption keywords. At the document level, the correspondence between the set of image blobs and the set of words is known, but the fine-grained alignment between individual image regions and specific words is unobserved. Learning the mapping between visual regions and semantic concepts is thereby equivalent to learning a bilingual translation lexicon from partially aligned parallel text.

  2. Knowl 2 — Likelihood Function for Discrete Word-Blob Translation

    equation

    Let an annotated dataset consist of NN images. For the nn-th image (n{1,,N}n \in \{1, \dots, N\}), let wn=(wn1,wn2,,wnMn)w_n = (w_{n1}, w_{n2}, \dots, w_{nM_n}) denote the sequence of MnM_n keyword tokens and bn=(bn1,bn2,,bnLn)b_n = (b_{n1}, b_{n2}, \dots, b_{nL_n}) denote the sequence of LnL_n visual blob tokens.

    The marginal conditional probability of observing the word sequence wnw_n given the blob sequence bnb_n across all images is defined by marginalizing over all possible latent alignments between words and blobs:

    p(wb)=n=1Nj=1Mni=1Lnp(anj=i)t(w=wnjb=bni)p(w \mid b) = \prod_{n=1}^N \prod_{j=1}^{M_n} \sum_{i=1}^{L_n} p(a_{nj} = i) t(w = w_{nj} \mid b = b_{ni})

    where anj{1,,Ln}a_{nj} \in \{1, \dots, L_n\} is the alignment index associating word token wnjw_{nj} with visual blob token bnib_{ni}, p(anj=i)p(a_{nj} = i) is the mixing probability of assigning the jj-th word position to the ii-th blob position in an image of length pair (Ln,Mn)(L_n, M_n), and t(w=wb=b)t(w = w^* \mid b = b^*) is the translation probability of emitting word ww^* given visual blob bb^*. The parameters satisfy the normalization constraints i=1Lnp(anj=i)=1\sum_{i=1}^{L_n} p(a_{nj} = i) = 1 for each (n,j)(n, j) and wt(wb)=1\sum_{w^*} t(w^* \mid b^*) = 1 for each blob type bb^*.

  3. Knowl 3 — Expectation-Maximization Algorithm for Translation Lexicon Learning

    algorithm

    The translation probability table t(wb)t(w \mid b) and the alignment mixing probabilities p(anj=i)p(a_{nj}=i) are estimated using the Expectation-Maximization (EM) algorithm by maximizing the likelihood of the observed parallel image-keyword data with alignment variables treated as missing data.

    Input: Dataset of NN images, where image nn contains words (wn1,,wnMn)(w_{n1}, \dots, w_{nM_n}) and blobs (bn1,,bnLn)(b_{n1}, \dots, b_{nL_n})
    Output: Translation probability table t(wb)t(w \mid b) and mixing probabilities p(anj=i)p(a_{nj} = i)
    Initialize parameters θ={p(anj=i),t(wb)}\theta = \{p(a_{nj} = i), t(w \mid b)\}
    repeat
        // E-step: Compute expected word-blob alignments for each image nn
        for n=1n = 1 to NN do
            for j=1j = 1 to MnM_n do
                for i=1i = 1 to LnL_n do
                    p~(anj=iwnj,bni,θ)=p(anj=i)t(wnjbni)\tilde{p}(a_{nj} = i \mid w_{nj}, b_{ni}, \theta) = p(a_{nj} = i) \cdot t(w_{nj} \mid b_{ni})
                end for
                for i=1i = 1 to LnL_n do
                    p(anj=iwnj,bni,θ)=p~(anj=iwnj,bni,θ)i=1Lnp~(anj=iwnj,bni,θ)p(a_{nj} = i \mid w_{nj}, b_{ni}, \theta) = \frac{\tilde{p}(a_{nj} = i \mid w_{nj}, b_{ni}, \theta)}{\sum_{i'=1}^{L_n} \tilde{p}(a_{nj} = i' \mid w_{nj}, b_{ni'}, \theta)}
                end for
            end for
        end for
        // M-step: Update mixing probabilities stratified by image lengths (l,m)(l, m)
        for each unique pair of lengths (l,m)(l, m) with image count Nl,mN_{l,m} do
            for j=1j = 1 to mm and i=1i = 1 to ll do
                p(anj=i)=1Nl,mn:Ln=l,Mn=mp(anj=iwnj,bni,θ)p(a_{nj} = i) = \frac{1}{N_{l,m}} \sum_{n: L_n=l, M_n=m} p(a_{nj} = i \mid w_{nj}, b_{ni}, \theta)
            end for
        end for
        // M-step: Accumulate unnormalized translation table counts
        for each distinct blob token bb^* and word token ww^* do
            t~(wb)=n=1Nj=1Mni=1Lnp(anj=iwnj,bni,θ)δ(wnj=w,bni=b)\tilde{t}(w^* \mid b^*) = \sum_{n=1}^N \sum_{j=1}^{M_n} \sum_{i=1}^{L_n} p(a_{nj} = i \mid w_{nj}, b_{ni}, \theta) \cdot \delta(w_{nj} = w^*, b_{ni} = b^*)
        end for
        // Normalize translation table per blob
        for each distinct blob token bb^* do
            for each distinct word token ww^* do
                t(wb)=t~(wb)wt~(wb)t(w^* \mid b^*) = \frac{\tilde{t}(w^* \mid b^*)}{\sum_{w'} \tilde{t}(w' \mid b^*)}
            end for
        end for
    until convergence
  4. Knowl 4 — Region Feature Extraction and Visual Blob Quantization

    model/method

    To bridge continuous image representations and discrete machine translation algorithms, images undergo segmentation, feature extraction, and vector quantization:

    1. Segmentation: Images are segmented into coherent regions using the Normalized Cuts graph partitioning algorithm. Regions smaller than an area threshold are discarded, yielding approximately 5 to 10 visual regions per image.
    2. Feature Representation: Each extracted region is represented by a 33-dimensional feature vector consisting of:
      • Average region color and color standard deviation.
      • Average orientation energy computed across 12 orientation filters.
      • Geometric and structural descriptors: region area (size), centroid location coordinates, convexity measure, first spatial moment, and the ratio of region area to boundary perimeter squared.
    3. Vector Quantization (Blob Construction): The 33-dimensional feature vectors aggregated across all regions in the training set are clustered into K=500K = 500 discrete clusters using kk-means. Each cluster identifier corresponds to a discrete visual token termed a "blob".
  5. Knowl 5 — Null Prediction Thresholding and Vocabulary Retraining

    model/method

    Given a test image region quantized to blob bb, candidate word annotations are ranked by p(wb)=t(wb)p(w \mid b) = t(w \mid b). Two refinement mechanisms improve lexicon precision:

    1. Null Word Prediction: For any blob bb, if the maximum translation probability falls below a designated threshold τnull\tau_{\text{null}}:

    maxwt(wb)τnull\max_{w} t(w \mid b) \le \tau_{\text{null}}

    the model refuses to predict and assigns a null label to that region. This prevents ambiguous, small, or featureless segments from generating spurious word predictions.

    1. Vocabulary Pruning and Retraining: In the base model, many words in the large initial vocabulary never achieve the highest probability argmaxwt(wb)\arg\max_w t(w \mid b) for any blob bb. Restricting the active vocabulary strictly to words that can be predicted as the maximum for at least one blob and rerunning the EM algorithm eliminates probability mass dispersion onto unpredicted words, substantially improving the probability calibration, precision, and recall of the retained words.
  6. Knowl 6 — Word Clustering via Symmetrized KL Divergence and Normalized Cuts

    model/method

    Certain word pairs are visually indistinguishable given a chosen feature set (e.g., cat vs. tiger, eagle vs. jet) or consistently co-occur in the dataset such that correspondences cannot be disentangled (e.g., polar with bear, mare with horses).

    To identify and merge these equivalent words, a word similarity matrix is constructed using the symmetrized Kullback-Leibler (KL) divergence between the conditional blob distributions given the words. For words w1w_1 and w2w_2, the divergence between p(bw1)p(b \mid w_1) and p(bw2)p(b \mid w_2) is given by:

    Dsym(w1,w2)=DKL(p(bw1)p(bw2))+DKL(p(bw2)p(bw1))D_{\text{sym}}(w_1, w_2) = D_{\text{KL}}(p(b \mid w_1) \parallel p(b \mid w_2)) + D_{\text{KL}}(p(b \mid w_2) \parallel p(b \mid w_1))

    where p(bw)=t(wb)p(b)p(w)p(b \mid w) = \frac{t(w \mid b) p(b)}{p(w)}. Normalized Cuts is applied to this similarity matrix to recursively cluster the vocabulary, setting the number of target clusters to 75%75\% of the vocabulary size at each clustering iteration. Words within a cluster are merged into a single compound token, allowing the model to accurately predict the underlying visual concept.

  7. Knowl 7 — Corel Image Benchmark and Annotation Evaluation Protocol

    experimental setup

    The translation model is evaluated on the Corel stock photograph dataset:

    • Dataset Size & Vocabulary: 5,000 images are used, split into 4,500 training images and 500 held-out test images. The vocabulary consists of 371 keywords, with each image containing 4 to 5 ground-truth keywords and 5 to 10 segmented regions mapped to 500 kk-means blob clusters.
    • Image Retrieval Evaluation: Each test image is automatically annotated by segmenting its regions, mapping each region to its nearest blob token, and assigning the most probable word w=argmaxwt(wb)w = \arg\max_{w'} t(w' \mid b) if t(wb)>τnullt(w \mid b) > \tau_{\text{null}}. Image retrieval is evaluated across keywords by querying the automatically annotated dataset and measuring precision and recall against ground-truth image annotations.
    • Region Correspondence Evaluation: Manual inspection is conducted on 100 test images to directly evaluate whether the predicted word for a given blob corresponds to the correct physical object region in the image.
  8. Knowl 8 — Base Model Annotation Performance across Null Thresholds

    data/table

    Out of the 371 total vocabulary words, only 80 words achieve the maximum translation probability argmaxwt(wb)\arg\max_w t(w \mid b) for at least one blob in the base translation model. For these 80 predictable words, increasing the null word probability threshold τnull\tau_{\text{null}} from 0.00.0 to 0.40.4 trades off recall for higher precision, as regions with low confidence are suppressed.

    Word th = 0 th = 0.1 th = 0.2 th = 0.3 th = 0.4
    rec prec rec prec rec prec rec prec rec prec
    petals 0.50 1.00 0.50 1.00 0.50 1.00 0.50 1.00 0.50 1.00
    sky 0.83 0.34 0.80 0.35 0.58 0.44
    flowers 0.67 0.21 0.67 0.21 0.44 0.24
    horses 0.58 0.27 0.58 0.27 0.50 0.26
    foals 0.56 0.29 0.56 0.29 0.56 0.29
    mare 0.78 0.23 0.78 0.23
    tree 0.77 0.20 0.74 0.20
    people 0.74 0.22 0.74 0.22
    water 0.74 0.24 0.74 0.24
    sun 0.70 0.28 0.70 0.28
    bear 0.59 0.20 0.55 0.20
    stone 0.48 0.18 0.48 0.18
    buildings 0.48 0.17 0.48 0.17
    snow 0.48 0.17 0.48 0.19

    The table demonstrates that frequent visual background classes (e.g., sky, tree, water) and distinct object classes (e.g., petals, horses) attain high recall (>0.50>0.50) at τnull=0\tau_{\text{null}}=0. Specific words such as petals achieve perfect precision (1.001.00) across all thresholds where predictions are made.

  9. Knowl 9 — Quantitative Evaluation of Lexicon Retraining and Word Clustering

    data/table

    Pruning the vocabulary to the 80 predictable words and refitting the EM translation model increases both the prediction probability of words and their precision and recall across higher thresholds. Hierarchically clustering visually or semantically related words further boosts retrieval performance.

    Refitted Word th = 0 th = 0.1 th = 0.2 th = 0.3
    rec prec rec prec rec prec rec prec
    petals 0.50 1.00 0.50 1.00 0.50 1.00 0.50 1.00
    sky 0.83 0.31 0.83 0.31 0.75 0.37 0.58 0.47
    people 0.78 0.26 0.78 0.26 0.68 0.27 0.51 0.31
    water 0.75 0.25 0.75 0.25 0.72 0.26 0.44 0.27
    mare 0.78 0.23 0.78 0.23 0.67 0.21
    tree 0.71 0.19 0.71 0.19 0.66 0.20
    sun 0.60 0.38 0.60 0.38 0.60 0.43
    grass 0.57 0.19 0.57 0.19 0.49 0.22
    stone 0.57 0.16 0.57 0.16 0.52 0.23
    foals 0.56 0.26 0.56 0.26 0.56 0.26
    coral 0.56 0.19 0.56 0.19 0.56 0.19
    scotland 0.55 0.20 0.55 0.20 0.45 0.19
    flowers 0.48 0.17 0.48 0.17 0.48 0.18
    buildings 0.44 0.16 0.44 0.16
    1st Clusters r p 2nd Clusters r p 3rd Clusters r p
    horses mare 0.83 0.18 kit horses mare foals 0.77 0.16 kit horses mare foals 0.77 0.27
    leaf flowers 0.69 0.22 leaf flowers plants veg. 0.63 0.25 leaf flowers plants veg. 0.60 0.19
    plane 0.12 0.14 jet plane arctic 0.46 0.18 jet plane arctic prop flight.. 0.43 0.17
    pool athlete 0.33 0.31 pool athlete vines 0.17 0.50 pool athlete vines swimmers 0.75 0.27
    sun ceiling 0.60 0.30 sun ceiling 0.70 0.30 sun ceiling cave store 0.62 0.35
    sky beach 0.83 0.30 sky beach cathedral 0.82 0.31 sky beach cathedral clouds.. 0.87 0.36
    water 0.77 0.26 water 0.72 0.25 water waves 0.70 0.26
    tree 0.73 0.20 tree 0.76 0.20 tree 0.58 0.20
    people 0.68 0.24 people 0.62 0.26 people 0.54 0.25

    In the retrained model, words remain predictable up to τnull=0.3\tau_{\text{null}} = 0.3 (where the base model produced no predictions for most words). In the clustered lexicon, compound concepts such as pool athlete vines swimmers (r=0.75,p=0.27r=0.75, p=0.27) and sky beach cathedral clouds mural arch waterfalls (r=0.87,p=0.36r=0.87, p=0.36) substantially outperform single-word predictions.

  10. Knowl 10 — Region-Word Correspondence Accuracy and Alignment Ambiguity

    empirical result

    Manual region-level evaluation on 100 test images confirms that the model learns accurate local correspondence between image segments and semantic words without localized ground-truth supervision:

    • High-Accuracy Recognition: Prominent concepts achieve up to 70%70\% correct region localization on the test set. When the word ocean is predicted for a blob, it falls on an actual ocean region in 70%70\% of test occurrences.
    • Co-occurrence Ambiguity: The principal structural failure mode arises from non-separable co-occurrence patterns in the training data. When two concepts almost perfectly co-occur (such as grass and foals, or polar and bear), the translation model cannot determine which region is the animal and which is the background without incorporating multi-region spatial grouping or compound phrase discovery.

Coverage note — No substantial contributed material was omitted.

References

  1. 1.K. Barnard, P. Duygulu and D. A. Forsyth. Clustering art. In IEEE Conf. on Computer Vision and Pattern Recognition, II: 434-441, 2001.
  2. 2.K. Barnard and D. A. Forsyth. Learning the semantics of words and pictures. In Int. Conf. on Computer Vision pages 408-15, 2001.
  3. 3.P. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 32(2):263-311, 1993.
  4. 4.D.A. Forsyth and J. Ponce. Computer Vision: a modern approach. Prentice-Hall 2001, in preparation.
  5. 5.D. Jurafsky and J. H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition. Prentice-Hall, 2000.
  6. 6.C. D. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 1999.
  7. 7.M. Markkula and E. Sormunen. End-user searching challenges indexing practices in the digital newspaper photo archive. Information retrieval, 1:259-285, 2000.
  8. 8.Y. Mori, H. Takahashi, R. Oka Image-to-word transformation based on dividing and vector quantizing images with words In First International Workshop on Multimedia Intelligent Storage and Retrieval Management (MISRM'99), 1999
  9. 9.O. Maron. Learning from Ambiguity. PhD thesis, MIT, 1998.
  10. 10.O. Maron and A. L. Ratan. Multiple-Instance Learning for Natural Scene Classification, In The Fifteenth International Conference on Machine Learning, 1998
  11. 11.I. Dan Melamed. Empirical Methods for Exploiting Parallel Texts. MIT Press, 2001.
  12. 12.S. Ornager. View a picture, theoretical image analysis and empirical user studies on indexing and retrieval. Swedis Library Research, 2-3:31-41, 1996.
  13. 13.J. Shi and J. Malik. Normalised cuts and image segmentation. In IEEE Conf. on Computer Vision and Pattern Recognition, pages 731-737, 1997.

Citation

MLA
Duygulu, P., et al. “Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary”. Lecture Notes in Computer Science, Springer Berlin Heidelberg, 2002, pp. 97–112, https://doi.org/10.1007/3-540-47979-1_7.
APA
Duygulu, P., Barnard, K., de Freitas, J. F. G., & Forsyth, D. A. (2002). Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. In Lecture Notes in Computer Science (pp. 97–112). Springer Berlin Heidelberg. https://doi.org/10.1007/3-540-47979-1_7
Chicago
Duygulu, P., K. Barnard, J. F. G. de Freitas, and D. A. Forsyth. 2002. “Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary”. In Lecture Notes in Computer Science. Springer Berlin Heidelberg. https://doi.org/10.1007/3-540-47979-1_7.
Harvard
Duygulu, P. et al. (2002) “Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary”, Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 97–112. Available at: https://doi.org/10.1007/3-540-47979-1_7.
Vancouver
1. Duygulu P, Barnard K, de Freitas JFG, Forsyth DA (2002) Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. In: Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp 97–112

BibTeX

@inbook{Duygulu_2002, title={Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary}, ISBN={9783540479796}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/3-540-47979-1_7}, DOI={10.1007/3-540-47979-1_7}, booktitle={Computer Vision — ECCV 2002}, publisher={Springer Berlin Heidelberg}, author={Duygulu, P. and Barnard, K. and de Freitas, J. F. G. and Forsyth, D. A.}, year={2002}, pages={97–112} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF