Adapting Visual Category Models to New Domains

Kate SaenkoBrian KulisMario FritzTrevor Darrell

article2010ECCV3,249 citations

Proposes a metric learning framework for visual domain adaptation that aligns feature distributions using cross-domain pairwise constraints and introduces the multi-domain Office benchmark dataset to evaluate object recognition across disparate imaging conditions.

Listen

Object recognition systems trained on large labeled image collections often fail when applied to images from new cameras, lighting conditions, or environments, because the underlying feature distributions change. This domain shift limits practical use in settings such as robotics, where labeled data for every new sensor or scene is scarce.

The work set out to develop and test a general method that learns a transformation mapping features from a source domain closer to those of a target domain, allowing classifiers trained on abundant source labels to perform well in the target domain even when few or no target labels exist for some categories.

The authors created a new benchmark dataset containing the same 31 object categories imaged in three distinct domains—web photographs, high-resolution DSLR images, and low-resolution webcam images—plus instance-level subsets. They formulated domain adaptation as the problem of learning a regularized linear or kernelized transformation from labeled pairs of similar and dissimilar cross-domain examples, then applied the resulting distance function inside a nearest-neighbor classifier. Experiments compared the approach against unadapted baselines and standard metric-learning methods across multiple source–target pairs and two settings: categories seen during adaptation and entirely new categories.

The learned transformation raised classification accuracy in every tested shift. Gains were largest for the most severe shift (web to webcam), where performance roughly doubled relative to the unadapted baseline. The method also improved accuracy on categories never labeled in the target domain, and the asymmetric variant outperformed the symmetric one when domain differences were large. Standard metric learning applied to pooled data performed worse than the domain-specific constraints used here.

These results show that a modest number of cross-domain labeled pairs can recover enough of the domain shift to make existing object models usable in new imaging conditions without collecting target labels for every category. The technique therefore reduces the labeling burden that currently prevents large public datasets from being deployed on robots or other specialized platforms.

Further gains are likely from richer constraint sets, automatic selection of which source categories to use for learning the transform, and integration with classifiers beyond nearest neighbors. The main limitations are reliance on at least some target-domain labels to build the transformation, restriction of the symmetric version to equal-dimensional features, and evaluation on only one classifier and one dataset; results should therefore be confirmed on additional domains and tasks before broad deployment.

International Computer Science Institute University of California Berkeley computer vision database systems object recognition Domain Adaptation Visual Category Model Cross-Domain Transform

Saenko et al (2010).pdf

Paper: A theory of learning from different domains, Shai Ben-David et al. (2010). This foundational theoretical study on domain adaptation provides the generalization bounds and divergence measures that motivate transform-based feature alignment.

Paper: Unsupervised Domain Adaptation by Backpropagation, Yaroslav Ganin et al. (2015). This work extends early explicit domain transformation techniques into an end-to-end adversarial framework using gradient reversal layers for deep networks.
Paper: Adversarial Discriminative Domain Adaptation, Eric Tzeng et al. (2017). This paper builds directly upon transfer methodologies by introducing Adversarial Discriminative Domain Adaptation to align source and target feature spaces.

Cover for Adapting Visual Category Models to New Domains

Abstract

Abstract. Domain adaptation is an important emerging topic in computer vision. In this paper, we present one of the first studies of domain shift in the context of object recognition. We introduce a method that adapts object models acquired in a particular visual domain to new imaging conditions by learning a transformation that minimizes the effect of domain-induced changes in the feature distribution. The transformation is learned in a supervised manner and can be applied to categories for which there are no labeled examples in the new domain. While we focus our evaluation on object recognition tasks, the transform-based adaptation technique we develop is general and could be applied to non-image data. Another contribution is a new multi-domain object database, freely available for download. We experimentally demonstrate the ability of our method to improve recognition on categories with few or no target domain labels and moderate to large changes in the imaging conditions.

Topics

computer vision database systems object recognition abstract domains

Keywords

Domain Adaptation Visual Category Model Cross-Domain Transform Information-Theoretic Metric Learning Symmetric Transform Asymmetric Transform Multi-Domain Object Database SURF Feature k-NN Classifier ITML

1 Introduction
2 Related Work
3 Domain Adaptation Using Regularized Cross-Domain Transforms
3.1 Domain Adaptation Using Metric Learning
4 A Database for Studying Effects of Domain Shift in Object Recognition
5 Experiments
6 Conclusion
References

Knowls

Knowl 1 — Symmetric Domain Adaptation via Information-Theoretic Metric Learning
model/method
When feature representations across domains share identical dimensionality ( $d_A = d_B = d$ ), visual domain shift can be modeled as a linear scaling and rotation parameterized by a symmetric positive semi-definite matrix $W \in \mathbb{R}^{d \times d}$ . Factoring $W = G^T G$ , the transformed Mahalanobis distance between a source point $x \in \mathcal{A}$ and a target point $y \in \mathcal{B}$ is:

$d_W(x, y) = (x - y)^T W (x - y) = (G x - G y)^T (G x - G y)$

This formulation projects data points from both domains into a shared invariant subspace via linear operator $G$ , where distance corresponds to squared Euclidean distance.

The symmetric adaptation objective is formulated using Information-Theoretic Metric Learning (ITML) with the LogDet divergence regularizer $r(W) = \text{tr}(W) - \log \det(W)$ , which preserves scale and rotation invariance:

$\min_{W \succeq 0} \left[ \text{tr}(W) - \log \det(W) \right]$

$\text{subject to} \quad d_W(x_i^A, x_j^B) \le u \quad \forall (i, j) \in \mathcal{S}$

$d_W(x_i^A, x_j^B) \ge \ell \quad \forall (i, j) \in \mathcal{D}$

where $\mathcal{S}$ is a set of cross-domain similarity pairs, $\mathcal{D}$ is a set of cross-domain dissimilarity pairs, and $u, \ell \in \mathbb{R}$ are upper and lower distance threshold bounds. Slack variables governed by trade-off parameter $\lambda$ are incorporated to guarantee feasibility.
Knowl 2 — Regularized Cross-Domain Transformation Formulation for Domain Adaptation
model/method
The cross-domain transformation framework maps feature representations between a source domain $\mathcal{A}$ and a target domain $\mathcal{B}$ . Given feature vectors $x \in \mathbb{R}^{d_A}$ from domain $\mathcal{A}$ and $y \in \mathbb{R}^{d_B}$ from domain $\mathcal{B}$ , a linear transformation matrix $W \in \mathbb{R}^{d_A \times d_B}$ maps data points from $\mathcal{B}$ to $\mathcal{A}$ (or equivalently $W^T \in \mathbb{R}^{d_B \times d_A}$ from $\mathcal{A}$ to $\mathcal{B}$ ).

The cross-domain inner product similarity function is defined as:

$\text{sim}_W(x, y) = x^T W y$

Let $X = [x_1, \dots, x_{n_A}] \in \mathbb{R}^{d_A \times n_A}$ denote the matrix of $n_A$ training examples from domain $\mathcal{A}$ , and let $Y = [y_1, \dots, y_{n_B}] \in \mathbb{R}^{d_B \times n_B}$ denote the matrix of $n_B$ training examples from domain $\mathcal{B}$ . To avoid overfitting, $W$ is learned by minimizing a regularization function $r(W)$ subject to $c$ cross-domain supervision constraints $c_i(X^T W Y) \ge 0$ :

$\min_W r(W) \quad \text{subject to} \quad c_i(X^T W Y) \ge 0, \quad 1 \le i \le c$

Allowing for potential constraint infeasibility via penalty trade-off parameter $\lambda > 0$ , the optimization problem is expressed in unconstrained form as:

$\min_W \left[ r(W) + \lambda \sum_{i=1}^c c_i(X^T W Y) \right]$
Knowl 3 — Cross-Domain Pairwise Constraint Construction
model/method
Supervision for learning domain-invariant transformations is established exclusively through cross-domain pairs $(x_i^A, x_j^B)$ consisting of one source sample $x_i^A \in \mathcal{A}$ and one target sample $x_j^B \in \mathcal{B}$ . Intra-domain pairs (where both samples belong to the same domain) are deliberately excluded, because including same-domain constraints biases the objective toward standard within-domain clustering rather than aligning the disparate distributions across domain boundaries.

Two types of cross-domain constraints are constructed:
1. Class-based constraints: For labeled source data $(x_i^A, l_i^A)$ and target data $(x_j^B, l_j^B)$ with category labels $l_i^A, l_j^B$ , cross-domain pairs are assigned distance bounds:
$d_W(x_i^A, x_j^B) \le u \quad \text{if } l_i^A = l_j^B \quad ((i, j) \in \mathcal{S})$

$d_W(x_i^A, x_j^B) \ge \ell \quad \text{if } l_i^A \ne l_j^B \quad ((i, j) \in \mathcal{D})$
1. Correspondence constraints: When the source and target datasets contain images of the same physical object instance (e.g., under different camera viewpoints or lighting conditions), constraints are established between instances of the same object in similar poses ( $d_W(x_i^A, x_j^B) \le u$ ). This enables transferring learned domain shift structures to novel object categories for which no target-domain labels exist.
Knowl 4 — Information-Theoretic Metric Learning Algorithm for Cross-Domain Adaptation
algorithm
The cross-domain metric learning adaptation algorithm optimizes a symmetric positive semi-definite matrix $W$ (or its associated kernel matrix $K$ ) by iteratively selecting the most violated cross-domain constraint and applying a rank-one Bregman projection update.
Input: Source training set $X \in \mathbb{R}^{d \times n_A}$ , target training set $Y \in \mathbb{R}^{d \times n_B}$ , similarity pair set $\mathcal{S}$ , dissimilarity pair set $\mathcal{D}$ , distance bounds $u$ and $\ell$ , tolerance $\epsilon$
Output: Learned kernel matrix $K$ and metric transformation $W$
Initialize $\bar{X} = [X \; Y] \in \mathbb{R}^{d \times (n_A + n_B)}$
Initialize kernel matrix $K_0 = \bar{X}^T \bar{X}$ (or compute via RBF kernel)
Set $t = 0$
while true do
    Find pair $(i, j) \in \mathcal{S} \cup \mathcal{D}$ with maximum constraint violation
    Compute current distance $d_t(x_i^A, x_j^B) = (e_i^A - e_j^B)^T K_t (e_i^A - e_j^B)$
    if maximum constraint violation $\le \epsilon$ then
        break
    end if
    Compute step parameter $\beta_t$ based on constraint bound ( $u$ or $\ell$ ) and violation magnitude
    Update kernel matrix:
         $K_{t+1} = K_t + \beta_t K_t (e_i^A - e_j^B)(e_i^A - e_j^B)^T K_t$
     $t = t + 1$
end while
return $K_t$
In the linear case, the corresponding parameter update is $W_{t+1} = W_t + \beta_t W_t (x_i^A - x_j^B)(x_i^A - x_j^B)^T W_t$ , starting from $W_0 = I$ . The vector $e_i^A$ represents the standard basis vector indexing $x_i^A$ in the concatenated data matrix $\bar{X}$ .
Knowl 5 — Multi-Domain Visual Category Benchmark Dataset (Office Dataset)
definition
The multi-domain Office object dataset comprises 4,652 images organized across 31 visual object categories collected in office environments under three distinct imaging domains:
1. Amazon (amazon): 31 categories with an average of 90 images per category (2,817 images total) collected from merchant listings on www.amazon.com. Images depict objects under studio lighting against clean backgrounds, primarily in canonical poses. An instance subset (amazonINS) contains 17 specific object instances with an average of 2 images each.
2. Digital SLR (dslr): 31 categories with 5 distinct physical objects per category (423 images total) captured in office environments using a high-end digital SLR camera at high resolution ( $4288 \times 2848$ ) with low noise under natural lighting (average 3 images per object across varying viewpoints). An instance subset (dslrINS) contains 534 images across 17 object instances in home environments.
3. Webcam (webcam): 31 categories with the identical 5 physical objects per category as the dslr domain (795 images total) captured using a low-resolution ( $640 \times 480$ ) consumer webcam, containing substantial sensor noise, motion blur, and color/white balance artifacts.
The 31 categories are: backpack, bike, bike helmet, bookcase, bottle, calculator, desk chair, desk lamp, computer, file cabinet, headphones, keyboard, laptop, letter tray, mobile phone, monitor, mouse, mug, notebook, pen, phone, printer, projector, puncher, ring binder, ruler, scissors, speaker, stapler, tape, and trash can.
Knowl 6 — Visual Feature Extraction and Bag-of-Words Representation Pipeline
experimental setup
Images are represented using a bag-of-visual-words representation built from local invariant feature descriptors:
1. Preprocessing: Images are resized to a uniform width and converted to grayscale.
2. Keypoint Detection: Scale-invariant interest points are detected using the Speeded Up Robust Features (SURF) detector with a Hessian blob response threshold of 1000.
3. Feature Description: An upright, non-rotationally invariant 64-dimensional SURF descriptor is computed over the local image patch surrounding each detected interest point.
4. Codebook Generation: A visual vocabulary of size 800 is constructed by applying $k$ -means clustering to a random subset of SURF descriptors extracted from the Amazon domain.
5. Histogram Quantization: Each image is converted into an 800-dimensional histogram of visual word occurrences by vector-quantizing its SURF descriptors against the 800-word codebook. No spatial pyramid pooling or color features are incorporated.

Knowl 7 — Classification Performance on Same-Category Visual Domain Adaptation

data/table

In the same-category domain adaptation setting, training data contains labeled examples for all 31 categories in both domains: 3 labeled examples per category in the target domain, and 8 (for webcam/dslr) or 20 (for amazon) examples per category in the source domain. Performance is evaluated using a 1-nearest-neighbor ( $1$ -NN) classifier with an RBF kernel ( $\sigma = 1.0$ , slack parameter $\lambda = 10^2$ ), reporting classification accuracy averaged over 10 random train/test splits.

Domain A	Domain B	knnAA	knnAB	knnBB	ITML(A+B)	ITML(B)	symm
webcam	dslr	0.34	0.14	0.20	0.18	0.23	0.27
dslr	webcam	0.31	0.25	0.23	0.23	0.28	0.31
amazon	webcam	0.33	0.03	0.43	0.41	0.43	0.44

The baseline methods comprise:

knnAA: $1$ -NN trained and tested entirely within domain A (representing upper-bound performance without domain shift).
knnAB: $1$ -NN trained on domain A and evaluated directly on domain B without adaptation.
knnBB: $1$ -NN trained and tested exclusively on domain B using only the 3 available target labels per class.
ITML(A+B): Standard Information-Theoretic Metric Learning trained on pooled data from both domains without domain distinction.
ITML(B): Standard ITML trained exclusively on domain B labeled data.

The symmetric cross-domain metric learning method (symm) outperforms direct cross-domain transfer (knnAB) across all domain shifts (e.g., improving webcam $\to$ dslr from 0.14 to 0.27, and amazon $\to$ webcam from 0.03 to 0.44) and outperforms standard pooled metric learning (ITML(A+B)), confirming the efficacy of restricting constraints strictly across domain boundaries.

Knowl 8 — Classification Performance on Novel-Category Visual Domain Adaptation

data/table

In the novel-category domain adaptation setting, target domain labeled data is available only for a training subset of categories (the first half), while evaluation is conducted on the remaining novel categories that possess zero target-domain training labels. For webcam $\to$ dslr (a 16-way classification task on novel categories), cross-domain correspondence constraints are formed on the first half of categories between images of matching physical objects in similar poses. For amazonINS $o$ dslrINS (a 17-way instance recognition task), correspondence constraints are formed across shared product instances. Classification is performed via a 1-NN classifier, reporting accuracy averaged over 10 train/test splits.

Baseline methods include knnAB (unadapted 1-NN trained on source and evaluated on target) and ITML(A+B) (standard metric learning on pooled data). For webcam $\to$ dslr, symm improves classification accuracy from 0.37 to 0.49, outperforming pooled metric learning (0.38). This confirms that cross-domain transformations capture domain shift properties that generalize to completely unseen object categories. On amazonINS $o$ dslrINS, symm achieves 0.25 compared to 0.23 for knnAB, limited by training data sparsity (only 2 Amazon images per instance).

Knowl 9 — Degradation of Standard Visual Classifiers Under Domain Shift
empirical result
Standard visual category models trained on one image domain exhibit severe performance degradation when evaluated directly on another visual domain without adaptation. On the 31-category Office dataset, classification accuracy evaluated across 5 random 80%-20% train/test splits demonstrates:
- A Support Vector Machine classifier trained on bag-of-words histograms (SVM-bow) achieves $54 \pm 2\%$ accuracy when trained and tested on the source domain (amazon), but drops to $20 \pm 1\%$ accuracy when trained on amazon and evaluated on the target domain (webcam).
- A Naive Bayes Nearest Neighbor (NBNN) classifier achieves $61 \pm 1\%$ accuracy when trained and tested on the source domain, but drops to $19 \pm 1\%$ accuracy when trained on amazon and tested on webcam.
This performance drop occurs despite using identical feature extraction (SURF interest points quantized into visual word histograms) and standard normalizations, showing that domain shifts (differences in viewpoint, resolution, lighting, and background) severely violate standard classifier assumptions.

Coverage note — Deliberately omitted detailed mathematical derivations for the asymmetric adaptation model (asymm) as the paper explicitly defers its formal description to a cited technical report, though comparative empirical results for asymm are included in the performance knowls.

References

1.Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006)
2.Blitzer, J., Dredze, M., Pereira, F.: Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In: ACL (2007)
3.Boiman, O., Shechtman, E., Irani, M.: In defense of nearest-neighbor based image classification. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos (2008)
4.Bosch, A., Zisserman, A., Munoz, X.: Representing shape with a spatial pyramid kernel. In: CIVR (2007)
5.Chechik, G., Sharma, V., Shalit, U., Bengio, S.: Large scale online learning of image similarity through ranking. In: Pattern Recognition and Image Analysis (2009)
6.Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: Proc. CVPR (2005)
7.Daume III, H.: Frustratingly easy domain adaptation. In: ACL (2007)
8.Davis, J., Kulis, B., Jain, P., Sra, S., Dhillon, I.: Information-theoretic metric learning. In: ICML (2007)
9.Duan, L., Tsang, I.W., Xu, D., Maybank, S.J.: Domain transfer svm for video concept detection. In: CVPR (2009)
10.Fink, M.: Object classification from a single example utilizing class relevance metrics. In: Proc. NIPS (2004)
11.Hertz, T., Bar-Hillel, A., Weinshall, D.: Learning distance functions for image retrieval. In: CVPR (2004)
12.Hertz, T., Hillel, A.B., Weinshall, D.: Learning a kernel function for classification with small training samples. In: International Conference on Machine Learning (ICML), pp. 401–408 (2006)
13.Jiang, W., Zavesky, E., Chang, S., Loui, A.: Cross-domain learning methods for high-level visual concept classification. In: ICIP (2008)
14.Kulis, B., Jain, P., Grauman, K.: Fast similarity search for learned metrics. IEEE PAMI 39(12), 2143–2157 (2009)
15.Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Visual domain adaptation using regularized cross-domain transforms. Technical Report UCB/EECS-2010-106, EECS Department, University of California, Berkeley (July 2010)
16.Stark, M., Goesele, M., Schiele, B.: A shape-based object class model for knowledge transfer. In: ICCV (2009)
17.Varma, M., Ray, D.: Learning the discriminative power-invariance trade-off. In: ICCV (2007)
18.Yang, J., Yan, R., Hauptmann, A.G.: Cross-domain video concept detection using adaptive svms. In: ACM Multimedia (2007)

Citation

MLA

Saenko, K., et al. “Adapting Visual Category Models to New Domains”. Lecture Notes in Computer Science, Springer Berlin Heidelberg, 2010, pp. 213–26, https://doi.org/10.1007/978-3-642-15561-1_16.

APA

Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010). Adapting Visual Category Models to New Domains. In Lecture Notes in Computer Science (pp. 213–226). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-15561-1_16

Chicago

Saenko, K., B. Kulis, M. Fritz, and T. Darrell. 2010. “Adapting Visual Category Models to New Domains”. In Lecture Notes in Computer Science. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-15561-1_16.

Harvard

Saenko, K. et al. (2010) “Adapting Visual Category Models to New Domains”, Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 213–226. Available at: https://doi.org/10.1007/978-3-642-15561-1_16.

Vancouver

1. Saenko K, Kulis B, Fritz M, Darrell T (2010) Adapting Visual Category Models to New Domains. In: Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp 213–226

BibTeX

@inbook{Saenko_2010, title={Adapting Visual Category Models to New Domains}, ISBN={9783642155611}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/978-3-642-15561-1_16}, DOI={10.1007/978-3-642-15561-1_16}, booktitle={Computer Vision – ECCV 2010}, publisher={Springer Berlin Heidelberg}, author={Saenko, Kate and Kulis, Brian and Fritz, Mario and Darrell, Trevor}, year={2010}, pages={213–226} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Adapting Visual Category Models to New Domains

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Symmetric Domain Adaptation via Information-Theoretic Metric Learning

Knowl 2 — Regularized Cross-Domain Transformation Formulation for Domain Adaptation

Knowl 3 — Cross-Domain Pairwise Constraint Construction

Knowl 4 — Information-Theoretic Metric Learning Algorithm for Cross-Domain Adaptation

Knowl 5 — Multi-Domain Visual Category Benchmark Dataset (Office Dataset)