Deep Metric Learning via Lifted Structured Feature Embedding

Hyun Oh SongYu XiangStefanie JegelkaSilvio Savarese

article2015CVPR1,803 citations

Proposes a lifted structured prediction loss that optimizes the entire matrix of pairwise distances within training batches for deep metric learning, substantially outperforming traditional triplet methods across visual retrieval benchmarks including the newly introduced Online Products dataset.

Listen

Visual search, product recommendation, and large-scale image categorization require systems to quickly determine whether two images show the same object or category. Traditional classification models struggle when faced with tens of thousands of categories and very few sample images per category. To address this challenge, deep metric learning maps images into a continuous feature space where similar items cluster together and dissimilar items remain separated. Existing training strategies, however, process isolated pairs or triplets within each training batch, leaving valuable structural relationships between the remaining samples unused and slowing down model convergence.

The article develops and evaluates a training methodology that utilizes all pairwise relationships across an entire mini-batch rather than analyzing isolated pairs or triplets. The objective is to demonstrate that lifting batch distance vectors into a dense pairwise matrix and optimizing a structured loss function significantly improves visual retrieval and clustering accuracy on previously unseen categories.

The authors implemented their structured learning algorithm within a standard deep convolutional neural network architecture. They evaluated performance across three distinct image collections: a fine-grained bird dataset, a fine-grained car dataset, and a newly assembled collection of 120,000 online e-commerce product images across roughly 23,000 distinct classes. In every benchmark, the training and testing sets contained entirely non-overlapping categories to verify general similarity learning. The evaluation measured clustering quality via standard statistical metrics and retrieval quality via top-match recall rates across various embedding dimensions.

The experimental findings show that the proposed approach consistently outperforms standard pair-based and triplet-based embedding baselines across all datasets and dimensionalities. On fine-grained car and bird benchmarks, the algorithm achieved substantially higher clustering scores and retrieval rates compared to standard methods. On the e-commerce product dataset, the model maintained high retrieval precision, with most misclassifications limited to subtle differences between very similar products. Furthermore, the approach simplifies network training by using a single network stream rather than multi-branch architectures.

These results indicate that organizations managing large visual catalogs can achieve superior search and discovery performance without retraining models as new inventory categories appear. By fully utilizing in-batch relationships, development teams can train higher-performing visual search engines with greater stability and fewer architectural complexities. Consequently, systems can handle extreme classification challenges and zero-shot retrieval tasks with higher confidence and reduced operational overhead.

Organizations developing visual search, digital asset management, or e-commerce discovery platforms should consider adopting dense batch-level structured loss formulations for their embedding pipelines. Initial deployments should benchmark the approach against existing retrieval pipelines to measure search relevance gains. Future investigations should examine how these embeddings scale under additional domain shifts and explore extending the dataset for broader fine-grained visual recognition tasks.

The empirical findings are well-supported across multiple benchmarks and standard embedding dimensions. Nevertheless, practitioners should account for potential limitations, including computational memory demands when scaling to exceptionally large batch sizes and visual degradation caused by highly ambiguous or near-identical fine-grained items.

Massachusetts Institute of Technology Stanford University convolutional neural networks network architecture online learning deep metric learning lifted structured feature embedding structured loss objective

arXiv: 1511.06452 rksltnl/Deep-Metric-Learning-CVPR16

Paper: Deep Metric Learning Using Triplet Network, Elad Hoffer et al. (2014). It introduces deep metric learning with triplet networks, establishing the relative comparison paradigm that the source generalizes to batch-wide lifted structured embeddings.
Paper: Dimensionality Reduction by Learning an Invariant Mapping, Raia Hadsell et al. (2006). It provides the foundational contrastive loss and Siamese network framework for learning distance mappings using pairwise relationships.
Paper: Distance Metric Learning for Large Margin Nearest Neighbor Classification, Kilian Q. Weinberger et al. (2005). It formulates margin-based distance metric learning for nearest-neighbor classification, which directly informs the margin and neighborhood structure utilized in deep embedding losses.
Paper: Learning a similarity metric discriminatively, with application to face verification, Sumit Chopra et al. (2005). It pioneers discriminative similarity metric learning with convolutional Siamese networks under contrastive objectives.
Paper: Distance Metric Learning with Application to Clustering with Side-Information, Eric P. Xing et al. (2002). It establishes the classic convex optimization framework for learning distance metrics from pairwise side-information constraints.

Paper: Improved Deep Metric Learning with Multi-class N-pair Loss Objective, Kihyuk Sohn (2016). It extends batch-aware deep metric learning by proposing an N-pair multi-class objective evaluated directly on the Stanford Online Products and fine-grained benchmarks introduced in the source.
Paper: In Defense of the Triplet Loss for Person Re-Identification, Alexander Hermans et al. (2017). It develops in-batch hard negative mining techniques (batch-hard loss) that build upon structured batch-level distance matrices for deep metric embeddings.
Paper: A Discriminative Feature Learning Approach for Deep Face Recognition, Yandong Wen et al. (2016). It proposes center loss to enhance feature compactness and discriminability for deep metric verification tasks, complementing pairwise and structured embedding methods.
Paper: SphereFace: Deep Hypersphere Embedding for Face Recognition, Weiyang Liu et al. (2017). It advances deep metric representation learning by introducing angular margin constraints on hyperspherical embeddings.
Paper: Unsupervised Feature Learning via Non-parametric Instance Discrimination, Zhirong Wu et al. (2018). It generalizes metric embedding principles to unsupervised representation learning using non-parametric instance discrimination and memory banks.

Cover for Deep Metric Learning via Lifted Structured Feature Embedding

Abstract

Learning the distance metric between pairs of examples is of great importance for learning and visual recognition. With the remarkable success from the state of the art convolutional neural networks, recent works have shown promising results on discriminatively training the networks to learn semantic feature embeddings where similar examples are mapped close to each other and dissimilar examples are mapped farther apart. In this paper, we describe an algorithm for taking full advantage of the training batches in the neural network training by lifting the vector of pairwise distances within the batch to the matrix of pairwise distances. This step enables the algorithm to learn the state of the art feature embedding by optimizing a novel structured prediction objective on the lifted problem. Additionally, we collected Online Products dataset: 120k images of 23k classes of online products for metric learning. Our experiments on the CUB-200-2011, CARS196, and Online Products datasets demonstrate significant improvement over existing deep feature embedding methods on all experimented embedding sizes with the GoogLeNet network.

Topics

convolutional neural networks network architecture online learning online products semantic information discriminative training

Keywords

deep metric learning lifted structured feature embedding structured loss objective convolutional neural networks pairwise distance matrix image similarity example retrieval

1 Introduction
2 Related works
3 Review
4 Deep metric learning via lifted structured feature embedding
5 Implementation details
6 Evaluation
7 Experiments
7.1 CUB-200-2011
7.2 CARS196
7.3 Online Products dataset
8 Conclusion
References

Knowls

Knowl 1 — Smooth Lifted Structured Loss Objective
equation
In deep metric learning on mini-batches, the lifted structured feature embedding optimizes a smooth upper-bound loss $\tilde{J}$ defined over all positive and negative pairs present within the mini-batch:

$\tilde{J} = \frac{1}{2|\mathcal{P}|} \sum_{(i,j) \in \mathcal{P}} \max\left(0, \tilde{J}_{i,j}\right)^2$

where $\mathcal{P}$ is the set of positive pairs (pairs of examples sharing the same class label) in the batch, and $\tilde{J}_{i,j}$ measures the margin-violation score for positive pair $(i, j)$ with respect to all negative pairs $\mathcal{N}$ in the batch:

$\tilde{J}_{i,j} = \log \left( \sum_{(i,k) \in \mathcal{N}} \exp\{\alpha - D_{i,k}\} + \sum_{(j,l) \in \mathcal{N}} \exp\{\alpha - D_{j,l}\} \right) + D_{i,j}$

Here, $f(x_i) \in \mathbb{R}^c$ denotes the $c$ -dimensional embedded feature vector for input $x_i$ , $D_{i,j} = \|f(x_i) - f(x_j)\|_2$ is the Euclidean distance between embeddings $f(x_i)$ and $f(x_j)$ , $\alpha > 0$ is a predefined distance margin parameter, and $\mathcal{N}$ is the set of negative pairs (pairs of examples with different class labels) in the batch. The log-sum-exp formulation acts as a smooth surrogate to taking the maximum negative margin violation $\max\left(\max_{(i,k) \in \mathcal{N}} (\alpha - D_{i,k}), \max_{(j,l) \in \mathcal{N}} (\alpha - D_{j,l})\right) + D_{i,j}$ , which prevents the network optimization from getting trapped in poor local optima.
Knowl 2 — Distance Gradients for Lifted Structured Loss
equation
The partial derivatives of the smooth lifted structured loss $\tilde{J}$ with respect to the pairwise Euclidean distances $D_{i,j}$ , $D_{i,k}$ , and $D_{j,l}$ for a positive pair $(i,j) \in \mathcal{P}$ and negative pairs $(i,k) \in \mathcal{N}$ and $(j,l) \in \mathcal{N}$ are:

$\frac{\partial \tilde{J}}{\partial D_{i,j}} = \frac{1}{|\mathcal{P}|} \tilde{J}_{i,j} \mathbf{1}\left[\tilde{J}_{i,j} > 0\right]$

$\frac{\partial \tilde{J}}{\partial D_{i,k}} = \frac{1}{|\mathcal{P}|} \tilde{J}_{i,j} \mathbf{1}\left[\tilde{J}_{i,j} > 0\right] \frac{-\exp\{\alpha - D_{i,k}\}}{\exp\left\{\tilde{J}_{i,j} - D_{i,j}\right\}}$

$\frac{\partial \tilde{J}}{\partial D_{j,l}} = \frac{1}{|\mathcal{P}|} \tilde{J}_{i,j} \mathbf{1}\left[\tilde{J}_{i,j} > 0\right] \frac{-\exp\{\alpha - D_{j,l}\}}{\exp\left\{\tilde{J}_{i,j} - D_{i,j}\right\}}$

where $\mathbf{1}[\cdot]$ is the indicator function evaluating to $1$ when the condition is true and $0$ otherwise, $\mathcal{P}$ is the set of positive pairs in the mini-batch, $\mathcal{N}$ is the set of negative pairs in the mini-batch, $\alpha$ is the margin hyperparameter, $D_{i,j} = \|f(x_i) - f(x_j)\|_2$ , and $\tilde{J}_{i,j} = \log \left( \sum_{(i,k) \in \mathcal{N}} \exp\{\alpha - D_{i,k}\} + \sum_{(j,l) \in \mathcal{N}} \exp\{\alpha - D_{j,l}\} \right) + D_{i,j}$ . These smooth derivatives supply non-zero learning signals for all negative examples that fall within the margin of any positive pair, stabilizing gradient descent compared to only updating the single hardest negative.
Knowl 3 — Backpropagation Gradient Computation for Lifted Structured Embedding
algorithm
The backpropagation gradients of the lifted structured loss with respect to all feature embedding vectors $f(x_1), \dots, f(x_m)$ in a mini-batch of $m$ examples are computed by accumulating the chain-rule gradients over all pairwise distance terms:
Input: Distance matrix $D \in \mathbb{R}^{m \times m}$ , margin $\alpha$ , positive pair set $\mathcal{P}$ , negative pair set $\mathcal{N}$
Output: Feature gradients $\frac{\partial \tilde{J}}{\partial f(x_i)}$ for all $i \in \{1, \dots, m\}$
Initialize: $\frac{\partial \tilde{J}}{\partial f(x_i)} = 0$ for all $i \in \{1, \dots, m\}$
for $i = 1, \dots, m$ do
    for $j = i + 1, \dots, m$ such that $(i, j) \in \mathcal{P}$ do
         $\frac{\partial \tilde{J}}{\partial f(x_i)} \leftarrow \frac{\partial \tilde{J}}{\partial f(x_i)} + \frac{\partial \tilde{J}}{\partial D_{i,j}} \frac{\partial D_{i,j}}{\partial f(x_i)}$
         $\frac{\partial \tilde{J}}{\partial f(x_j)} \leftarrow \frac{\partial \tilde{J}}{\partial f(x_j)} + \frac{\partial \tilde{J}}{\partial D_{i,j}} \frac{\partial D_{i,j}}{\partial f(x_j)}$
        for $k = 1, \dots, m$ such that $(i, k) \in \mathcal{N}$ do
             $\frac{\partial \tilde{J}}{\partial f(x_i)} \leftarrow \frac{\partial \tilde{J}}{\partial f(x_i)} + \frac{\partial \tilde{J}}{\partial D_{i,k}} \frac{\partial D_{i,k}}{\partial f(x_i)}$
             $\frac{\partial \tilde{J}}{\partial f(x_k)} \leftarrow \frac{\partial \tilde{J}}{\partial f(x_k)} + \frac{\partial \tilde{J}}{\partial D_{i,k}} \frac{\partial D_{i,k}}{\partial f(x_k)}$
        end
        for $l = 1, \dots, m$ such that $(j, l) \in \mathcal{N}$ do
             $\frac{\partial \tilde{J}}{\partial f(x_j)} \leftarrow \frac{\partial \tilde{J}}{\partial f(x_j)} + \frac{\partial \tilde{J}}{\partial D_{j,l}} \frac{\partial D_{j,l}}{\partial f(x_j)}$
             $\frac{\partial \tilde{J}}{\partial f(x_l)} \leftarrow \frac{\partial \tilde{J}}{\partial f(x_l)} + \frac{\partial \tilde{J}}{\partial D_{j,l}} \frac{\partial D_{j,l}}{\partial f(x_l)}$
        end
    end
end
return $\frac{\partial \tilde{J}}{\partial f(x_1)}, \dots, \frac{\partial \tilde{J}}{\partial f(x_m)}$
where $\frac{\partial D_{a,b}}{\partial f(x_a)} = \frac{f(x_a) - f(x_b)}{\|f(x_a) - f(x_b)\|_2}$ and $\frac{\partial D_{a,b}}{\partial f(x_b)} = \frac{f(x_b) - f(x_a)}{\|f(x_a) - f(x_b)\|_2}$ .
Knowl 4 — Dense Pairwise Batch Matrix Construction and Biased Sampling
model/method
Traditional Siamese or triplet networks sample individual pairs or triplets uniformly at random, utilizing only $O(m)$ distance constraints for a batch of $m$ images. The lifted structured embedding transforms the batch feature matrix $X \in \mathbb{R}^{m \times c}$ into a fully connected $O(m^2)$ pairwise squared distance matrix $D^2 \in \mathbb{R}^{m \times m}$ via:

$D^2 = \tilde{x}\mathbf{1}^T + \mathbf{1}\tilde{x}^T - 2XX^T$

where $\tilde{x} = [\|f(x_1)\|_2^2, \dots, \|f(x_m)\|_2^2]^T \in \mathbb{R}^m$ and $\mathbf{1} \in \mathbb{R}^m$ is the all-ones column vector.

To ensure that the batch contains informative negative edges rather than trivial random pairs, the mini-batch sampling incorporates importance sampling: a few positive pairs are sampled at random, and difficult (close) negative neighbors are actively added to the training mini-batch. Furthermore, negative mining is performed symmetrically with respect to both images $(x_i, x_j)$ of each positive pair, avoiding the rigid anchor-only constraint used in standard triplet formulations.
Knowl 5 — Online Products Dataset Specification
definition
The Online Products dataset is a large-scale visual metric learning dataset collected via the eBay.com web crawling API, with duplicate and irrelevant images (such as contact phone numbers and logos) filtered out. It consists of 120,053 images across 22,634 online product categories (classes), averaging approximately 5.3 images per class.

For zero-shot metric learning benchmarks where train and test class sets are strictly disjoint:
- The training set contains 59,551 images belonging to 11,318 classes.
- The test set contains 60,502 images belonging to the remaining 11,316 classes.
Knowl 6 — Zero-Shot Deep Metric Learning Benchmark and Training Setup
experimental setup
The metric learning models are trained and evaluated under a zero-shot generalization setting where classes in the test split have zero intersection with classes in the training split.
- Network Architecture: GoogLeNet architecture pretrained on ImageNet ILSVRC; the final fully connected classification layer is replaced with a randomly initialized projection layer producing embeddings of dimension $c \in \{64, 128, 256, 512\}$ . The learning rate for the randomly initialized layer is multiplied by $10.0$ .
- Optimization: Trained using Caffe for a maximum of 20,000 iterations. Batch size is set to 128 for contrastive and lifted structured losses, and 120 for triplet loss. The margin parameter is set to $\alpha = 1.0$ .
- Data Augmentation and Input: All training and test images are normalized to $256 \times 256$ . Training images are randomly cropped to $227 \times 227$ and randomly mirrored horizontally.
- Clustering Evaluation: Affinity propagation clustering combined with a bisection method to obtain a number of clusters equal to the ground-truth class count in the test set. Performance is measured by $F_1$ score (harmonic mean of precision and recall) and Normalized Mutual Information (NMI).
- Retrieval Evaluation: Evaluated using the Recall@K metric, where a query image is scored 1 if at least one image of the same class is retrieved among the $K$ nearest neighbors, and 0 otherwise, averaged over all test query images.
Knowl 7 — Clustering and Retrieval Performance on CUB-200-2011
empirical result
On the CUB-200-2011 dataset (100 training classes with 5,864 images; 100 test classes with 5,924 images), the lifted structured embedding method consistently outperforms contrastive embedding, triplet embedding, and un-finetuned GoogLeNet pool5 activations across all tested embedding dimensions $\{64, 128, 256, 512\}$ :
- $F_1$ Score: Lifted structured embedding achieves approximately $19.0\%$ at 64 dimensions and increases to approximately $21.0\%$ at 512 dimensions, compared to triplet embedding (ranging from $\sim 15.5\%$ to $16.0\%$ ) and contrastive embedding (ranging from $\sim 12.0\%$ to $13.0\%$ ).
- NMI Score: Lifted structured embedding reaches $\sim 54.0\%$ at 64 dimensions and increases to $\sim 56.0\%$ at 512 dimensions, compared to triplet embedding ( $\sim 49.5\%$ to $50.0\%$ ) and contrastive embedding ( $\sim 46.0\%$ to $47.0\%$ ).
- Recall@K (at 64 dimensions): Lifted structured embedding achieves Recall@1 of $\sim 47\%$ , Recall@2 of $\sim 58\%$ , Recall@4 of $\sim 70\%$ , Recall@8 of $\sim 79\%$ , Recall@16 of $\sim 87\%$ , and Recall@32 of $\sim 92\%$ , substantially exceeding triplet embedding (Recall@1 $\sim 36\%$ ) and contrastive embedding (Recall@1 $\sim 27\%$ ).
Knowl 8 — Clustering and Retrieval Performance on CARS196
empirical result
On the CARS196 dataset (98 training classes with 8,054 images; 98 test classes with 8,131 images), the lifted structured feature embedding outperforms contrastive, triplet, and GoogLeNet pool5 baselines across all embedding dimensions $\{64, 128, 256, 512\}$ :
- $F_1$ Score: Lifted structured embedding achieves $\sim 22.0\%$ to $25.0\%$ , whereas triplet embedding achieves $\sim 16.5\%$ to $17.5\%$ and contrastive embedding achieves $\sim 14.0\%$ to $16.0\%$ .
- NMI Score: Lifted structured embedding achieves $\sim 56.0\%$ to $58.0\%$ , compared to triplet embedding at $\sim 51.0\%$ to $52.0\%$ and contrastive embedding at $\sim 44.0\%$ to $46.0\%$ .
- Recall@K (at 64 dimensions): Lifted structured embedding achieves Recall@1 of $\sim 48\%$ , Recall@2 of $\sim 60\%$ , Recall@4 of $\sim 71\%$ , Recall@8 of $\sim 81\%$ , Recall@16 of $\sim 89\%$ , and Recall@32 of $\sim 94\%$ , significantly outperforming triplet embedding (Recall@1 $\sim 39\%$ ) and contrastive embedding (Recall@1 $\sim 30\%$ ).
Knowl 9 — Clustering and Retrieval Performance on Online Products Dataset
empirical result
On the Online Products dataset (11,318 training classes with 59,551 images; 11,316 test classes with 60,502 images), the lifted structured embedding provides large performance gains over contrastive embedding, triplet embedding, and pretrained GoogLeNet pool5 activations across all embedding dimensions $\{64, 128, 256, 512\}$ :
- $F_1$ Score: Lifted structured embedding scores $\sim 22.0\%$ to $23.0\%$ , compared to triplet embedding ( $\sim 14.0\%$ to $15.0\%$ ) and contrastive embedding ( $\sim 9.5\%$ to $11.0\%$ ).
- NMI Score: Lifted structured embedding achieves $\sim 87.5\%$ to $88.0\%$ , outperforming triplet embedding ( $\sim 85.0\%$ ) and contrastive embedding ( $\sim 83.0\%$ to $84.0\%$ ).
- Recall@K (at 512 dimensions): Lifted structured embedding achieves Recall@1 of $\sim 62.5\%$ , Recall@10 of $\sim 81.0\%$ , Recall@100 of $\sim 92.5\%$ , and Recall@1000 of $\sim 97.5\%$ , substantially higher than triplet embedding (Recall@1 $\sim 46\%$ ) and contrastive embedding (Recall@1 $\sim 40\%$ ).
Knowl 10 — Geometric Failure Modes of Contrastive and Triplet Losses vs. Lifted Embedding
model/method
Pair-based and triplet-based loss functions are prone to optimization failure modes when trained on randomly sampled mini-batches:
1. Contrastive Loss Failure: If a randomly sampled negative example $x_j$ is collinear with examples from an irrelevant third class, pushing $x_i$ away from $x_j$ inadvertently moves $x_i$ toward the cluster of the third class.
2. Triplet Loss Failure: If a sampled negative example $x_n$ falls within the margin bound with respect to an anchor $x_a$ and positive $x_p$ , the update pushes $x_a$ away from $x_n$ and toward the third class cluster.
Lifted structured feature embedding overcomes these failure modes by simultaneously mining and pulling/pushing against all hard negative examples within the margin bound for both items in a positive pair across the entire mini-batch. The resulting joint force vector points toward the true cluster centroid rather than drifting into adjacent class regions.

Coverage note — None was omitted; all key contributions including the loss formulation, distance gradients, backpropagation algorithm, batch construction/mining technique, dataset creation, experimental protocol, and quantitative results on CUB-200-2011, CARS196, and Online Products datasets have been fully captured.

References

1.S. Bell and K. Bala. Learning visual similarity for product design with convolutional neural networks. In SIGGRAPH, 2015. 1, 2, 3, 6
2.Y. Bengio, J. Paiement, and P. Vincent. Out-of-sample extensions for lle, isomap, mds, eigenmaps, and spectral clustering. In NIPS, 2004. 1
3.J. Bromley, I. Guyon, Y. Lecun, E. SŁckinger, and R. Shah. Signature verification using a “siamese” time delay neural network. In NIPS, 1994. 2
4.G. Chechik, V. Sharma, U. Shalit, and S. Bengio. Large scale online learning of image similarity through ranking. JMLR, 11, 2010. 2
5.S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. In CVPR, June 2005. 2
6.A. Choromanska, A. Agarwal, and J. Langford. Extreme multi class classification. In NIPS, 2013. 1
7.T. Cox and M. Cox. Multidimensional scaling. In London: Chapman and Hill, 1994. 1
8.I. S. Data. https://sites.google.com/site/imagesimilaritydata/, 2014. 2
9.eBay Developers Program. http://go.developer.ebay.com/what-ebay-api, 2015. 8
10.B. J. Frey and D. Dueck. apclusterk.m. http://www.psi.toronto.edu/affinitypropagation/apclusterK.m, 2007. 6
11.B. J. Frey and D. Dueck. Clustering by passing messages between data points. Science, 2007. 6
12.A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov. Devise: A deep visual-semantic embedding model. In NIPS, 2013. 2
13.J. Goldberger, S. Roweis, G. Hinton, and R. Salakhutdinov. Neighbourhood component analysis. In NIPS, 2004. 1
14.R. Hadsell, S. Chopra, and Y. Lecun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006. 2, 3, 5, 6
15.H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. In PAMI, 2011. 6
16.Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 5
17.T. Joachims, T. Finley, and C.-N. Yu. Cutting-plane training of structural svms. JMLR, 2009. 4
18.T. I. Jolliffe. Principal component analysis. In New York: Springer-Verlag, 1986. 1
19.J. Krause, M. Stark, J. Deng, and F.-F. Li. 3d object representations for fine-grained categorization. ICCV 3dRR-13, 2013. 1, 2, 6, 8, 10
20.A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 1, 2
21.C. H. Lampert, H. Nickisch, and S. Harmeling. Attribute-based classification for zero-shot visual object categorization. In TPAMI, 2014. 2
22.Y. Li, H. Su, C. Qi, N. Fish, D. Cohen-Or, and L. Guibas. Joint embeddings of shapes and images via cnn image purification. In SIGGRAPH Asia, 2015. 2, 3
23.C. D. Manning, P. Raghavan, and H. Schtze. Introduction to Information Retrieval. Cambridge university press, 2008. 2, 6
24.T. Mensink, J. Verbeek, F. Perronnin, and G. Csurk. Metric learning for large scale image classification: Generalizaing to new classes at near-zero cost. In ECCV, 2012. 2
25.M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell. Zero-shot learning with semantic output codes. In NIPS, 2009. 2
26.Y. Prabhu and M. Varma. Fastxml: A fast, accurate and stable tree-classifier for extreme multi-label learning. In SIGKDD, 2014. 1
27.Q. Qian, R. Jin, S. Zhu, and Y. Lin. Fine-grained visual categorization via multi-stage metric learning. In CVPR, 2015. 2
28.M. Rohrbach, M. Stark, and B. Schiel. Evaluating knowledge transfer and zero-shot learn- ing in a large-scale setting. In CVPR, 2011. 2
29.S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. In Science, 290. 1
30.O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015. 5, 6, 8
31.F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015. 1, 2, 3, 4, 5, 6
32.R. Socher, C. D. M. M. Ganjoo H. Sridhar, O. Bastani, and A. Y. Ng. Zero-shot learning through cross-modal transfer. In ICLR, 2013. 2
33.C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. 1, 2, 6, 8
34.G. Taylor, R. Fergus, G. Williams, I. Spiro, and C. Bregler. Pose-sensitive embedding by nonlinear nca regression. In NIPS, 2010. 1
35.I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun. Support vector machine learning for interdependent and structured output spaces. In ICML, 2004. 4
36.L. van der maaten. Accelerating t-sne using tree-based algorithms. In JMLR, 2014. 6, 7, 8, 9
37.C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 1, 2, 6, 8, 10
38.J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu. Learning fine-grained image similarity with deep ranking. In CVPR, 2014. 2
39.K. Q. Weinberger, J. Blitzer, and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. In NIPS, 2006. 1, 2, 3, 5, 6
40.J. Weston, S. Bengio, and N. Usurer. Wsabi: Scaling up to large vocabulary image annotation. In IJCAI, 2011. 2

Citation

MLA

Song, H. O., et al. “Deep Metric Learning via Lifted Structured Feature Embedding”. arXiv, 2015, http://arxiv.org/abs/1511.06452v1.

APA

Song, H. O., Xiang, Y., Jegelka, S., & Savarese, S. (2015). Deep Metric Learning via Lifted Structured Feature Embedding. arXiv. http://arxiv.org/abs/1511.06452v1

Chicago

Song, H. O., Y. Xiang, S. Jegelka, and S. Savarese. 2015. “Deep Metric Learning via Lifted Structured Feature Embedding”. arXiv. http://arxiv.org/abs/1511.06452v1.

Harvard

Song, H.O. et al. (2015) “Deep Metric Learning via Lifted Structured Feature Embedding”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1511.06452v1.

Vancouver

1. Song HO, Xiang Y, Jegelka S, Savarese S (2015) Deep Metric Learning via Lifted Structured Feature Embedding. arXiv

BibTeX

@article{song2015deep,
  title = {Deep Metric Learning via Lifted Structured Feature Embedding},
  author = {Song, Hyun Oh and Xiang, Yu and Jegelka, Stefanie and Savarese, Silvio},
  year = {2015},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1511.06452v1},
  eprint = {1511.06452}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Deep Metric Learning via Lifted Structured Feature Embedding

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Smooth Lifted Structured Loss Objective

Knowl 2 — Distance Gradients for Lifted Structured Loss

Knowl 3 — Backpropagation Gradient Computation for Lifted Structured Embedding

Knowl 4 — Dense Pairwise Batch Matrix Construction and Biased Sampling

Knowl 5 — Online Products Dataset Specification

Knowl 6 — Zero-Shot Deep Metric Learning Benchmark and Training Setup

Knowl 7 — Clustering and Retrieval Performance on CUB-200-2011

Knowl 8 — Clustering and Retrieval Performance on CARS196

Knowl 9 — Clustering and Retrieval Performance on Online Products Dataset

Knowl 10 — Geometric Failure Modes of Contrastive and Triplet Losses vs. Lifted Embedding

References

Citation

BibTeX

Source Code

Access the Paper