Deep Clustering for Unsupervised Learning of Visual Features

Mathilde CaronPiotr BojanowskiArmand JoulinMatthijs Douze

article2018ECCV2,387 citations

Introduces DeepCluster, an unsupervised training approach that iteratively uses standard k-means clustering to generate pseudo-labels, enabling end-to-end visual feature learning on massive unlabeled image datasets.

Listen

Modern computer vision relies heavily on deep neural networks trained on large, human-annotated datasets. However, manual annotation is costly, time-consuming, and difficult to scale to billions of images, while substituting annotations with uncurated metadata introduces unpredictable biases. Unsupervised visual feature learning offers a solution by extracting useful visual representations from unlabelled data, but adapting standard clustering techniques directly to large neural networks has historically caused degenerate solutions, such as feature collapse into a single cluster.

The article introduces and evaluates DeepCluster, a scalable clustering method for the end-to-end unsupervised training of convolutional neural networks on large-scale image datasets.

The approach iteratively groups extracted visual features using a standard clustering algorithm (k-means) and uses these cluster assignments as artificial labels to update the network's parameters via standard supervised optimization. To maintain stability, the method incorporates simple mechanisms to prevent trivial solutions, including reassigning empty clusters and reweighting samples from unbalanced clusters. The researchers evaluated the framework using standard convolutional architectures (AlexNet and VGG-16) across major visual benchmarks, pre-training models on both ImageNet (over 1.2 million curated images without labels) and an uncurated set of 1 million random Flickr images from the YFCC100M dataset.

The experimental findings show significant performance improvements across visual tasks. First, DeepCluster outperformed all previous unsupervised methods across standard transfer benchmarks, achieving 73.7% accuracy on Pascal VOC classification and 45.1% mean intersection-over-union in semantic segmentation, representing a 7.5% improvement over prior state-of-the-art results. Second, the method exhibited strong robustness when pre-trained on uncurated Flickr data, maintaining top-tier performance and achieving 69.3% classification and 42.2% segmentation accuracy despite the heavy class imbalance in raw web images. Third, scaling the network from AlexNet to a deeper VGG-16 architecture substantially improved performance, narrowing the gap to fully supervised models to just 1.4 percentage points on object detection. Finally, on instance-level image retrieval benchmarks (Oxford and Paris datasets), the learned representations outperformed prior unsupervised approaches by an average margin of roughly 19 points.

These findings demonstrate that high-performance visual models do not require expensive manual annotations or specialized, domain-specific pretext tasks. By demonstrating that unsupervised clustering approaches the effectiveness of fully supervised pre-training, the method significantly reduces data labeling costs and operational bottlenecks. It allows organizations to exploit large, unstructured image repositories directly, enabling computer vision deployment in specialized domains—such as medical imaging or satellite analysis—where labeled data is scarce.

Organizations developing computer vision systems should consider adopting scalable unsupervised clustering pipelines to pre-train models on their uncurated domain data before fine-tuning on limited downstream targets. Teams should prioritize deeper architectures such as VGG or ResNet, as the empirical evidence indicates that larger models derive the greatest benefit from unsupervised pre-training. Further engineering work is recommended to optimize clustering turnaround times and explore dynamic cluster reassignment intervals on larger datasets.

The results carry high confidence across standard transfer benchmarks, but practitioners should note certain limitations. The method requires significant computing time (taking approximately 12 days to train on a single graphics processing unit), and preprocessing inputs with edge-detection filters discards color information, leading to reduced discriminative power in the earliest network layers.

Meta clustering methods computer vision convolutional neural networks Learning Features Unsupervised

arXiv: 1807.05520 facebookresearch/deepcluster

Paper: Unsupervised Deep Embedding for Clustering Analysis, Junyuan Xie et al. (2015). Reading this foundational work on deep embedded clustering helps clarify how joint representation learning and k-means assignments can be integrated before seeing the iterative feature-clustering approach of DeepCluster.
Paper: A tutorial on spectral clustering, Ulrike von Luxburg (2007). This tutorial on spectral clustering provides essential background on graph Laplacians and clustering formulations that underpin deep unsupervised clustering techniques.

Paper: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, Mathilde Caron et al. (2020). This paper builds on DeepCluster by introducing SwAV, which adapts online clustering to contrastive learning for even more scalable unsupervised visual representation.
Paper: Improved Baselines with Momentum Contrastive Learning, Xinlei Chen et al. (2020). This work extends contrastive self-supervised learning principles to achieve superior performance and efficiency baselines in visual representation learning.
Paper: A Simple Framework for Contrastive Learning of Visual Representations, Ting Chen et al. (2020). SimCLR extends unsupervised visual feature learning through a streamlined contrastive framework that serves as a direct point of comparison for clustering-based approaches.

Cover for Deep Clustering for Unsupervised Learning of Visual Features

Abstract

Clustering is a class of unsupervised learning methods that has been extensively applied and studied in computer vision. Little work has been done to adapt it to the end-to-end training of visual features on large scale datasets. In this work, we present DeepCluster, a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm, k-means, and uses the subsequent assignments as supervision to update the weights of the network. We apply DeepCluster to the unsupervised training of convolutional neural networks on large datasets like ImageNet and YFCC100M. The resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks.

1 Introduction
2 Related Work
3 Method
3.1 Preliminaries
3.2 Unsupervised learning by clustering
3.3 Avoiding trivial solutions
3.4 Implementation details
4 Experiments
4.1 Preliminary study
4.2 Visualizations
4.3 Linear classification on activations
4.4 Pascal VOC 2007
5 Discussion
5.1 ImageNet versus YFCC100M
5.2 AlexNet versus VGG
5.3 Evaluation on instance retrieval
6 Conclusion
References
7 Additional results
7.1 Classification on ImageNet
7.2 Stopping criterion
8 Further discussion
8.1 Alternative clustering algorithm
8.2 Variants of DeepCluster
9 Additional visualisation
9.1 Visualise VGG-1616 features
9.2 AlexNet
10 Erratum [18/03/2019]

Knowls

Knowl 1 — DeepCluster Optimization Framework
model/method
DeepCluster is an unsupervised representation learning method that jointly optimizes the parameters of a convolutional neural network (convnet) and groups the extracted features into clusters. Given a training set of $N$ unlabeled images $X = \{x_1, x_2, \dots, x_N\}$ , the convnet mapping $f_\theta: \mathcal{X} \to \mathbb{R}^d$ maps each image $x_n$ to a $d$ -dimensional feature vector $f_\theta(x_n)$ .

DeepCluster alternates between two optimization sub-problems:
1. Feature Clustering and Pseudo-Label Assignment: With the convnet parameters $\theta$ fixed, the features $\{f_\theta(x_n)\}_{n=1}^N$ are clustered into $k$ groups using standard $k$ -means. This jointly optimizes a centroid matrix $C \in \mathbb{R}^{d \times k}$ and 1-of- $k$ binary cluster assignment vectors $y_n \in \{0, 1\}^k$ :
$\min_{C \in \mathbb{R}^{d \times k}} \frac{1}{N} \sum_{n=1}^N \min_{y_n \in \{0, 1\}^k} \|f_\theta(x_n) - C y_n\|_2^2 \quad \text{subject to } y_n^\top \mathbf{1}_k = 1$

The resulting optimal assignment vectors $y_n^*$ serve as discrete pseudo-labels for each image $x_n$ .
1. Convnet Representation Learning: With the pseudo-labels $y_n^*$ fixed, a parameterized linear classifier $g_W$ with weights $W$ is placed on top of $f_\theta$ . The convnet weights $\theta$ and classifier weights $W$ are updated by minimizing the multinomial logistic (negative log-softmax cross-entropy) loss $\ell$ via mini-batch stochastic gradient descent:
$\min_{\theta, W} \frac{1}{N} \sum_{n=1}^N \ell(g_W(f_\theta(x_n)), y_n^*)$

This alternating cycle bootstraps discriminative visual representations directly from the inductive bias of convolutional architectures without human supervision.
Knowl 2 — Degeneracy Avoidance in Deep Clustering
model/method
Jointly optimizing a discriminative neural network and cluster assignments is susceptible to degenerate trivial solutions. DeepCluster prevents two principal failure modes using lightweight algorithmic corrections:
1. Empty Cluster Elimination: A discriminative model can achieve trivial optimal boundaries by assigning all data points to a single cluster, leaving other clusters empty. When an empty cluster is detected during $k$ -means optimization, an active non-empty cluster centroid is randomly chosen and assigned a small random perturbation to form a new centroid for the empty cluster. The data points of the original non-empty cluster are then redistributed between the two resulting centroids.
2. Trivial Parameterization via Dominant Clusters: If the vast majority of dataset instances are assigned to a small subset of clusters, the convnet minimizes classification loss by predicting a constant output regardless of the input. To prevent this collapse, training instances are sampled according to a uniform distribution over the assigned pseudo-label classes during mini-batch stochastic gradient descent. This uniform sampling scheme is mathematically equivalent to reweighting the loss contribution of image $x_n$ inversely proportional to the cardinality of its assigned cluster: $w_n \propto 1 / |S_{y_n}|$ , where $S_{y_n} = \{m \in \{1, \dots, N\} : y_m = y_n\}$ .
Knowl 3 — DeepCluster Training Pipeline
algorithm
The complete DeepCluster representation learning procedure executes an iterative loop over training epochs, alternating between dataset-level feature extraction/clustering and stochastic gradient descent updates on pseudo-labels.
Input: Unlabeled image dataset $X = \{x_1, \dots, x_N\}$ , convnet $f_\theta$ , classifier head $g_W$ , number of clusters $k$ , total epochs $E$
Output: Trained convnet feature extractor parameters $\theta$
Initialize convnet weights $\theta$ and classifier weights $W$
Apply fixed Sobel filtering to all images in $X$ to discard raw color cues
for epoch = 1 to $E$ do
    For each image $x_n \in X$ , extract feature representation $z_n = f_\theta(\text{central\_crop}(x_n))$
    Reduce $\{z_n\}_{n=1}^N$ to 256 dimensions using PCA, apply whitening, and $\ell_2$ -normalize
    Cluster transformed features into $k$ clusters via $k$ -means, reassigning empty cluster centroids
    Assign pseudo-label $y_n^* \in \{1, \dots, k\}$ to each image $x_n$
    Re-initialize classifier head $g_W \in \mathbb{R}^{k \times d}$
    for mini-batch $B \subset X$ sampled uniformly across pseudo-label classes do
        Apply data augmentations (random resized crops, horizontal flips) to each image in $B$
        Compute cross-entropy loss $\mathcal{L} = \frac{1}{|B|} \sum_{x_n \in B} \ell(g_W(f_\theta(x_n)), y_n^*)$
        Update $\theta$ and $W$ with SGD using momentum 0.9 and $\ell_2$ weight regularization
    end for
end for
return $\theta$

Knowl 4 — Linear Classification Probing Across Convolutional Layers

data/table

To analyze representation quality across depths, an AlexNet is pre-trained without supervision on ImageNet (using Sobel preprocessing), and linear classifiers are trained on frozen convolutional activations (conv1 to conv5) evaluated on ImageNet and Places datasets (top-1 accuracy %, 10-crop averaged):

Method	ImageNet					Places
	conv1	conv2	conv3	conv4	conv5	conv1	conv2	conv3	conv4	conv5
Places labels	–	–	–	–	–	22.1	35.1	40.2	43.3	44.6
ImageNet labels	19.3	36.3	44.2	48.3	50.5	22.7	34.8	38.4	39.4	38.7
Random	11.6	17.1	16.9	16.3	14.1	15.7	20.3	19.8	19.1	17.5
Pathak et al.	14.1	20.7	21.0	19.8	15.5	18.2	23.2	23.4	21.9	18.4
Doersch et al.	16.2	23.3	30.2	31.7	29.6	19.7	26.7	31.9	32.7	30.9
Zhang et al. (Colorization)	12.5	24.5	30.4	31.5	30.3	16.0	25.7	29.6	30.3	29.7
Donahue et al. (BiGAN)	17.7	24.5	31.0	29.9	28.0	21.4	26.2	27.1	26.1	24.0
Noroozi and Favaro (Jigsaw)	18.2	28.8	34.0	33.9	27.1	23.0	32.1	35.5	34.8	31.3
Noroozi et al. (Counting)	18.0	30.6	34.3	32.5	25.7	23.3	33.9	36.3	34.7	29.6
Zhang et al. (Split-Brain)	17.7	29.3	35.4	35.2	32.8	21.3	30.7	34.0	34.1	32.5
DeepCluster	13.4	32.3	41.0	39.6	38.2	19.6	33.2	39.2	39.8	34.7

DeepCluster outperforms prior unsupervised methods across conv2 through conv5 on ImageNet. On Places, DeepCluster conv3 (39.2%) and conv4 (39.8%) features match or surpass supervised ImageNet-trained features (38.4% and 39.4%), demonstrating strong out-of-domain transferability.

Knowl 5 — Transfer Learning Performance on PASCAL VOC Benchmarks

data/table

AlexNet features trained unsupervised on ImageNet are evaluated on PASCAL VOC 2007 classification (mAP %), VOC 2007 object detection (Fast R-CNN mAP %), and VOC 2012 semantic segmentation (mean IoU %). Results compare linear adaptation on frozen convolutional features (fc6-8) against end-to-end fine-tuning (all):

Method	Classification		Detection		Segmentation
	fc6-8	all	fc6-8	all	fc6-8	all
ImageNet labels	78.9	79.9	–	56.8	–	48.0
Random-rgb	33.2	57.0	22.2	44.5	15.2	30.1
Random-sobel	29.0	61.9	18.9	47.9	13.0	32.0
Pathak et al.	34.6	56.5	–	44.5	–	29.7
Donahue et al.	52.3	60.1	–	46.9	–	35.2
Pathak et al.	–	61.0	–	52.2	–	–
Owens et al.	52.3	61.3	–	–	–	–
Wang and Gupta	55.6	63.1	32.8	47.2	26.0	35.4
Doersch et al.	55.1	65.3	–	51.1	–	–
Bojanowski and Joulin	56.7	65.3	33.7	49.4	26.7	37.1
Zhang et al. (Colorization)	61.5	65.9	43.4	46.9	35.8	35.6
Zhang et al. (Split-Brain)	63.0	67.1	–	46.7	–	36.0
Noroozi and Favaro	–	67.6	–	53.2	–	37.6
Noroozi et al.	–	67.7	–	51.4	–	36.6
DeepCluster	72.0	73.7	51.4	55.4	43.2	45.1

DeepCluster outperforms all prior unsupervised methods across all three tasks in both fine-tuned and frozen settings, achieving substantial gains on frozen fc6-8 classification (+9.0% over prior best) and fine-tuned segmentation (+7.5% mIoU over prior best).

Knowl 6 — Impact of Cluster Count and Over-Segmentation
empirical result
The number of clusters $k$ chosen for the $k$ -means step significantly governs the quality of visual representations learned by DeepCluster. When evaluating an AlexNet on PASCAL VOC 2007 classification (validation mAP) across cluster counts ranging from $k = 10^2$ to $k = 10^5$ on ImageNet:
1. Low values of $k$ (e.g., $k = 10^2$ or $10^3$ ) yield sub-optimal performance (mAP below 64%).
2. Performance peaks at $k = 10,000$ (mAP $\approx 67.5\%$ after 300 epochs).
3. Performance gently decreases as $k$ increases to $100,000$ .
Even though the underlying ImageNet dataset contains exactly 1,000 human-annotated ground truth classes, setting $k = 10,000$ yields superior representations compared to $k = 1,000$ . This indicates that over-segmentation—grouping instances into fine-grained sub-categories—helps bootstrap feature learning by reducing intra-cluster visual variance.
Knowl 7 — Quantifying Representation Stability and Semantic Emergence via NMI
empirical result
The information shared between two categorical labelings $A$ and $B$ over the dataset is measured using the Normalized Mutual Information (NMI):

$\text{NMI}(A; B) = \frac{I(A; B)}{\sqrt{H(A) H(B)}}$

where $I(A; B)$ is the mutual information and $H(\cdot)$ denotes entropy. Evaluating DeepCluster using NMI reveals two fundamental training dynamics:
1. Semantic Alignment with Ground Truth: The NMI between the unsupervised cluster assignments at epoch $t$ and the true human ImageNet labels steadily increases throughout training from $\approx 0.25$ to over $0.43$ . This confirms that minimizing unsupervised cross-entropy over clustering pseudo-labels forces the convnet to self-organize representations aligned with high-level object classes.
2. Cluster Stability Across Epochs: The NMI between cluster assignments at consecutive epochs $t-1$ and $t$ increases from $\approx 0.64$ and plateaus below $0.80$ . Although a fraction of images are continually reassigned between epochs, this level of dynamic assignment stabilizes sufficiently to prevent representation collapse or optimization divergence.

Knowl 8 — Robustness to Uncurated Web Image Distributions

data/table

To test whether DeepCluster depends on the balanced, object-centric distribution of ImageNet, an AlexNet is pre-trained on an uncurated, unfiltered subset of 1 million random Flickr images from YFCC100M (which exhibits a heavy-tailed Zipf distribution of concepts). Transfer performance is evaluated on PASCAL VOC:

Method	Training set	Classification		Detection		Segmentation
		fc6-8	all	fc6-8	all	fc6-8	all
Best competitor	ImageNet	63.0	67.7	43.4	53.2	35.8	37.7
DeepCluster	ImageNet	72.0	73.7	51.4	55.4	43.2	45.1
DeepCluster	YFCC100M	67.3	69.3	45.6	53.0	39.2	42.2

Despite the distribution shift from curated object images to raw Flickr photos, DeepCluster trained on YFCC100M maintains performance above all prior unsupervised methods pre-trained on ImageNet across classification (69.3% vs 67.7% mAP), detection (53.0% vs 53.2% mAP), and semantic segmentation (42.2% vs 37.7% mean IoU).

Knowl 9 — Convnet Architecture Scaling and Instance Retrieval Performance

data/table

DeepCluster scales effectively to deeper network architectures and transfers successfully to instance-level image retrieval benchmarks without requiring class-level supervision.

Architecture Scaling on PASCAL VOC 2007 Detection (Fast R-CNN mAP %):

Method	AlexNet	VGG-16
ImageNet labels (Supervised)	56.8	67.3
Random	47.8	39.7
Doersch et al.	51.1	61.5
Wang and Gupta	47.2	60.2
Wang et al.	–	63.2
DeepCluster	55.4	65.9

Switching from AlexNet to VGG-16 boosts DeepCluster detection performance by +10.5% mAP to 65.9%, narrowing the gap with fully supervised VGG-16 pre-training to 1.4%.

Instance-Level Image Retrieval (R-MAC, 1024px, 3 grid levels, mAP %):

On instance-level retrieval, DeepCluster substantially outperforms other self-supervised and unsupervised approaches (+18.7% on Oxford5K and +14.0% on Paris6K over prior best).

Knowl 10 — Color Invariance via Fixed Sobel Preprocessing
empirical result
Training deep clustering models directly on raw RGB images leads to trivial feature representations where convolutional filters primarily capture low-level color histograms rather than semantic and structural shapes. Applying a fixed, non-trainable linear transformation based on Sobel filters to input images discards color cues and increases local contrast.

Empirical filter visualizations demonstrate:
1. AlexNet trained on raw RGB inputs produces first-layer (conv1) filters that almost exclusively represent uniform color gradients and color blobs.
2. AlexNet trained with Sobel preprocessing produces oriented Gabor-like edge detectors and frequency-selective patterns in conv1, encouraging deeper layers (conv3 to conv5) to learn textures, object parts, and semantic object boundaries.

Coverage note — None was omitted; all key contributions—including the alternating optimization formulation, degeneracy remedies, full algorithmic pipeline, empirical probe evaluations on ImageNet/Places/VOC, architectural scaling to VGG-16, uncurated YFCC100M data distribution experiments, and instance retrieval results—are represented.

References

1.Agrawal, P., Carreira, J., Malik, J.: Learning to see by moving. In: ICCV (2015)
2.Bach, F.R., Harchaoui, Z.: Diffrac: a discriminative and flexible framework for clustering. In: NIPS (2008)
3.Bautista, M.A., Sanakoyeu, A., Tikhoncheva, E., Ommer, B.: Cliquecnn: deep unsupervised exemplar learning. In: Advances in Neural Information Processing Systems, pp. 3846–3854 (2016)
4.Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: NIPS (2007)
5.Bojanowski, P., Joulin, A.: Unsupervised learning by predicting noise. In: ICML (2017)
6.Bojanowski, P., Joulin, A., Lopez-Paz, D., Szlam, A.: Optimizing the latent space of generative networks. arXiv preprint arXiv:1707.05776 (2017)
7.Bottou, L.: Stochastic Gradient Descent Tricks. In: Montavon, G., Orr, G.B., M¨uller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 421–436. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_25
8.Carreira, J., Agrawal, P., Fragkiadaki, K., Malik, J.: Human pose estimation with iterative error feedback. In: CVPR (2016)
9.Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. arXiv preprint arXiv:1606.00915 (2016)
10.Coates, A., Ng, A.Y.: Learning feature representations with k-means. In: Montavon, G., Orr, G.B., M¨uller, K.R. (eds.) NN: Tricks of the Trade. LNCS, vol. 7700, pp. 561–580. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_30
11.Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Workshop on Satistical Learning in Computer Vision ECCV, vol. 1, pp. 1–2. Prague (2004)
12.Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR (2009)
13.Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: ICCV (2015)
14.Doersch, C., Zisserman, A.: Multi-task self-supervised visual learning (2017)
15.Donahue, J., Kr¨ahenb¨uhl, P., Darrell, T.: Adversarial feature learning. arXiv preprint arXiv:1605.09782 (2016)
16.Dosovitskiy, A., Springenberg, J.T., Riedmiller, M., Brox, T.: Discriminative unsupervised feature learning with convolutional neural networks. In: NIPS (2014)
17.Dumoulin, V., et al.: Adversarially learned inference. arXiv preprint arXiv:1606.00704 (2016)
18.Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Univ. Montr. 1341, 3 (2009)
19.Friedman, J., Hastie, T., Tibshirani, R.: The Elements of Statistical Learning, vol. 1. Springer, New York (2001). https://doi.org/10.1007/978-0-387-21606-5
20.Goodfellow, I., et al.: Generative adversarial nets. In: NIPS (2014)
21.He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCV (2015)
22.Huang, F.J., Boureau, Y.L., LeCun, Y., et al.: Unsupervised learning of invariant feature hierarchies with applications to object recognition. In: CVPR (2007)
23.Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely connected convolutional networks. arXiv preprint arXiv:1608.06993 (2016)
24.Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015)
25.Johnson, J., Douze, M., J´egou, H.: Billion-scale similarity search with GPUs. arXiv preprint arXiv:1702.08734 (2017)
26.Joulin, A., Bach, F.: A convex relaxation for weakly supervised classifiers. arXiv preprint arXiv:1206.6413 (2012)
27.Joulin, A., Bach, F., Ponce, J.: Discriminative clustering for image co-segmentation. In: CVPR (2010)
28.Joulin, A., van der Maaten, L., Jabri, A., Vasilache, N.: Learning visual features from large weakly supervised data. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 67–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_5
29.Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
30.Kovashka, A., Russakovsky, O., Fei-Fei, L., Grauman, K.: Crowdsourcing in computer vision. Found. Trends® Comput. Graph. Vis. 10(3), 177–243 (2016)
31.Kr¨ahenb¨uhl, P., Doersch, C., Donahue, J., Darrell, T.: Data-dependent initializations of convolutional neural networks. arXiv preprint arXiv:1511.06856 (2015)
32.Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)
33.Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 577–593. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_35
34.LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
35.Liao, R., Schwing, A., Zemel, R., Urtasun, R.: Learning deep parsimonious representations. In: NIPS (2016)
36.Lin, F., Cohen, W.W.: Power iteration clustering. In: ICML (2010)
37.Linsker, R.: Towards an organizing principle for a layered perceptual network. In: NIPS (1988)
38.Mairal, J., Koniusz, P., Harchaoui, Z., Schmid, C.: Convolutional kernel networks. In: NIPS (2014)
39.Malisiewicz, T., Gupta, A., Efros, A.A.: Ensemble of exemplar-SVMS for object detection and beyond. In: ICCV (2011)
40.Masci, J., Meier, U., Cire¸san, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6791, pp. 52–59. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21735-7_7
41.Misra, I., Zitnick, C.L., Mitchell, M., Girshick, R.: Seeing through the human reporting bias: visual classifiers from noisy human-centric labels. In: CVPR (2016)
42.Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by Solving Jigsaw Puzzles. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 69–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_5
43.Noroozi, M., Pirsiavash, H., Favaro, P.: Representation learning by learning to count. In: ICCV (2017)
44.Owens, A., Wu, J., McDermott, J.H., Freeman, W.T., Torralba, A.: Ambient sound provides supervision for visual learning. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 801–816. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_48
45.Pathak, D., Girshick, R., Doll´ar, P., Darrell, T., Hariharan, B.: Learning features by watching objects move. In: CVPR (2017)
46.Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR (2016)
47.Paulin, M., Douze, M., Harchaoui, Z., Mairal, J., Perronin, F., Schmid, C.: Local convolutional features with unsupervised training for image retrieval. In: ICCV (2015)
48.Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: CVPR (2007)
49.Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization: improving particular object retrieval in large scale image databases. In: CVPR (2008)
50.Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: NIPS (2015)
51.Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. IJCV 115(3), 211–252 (2015)
52.de Sa, V.R.: Learning classification with unlabeled data. In: NIPS (1994)
53.Sharif Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: CVPR workshops (2014)
54.Shi, J., Malik, J.: Normalized cuts and image segmentation. TPAMI 22(8), 888–905 (2000)
55.Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
56.Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. JMLR 15(1), 1929–1958 (2014)
57.Stock, P., Cisse, M.: Convnets and imagenet beyond accuracy: explanations, bias detection, adversarial examples and model criticism. arXiv preprint arXiv:1711.11443 (2017)
58.Thomee, B., et al.: The new data and new challenges in multimedia research. arXiv preprint arXiv:1503.01817 (2015)
59.Tolias, G., Sicre, R., J´egou, H.: Particular object retrieval with integral max-pooling of CNN activations. arXiv preprint arXiv:1511.05879 (2015)
60.Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: CVPR (1991)
61.Van De Sande, K., Gevers, T., Snoek, C.: Evaluating color descriptors for object and scene recognition. TPAMI 32(9), 1582–1596 (2010)
62.Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. JMLR 11(Dec), 3371–3408 (2010)
63.Wang, X., Gupta, A.: Unsupervised learning of visual representations using videos. In: ICCV (2015)
64.Wang, X., He, K., Gupta, A.: Transitive invariance for self-supervised visual representation learning. arXiv preprint arXiv:1708.02901 (2017)
65.Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: Deepflow: Large displacement optical flow with deep matching. In: ICCV (2013)
66.Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: ICML (2016)
67.Xu, L., Neufeld, J., Larson, B., Schuurmans, D.: Maximum margin clustering. In: NIPS (2005)
68.Yang, J., Parikh, D., Batra, D.: Joint unsupervised learning of deep representations and image clusters. In: CVPR (2016)
69.Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015)
70.Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
71.Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
72.Zhang, R., Isola, P., Efros, A.A.: Split-brain autoencoders: unsupervised learning by cross-channel prediction. arXiv preprint arXiv:1611.09842 (2016)
73.Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: NIPS (2014)

Citation

MLA

Caron, M., et al. “Deep Clustering for Unsupervised Learning of Visual Features”. Lecture Notes in Computer Science, Springer International Publishing, 2018, pp. 139–56, https://doi.org/10.1007/978-3-030-01264-9_9.

APA

Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018). Deep Clustering for Unsupervised Learning of Visual Features. In Lecture Notes in Computer Science (pp. 139–156). Springer International Publishing. https://doi.org/10.1007/978-3-030-01264-9_9

Chicago

Caron, M., P. Bojanowski, A. Joulin, and M. Douze. 2018. “Deep Clustering for Unsupervised Learning of Visual Features”. In Lecture Notes in Computer Science. Springer International Publishing. https://doi.org/10.1007/978-3-030-01264-9_9.

Harvard

Caron, M. et al. (2018) “Deep Clustering for Unsupervised Learning of Visual Features”, Lecture Notes in Computer Science. Springer International Publishing, pp. 139–156. Available at: https://doi.org/10.1007/978-3-030-01264-9_9.

Vancouver

1. Caron M, Bojanowski P, Joulin A, Douze M (2018) Deep Clustering for Unsupervised Learning of Visual Features. In: Lecture Notes in Computer Science. Springer International Publishing, pp 139–156

BibTeX

@inbook{Caron_2018, title={Deep Clustering for Unsupervised Learning of Visual Features}, ISBN={9783030012649}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/978-3-030-01264-9_9}, DOI={10.1007/978-3-030-01264-9_9}, booktitle={Computer Vision – ECCV 2018}, publisher={Springer International Publishing}, author={Caron, Mathilde and Bojanowski, Piotr and Joulin, Armand and Douze, Matthijs}, year={2018}, pages={139–156} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Deep Clustering for Unsupervised Learning of Visual Features

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — DeepCluster Optimization Framework

Knowl 2 — Degeneracy Avoidance in Deep Clustering

Knowl 3 — DeepCluster Training Pipeline

Knowl 4 — Linear Classification Probing Across Convolutional Layers

Knowl 5 — Transfer Learning Performance on PASCAL VOC Benchmarks

Knowl 6 — Impact of Cluster Count and Over-Segmentation

Knowl 7 — Quantifying Representation Stability and Semantic Emergence via NMI

Knowl 8 — Robustness to Uncurated Web Image Distributions

Knowl 9 — Convnet Architecture Scaling and Instance Retrieval Performance

Knowl 10 — Color Invariance via Fixed Sobel Preprocessing

References

Citation

BibTeX

Source Code

Access the Paper

Method	Oxford5K	Paris6K
ImageNet labels	72.4	81.5
Random	6.9	22.0
Doersch et al.	35.4	53.1
Wang et al.	42.3	58.0
DeepCluster	61.0	72.0