Back

Challenges in representation learning: A report on three machine learning contests

Ian J. GoodfellowDumitru ErhanPierre Luc CarrierAaron CourvilleMehdi MirzaBen HamnerWill CukierskiYichuan TangDavid ThalerDong-Hyun Lee

article2013Neural Networks2,073 citations

Examines the top-performing representation learning methods across three machine learning competitions—covering black-box, facial expression, and multimodal data—while providing actionable insights on algorithm performance and effective benchmark design.

expression recognitionrepresentation learningfacial expression analysisLearningContestRepresentation Learning
arXiv: 1307.0414
  • Paper: Multimodal Machine Learning: A Survey and Taxonomy, Tadas Baltrušaitis et al. (2017). This comprehensive taxonomy systematizes multimodal representation, translation, and fusion, substantially generalizing beyond the early multimodal contest paradigms.
  • Paper: Deep Learning Face Representation by Joint Identification-Verification, Yi Sun et al. (2014). It advances deep facial representation learning by formulating joint identification-verification objectives, extending the facial analysis benchmarks introduced in the contest.
  • Paper: Deep Face Recognition, Omkar M. Parkhi et al. (2015). It scales up deep facial representation and metric learning architectures, building upon the baseline visual feature representations evaluated in the workshop.
  • Paper: Multimodal Transformer for Unaligned Multimodal Language Sequences, Yao-Hung Hubert Tsai et al. (2019). It presents modern crossmodal attention mechanisms for unaligned multimodal sequences, significantly advancing the fusion approaches benchmarked in early multimodal challenges.
  • Paper: Contrastive Multiview Coding, Yonglong Tian et al. (2019). It extends multimodal and multi-view representation learning into modern self-supervised contrastive frameworks across diverse visual and sensory channels.
  • Paper: ImageBind One Embedding Space to Bind Them All, Rohit Girdhar et al. (2023). It scales the multimodal representation challenge to unify six disparate sensory modalities within a single joint embedding space.
Cover for Challenges in representation learning: A report on three machine learning contests

Abstract

The ICML 2013 Workshop on Challenges in Representation Learning focused on three challenges: the black box learning challenge, the facial expression recognition challenge, and the multimodal learning challenge. We describe the datasets created for these challenges and summarize the results of the competitions. We provide suggestions for organizers of future challenges and some comments on what kind of knowledge can be gained from machine learning competitions.

Topics

expression recognitionrepresentation learningfacial expression analysisfacial imagesonline learning

Keywords

LearningContestRepresentation LearningMachine LearningChallengesContestsThree Machine LearningMachine Learning ContestsReportDataset

Table of Contents

  • 1 Introduction
  • 2 The black box learning challenge
  • 3 The facial expression recognition challenge
  • 4 The multimodal learning challenge
  • 5 Advice to contest organizers
  • 6 Discussion and conclusion
  • References

Knowls

  1. Knowl 1 — BBL-2013 Dataset Creation and Obfuscation Protocol

    experimental setup

    The Black Box Learning 2013 (BBL-2013) dataset was created to benchmark semi-supervised representation learning without human-in-the-loop domain knowledge or filter visualization. It is derived from the second format (MNIST-like 32×3232 \times 32 cropped images) of the Street View House Numbers (SVHN) dataset. The original 3,072 pixel features were linearly projected down to 1,875 continuous features via multiplication by a fixed random projection matrix, and images of the digit class "4" were removed, leaving a 9-class classification problem.

    The dataset partition consists of:

    • Labeled Training Set: 1,000 examples with labels.
    • Public Leaderboard Set: 5,000 examples without labels provided to participants (features only), used for validation and daily public leaderboard feedback.
    • Private Test Set: 5,000 examples without labels provided to participants, evaluated once after the competition close to determine final rankings.
    • Unlabeled Auxiliary Set: 130,000 unlabeled examples drawn from the subset designated as "less difficult" by the creators of SVHN.

  2. Knowl 2 — Methods and Semi-Supervised Results in the Black Box Learning Challenge

    empirical result

    In the Black Box Learning 2013 challenge (218 participating teams, 1,963 submissions), 75 teams outperformed the organizer-provided 3-layer multilayer perceptron (MLP) baseline.

    • 1st Place: David Thaler achieved the winning classification accuracy of 70.22% on the private test set by blending three models that combined sparse filtering for unsupervised feature learning, random forests for feature selection, and support vector machines (SVMs) for classification.
    • 2nd Place: Dong-Hyun Lee achieved second place using pseudo-labeling / entropy regularization for deep neural networks.
    • Unsupervised Data Dynamics: Multiple top competitors (including David Thaler and Lukasz Romaszko) reported that training sparse filtering representations jointly on both labeled and unlabeled data yielded lower validation performance than training sparse filtering on the 1,000 labeled examples alone. This counterintuitive effect was attributed to the 1,000 labeled samples being drawn from the more difficult portion of the SVHN dataset compared to the 130,000 unlabeled samples.

  3. Knowl 3 — FER-2013 Facial Expression Dataset Specification

    experimental setup

    The Facial Expression Recognition 2013 (FER-2013) dataset was designed to benchmark feature learning against hand-engineered visual descriptors on facial emotion classification. The dataset was collected by querying the Google Image Search API with 184 emotion-related keywords (e.g., "blissful", "enraged") combined with demographic descriptors (gender, age, ethnicity) to produce approximately 600 query strings.

    The top 1,000 returned images per query were processed with an OpenCV face detector to extract face bounding boxes. Human annotators removed incorrectly classified images, adjusted bounding box crops, and removed duplicates. Approved face crops were resized to 48×4848 \times 48 grayscale images and categorized into the 7 basic emotion classes used in the Toronto Face Database (TFD):

    • Anger: 4,953 images
    • Disgust: 547 images
    • Fear: 5,121 images
    • Happiness: 8,989 images
    • Sadness: 6,077 images
    • Surprise: 4,002 images
    • Neutral: 6,198 images

    The final curated FER-2013 dataset contains a total of 35,887 images.

  4. Knowl 4 — Human and Null-Model Performance Benchmarks on FER-2013

    empirical result

    Baseline human recognition accuracy on the FER-2013 dataset is estimated at 65%±5%65\% \pm 5\%. For comparison, human accuracy on a noise-free dataset of 1,500 images of laboratory actors posed in the same seven expressions is 68%±5%68\% \pm 5\%, indicating that web-scraping label noise in FER-2013 does not substantially impede human emotion recognition.

    An untrained "null" model benchmark—defined as a convolutional neural network with fixed, unlearned feature extraction layers and training restricted solely to the final linear classification layer—was optimized over network hyperparameters using the Tree-structured Parzen Estimator (TPE) algorithm. A single optimized null convolutional network achieved 60.0% accuracy, while an ensemble of such null models achieved 65.5% accuracy on FER-2013, matching estimated human-level performance without end-to-end representation learning.

  5. Knowl 5 — Private Test Accuracy on the FER-2013 Facial Expression Recognition Challenge

    data/table

    Fifty-six teams submitted predictions on the FER-2013 private test set. The top four systems surpassed the 65.5% accuracy of the tuned untrained convolutional network ensemble baseline.

    Team Members Accuracy
    RBM Yichuan Tang 71.162%
    Unsupervised Yingbo Zhou, Chetan Ramaiah 69.267%
    Maxim Milakov Maxim Milakov 68.821%
    Radu + Marius + Cristi Radu Ionescu, Marius Popescu, Cristian Grozea 67.484%

    The top three entries utilized convolutional neural networks trained discriminatively with data augmentation transformations. The winning entry by Yichuan Tang trained a CNN by optimizing the L2-SVM primal objective loss function instead of the conventional softmax cross-entropy loss. The strongest submission avoiding representation learning (Radu Ionescu, Marius Popescu, and Cristian Grozea) used dense SIFT descriptors with Multiple Kernel Learning (MKL), achieving 67.484%. This demonstrates that learned deep convolutional representations outperform engineered vision descriptors on this task, although the margin remains moderate.

  6. Knowl 6 — Multimodal Classification Formulation and Global Matching Vulnerability

    limitation

    The Multimodal Learning Challenge evaluated joint image-text representation learning using a test set of 1,000 manually annotated web images paired with candidate text labels from the ESP game vocabulary. The competition framed the evaluation as a binary choice classification task where each test image was presented with two candidate label options (one correct label from the test set and one distractor label).

    Because the pairs were constructed from a closed set of test labels forming an exact bipartite match across the test collection, competitors bypassed image-level classification by feeding pairwise model scores into the Hungarian matching algorithm. Computing the global optimal assignment yielded 100% classification accuracy despite underlying base classifiers having low individual accuracy, resulting in a three-way tie for first place (RBM, MMDL, and AlbinoSnowman).

    To prevent global assignment shortcuts in paired binary-choice tasks, challenge datasets must be constructed by generating twice as many labeled samples as needed, discarding half of the images, and using their true labels as negative distractors for the remaining images. This eliminates the closed-world 1-to-1 matching structure and forces models to evaluate each image independently.

  7. Knowl 7 — Contest Design Principles for Machine Learning Benchmarks

    model/method

    Evidence from organizing machine learning representation learning challenges provides four operational guidelines for future benchmark competitions:

    1. Anti-Cheating and Model Freezing Protocol: When using data that humans can easily label, prevent test set labeling and hyperparameter tuning on test sets by requiring competitors to submit their frozen, trained model binaries before the test inputs are released. Organizers release test inputs only after all models are locked, and then independently reproduce predictions from the winning submissions using the submitted code/weights.
    2. Difficulty Calibration: Tasks must be calibrated with sufficient intrinsic classification difficulty to avoid saturated scores or exploit vulnerabilities (such as global bipartite matching).
    3. Simultaneous Contest Portfolios: Running multiple distinct tracks simultaneously leverages fixed administrative costs and insures against task failure or design oversights in any single track.
    4. Baseline Dissemination: Providing functioning starter code and baseline pipelines substantially increases competitive participation by eliminating data ingestion and environment setup overhead.

Coverage note — None was omitted; all three competition setups, datasets, baseline metrics, winning technical methods, results tables, and contest design recommendations have been fully represented.

References

  1. 1.Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. Technical Report arXiv:1206.5538, U. Montreal, 2012. URL http://arxiv.org/abs/1206.5538.
  2. 2.Isabelle Guyon, Gideon Dror, Vincent Lemaire, Graham Taylor, and David W. Aha. Unsupervised and transfer learning challenge. In Proc. Int. Joint Conf. on Neural Networks, 2011.
  3. 3.Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. Deep Learning and Unsupervised Feature Learning Workshop, NIPS, 2011.
  4. 4.Jiquan Ngiam, Pang Wei W. Koh, Zhenghao Chen, Sonia A. Bhaskar, and Andrew Y. Ng. Sparse filtering. In J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F.C.N. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 1125–1133. 2011.
  5. 5.Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
  6. 6.Corinna Cortes and Vladimir Vapnik. Support vector networks. Machine Learning, 20:273–297, 1995.
  7. 7.Lukasz Romaszko. A deep learning approach with an ensemble-based neural network classifier for black box icml 2013 contest. Workshop on Challenges in Representation Learning, ICML, 2013.
  8. 8.Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. Workshop on Challenges in Representation Learning, ICML, 2013.
  9. 9.Yves Grandvalet and Yoshua Bengio. Semi-supervised Learning by Entropy Minimization. In NIPS’04, Cambridge, MA, December 2005. MIT Press.
  10. 10.Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML 2008, 2008.
  11. 11.Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. ICML, 2013. URL http://icml.cc/2013/.
  12. 12.Joshua Susskind, Adam Anderson, and Geoffrey E. Hinton. The Toronto face dataset. Technical Report UTML TR 2010-001, U. Toronto, 2010.
  13. 13.James Bergstra and David D. Cox. Hyperparameter optimization and boosting for classifying facial expressions: How good can a “null” model be? Workshop on Challenges in Representation Learning, ICML, 2013.
  14. 14.K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36[4], pp. 193-202, 1980.
  15. 15.D. Lowe. Object recognition from local scale invariant features. ICCV’99, 1999.
  16. 16.Yichuan Tang. Deep learning using linear support vector machines. Workshop on Challenges in Representation Learning, ICML, 2013.
  17. 17.Radu Tudor Ionescu, Marius Popescu, and Cristian Grozea. Local learning to improve bag of visual words model for facial expression recognition. Workshop on Challenges in Representation Learning, ICML, 2013.
  18. 18.Luis von Ahn and Laura Dabbish. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, pages 319–326, New York, NY, USA, 2004. ACM. ISBN 1-58113-702-8. doi: 10.1145/985692.985733. URL http://doi.acm.org/10.1145/985692.985733.
  19. 19.Fangxiang Feng, Ruifan Li, and Xiaojie Wang. Constructing hierarchical image-tags bimodal representations for word tags alternative choice. Workshop on Challenges in Representation Learning, ICML, 2013.
  20. 20.Quoc V. Le, Marc’Aurelio Ranzato, Ruslan Salakhutdinov, Andrew Ng, and Josh Tenenbaum. NIPS Workshop on Challenges in Learning Hierarchical Models: Transfer Learning and Optimization, 2011. https://sites.google.com/site/nips2011workshop.
  21. 21.Ian Goodfellow, Aaron Courville, and Yoshua Bengio. Large-scale feature learning with spike-and-slab sparse coding. ICML, 2012. URL http://icml.cc/discuss/2012/590.html.

Citation

MLA
Goodfellow, I. J., et al. “Challenges in Representation Learning: A Report on Three Machine Learning Contests”. arXiv, 2013, http://arxiv.org/abs/1307.0414v1.
APA
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., … Bengio, Y. (2013). Challenges in Representation Learning: A report on three machine learning contests. arXiv. http://arxiv.org/abs/1307.0414v1
Chicago
Goodfellow, I. J., D. Erhan, P. L. Carrier, et al. 2013. “Challenges in Representation Learning: A Report on Three Machine Learning Contests”. arXiv. http://arxiv.org/abs/1307.0414v1.
Harvard
Goodfellow, I.J. et al. (2013) “Challenges in Representation Learning: A report on three machine learning contests”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1307.0414v1.
Vancouver
1. Goodfellow IJ, Erhan D, Carrier PL, et al (2013) Challenges in Representation Learning: A report on three machine learning contests. arXiv

BibTeX

@article{goodfellow2013challenges,
  title = {Challenges in Representation Learning: A report on three machine learning contests},
  author = {Goodfellow, Ian J. and Erhan, Dumitru and Carrier, Pierre Luc and Courville, Aaron and Mirza, Mehdi and Hamner, Ben and Cukierski, Will and Tang, Yichuan and Thaler, David and Lee, Dong-Hyun and Zhou, Yingbo and Ramaiah, Chetan and Feng, Fangxiang and Li, Ruifan and Wang, Xiaojie and Athanasakis, Dimitris and Shawe-Taylor, John and Milakov, Maxim and Park, John and Ionescu, Radu and Popescu, Marius and Grozea, Cristian and Bergstra, James and Xie, Jingjing and Romaszko, Lukasz and Xu, Bing and Chuang, Zhang and Bengio, Yoshua},
  year = {2013},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1307.0414v1},
  eprint = {1307.0414}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF