Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Chen SunAbhinav ShrivastavaSaurabh SinghAbhinav Gupta
Demonstrates through experiments on the 300-million-image JFT dataset that visual task performance scales logarithmically with data volume, proving that scaling pre-training data directly improves downstream accuracy across classification, detection, and segmentation.
The article addresses a key question in computer vision: while model capacity and GPU power have grown substantially since 2012, the largest widely used training dataset has stayed fixed at roughly one million ImageNet images. This stagnation raises the practical issue of whether performance on core tasks would continue to improve if training data were scaled by factors of ten or one hundred.
The article set out to measure how visual representations learned from a much larger, automatically labeled collection affect downstream performance. Researchers pre-trained standard ResNet architectures on the JFT-300M dataset of 300 million images carrying 375 million noisy labels across 18,000 categories, then evaluated the resulting models on image classification, object detection, semantic segmentation, and human pose estimation.
Experiments compared models trained from scratch on JFT-300M, models initialized from ImageNet and then trained on JFT-300M, and conventional ImageNet-only baselines. Subsets of 10 million, 30 million, and 100 million images were also tested, along with variations in model depth and label vocabulary size.
Performance on every task rose steadily with data volume, following a clear logarithmic relationship. Pre-training on the full 300 million images produced new state-of-the-art numbers, including a 3.1-point gain in COCO average precision (37.4 versus 34.3) and comparable lifts on PASCAL VOC detection and segmentation. Larger-capacity networks captured more benefit from the added data, while label noise of roughly 20 percent and a long-tailed category distribution did not prevent convergence or gains. Increasing the number of images mattered more than expanding the label vocabulary.
These results indicate that representation learning remains a high-leverage direction and that further scaling of training data can still deliver measurable improvements even after models have grown deeper. Because the observed gains are logarithmic, organizations can expect continued but diminishing returns that must be weighed against the cost of data collection and training.
The article recommends renewed collective investment in larger, diverse datasets and renewed attention to unsupervised or self-supervised methods that could exploit similar scale without exhaustive labeling. It also notes that the reported numbers likely underestimate potential gains, because training schedules were carried over from the 1-million-image regime without extensive retuning. Readers should treat the precise magnitudes as directional rather than definitive until larger-scale hyper-parameter studies are performed.
- Paper: Deep Residual Learning for Image Recognition, Kaiming He et al. (2016). This foundational work introduces the deep ResNet architectures that serve as the primary model backbones tested across the 300-million image dataset in the source paper.
- Paper: ImageNet Large Scale Visual Recognition Challenge, Olga Russakovsky et al. (2014). This paper establishes the 1-million-image ImageNet benchmark, providing the baseline dataset regime whose stagnation motivated the source paper's scaling study.
- Paper: ImageNet Classification with Deep Convolutional Neural Networks, Alex Krizhevsky et al. (2012). It initiated the modern deep learning era by demonstrating the power of training deep convolutional neural networks on ImageNet scale data.
- Paper: ImageNet: A large-scale hierarchical image database, Jia Deng et al. (2009). This paper details the construction and ontology of ImageNet, which defines the standard pre-training paradigm evaluated against massive web-scale pre-training.
- Paper: Unbiased look at dataset bias, A. Torralba et al. (2011). It provides crucial groundwork on dataset bias and cross-dataset generalization limits, contextualizing why larger and more diverse datasets are necessary.
- Paper: Very Deep Convolutional Networks for Large-Scale Image Recognition, Karen Simonyan et al. (2015). It demonstrated the importance of architectural depth in convolutional networks, establishing principles of model capacity explored further under extreme data scaling.
- Paper: LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop, Fisher Yu et al. (2015). This work explores scaling dataset construction through human-in-the-loop and semi-automated techniques, preceding fully automated noisy dataset curation.
- Paper: Fully convolutional networks for semantic segmentation, Jonathan Long et al. (2015). It introduces end-to-end fully convolutional networks for semantic segmentation, one of the primary downstream transfer tasks evaluated in the source paper.
- Paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy et al. (2021). This seminal work uses the JFT-300M dataset to demonstrate that Vision Transformers surpass convolutional networks when pre-trained on massive web-scale data.
- Paper: Self-Training With Noisy Student Improves ImageNet Classification, Qizhe Xie et al. (2019). It leverages the JFT-300M dataset within a semi-supervised noisy student framework, continuing the line of research on harvesting massive weakly labeled collections.
- Paper: Large Scale GAN Training for High Fidelity Natural Image Synthesis, Andrew Brock et al. (2019). This work explores scaling generative adversarial networks to extreme batch sizes and parameters, extending validation onto the JFT-300M dataset.
- Paper: Scaling Vision with Sparse Mixture of Experts, Carlos Riquelme et al. (2021). It scales Vision Transformers up to 15 billion parameters using mixture of experts pre-trained directly on JFT-300M.
- Paper: Learning Transferable Visual Models From Natural Language Supervision, Alec Radford et al.. It shifts from noisy discrete class labels to 400 million internet image-text pairs, extending data-scaling insights to multimodal zero-shot transfer.
- Paper: LAION-5B: An open large-scale dataset for training next generation image-text models, Christoph Schuhmann et al. (2022). It scales open multimodal datasets to over 5 billion image-text pairs, addressing the demand for public ultra-large-scale pre-training corpora.
- Paper: Do ImageNet Classifiers Generalize to ImageNet?, Benjamin Recht et al. (2019). This study critically investigates the generalization and reproduction limits of ImageNet-trained models, complementing findings on dataset scale and saturation.
- Paper: Big Self-Supervised Models are Strong Semi-Supervised Learners, Ting Chen et al. (2020). It demonstrates how scaling model size and self-supervised pre-training enables exceptional downstream transfer even with very few human labels.
- Paper: Masked Autoencoders Are Scalable Vision Learners, Kaiming He et al. (2022). It realizes the source's recommendation to explore self-supervised visual representation learning at scale without relying on supervised categorical labels.
- Paper: Training data-efficient image transformers & distillation through attention, Hugo Touvron et al. (2021). It directly tackles the heavy data appetite of vision transformers by introducing data-efficient training and distillation without needing hundreds of millions of images.
