LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
Fisher YuAri SeffYinda ZhangShuran SongThomas FunkhouserJianxiong Xiao
Presents an efficient human-in-the-loop labeling framework that scales dataset construction using deep model predictions, producing the massive LSUN visual recognition benchmark spanning millions of scene and object images.
The article addresses the challenge of creating sufficiently large and dense labeled image datasets to train high-capacity deep learning models for visual recognition. Existing collections such as ImageNet and Places have become too small relative to modern network sizes, and fully manual labeling cannot scale fast enough to close the gap.
The work set out to build a new dataset, LSUN, containing roughly one million labeled images in each of ten scene categories and twenty object categories, while keeping human effort practical.
Researchers collected more than one hundred million candidate images per category through web search, then applied an iterative pipeline. In each round a modest number of images received manual labels from Amazon Mechanical Turk workers; those labels trained a convolutional classifier that confidently labeled easy positives and negatives and passed only ambiguous images to the next round. The process continued until the remaining set was small enough for exhaustive manual review. Quality was maintained through redundant labeling, tutorial checks, and hidden ground-truth images.
The resulting dataset contains ten million scene images and fifty-nine million object images. Average labeling precision reached approximately ninety percent. Human effort was amplified by a factor of roughly forty. When AlexNet was fine-tuned on the new data, classification error on a Places-derived test set fell from 28.6 percent to 22.2 percent. Pre-training the same architecture on LSUN object images raised mean average precision on PASCAL VOC 2012 from 0.80 to 0.87; a deeper VGG network showed a similar gain.
These outcomes indicate that training on larger but slightly noisy data can outperform training on smaller clean collections, and that first-layer filters learned from the denser set appear cleaner and more coherent. The gains matter because they demonstrate a practical route to continued progress in visual recognition without proportional increases in manual annotation cost.
The authors plan to keep expanding LSUN and to release it publicly. Practitioners should consider using the data for pre-training or fine-tuning when category coverage aligns with their needs, while monitoring for residual label noise on categories that contain many toys or rendered images. Additional validation on new tasks and architectures would strengthen before large-scale deployment.
The main limitations are the ninety-percent precision ceiling, possible dataset bias relative to other collections, and the focus on only two model families. Results should be treated as indicative rather than definitive for every downstream application.
- Paper: Learning Deep Features for Scene Recognition using Places Database, Bolei Zhou et al. (2014). It introduces the Places database and demonstrates how dedicated large-scale scene datasets enable convolutional networks to learn scene-centric features, directly motivating LSUN's creation and evaluation protocol.
- Paper: ImageNet: A large-scale hierarchical image database, Jia Deng et al. (2009). It establishes the foundational methodology of using Amazon Mechanical Turk crowdsourcing to construct large-scale image ontologies that LSUN scales up through active human-in-the-loop filtering.
- Paper: ImageNet Classification with Deep Convolutional Neural Networks, Alex Krizhevsky et al. (2012). It provides the baseline AlexNet architecture and pretraining methodology that LSUN fine-tunes to demonstrate the empirical benefits of large-scale scene data.
- Paper: Unbiased look at dataset bias, A. Torralba et al. (2011). It formalizes the problem of dataset bias across visual recognition benchmarks, a key analytical limitation and motivation evaluated in the LSUN paper.
- Paper: Microsoft COCO: Common Objects in Context, Tsung-Yi Lin et al. (2014). It provides the baseline pipeline for multi-stage crowdsourced visual data collection and standard object detection benchmarks used for downstream validation in LSUN.
- Paper: Places: A 10 Million Image Database for Scene Recognition, Bolei Zhou et al. (2018). It scales dedicated scene-centric recognition to 10 million images across 434 categories, building directly on the large-scale scene learning paradigms explored by LSUN.
- Paper: Revisiting Unreasonable Effectiveness of Data in Deep Learning Era, Chen Sun et al. (2017). It investigates whether scaling noisy, weakly labeled datasets by orders of magnitude continues to drive downstream transfer performance, extending LSUN's core findings.
- Paper: LAION-5B: An open large-scale dataset for training next generation image-text models, Christoph Schuhmann et al. (2022). It massively scales automated web-scraping and filtering pipelines to billions of image-text pairs, extending the iterative, data-driven collection principles established in LSUN.
- Paper: Semantic Understanding of Scenes Through the ADE20K Dataset, Bolei Zhou et al. (2016). It advances large-scale scene understanding from image-level category recognition to dense pixel-level parsing across complex visual environments.
- Paper: Matterport3D: Learning from RGB-D Data in Indoor Environments, Angel Chang et al. (2017). It extends large-scale indoor scene dataset collection into full 3D spatial and RGB-D reconstructions for complex indoor visual reasoning.
