Learning Deep Features for Scene Recognition using Places Database
Bolei ZhouÀgata LapedrizaJianxiong XiaoA. TorralbaA. Oliva
Introduces the 7-million-image Places database and demonstrates that convolutional neural networks trained on scene-centric data develop distinct visual representations that significantly outperform object-trained models on scene recognition tasks.
Scene recognition has long trailed object recognition in computer vision because available scene datasets were too small to train modern deep networks effectively. Researchers addressed this gap by building Places, a scene-centric collection of more than seven million labeled images across 476 categories, and by training convolutional neural networks on it to learn dedicated scene features.
The work set out to test whether a large, dedicated scene database could produce deep features that outperform those derived from the object-centric ImageNet dataset on scene classification tasks. The team gathered images through web searches that combined scene names with descriptive adjectives, removed duplicates, and used two rounds of Amazon Mechanical Turk verification to ensure label quality. They then trained a standard convolutional network architecture on a 205-category subset of Places and compared its performance and internal representations against an identical network trained on ImageNet. Relative density and diversity were measured through controlled human similarity judgments on the 88 categories shared with ImageNet and SUN.
Places proved comparable in density yet substantially more diverse than prior scene collections. The resulting Places-trained network reached 50 percent top-1 accuracy on the Places 205 test set and 66 percent on SUN 205, well above the accuracy obtained with ImageNet features. On established scene benchmarks it set new records, including 54.3 percent on SUN397 and 68.2 percent on MIT Indoor67, while ImageNet features remained stronger on object-centric tasks such as Caltech101. Visualization of network units showed that scene training produced receptive fields dominated by spatial layout and landscape structure, whereas object training produced more localized object-like patterns.
These outcomes indicate that scene and object recognition benefit from distinct high-level features and that large-scale scene data can close the performance gap that previously existed between the two domains. The two networks therefore offer complementary strengths that can be combined for broader visual recognition systems.
Further gains are likely from hybrid training sets that merge Places and ImageNet, from fine-tuning on additional scene benchmarks, and from relating the learned representations to known scene-processing pathways in the human brain. Results rest on the 205- and 476-category subsets released so far; performance on rarer or fine-grained scene distinctions remains untested, and all models inherit the biases present in web-sourced imagery.
- Paper: ImageNet Classification with Deep Convolutional Neural Networks, Alex Krizhevsky et al. (2012). This landmark paper introduced the deep convolutional neural network architectures and training protocols that the source study adapts and scales for large-scale scene recognition.
- Paper: Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, Svetlana Lazebnik et al. (2006). This foundational work establishes spatial pyramid matching for scene categorization, providing critical context on pre-deep-learning representations that the source study supersedes with CNN features.
- Paper: Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope, Aude Oliva et al. (2001). This classical study on holistic spatial properties of scenes defines the core perceptual dimensions of environments that the source study's deep network learns to represent implicitly.
- Paper: ImageNet: A large-scale hierarchical image database, Jia Deng et al. (2009). This foundational paper introduces the ImageNet database, whose object-centric design and hierarchical structure motivated the creation of the scene-centric Places database presented in the source study.
- Paper: Places: A 10 Million Image Database for Scene Recognition, Bolei Zhou et al. (2018). This later work directly extends the source study by scaling the Places database to over 10 million images and establishing comprehensive benchmark evaluations across multiple architectures.
- Paper: NetVLAD: CNN Architecture for Weakly Supervised Place Recognition, Relja Arandjelović et al. (2015). This study builds upon the feature representations established by scene- and object-centric networks to develop an end-to-end differentiable architecture for visual place recognition.
- Paper: Learning Deep Features for Discriminative Localization, Bolei Zhou et al. (2016). This subsequent work builds on the convolutional feature maps analyzed in the source study to introduce class activation mapping for weakly supervised localization.
- Paper: Scene Parsing through ADE20K Dataset, Bolei Zhou et al. (2017). This later paper extends the scene-centric data collection paradigm pioneered in the source study to build ADE20K for exhaustive dense scene parsing.
