Multi-view Convolutional Neural Networks for 3D Shape Recognition
Hang SuSubhransu MajiEvangelos KalogerakisErik Learned-Miller
Introduces a multi-view convolutional neural network architecture that pools standard 2D rendered views into a compact descriptor, demonstrating that 2D image representations outperform native 3D voxel and mesh models on 3D shape and sketch recognition.
A longstanding challenge in computer vision is how to represent three-dimensional shapes for accurate recognition when only two-dimensional images are typically available. Large online repositories now supply thousands of 3D models, yet prior classifiers built directly on voxel or mesh representations have achieved only modest accuracy. The paper examines whether collections of 2D renderings can serve as a stronger basis for recognition than native 3D descriptors.
The work set out to measure the recognition performance obtainable from rendered views alone and to develop a compact descriptor that fuses information across multiple views. Researchers rendered twelve fixed viewpoints of each model in the ModelNet40 benchmark, passed the images through convolutional networks pre-trained on ImageNet and fine-tuned on the rendered views, and compared results against established 3D shape descriptors and against simpler view-averaging baselines.
A single rendered view classified by a fine-tuned network already reached 84 percent accuracy, exceeding the previous best 3D method by roughly seven points. Supplying all twelve views raised accuracy to 87–89 percent and mean average precision for retrieval to 69 percent; a learned low-rank metric further lifted retrieval to 79 percent while shrinking the descriptor to 128 dimensions. The same multi-view architecture improved sketch recognition from 79 percent to 87 percent, approaching human performance on cleaned data, and enabled direct sketch-to-model retrieval at 36 percent mAP without additional domain-specific training.
These gains indicate that high-resolution two-dimensional projections, combined with existing image networks, capture category-discriminating structure more effectively than coarse voxel grids. The resulting descriptors are compact, fast to compare, and immediately usable for both classification and retrieval tasks.
Further gains will likely require systematic study of viewpoint selection and the extension of the same aggregation approach to video or real-world multi-view captures. The current results rest on upright-oriented synthetic models rendered under controlled lighting; performance on unoriented or noisy scans remains untested. Overall the evidence strongly supports view-based convolutional pipelines for 3D shape tasks, with the largest practical advantages appearing in retrieval and cross-domain search.
- Paper: An Introduction to Convolutional Neural Networks, Keiron O'Shea et al. (2015). Reading this foundational tutorial on convolutional neural networks provides the core architectural background necessary to understand how the source applies CNNs to 2D views of 3D objects.
- Paper: CNN Features Off-the-Shelf: An Astounding Baseline for Recognition, Ali Sharif Razavian et al. (2014). This paper establishes the powerful baseline of using pre-trained image classification CNNs off-the-shelf, which directly motivates the source's investigation into applying standard CNN architectures to rendered 2D views.
- Paper: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Ben Mildenhall et al. (2020). Taking inspiration from multi-view representations, this paper advances 3D scene understanding by introducing neural radiance fields to synthesize continuous novel views.
- Paper: Dynamic Graph CNN for Learning on Point Clouds, Yue Wang et al. (2018). Extending beyond the source's view-based and volumetric approaches, this work introduces dynamic graph convolutions to learn local geometric structures directly on raw 3D point clouds.
