PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

Yu XiangTanner SchmidtVenkatraman NarayananDieter Fox

article2017Robotics: Science and Systems Conference2,385 citations

Presents PoseCNN, a convolutional network that estimates full 6D object poses from color images by decoupling 3D translation and quaternion rotation to handle symmetric objects and severe occlusions, while introducing the standard YCB-Video benchmark.

Listen

For autonomous robotic systems to effectively manipulate items and interact with their surroundings, they must accurately determine an object's precise position and orientation in three-dimensional space, known as its six-dimensional pose. This capability is critical for practical automation in environments like logistics warehouses and manufacturing facilities. However, existing computer vision approaches struggle in real-world conditions where scenes are cluttered, objects partially block one another from view, surfaces lack distinctive visual textures, or shapes exhibit geometric symmetry.

The article set out to introduce and evaluate a deep learning framework, named PoseCNN, designed to robustly estimate six-dimensional object poses in cluttered and occluded scenes using standard color imagery, with optional depth-based refinement. Additionally, it aimed to demonstrate a training loss function tailored for symmetric objects and establish a large-scale video benchmark to advance research in this area.

The authors designed a neural network architecture that decouples pose estimation into three distinct steps: identifying object pixels, localizing the object center and its distance via a scale-invariant voting mechanism, and directly predicting the three-dimensional rotation. To address the ambiguity of symmetric objects, the authors introduced ShapeMatch-Loss, a training function that measures how closely estimated shapes align with physical targets rather than enforcing a single rigid orientation. The system was trained and evaluated on 80,000 synthetic images, the challenging OccludedLINEMOD benchmark, and the newly created YCB-Video dataset—a benchmark comprising 133,827 video frames across 92 video sequences covering 21 distinct objects.

The evaluation revealed several key findings. First, when relying strictly on standard color images, PoseCNN significantly outperformed previous state-of-the-art methods and alternative regression baselines on cluttered scenes, achieving an area under the accuracy curve of 75.9% on the YCB-Video dataset compared to 29.8% for standard coordinate regression. Second, when depth sensor data was incorporated for pose refinement, PoseCNN achieved a state-of-the-art mean accuracy of 78.0% on the OccludedLINEMOD dataset, surpassing established competing approaches. Third, the ShapeMatch-Loss successfully resolved rotational ambiguities for symmetric items, eliminating large orientation errors on objects like wooden blocks and clamps that previously caused training failures.

These findings demonstrate that automated robotic perception can achieve high reliability even under significant visual obstruction and without relying on highly textured surfaces. In practical applications, this reduces hardware costs and sensor complexity by enabling effective pose estimation using standard camera images, while allowing high-precision tasks to be completed when depth data is integrated for final alignment. It also eliminates the manual overhead previously required to explicitly program geometric symmetry rules for every manufactured part.

Organizations developing automated manipulation pipelines should consider adopting decoupled pose estimation architectures to handle occlusion and symmetry in complex operational environments. For tasks demanding high spatial precision under strict tolerances, implementing depth-based refinement alongside the network provides optimal reliability. Future development should focus on addressing remaining algorithmic limitations, such as optimizing symmetry loss functions to prevent convergence to local error minimums and improving the discrimination of identical objects that differ only in scale.

Confidence in these findings is high given the extensive validation across tens of thousands of real and synthetic frames. However, decision-makers should note that color-only pose accuracy experiences noticeable drops under extremely tight spatial thresholds (such as sub-two-centimeter tolerances), and performance can degrade on very small, texture-free items or visually identical objects of different sizes. In addition, the video dataset ground-truth annotations carry minor sensor artifacts, including camera rolling shutter and minor synchronization offsets between color and depth feeds.

Carnegie Mellon University NVIDIA University of Washington color images convolutional neural networks frames 6D object pose estimation object coordinate regression Convolutional Neural Network

arXiv: 1711.00199 yuxng/PoseCNN

Paper: You Only Look Once: Unified, Real-Time Object Detection, Joseph Redmon et al. (2016). You Only Look Once establishes the foundational real-time single-pass regression framework that PoseCNN adapts to predict 3D bounding coordinates and object classes.
Paper: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Charles R. Qi et al. (2017). PointNet introduces the direct deep learning principles for processing raw geometric coordinates that inform PoseCNN's approach to spatial regression and feature extraction.

Paper: Frustum PointNets for 3D Object Detection from RGB-D Data, Charles R. Qi et al. (2018). Frustum PointNets builds directly upon PoseCNN's 2D object localization approach by lifting detections into viewing frustums for improved 3D bounding-box regression.

Cover for PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

Abstract

Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at this https URL.

Topics

color images convolutional neural networks frames object pose clutter background video streams robotic grasping digital image

Keywords

6D object pose estimation object coordinate regression Convolutional Neural Network semantic segmentation textureless objects cluttered scenes

I INTRODUCTION
II RELATED WORK
III PoseCNN
III-A Overview of the Network
III-B Semantic Labeling
III-C 3D Translation Estimation
III-D 3D Rotation Regression
IV The YCB-Video Dataset
IV-A 6D Pose Annotation
IV-B Dataset Characteristics
V EXPERIMENTS
V-A Datasets
V-B Evaluation Metrics
V-C Implementation Details
V-D Baselines
V-E Analysis on the Rotation Regress Losses
V-F Results on the YCB-Video Dataset
V-G Results on the OccludedLINEMOD Dataset
VI CONCLUSIONS
References

Knowls

Knowl 1 — PoseCNN Architecture for Decoupled 6D Object Pose Estimation
model/method
PoseCNN is an end-to-end convolutional neural network architecture for estimating the 6D pose of rigid objects ( $3\text{D}$ rotation $R \in \mathrm{SO}(3)$ and $3\text{D}$ translation $T \in \mathbb{R}^3$ ) from RGB images. It decouples the pose estimation task into three distinct branches sharing a common feature extractor:
1. Feature Extraction Backbone: 13 convolutional layers and 4 max-pooling layers (parameterized by a VGG16 backbone) produce shared multi-scale feature maps at $1/8$ and $1/16$ of the original image resolution.
2. Semantic Labeling Branch: Embeds the backbone features to 64 channels, upsamples the $1/16$ map by $2\times$ via deconvolution, sums it with the $1/8$ map, and upsamples by $8\times$ via deconvolution to original image resolution. A convolutional layer outputs $n$ pixel-wise class logits trained via softmax cross-entropy loss ( $n$ is the number of target classes).
3. 3D Translation (Center Direction and Depth) Branch: Follows the same embedding structure as the semantic branch with 128 channels, outputting $3n$ channels ( $2\text{D}$ unit vector direction $(n_x, n_y)^T$ towards the projected $2\text{D}$ center and distance $T_z$ per class), trained using a smoothed $L_1$ loss. A Hough voting layer aggregates pixel votes to locate the $2\text{D}$ center $c=(c_x, c_y)^T$ , estimates $T_z$ , and outputs $2\text{D}$ bounding boxes.
4. 3D Rotation Regression Branch: Using the estimated bounding boxes, RoI pooling layers crop and pool features from the backbone. The pooled features are summed and passed through three fully connected layers (dimensions 4096, 4096, and $4n$ ) to output a unit quaternion representation of $R$ for each class.
Knowl 2 — 3D Translation Recovery from 2D Center Localization and Depth
equation
Given known camera intrinsic parameters (focal lengths $f_x, f_y$ and principal point $(p_x, p_y)^T$ ), the $3\text{D}$ translation $T = (T_x, T_y, T_z)^T$ of the object coordinate origin in the camera coordinate frame is recovered from its projected $2\text{D}$ image center $c = (c_x, c_y)^T$ and depth $T_z$ via pinhole projection:

$\begin{bmatrix} c_x \\ c_y \end{bmatrix} = \begin{bmatrix} f_x \frac{T_x}{T_z} + p_x \\ f_y \frac{T_y}{T_z} + p_y \end{bmatrix}$

Solving for horizontal and vertical translations yields:

$T_x = \frac{(c_x - p_x) T_z}{f_x}, \quad T_y = \frac{(c_y - p_y) T_z}{f_y}$

To predict $c$ , each pixel $p = (x, y)^T$ regresses a scale-invariant unit direction vector $n = (n_x, n_y)^T$ towards the center along with the depth $T_z$ :

$(x, y) \to \left( n_x = \frac{c_x - x}{\|c - p\|}, \; n_y = \frac{c_y - y}{\|c - p\|}, \; T_z \right)$
Knowl 3 — Hough Voting Algorithm for Object Center Localization and Bounding Box Extraction
algorithm
The Hough voting layer aggregates pixel-level predictions to localize object centers and bounding boxes. No gradients are back-propagated through this layer during training.
Input: Semantic segmentation mask $S$ , predicted 2D unit center vectors $n(p)$ , predicted depths $T_z(p)$ for all pixels $p$ in image of dimensions $W \times H$ , target class $k$
Output: Estimated 2D center $c^*$ , estimated depth $T_z^*$ , 2D bounding box $B$
Initialize accumulator array $V(x, y) \leftarrow 0$ for all $x \in [1, W], y \in [1, H]$
for each pixel $p=(x, y)$ such that $S(p) == k$ do
    for each integer step $r > 0$ along ray $p + r \cdot n(p)$ within image boundaries do
         $V(p + r \cdot n(p)) \leftarrow V(p + r \cdot n(p)) + 1$
    end for
end for
Apply non-maximum suppression to voting grid $V$
$c^* \leftarrow \arg\max_{(x, y)} V(x, y)$
$I \leftarrow \{p \mid S(p) == k \text{ and pixel } p \text{ voted for } c^*\}$
$T_z^* \leftarrow \frac{1}{|I|} \sum_{p \in I} T_z(p)$
$B \leftarrow [\min_{p \in I} x(p), \; \min_{p \in I} y(p), \; \max_{p \in I} x(p), \; \max_{p \in I} y(p)]$
return $c^*, T_z^*, B$
Knowl 4 — PoseLoss for Asymmetric 3D Rotation Regression
equation
For asymmetric objects, PoseCNN trains rotation regression using PoseLoss ( $PLOSS$ ). Let $M$ denote the set of $m$ $3\text{D}$ points sampled from the target object mesh model. Given the ground truth quaternion $q$ and estimated quaternion $\tilde{q}$ , which determine $3\times 3$ rotation matrices $R(q)$ and $R(\tilde{q})$ respectively, PoseLoss computes the average squared Euclidean distance between corresponding points on the object model:

$PLOSS(\tilde{q}, q) = \frac{1}{2m} \sum_{x \in M} \|R(\tilde{q})x - R(q)x\|^2$

$PLOSS$ attains a unique global minimum when $R(\tilde{q}) = R(q)$ .
Knowl 5 — ShapeMatch-Loss for Symmetric Object 6D Rotation Regression
equation
To train rotation regression on objects with geometric symmetries without requiring explicit manual specification of symmetry groups, ShapeMatch-Loss ( $SLOSS$ ) measures the average squared distance from each point on the estimated model orientation to the closest point on the ground truth model:

$SLOSS(\tilde{q}, q) = \frac{1}{2m} \sum_{x_1 \in M} \min_{x_2 \in M} \|R(\tilde{q})x_1 - R(q)x_2\|^2$

where $M$ denotes the set of $m$ $3\text{D}$ model points, $R(\tilde{q})$ is the rotation matrix of the predicted quaternion $\tilde{q}$ , and $R(q)$ is the rotation matrix of the ground truth quaternion $q$ . $SLOSS$ produces zero loss whenever the predicted $3\text{D}$ shape aligns with the ground truth shape, preventing inconsistent gradient penalties caused by ambiguous rotational orientations.
Knowl 6 — Depth-Based 6D Pose Refinement via Projective ICP
model/method
When depth observations are available, the initial $6\text{D}$ pose $(\tilde{R}, \tilde{T})$ estimated by PoseCNN is refined using an Iterative Closest Point (ICP) algorithm with projective data association and a point-to-plane residual:
1. Data Association: Using the predicted semantic mask, observed $3\text{D}$ points are cropped from the depth image. A point cloud is rendered using the $3\text{D}$ object model placed at the estimated pose $(\tilde{R}, \tilde{T})$ . Each observed depth pixel is paired with the rendered model point at the same pixel location.
2. Point-to-Plane Residual Minimization: Residuals exceeding a threshold are discarded as outliers. The remaining point-to-plane residuals $(p_{\text{obs}} - p_{\text{rend}}) \cdot n_{\text{rend}}$ (where $n_{\text{rend}}$ is the rendered surface normal) are minimized using gradient descent.
3. Multiple Hypotheses Perturbation: To avoid convergence to local minima, multiple initial poses are generated by perturbing the PoseCNN pose, each is refined via ICP, and the best refined pose is selected using an alignment metric.
Knowl 7 — The YCB-Video Dataset for 6D Object Pose Estimation
experimental setup
The YCB-Video dataset contains 92 RGB-D video sequences with a total of 133,827 frames at $640 \times 480$ resolution captured at $30\text{ FPS}$ using an Asus Xtion Pro Live camera. It features 21 distinct objects selected from the YCB object set exhibiting varying textures, geometries, and rotational symmetries.

Key characteristics of the dataset include:
- Train/Test Split: 80 video sequences for training and 12 held-out video sequences providing 2,949 keyframes for testing.
- Clutter and Occlusion: Scenes contain between 3 and 9 objects per frame (mean of 4.47 objects) in dense configurations.
- Ground Truth Pose Annotation: Poses are manually initialized in the first frame, refined using Signed Distance Function (SDF) representations on the initial depth frame, tracked across frames via depth video tracking, and refined using a global trajectory and pose optimization step.

Knowl 8 — Pose Estimation Accuracy on the YCB-Video Dataset

data/table

Evaluation of $6\text{D}$ pose accuracy across 2,949 test keyframes for 21 objects on the YCB-Video dataset. Metrics report the Area Under the Curve (AUC) for the ADD metric (average distance for asymmetric objects) and ADD-S metric (closest-point distance for symmetric objects), up to a maximum error threshold of $10\text{ cm}$ .

	RGB (3D Coord)		RGB (PoseCNN)		RGB-D (PoseCNN+ICP)
Object	ADD	ADD-S	ADD	ADD-S	ADD	ADD-S
002_master_chef_can	12.3	34.4	50.9	84.0	69.0	95.8
003_cracker_box	16.8	40.0	51.7	76.9	80.7	91.8
004_sugar_box	28.7	48.9	68.6	84.3	97.2	98.2
005_tomato_soup_can	27.3	42.2	66.0	80.9	81.6	94.5
006_mustard_bottle	25.9	44.8	79.9	90.2	97.0	98.4
007_tuna_fish_can	5.4	10.4	70.4	87.9	83.1	97.1
008_pudding_box	14.9	26.3	62.9	79.0	96.6	97.9
009_gelatin_box	25.4	36.7	75.2	87.1	98.2	98.8
010_potted_meat_can	18.7	32.3	59.6	78.5	83.8	92.8
011_banana	3.2	8.8	72.3	85.9	91.6	96.9
019_pitcher_base	27.3	54.3	52.5	76.8	96.7	97.8
021_bleach_cleanser	25.2	44.3	50.5	71.9	92.3	96.8
024_bowl	2.7	25.4	6.5	69.7	17.5	78.3
025_mug	9.0	20.0	57.7	78.0	81.4	95.1
035_power_drill	18.0	36.1	55.1	72.8	96.9	98.0
036_wood_block	1.2	19.6	31.8	65.8	79.2	90.5
037_scissors	1.0	2.9	35.8	56.2	78.4	92.2
040_large_marker	0.2	0.3	58.0	71.4	85.4	97.2
051_large_clamp	6.9	14.6	25.0	49.9	52.6	75.4
052_extra_large_clamp	2.7	14.0	15.8	47.0	28.7	65.3
061_foam_brick	0.6	1.2	40.4	87.8	48.3	97.1
ALL (Mean)	15.1	29.8	53.7	75.9	79.3	93.0

Using only RGB inputs, PoseCNN achieves an overall ADD AUC of 53.7 and ADD-S AUC of 75.9, significantly outperforming the 3D coordinate regression baseline (15.1 ADD, 29.8 ADD-S). Adding ICP refinement on depth data increases PoseCNN's performance to 79.3 ADD and 93.0 ADD-S.

Knowl 9 — 6D Pose Estimation Accuracy on the OccludedLINEMOD Dataset

data/table

Pose estimation accuracy on the OccludedLINEMOD benchmark (1,214 test frames with 8 annotated objects). A predicted pose is considered correct if the average model point distance is within $10\%$ of the $3\text{D}$ object model diameter (ADD for asymmetric objects, ADD-S for symmetric objects: Eggbox and Glue).

Object	Michel et al.	Hinterstoisser et al.	Krull et al.	Brachmann et al.	PoseCNN (Color)	PoseCNN+ICP (RGB-D)
Ape	80.7	81.4	68.0	53.1	9.6	76.2
Can	88.5	94.7	87.9	79.9	45.2	87.4
Cat	57.8	55.2	50.6	28.2	0.93	52.2
Driller	94.7	86.0	91.2	82.0	41.4	90.3
Duck	74.4	79.7	64.7	64.3	19.6	77.7
Eggbox	47.6	65.5	41.5	9.0	22.0	72.2
Glue	73.8	52.1	65.3	44.5	38.5	76.7
Holepuncher	96.3	95.5	92.9	91.6	22.1	91.4
MEAN	76.7	76.3	70.3	56.6	24.9	78.0

PoseCNN+ICP achieves an overall mean accuracy of $78.0\%$ , establishing state-of-the-art results on RGB-D input. Its largest advantages over previous methods occur on the symmetric objects Eggbox ( $72.2\%$ ) and Glue ( $76.7\%$ ), directly enabled by training with ShapeMatch-Loss.

Knowl 10 — Impact of ShapeMatch-Loss on Symmetric Object Orientation Learning
empirical result
When training rotation regression with standard PoseLoss ( $PLOSS$ ) on symmetric objects (such as the wood block and large clamp in the YCB-Video dataset), orientation prediction errors are spread broadly across $0^\circ$ to $180^\circ$ due to conflicting loss signals from geometrically identical configurations. Training with ShapeMatch-Loss ( $SLOSS$ ) eliminates penalties for symmetry-equivalent orientations, concentrating prediction errors tightly at the true symmetry modes ( $180^\circ$ for the wood block, and $0^\circ$ and $180^\circ$ for the large clamp, reflecting their $180^\circ$ rotational symmetries around coordinate axes).
Knowl 11 — Limitations of PoseCNN and ShapeMatch-Loss
limitation
The PoseCNN approach has four primary limitations identified in experimental evaluation:
1. Optimization Local Minima: ShapeMatch-Loss ( $SLOSS$ ) can suffer from local minima in pose space during network training, analogous to local minimum traps in standard ICP.
2. Scale Ambiguity for Identical Appearances: The network confuses objects sharing identical shape and texture appearances that differ only by physical scale, such as the large clamp and extra large clamp.
3. Textureless/Small Object Difficulty: Small objects with minimal surface texture (e.g., tuna fish can, scissors) exhibit lower pose estimation accuracy from RGB alone.
4. Color-Only Accuracy Under Strict Thresholds: Color-only pose estimation accuracy drops substantially when evaluated under strict spatial error thresholds (e.g., $<2\text{ cm}$ or $10\%$ model diameter under severe occlusions).

Coverage note — None was omitted; all key architectural components, mathematical loss functions, datasets, empirical evaluations, and failure modes are covered.

References

1.Mart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. TensorFlow: A system for large-scale machine learning. In OSDI, volume 16, pages 265–283, 2016.
2.Liefeng Bo, Xiaofeng Ren, and Dieter Fox. Learning hierarchical sparse features for RGB-D object recognition. International Journal of Robotics Research (IJRR), 33(4):581–599, 2014.
3.Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6D object pose estimation using 3D object coordinates. In European Conference on Computer Vision (ECCV), pages 536–551, 2014.
4.Eric Brachmann, Frank Michel, Alexander Krull, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3364–3372, 2016.
5.Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The YCB object and model set: Towards common benchmarks for manipulation research. In International Conference on Advanced Robotics (ICAR), pages 510–517, 2015.
6.Zhe Cao, Yaser Sheikh, and Natasha Kholgade Banerjee. Real-time scalable 6DOF pose estimation for textureless objects. In IEEE International Conference on Robotics and Automation (ICRA), pages 2441–2448, 2016.
7.Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
8.Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa. The MOPED framework: Object recognition and pose estimation for manipulation. International Journal of Robotics Research (IJRR), 30(10):1284–1306, 2011.
9.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009.
10.Andreas Doumanoglou, Vassileios Balntas, Rigas Kouskouridas, and Tae-Kyun Kim. Siamese regression networks with efficient mid-level feature extraction for 3D object pose estimation. arXiv preprint arXiv:1607.02257, 2016.
11.Ross Girshick. Fast R-CNN. In IEEE International Conference on Computer Vision (ICCV), pages 1440–1448, 2015.
12.Stefan Hinterstoisser, Cedric Cagniart, Slobodan Ilic, Peter Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit. Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 34(5):876–888, 2012.
13.Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In Asian Conference on Computer Vision (ACCV), pages 548–562, 2012.
14.Stefan Hinterstoisser, Vincent Lepetit, Naresh Rajkumar, and Kurt Konolige. Going further with point pair features. In European Conference on Computer Vision (ECCV), pages 834–848, 2016.
15.Wadim Kehl, Fausto Milletari, Federico Tombari, Slobodan Ilic, and Nassir Navab. Deep learning of local RGB-D patches for 3D object detection and 6D pose estimation. In European Conference on Computer Vision (ECCV), pages 205–220, 2016.
16.Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. In IEEE International Conference on Computer Vision (ICCV), pages 1521–1529, 2017.
17.Alexander Krull, Eric Brachmann, Frank Michel, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Learning analysis-by-synthesis for 6D pose estimation in RGB-D images. In IEEE International Conference on Computer Vision (ICCV), pages 954–962, 2015.
18.Bastian Leibe, Ales Leonardis, and Bernt Schiele. Combined object categorization and segmentation with an implicit shape model. In ECCV Workshop on statistical learning in computer vision, 2004.
19.Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431–3440, 2015.
20.David G Lowe. Object recognition from local scale-invariant features. In IEEE International Conference on Computer Vision (ICCV), volume 2, pages 1150–1157, 1999.
21.Frank Michel, Alexander Kirillov, Erix Brachmann, Alexander Krull, Stefan Gumhold, Bogdan Savchynskyy, and Carsten Rother. Global hypothesis generation for 6D object pose estimation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
22.Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis. 6-DOF object pose from semantic keypoints. IEEE International Conference on Robotics and Automation (ICRA), 2017.
23.Mahdi Rad and Vincent Lepetit. BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth. In IEEE International Conference on Computer Vision (ICCV), 2017.
24.Colin Rennie, Rahul Shome, Kostas E Bekris, and Alberto F De Souza. A dataset for improved RGBD-based object detection and pose estimation for warehouse pick-and-place. IEEE Robotics and Automation Letters, 1(2):1179–1185, 2016.
25.Fred Rothganger, Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 3D object modeling and recognition using local affine-invariant image descriptors and multi-view spatial constraints. International Journal of Computer Vision (IJCV), 66(3):231–259, 2006.
26.Max Schwarz, Hannes Schulz, and Sven Behnke. RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features. In IEEE International Conference on Robotics and Automation (ICRA), pages 1329–1335, 2015.
27.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
28.Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas. Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views. In IEEE International Conference on Computer Vision (ICCV), pages 2686–2694, 2015.
29.Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6D object pose prediction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
30.Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1510–1519, 2015.
31.Gert-Jan van den Braak, Cedric Nugteren, Bart Mesman, and Henk Corporaal. Fast Hough transform on GPUs: Exploration of algorithm trade-offs. In International Conference on Advanced Concepts for Intelligent Vision Systems, pages 611–622, 2011.
32.Paul Wohlhart and Vincent Lepetit. Learning descriptors for object recognition and 3D pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3109–3118, 2015.
33.Jay M Wong, Vincent Kee, Tiffany Le, Syler Wagner, Gian-Luca Mariottini, Abraham Schneider, Lei Hamilton, Rahul Chipalkatty, Mitchell Hebert, David Johnson, et al. SegICP: Integrated deep semantic segmentation and pose estimation. arXiv preprint arXiv:1703.01661, 2017.
34.Yu Xiang and Dieter Fox. DA-RNN: Semantic mapping with data associated recurrent neural networks. In Robotics: Science and Systems (RSS). 2017.
35.Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6D pose estimation in the amazon picking challenge. In IEEE International Conference on Robotics and Automation (ICRA), pages 1386–1383, 2017.

Citation

MLA

Xiang, Y., et al. “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes”. Robotics: Science and Systems XIV, 2018, https://doi.org/10.15607/RSS.2018.XIV.019.

APA

Xiang, Y., Schmidt, T., Narayanan, V., & Fox, D. (2018). PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Robotics: Science and Systems XIV. https://doi.org/10.15607/RSS.2018.XIV.019

Chicago

Xiang, Y., T. Schmidt, V. Narayanan, and D. Fox. 2018. “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes”. Robotics: Science and Systems XIV. https://doi.org/10.15607/RSS.2018.XIV.019.

Harvard

Xiang, Y. et al. (2018) “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes”, Robotics: Science and Systems XIV. Robotics: Science and Systems Foundation. Available at: https://doi.org/10.15607/RSS.2018.XIV.019.

Vancouver

1. Xiang Y, Schmidt T, Narayanan V, Fox D (2018) PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Robotics: Science and Systems XIV. https://doi.org/10.15607/RSS.2018.XIV.019

BibTeX

@inproceedings{Xiang_2018, series={RSS2018}, title={PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes}, url={http://dx.doi.org/10.15607/RSS.2018.XIV.019}, DOI={10.15607/rss.2018.xiv.019}, booktitle={Robotics: Science and Systems XIV}, publisher={Robotics: Science and Systems Foundation}, author={Xiang, Yu and Schmidt, Tanner and Narayanan, Venkatraman and Fox, Dieter}, year={2018}, month=June, collection={RSS2018} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — PoseCNN Architecture for Decoupled 6D Object Pose Estimation

Knowl 2 — 3D Translation Recovery from 2D Center Localization and Depth

Knowl 3 — Hough Voting Algorithm for Object Center Localization and Bounding Box Extraction

Knowl 4 — PoseLoss for Asymmetric 3D Rotation Regression

Knowl 5 — ShapeMatch-Loss for Symmetric Object 6D Rotation Regression

Knowl 6 — Depth-Based 6D Pose Refinement via Projective ICP

Knowl 7 — The YCB-Video Dataset for 6D Object Pose Estimation