Convolutional Pose Machines

Shih-En WeiVarun RamakrishnaTakeo KanadeYaser Sheikh

article2016CVPR2,935 citations

Proposes Convolutional Pose Machines, an end-to-end multi-stage architecture that captures long-range spatial context for articulated human pose estimation by iteratively refining part belief maps using intermediate supervision without requiring explicit graphical models.

Listen

Convolutional Pose Machines introduce a sequential prediction architecture that integrates convolutional networks to estimate the locations of human body joints from single images. The work targets the long-standing challenge of articulated pose estimation, where parts exhibit large appearance variation, occlusions, and complex spatial dependencies that defeat both simple local detectors and tree-structured graphical models. Accurate pose estimation matters now because it underpins applications in activity recognition, humancomputer interaction, and video analysis, yet prior methods either required hand-designed spatial priors or suffered from training difficulties in deep networks.

The paper set out to demonstrate that a fully differentiable sequence of convolutional stages can implicitly learn both image features and image-dependent spatial relationships, eliminating the need for explicit graphical-model inference while avoiding vanishing gradients during training. The approach replaces the predictors inside the original pose-machine framework with convolutional networks that operate directly on intermediate belief maps. Each stage receives image evidence plus the belief maps from the preceding stage, and the network is sized to produce a large receptive field that captures long-range part interactions. Intermediate loss layers enforce supervision after every stage, replenishing gradients and guiding successive refinements. The resulting multi-stage model was trained end-to-end on standard benchmarks and evaluated with the PCK and PCKh metrics on the MPII, LSP, and FLIC datasets.

The central finding is that the architecture reaches state-of-the-art accuracy: 87.95 % PCKh at the 0.5 threshold on MPII (rising to 88.52 % when LSP data are added) and 84.32 % PCK on LSP, with the largest gains on difficult extremities such as ankles (roughly 10 percentage points above prior best results). Accuracy improves monotonically with the number of stages and with receptive-field size up to approximately 400 pixels, confirming that long-range context is effectively encoded. Intermediate supervision measurably prevents gradient collapse across deep layers and yields higher final performance than stage-wise training or joint training without intermediate losses. The method also delivers consistent gains across all viewpoints, including challenging non-frontal poses, without pre-training on external data or post-processing with hand-crafted priors.

These results indicate that pose estimation can be treated as a learned, feed-forward structured-prediction task rather than one requiring separate inference machinery, thereby simplifying deployment and improving precision at both coarse and fine scales. The approach removes the accuracyefficiency trade-off that previously forced designers to limit model expressiveness for tractable inference. Because the architecture is modular and fully differentiable, it can be extended to related spatial-prediction problems such as semantic labeling or depth estimation.

Further work is needed to handle scenes containing multiple overlapping people within a single end-to-end network; current failure cases occur mainly when individuals are in close proximity. Additional gains may also come from incorporating residual connections or testing on video sequences, yet the core design already provides a reproducible, high-accuracy baseline that future systems can build upon. The reported gains rest on standard academic benchmarks whose labeling quality varies; users should therefore validate performance on their own imagery before critical deployment.

  • Paper: DeepPose: Human Pose Estimation via Deep Neural Networks, Alexander Toshev et al. (2014). DeepPose introduced the formulation of human pose estimation as a deep regression task solved by convolutional neural networks, establishing the foundational architecture that Convolutional Pose Machines directly build upon and refine.
Cover for Convolutional Pose Machines

Abstract

Pose Machines provide a sequential prediction framework for learning rich implicit spatial models. In this work we show a systematic design for how convolutional networks can be incorporated into the pose machine framework for learning image features and image-dependent spatial models for the task of pose estimation. The contribution of this paper is to implicitly model long-range dependencies between variables in structured prediction tasks such as articulated pose estimation. We achieve this by designing a sequential architecture composed of convolutional networks that directly operate on belief maps from previous stages, producing increasingly refined estimates for part locations, without the need for explicit graphical model-style inference. Our approach addresses the characteristic difficulty of vanishing gradients during training by providing a natural learning objective function that enforces intermediate supervision, thereby replenishing back-propagated gradients and conditioning the learning procedure. We demonstrate state-of-the-art performance and outperform competing methods on standard benchmarks including the MPII, LSP, and FLIC datasets.

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Method
  • 3.1 Pose Machines
  • 3.2 Convolutional Pose Machines
  • 3.2.1 Keypoint Localization Using Local Image Evidence
  • 3.2.2 Sequential Prediction with Learned Spatial Context Features
  • 3.3 Learning in Convolutional Pose Machines
  • 4 Evaluation
  • 4.1 Analysis
  • 4.2 Datasets and Quantitative Analysis
  • 5 Discussion
  • References

Knowls

  1. Knowl 1 — Multi-Stage Convolutional Pose Machine Architecture

    model/method

    A Convolutional Pose Machine (CPM) is a multi-stage fully convolutional architecture for 2D articulated pose estimation that sequentially produces and refines non-parametric belief maps. Let ZR2\mathcal{Z} \subset \mathbb{R}^2 be the set of pixel coordinates (u,v)(u, v) in an image, and let PP denote the number of anatomical parts (keypoints). The goal is to estimate the part coordinates Y=(Y1,,YP)Y = (Y_1, \dots, Y_P), where YpZY_p \in \mathcal{Z}.

    The CPM operates in TT sequential stages:

    1. Stage 1: Operates strictly on local image evidence. A convolutional sub-network takes the input image xx and extracts local image features xzx_z, producing initial belief maps b1Rw×h×(P+1)b_1 \in \mathbb{R}^{w' \times h' \times (P+1)} across grid locations zZz \in \mathcal{Z}': g1(xz){b1p(Yp=z)}p{0,,P}g_1(x_z) \rightarrow \left\{ b_1^p(Y_p = z) \right\}_{p \in \{0, \dots, P\}} where p=0p=0 corresponds to the background class and p{1,,P}p \in \{1, \dots, P\} corresponds to body parts. Stage 1 consists of five convolutional layers with pooling layers that downsample spatial dimensions by a factor of 8 (stride 8) to establish an effective receptive field of 160×160160 \times 160 pixels on a 368×368368 \times 368 input image, followed by two 1×11 \times 1 convolutions.

    2. Subsequent Stages (t2t \ge 2): At each subsequent stage t{2,,T}t \in \{2, \dots, T\}, the predictor gtg_t operates on both the image feature representation xzx'_z (shared across stages t2t \ge 2) and spatial context features computed from the belief maps bt1b_{t-1} of the previous stage: gt(xz,ψt(z,bt1)){btp(Yp=z)}p{0,,P}g_t(x'_z, \psi_t(z, b_{t-1})) \rightarrow \left\{ b_t^p(Y_p = z) \right\}_{p \in \{0, \dots, P\}} The contextual feature function ψt\psi_t is implemented implicitly via the receptive field of convolutional layers acting on the previous belief maps bt1b_{t-1}. In each stage t2t \ge 2, bt1b_{t-1} and xx' are concatenated along the channel dimension and processed through convolutional layers to output refined belief maps btb_t.

  2. Knowl 2 — Intermediate Supervision and Loss Formulation in Convolutional Pose Machines

    equation

    In a TT-stage Convolutional Pose Machine, each stage t{1,,T}t \in \{1, \dots, T\} outputs a predicted belief map tensor btRw×h×(P+1)b_t \in \mathbb{R}^{w' \times h' \times (P+1)} across spatial locations zZz \in \mathcal{Z} for PP body parts and one background class. To train the network end-to-end and avoid vanishing gradients across stages, an L2L_2 loss is defined at the output of every stage tt: ft=p=1P+1zZbtp(z)bp(z)22f_t = \sum_{p=1}^{P+1} \sum_{z \in \mathcal{Z}} \left\| b_t^p(z) - b_*^p(z) \right\|_2^2 where btp(z)b_t^p(z) is the predicted belief score for part pp at pixel location zz, and bp(z)b_*^p(z) is the ground-truth ideal belief map for part pp. The ideal belief map bp(z)b_*^p(z) is generated by placing a 2D Gaussian peak centered at the ground-truth pixel coordinate YpY_p^* of part pp: bp(z)=exp(zYp22σ2)b_*^p(z) = \exp\left( -\frac{\|z - Y_p^*\|_2^2}{\sigma^2} \right) where σ\sigma controls the spatial spread of the target Gaussian distribution.

    The global loss function FF optimized during training across all stages is the sum of all intermediate losses: F=t=1TftF = \sum_{t=1}^T f_t The complete multi-stage model is trained jointly using stochastic gradient descent, with each intermediate loss ftf_t directly replenishing gradients back into stage tt.

  3. Knowl 3 — Large Receptive Field Design for Implicit Spatial Context

    model/method

    To capture long-range geometric dependencies between human body parts (such as relationships between the head and feet) without explicit graphical model inference, subsequent stages (t2t \ge 2) in a Convolutional Pose Machine are designed with large effective receptive fields on both the image features and previous belief maps.

    1. Downscaling and Layer Stacking: Feature representations and belief maps are processed at a stride of 8 (8×8\times downscaled spatial resolution). Stacking multiple small convolutional filters on stride-8 maps yields large effective receptive fields with significantly fewer parameters compared to using very large filter kernels or aggressive pooling.
    2. Effective Receptive Field Extent: For an input crop normalized to 368×368368 \times 368 pixels, Stage 1 possesses an effective receptive field of 160×160160 \times 160 pixels around each local prediction. In stages t2t \ge 2, the convolutional layers operating on the previous stage's belief maps achieve a receptive field of 31×3131 \times 31 on the stride-8 heatmap, corresponding to an effective receptive field of 400×400400 \times 400 pixels on the input image. This radius is sufficient to cover any pair of body parts across a normalized person.
  4. Knowl 4 — Dual Ground-Truth Belief Map Strategy for Proximity Disambiguation

    model/method

    When training Convolutional Pose Machines on images containing multiple people in proximity to a primary target subject (as in the MPII Human Pose dataset), two distinct ground-truth belief map configurations are used across stages:

    1. Stage 1 Loss Targets: Because Stage 1 operates on a local receptive field and lacks the global context required to disambiguate the target individual from bystanders, its target belief maps bpb_*^p contain Gaussian peaks for every person visible within the cropped image region.
    2. Subsequent Stage Loss Targets (t2t \ge 2): In subsequent stages where the receptive field spans the entire person, the ground-truth belief maps bpb_*^p contain Gaussian peaks only for the designated primary subject.
    3. Target Center Conditioning: To condition subsequent stages on the target person, an auxiliary Gaussian heatmap channel centered at the rough ground-truth center position of the primary subject is concatenated with the feature and belief maps entering stages t2t \ge 2.
  5. Knowl 5 — Vanishing Gradient Mitigation via Periodic Intermediate Supervision

    empirical result

    In deep multi-stage structured prediction networks, tracking gradient magnitude distributions across layers and training epochs demonstrates the necessity of intermediate supervision:

    1. Without Intermediate Supervision: In a model trained with only a terminal loss at the final stage, layers closest to the output show high gradient magnitude variance, but gradient distributions in layers closer to the input peak tightly at zero with near-zero variance during early training epochs, confirming that backpropagated gradients vanish.
    2. With Intermediate Supervision: Adding intermediate loss functions ftf_t after each stage replenishes the backpropagated gradient signal throughout the entire depth of the network. Gradient magnitudes maintain substantial variance across all layers from stage 1 to the final stage during initial epochs, allowing all stages to train simultaneously. As training progresses to convergence, gradient variance diminishes uniformly across all layers.
  6. Knowl 6 — Empirical Comparison of CPM Training Schemes

    empirical result

    On the Leeds Sports Pose (LSP) dataset with person-centric annotations, four training regimes were evaluated on a 3-stage Convolutional Pose Machine across Percentage of Correct Keypoints (PCK) thresholds:

    1. Joint Training from Scratch with Intermediate Supervision: Training all stages jointly from scratch using the summed multi-stage loss F=t=1TftF = \sum_{t=1}^T f_t achieved the highest PCK detection rates across all normalized distance thresholds.
    2. Stagewise Training: Training each stage independently in sequence and stacking them without joint backpropagation saturated at an inferior, suboptimal accuracy curve.
    3. Stagewise Pretraining Followed by Joint Fine-Tuning: Initializing stage weights from stagewise training and subsequent joint fine-tuning achieved accuracy comparable to joint training from scratch, but required substantially more overall training iterations.
    4. Joint Training without Intermediate Supervision: Training all stages end-to-end with a loss applied solely at the final output suffered from vanishing gradients and resulted in inferior performance across all distance thresholds.
  7. Knowl 7 — Accuracy Progression Across CPM Sequential Stages

    empirical result

    Evaluating keypoint localization accuracy stage-by-stage (t{1,,6}t \in \{1, \dots, 6\}) on the Leeds Sports Pose (LSP) benchmark shows the following dynamics:

    • Part detection accuracy improves monotonically from stage 1 through stage 5 as subsequent stages utilize the large receptive field over prior belief maps to eliminate background false positives and resolve left/right part ambiguities.
    • The largest single accuracy gain occurs between stage 1 (which relies solely on local appearance cues) and stage 2 (which incorporates spatial context from stage 1 belief maps).
    • Marginal performance gains exhibit diminishing returns beyond stage 5, with stage 6 providing minimal additional improvement, making a 6-stage architecture the standard operating configuration.
  8. Knowl 8 — Articulated Pose Estimation Performance on the MPII Dataset

    empirical result

    On the MPII Human Pose benchmark evaluated using the PCKh metric at a normalized distance threshold of 0.5 ([email protected]):

    • A 6-stage Convolutional Pose Machine (CPM) achieves a total [email protected] score of 87.95% when trained on MPII data alone, and 88.52% when augmented with LSP training data, outperforming prior state-of-the-art methods by 6.11 percentage points.
    • On the most challenging keypoint landmark, the ankle (which is farthest from the head along the kinematic chain), the CPM achieves a [email protected] score of 78.28% (79.41% with LSP training data), surpassing the closest competing method by 10.76 percentage points.
    • The model achieves consistently higher accuracy across all 15 viewpoint clusters in the benchmark, including non-frontal view angles, without requiring graphical-model post-processing, explicit kinematic tree priors, or separate precision refinement modules.
  9. Knowl 9 — Articulated Pose Estimation Performance on LSP and FLIC Datasets

    empirical result

    A multi-stage Convolutional Pose Machine (CPM) achieves state-of-the-art results on the Leeds Sports Pose (LSP) and Frames Labeled In Cinema (FLIC) datasets:

    1. LSP Dataset (PCK Metric): A 6-stage CPM achieves a total PCK score of 84.32% under person-centric annotations, increasing to 90.5% when augmented with MPII training data. Compared to the non-convolutional Pose Machine baseline, the CPM improves keypoint detection by 42.4 percentage points at [email protected] (high-precision regime) and 30.9 percentage points at [email protected] (low-precision regime).
    2. FLIC Dataset (PCK Metric): A 4-stage CPM achieves [email protected] accuracies of 97.59% on elbows and 95.03% on wrists. In the high-precision regime, the CPM achieves substantial improvements over prior methods: a 14.8 percentage point increase on wrists and 12.7 percentage points on elbows at [email protected], as well as an 8.9 percentage point increase on wrists and 9.3 percentage points on elbows at [email protected].
  10. Knowl 10 — Limitation in Multi-Person Close-Proximity Scenarios

    limitation

    The single-person Convolutional Pose Machine formulation experiences degraded accuracy and failure cases when multiple people are located in close physical proximity. In such scenarios, overlapping body parts and visual evidence from nearby individuals can cause spatial confusion or false part detections that the single-person context conditioning cannot fully disentangle within a single feed-forward pass.

Coverage note — Specific deep learning framework implementation details and routine dataset-specific bounding box cropping heuristics were omitted as standard engineering procedures.

References

  1. 1.M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2D human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.
  2. 2.M. Andriluka, S. Roth, and B. Schiele. Pictorial structures revisited: People detection and articulated pose estimation. In CVPR, 2009.
  3. 3.M. Andriluka, S. Roth, and B. Schiele. Monocular 3D pose estimation and tracking by detection. In CVPR, 2010.
  4. 4.Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 1994.
  5. 5.D. Bradley. Learning In Modular Systems. PhD thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, 2010.
  6. 6.J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback. arXiv preprint arXiv:1507.06550, 2015.
  7. 7.X. Chen and A. Yuille. Articulated pose estimation by a graphical model with image dependent pairwise relations. In NIPS, 2014.
  8. 8.M. Dantone, J. Gall, C. Leistner, and L. Van Gool. Human pose estimation using body parts dependent joint regressors. In CVPR, 2013.
  9. 9.P. Felzenszwalb and D. Huttenlocher. Pictorial structures for object recognition. In IJCV, 2005.
  10. 10.X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
  11. 11.K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
  12. 12.S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. A Field Guide to Dynamical Recurrent Neural Networks, IEEE Press, 2001.
  13. 13.Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
  14. 14.S. Johnson and M. Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010.
  15. 15.S. Johnson and M. Everingham. Learning effective human pose estimation from inaccurate annotation. In CVPR, 2011.
  16. 16.L. Karlinsky and S. Ullman. Using linking features in learning non-parametric part models. In ECCV, 2012.
  17. 17.M. Kiefel and P. V. Gehler. Human pose estimation with fields of parts. In ECCV. 2014.
  18. 18.A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  19. 19.X. Lan and D. Huttenlocher. Beyond trees: Common-factor models for 2D human pose recovery. In ICCV, 2005.
  20. 20.C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In AISTATS, 2015.
  21. 21.J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
  22. 22.D. Munoz, J. Bagnell, and M. Hebert. Stacked hierarchical labeling. In ECCV, 2010.
  23. 23.W. Ouyang, X. Chu, and X. Wang. Multi-source deep learning for human pose estimation. In CVPR, 2014.
  24. 24.T. Pfister, J. Charles, and A. Zisserman. Flowing convnets for human pose estimation in videos. In ICCV, 2015.
  25. 25.P. Pinheiro and R. Collobert. Recurrent convolutional neural networks for scene labeling. In ICML, 2014.
  26. 26.L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Poselet conditioned pictorial structures. In CVPR, 2013.
  27. 27.L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Strong appearance and expressive spatial models for human pose estimation. In ICCV, 2013.
  28. 28.L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. Gehler, and B. Schiele. Deepcut: Joint subset partition and labeling for multi person pose estimation. arXiv preprint arXiv:1511.06645, 2015.
  29. 29.V. Ramakrishna, D. Munoz, M. Hebert, J. Bagnell, and Y. Sheikh. Pose Machines: Articulated Pose Estimation via Inference Machines. In ECCV, 2014.
  30. 30.D. Ramanan, D. A. Forsyth, and A. Zisserman. Strike a Pose: Tracking people by finding stylized poses. In CVPR, 2005.
  31. 31.S. Ross, D. Munoz, M. Hebert, and J. Bagnell. Learning message-passing inference machines for structured prediction. In CVPR, 2011.
  32. 32.B. Sapp and B. Taskar. MODEC: Multimodal Decomposable Models for Human Pose Estimation. In CVPR, 2013.
  33. 33.L. Sigal and M. Black. Measure locally, reason globally: Occlusion-sensitive articulated pose estimation. In CVPR, 2006.
  34. 34.R. Stewart and M. Andriluka. End-to-end people detection in crowded scenes. arXiv preprint arXiv:1506.04878, 2015.
  35. 35.M. Sun and S. Savarese. Articulated part-based model for joint object detection and pose estimation. In ICCV, 2011.
  36. 36.C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014.
  37. 37.Y. Tian, C. L. Zitnick, and S. G. Narasimhan. Exploring the spatial hierarchy of mixture models for human pose estimation. In ECCV. 2012.
  38. 38.J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. Efficient object localization using convolutional networks. In CVPR, 2015.
  39. 39.J. Tompson, A. Jain, Y. LeCun, and C. Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In NIPS, 2014.
  40. 40.A. Toshev and C. Szegedy. DeepPose: Human pose estimation via deep neural networks. In CVPR, 2013.
  41. 41.Z. Tu and X. Bai. Auto-context and its application to high-level vision tasks and 3d brain image segmentation. In TPAMI, 2010.
  42. 42.Y. Wang and G. Mori. Multiple tree models for occlusion and spatial constraints in human pose estimation. In ECCV, 2008.
  43. 43.Y. Yang and D. Ramanan. Articulated pose estimation with flexible mixtures-of-parts. In CVPR, 2011.
  44. 44.Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. In TPAMI, 2013.

Citation

MLA
Wei, S.-E., et al. “Convolutional Pose Machines”. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4724–32, https://doi.org/10.1109/CVPR.2016.511.
APA
Wei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016). Convolutional Pose Machines. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4724–4732. https://doi.org/10.1109/CVPR.2016.511
Chicago
Wei, S.-E., V. Ramakrishna, T. Kanade, and Y. Sheikh. 2016. “Convolutional Pose Machines”. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4724–32. https://doi.org/10.1109/CVPR.2016.511.
Harvard
Wei, S.-E. et al. (2016) “Convolutional Pose Machines”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 4724–4732. Available at: https://doi.org/10.1109/CVPR.2016.511.
Vancouver
1. Wei S-E, Ramakrishna V, Kanade T, Sheikh Y (2016) Convolutional Pose Machines. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 4724–4732

BibTeX

@inproceedings{Wei_2016, title={Convolutional Pose Machines}, url={http://dx.doi.org/10.1109/CVPR.2016.511}, DOI={10.1109/cvpr.2016.511}, booktitle={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, publisher={IEEE}, author={Wei, Shih-En and Ramakrishna, Varun and Kanade, Takeo and Sheikh, Yaser}, year={2016}, month=June, pages={4724–4732} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE