The Recognition of Human Movement Using Temporal Templates
A. BobickJames W. Davis
Introduces temporal templates—combining Motion-Energy and Motion-History Images—to capture the spatial layout and recency of movement for efficient, real-time human action recognition without requiring 3D body reconstruction.
The paper introduces a view-based method for representing and recognizing human movements directly from image sequences, without first reconstructing three-dimensional body models. This approach addresses the growing need in computer vision to label actions in scenes for applications such as wireless interfaces and interactive environments, where traditional geometry-focused techniques fall short for consistent, view-dependent motions.
The work set out to demonstrate that a compact, two-component temporal template—capturing both the spatial extent of motion and its recency—could discriminate among a range of whole-body movements when matched against stored view-specific examples. The method constructs a binary motion-energy image to show where motion occurred and a motion-history image whose pixel intensity encodes how recently motion took place at each location. These templates are summarized by translation- and scale-invariant Hu moments, then compared via a Mahalanobis distance; a backward-looking search over integration windows enables automatic temporal segmentation and linear speed invariance.
Experiments recorded seven views of 18 aerobics exercises performed by an instructor, with a second subject providing test sequences captured at roughly 30° and 60° offsets. A single-camera match correctly identified 12 of the 18 movements, and the distance gap between the nearest incorrect match and the true class was typically small relative to the median distance across the set. Adding a second orthogonal camera reduced errors to three. A separate real-time implementation, trained on four people performing sitting, waving, and crouching from 45° view increments, ran at 9 Hz on standard hardware and produced few misclassifications when background subtraction replaced simple differencing.
These results indicate that coarse global shape statistics from the paired templates supply sufficient discrimination for many repetitive or articulated actions, while remaining computationally light enough for live deployment. The approach therefore supports low-cost recognition pipelines in constrained settings such as interactive rooms or monitoring stations, where full three-dimensional recovery is unnecessary and often impractical.
Further work should focus on isolating relevant motion when multiple people or camera movement are present, perhaps by combining overhead tracking with body-centered compensation. Extending the representation to automatically mask variable body parts (for example, legs during throwing) would broaden applicability. The current evidence is strongest for isolated, well-lit actions performed against static backgrounds; performance in cluttered or multi-person scenes remains untested at scale and would benefit from additional subjects and more robust motion detection before operational use.
- Paper: Statistical Pattern Recognition: A Review, Anil K. Jain et al. (2000). Reviewing foundational statistical pattern recognition principles provides essential context for the classification and decision-theoretic tools used in temporal template matching.
- Paper: Lucas-Kanade 20 Years On: A Unifying Framework, Simon Baker et al. (2004). Understanding advanced image alignment and tracking frameworks clarifies the motion estimation and temporal registration steps required for template-based video analysis.
- Paper: On Space-Time Interest Points, I. Laptev (2005). This paper extends local spatial interest points into three-dimensional space-time events, providing a more sparse and invariant alternative to holistic temporal templates.
- Paper: UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild, Khurram Soomro et al. (2012). The introduction of the UCF101 benchmark directly continues this line of action recognition research by supplying a massive unconstrained video dataset for evaluating subsequent movement models.
- Paper: Temporal Segment Networks: Towards Good Practices for Deep Action Recognition, Limin Wang et al. (2016). Temporal segment networks build directly upon the foundational goal of modeling long-range video dynamics introduced by early template and snippet-based approaches.
- Paper: MoCoGAN: Decomposing Motion and Content for Video Generation, Sergey Tulyakov et al. (2018). This work advances the decomposition of motion and content from template matching into generative adversarial frameworks capable of synthesizing novel human action sequences.
- Paper: Two-Stream Convolutional Networks for Action Recognition in Videos, Karen Simonyan et al. (2014). Two-stream convolutional networks generalize explicit motion representations like temporal templates into deep learning architectures that combine appearance and optical flow.
- Paper: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, João Carreira et al. (2017). The inflated 3D ConvNet architecture supersedes static template matching by learning spatio-temporal representations end-to-end from large-scale video collections.
- Paper: Learning Spatiotemporal Features with 3D Convolutional Networks, Du Tran et al. (2015). C3D models extend action recognition from handcrafted or template-based representations to scalable 3D convolutional networks trained directly on massive video datasets.
