Back

Convolutional Two-Stream Network Fusion for Video Action Recognition

Christoph FeichtenhoferAxel PinzAndrew Zisserman

article2016CVPR2,786 citations

Proposes a two-stream ConvNet fusion architecture that combines appearance and motion features at convolutional layers rather than the softmax classifier, substantially reducing parameter counts while achieving state-of-the-art accuracy in video action recognition.

Listen

Video action recognition has lagged behind other computer vision tasks because current methods struggle to combine appearance cues from single frames with motion information over time, and available training datasets are either small or noisy. The article addresses this by developing improved ways to fuse two separate convolutional networks—one for spatial appearance and one for temporal motion—so that the system can register what is moving where and track how actions evolve.

The work set out to test where and how to merge the two networks, both across space at different layers of abstraction and across time, while keeping the total number of model parameters low. Researchers ran systematic experiments on standard benchmarks, comparing fusion operations such as summation, concatenation, learned convolution, and bilinear pooling at various depths, then extended the best options with three-dimensional pooling and convolution to integrate information over longer video clips.

The clearest results are that convolutional fusion performed at the final convolutional layer matches or exceeds the accuracy of simply averaging the networks’ final predictions, yet cuts the parameter count roughly in half; adding a second fusion at the class-prediction layer yields a small further gain; and replacing two-dimensional pooling with three-dimensional spatiotemporal pooling and convolution improves accuracy on both datasets. The resulting architecture reaches 92.5 percent mean accuracy on UCF101 and 65.4 percent on HMDB51, several points above prior two-stream models, and combining its output with hand-crafted improved dense trajectories pushes performance to 93.5 percent and 69.2 percent.

These gains matter because they show that explicit spatial and temporal registration of abstract features can be learned efficiently without exploding model size, making real-time or large-scale deployment more practical. At the same time, the continued benefit from combining the learned representation with traditional features indicates that end-to-end networks have not yet fully captured all useful signals when training data remain limited.

The authors recommend exploring larger and cleaner video datasets to reduce reliance on complementary hand-crafted descriptors, and they caution that findings may shift once substantially more training data become available. Overall the evidence supports adopting the proposed fusion layers as a drop-in improvement to existing two-stream pipelines, provided practitioners verify performance on their specific domain and data volume.

Graz University of TechnologyUniversity of Oxfordconvolutional neural networkshuman-action recognitionmotion informationvideo action recognitionconvolutional two-stream networkspatiotemporal fusion
arXiv: 1604.06573feichtenhofer/twostreamfusion
Cover for Convolutional Two-Stream Network Fusion for Video Action Recognition

Abstract

Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters; (ii) that it is better to fuse such networks spatially at the last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy; finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves state-of-the-art results.

Topics

convolutional neural networkshuman-action recognitionmotion informationvideo streamshuman motion recognitionnetwork architecturedata streamarchitecture designs

Keywords

video action recognitionconvolutional two-stream networkspatiotemporal fusiontemporal fusion layeroptical flowtemporal poolingtemporal receptive fieldspatiotemporal architecture

Table of Contents

  • 1 Introduction
  • 2 Related work
  • 3 Approach
  • 3.1 Spatial fusion
  • 3.2 Where to fuse the networks
  • 3.3 Temporal fusion
  • 3.4 Proposed architecture
  • 3.5 Implementation details
  • 4 Evaluation
  • 4.1 Datasets and experimental protocols
  • 4.2 How to fuse the two streams spatially?
  • 4.3 Where to fuse the two streams spatially?
  • 4.4 Going from deep to very deep models
  • 4.5 How to fuse the two streams temporally?
  • 4.6 Comparison with the state-of-the-art
  • 5 Conclusion
  • References

Knowls

  1. Knowl 1 — Spatiotemporal Two-Stream ConvNet Architecture

    model/method

    The spatiotemporal two-stream architecture fuses appearance (spatial stream, operating on single RGB frames) and motion (temporal stream, operating on stacks of L=10L = 10 optical flow frames) representations across both spatial coordinates and time.

    Rather than fusing streams solely at the final softmax layer, the network performs spatial and temporal fusion at the last convolutional layer after ReLU activation (e.g., ReLU5). The spatial and temporal feature maps from TT time steps (sampled with a temporal stride τ\tau at timestamps t,t+τ,…,t+(T−1)τt, t+\tau, \dots, t+(T-1)\tau) are concatenated along channel dimensions and passed through a 3D convolutional fusion layer. The resulting representations are subsequently downsampled using 3D max pooling across space and time (W′×H′×T′W' \times H' \times T'), converting the spatial tower into a hybrid spatiotemporal stream.

    Concurrently, the temporal stream is preserved: its convolutional feature maps across the TT time steps are also aggregated using 3D max pooling. During training, both the spatiotemporal stream and the temporal stream are optimized with separate cross-entropy classification losses. During inference, action classification probabilities are obtained by averaging the predictions from both streams.

  2. Knowl 2 — Convolutional Fusion of Spatial and Temporal Feature Maps

    equation

    Let xa∈RH×W×D\mathbf{x}^a \in \mathbb{R}^{H \times W \times D} and xb∈RH×W×D\mathbf{x}^b \in \mathbb{R}^{H \times W \times D} denote the 3D feature maps produced at a given layer (such as the final rectified convolutional layer ReLU5) by the spatial stream and temporal stream, respectively, where HH is the feature height, WW is the feature width, and DD is the channel depth.

    Concatenation fusion stacks the feature maps across channels at each spatial coordinate (i,j)(i,j) for 1≤i≤H1 \le i \le H, 1≤j≤W1 \le j \le W:

    yi,j,2dcat=xi,j,da,yi,j,2d−1cat=xi,j,db(1≤d≤D)\mathbf{y}^{\text{cat}}_{i,j,2d} = \mathbf{x}^a_{i,j,d}, \quad \mathbf{y}^{\text{cat}}_{i,j,2d-1} = \mathbf{x}^b_{i,j,d} \quad (1 \le d \le D)

    producing ycat∈RH×W×2D\mathbf{y}^{\text{cat}} \in \mathbb{R}^{H \times W \times 2D}.

    Convolutional fusion applies a trainable 1×11 \times 1 convolution kernel f∈R1×1×2D×D\mathbf{f} \in \mathbb{R}^{1 \times 1 \times 2D \times D} and bias b∈RD\mathbf{b} \in \mathbb{R}^D over the concatenated channels:

    yconv=ycat∗f+b\mathbf{y}^{\text{conv}} = \mathbf{y}^{\text{cat}} * \mathbf{f} + \mathbf{b}

    where yconv∈RH×W×D\mathbf{y}^{\text{conv}} \in \mathbb{R}^{H \times W \times D}. This operation learns linear combinations and correspondences between spatial and temporal channels at identical pixel locations (i,j)(i,j) while reducing the channel dimension from 2D2D back to DD.

  3. Knowl 3 — Spatiotemporal Fusion via 3D Convolution and 3D Pooling

    equation

    To capture temporal dynamics across video snippets, feature representations are combined over TT temporal steps separated by stride τ\tau. Stacking the 3D feature maps xt∈RH×W×D\mathbf{x}_t \in \mathbb{R}^{H \times W \times D} across time t∈{1,…,T}t \in \{1, \dots, T\} forms a 4D tensor X∈RH×W×T×D\mathbf{X} \in \mathbb{R}^{H \times W \times T \times D}.

    Spatiotemporal convolutional fusion applies a bank of D′D' 3D filters f∈RW′′×H′′×T′′×D×D′\mathbf{f} \in \mathbb{R}^{W'' \times H'' \times T'' \times D \times D'} and biases b∈RD′\mathbf{b} \in \mathbb{R}^{D'}:

    Y=X∗f+b\mathbf{Y} = \mathbf{X} * \mathbf{f} + \mathbf{b}

    where kernel dimensions are spatial width W′′W'', height H′′H'', temporal depth T′′T'', input channels DD, and output channels D′D' (e.g., 3×3×3×1024×5123 \times 3 \times 3 \times 1024 \times 512 when fusing concatenated spatial and temporal streams of depth D=512D=512 each into D′=512D'=512).

    3D max pooling is subsequently applied over local spatiotemporal volumes of size W′×H′×T′W' \times H' \times T' (e.g., 3×3×33 \times 3 \times 3):

    Yi,j,k,dpool=max⁡0≤δi<H′0≤δj<W′0≤δk<T′Yi⋅sH+δi, j⋅sW+δj, k⋅sT+δk, d\mathbf{Y}^{\text{pool}}_{i,j,k,d} = \max_{\substack{0 \le \delta_i < H' \\ 0 \le \delta_j < W' \\ 0 \le \delta_k < T'}} \mathbf{Y}_{i \cdot s_H + \delta_i, \, j \cdot s_W + \delta_j, \, k \cdot s_T + \delta_k, \, d}

    where sH,sW,sTs_H, s_W, s_T are the pooling strides along the vertical, horizontal, and temporal axes. This operation provides local translational invariance over feature position shifts and temporal motion variations.

  4. Knowl 4 — Filter Initialization for 2D and 3D Convolutional Fusion

    model/method

    Appropriate parameter initialization of the injected convolutional fusion layers is required to facilitate stable finetuning of pre-trained two-stream networks:

    1. 2D Convolutional Fusion (1×1×2D×D1 \times 1 \times 2D \times D): The filter weights f\mathbf{f} are initialized by concatenating two identity matrices I∈RD×D\mathbf{I} \in \mathbb{R}^{D \times D}. This sets the initial operation of the fusion layer to element-wise summation between corresponding spatial and temporal channels (y=xa+xb\mathbf{y} = \mathbf{x}^a + \mathbf{x}^b).

    2. 3D Spatiotemporal Convolutional Fusion (3×3×3×2D×D′3 \times 3 \times 3 \times 2D \times D'):

    • Channel mapping: Stacking two identity matrices maps the 2D=10242D = 1024 input channels (from concatenated spatial and temporal ReLU5 feature maps) to D′=512D' = 512 output channels.
    • Stream magnitude balancing: Because feature activations at the final convolutional layer of the optical flow stream are approximately 3 times smaller in magnitude than those of the spatial stream, the temporal identity submatrix in f\mathbf{f} is scaled by a factor of 3.
    • Spatiotemporal filter weights: The 3×3×33 \times 3 \times 3 spatiotemporal filter component is initialized using a 3D Gaussian distribution with standard deviation σ=1\sigma = 1.

  5. Knowl 5 — Mathematical Formulations of Spatial Fusion Operators

    definition

    Given two spatially registered feature maps xa∈RH×W×D\mathbf{x}^a \in \mathbb{R}^{H \times W \times D} and xb∈RH×W×D\mathbf{x}^b \in \mathbb{R}^{H \times W \times D} at spatial coordinates (i,j)(i,j) (1≤i≤H,1≤j≤W1 \le i \le H, 1 \le j \le W) and channel index dd (1≤d≤D1 \le d \le D), elementary fusion functions f(xa,xb)f(\mathbf{x}^a, \mathbf{x}^b) define correspondences between streams:

    • Sum Fusion (ysum∈RH×W×D\mathbf{y}^{\text{sum}} \in \mathbb{R}^{H \times W \times D}):
    yi,j,dsum=xi,j,da+xi,j,db\mathbf{y}^{\text{sum}}_{i,j,d} = \mathbf{x}^a_{i,j,d} + \mathbf{x}^b_{i,j,d}
    • Max Fusion (ymax∈RH×W×D\mathbf{y}^{\text{max}} \in \mathbb{R}^{H \times W \times D}):
    yi,j,dmax=max⁡(xi,j,da,xi,j,db)\mathbf{y}^{\text{max}}_{i,j,d} = \max\left(\mathbf{x}^a_{i,j,d}, \mathbf{x}^b_{i,j,d}\right)
    • Concatenation Fusion (ycat∈RH×W×2D\mathbf{y}^{\text{cat}} \in \mathbb{R}^{H \times W \times 2D}):
    yi,j,2dcat=xi,j,da,yi,j,2d−1cat=xi,j,db\mathbf{y}^{\text{cat}}_{i,j,2d} = \mathbf{x}^a_{i,j,d}, \quad \mathbf{y}^{\text{cat}}_{i,j,2d-1} = \mathbf{x}^b_{i,j,d}
    • Bilinear Fusion (ybil∈RD2\mathbf{y}^{\text{bil}} \in \mathbb{R}^{D^2}):
    ybil=∑i=1H∑j=1W(xi,ja)⊤xi,jb\mathbf{y}^{\text{bil}} = \sum_{i=1}^H \sum_{j=1}^W \left(\mathbf{x}^a_{i,j}\right)^\top \mathbf{x}^b_{i,j}

    where xi,ja,xi,jb∈R1×D\mathbf{x}^a_{i,j}, \mathbf{x}^b_{i,j} \in \mathbb{R}^{1 \times D} are feature vectors at location (i,j)(i,j). Bilinear fusion computes multiplicative interactions across all channel pairs and sum-pools over all spatial positions, eliminating spatial dimensions.

  6. Knowl 6 — Training and Data Augmentation Protocol for Spatiotemporal Two-Stream Fusion

    experimental setup

    Training 3D convolutional fusion networks on video action datasets utilizes the following temporal sampling and data augmentation protocols:

    • Temporal Sampling: At each training iteration, T=5T = 5 frames are sampled from each video in a mini-batch of 96 videos. The starting frame is randomly sampled, and the temporal stride between consecutive sampled frames is randomly drawn as τ∈[1,10]\tau \in [1, 10] frames (spanning between 15 and 50 raw video frames given an optical flow stack of L=10L = 10).
    • Spatial Data Augmentation: Instead of fixed cropping, the bounding box width and height are randomly jittered by up to ±25%\pm 25\% (allowing aspect-ratio changes) and cropped within a maximum distance of 25% from image boundaries before being rescaled to 224×224224 \times 224 pixels. The spatial crop parameters and horizontal flipping chosen on the first frame are applied identically across all frames in the multi-frame stack.
    • Optimization: Networks are finetuned with an initial learning rate of 10−310^{-3} (5×10−45 \times 10^{-4} for VGG-16), reduced by a factor of 10 upon validation error saturation. Gradients are backpropagated only up to the injected fusion layer, leaving earlier convolutional layers frozen. Prediction-layer fusion is omitted during training to avoid biasing the loss toward the faster-adapting temporal tower.

  7. Knowl 7 — Empirical Comparison of Spatial Fusion Operators and Fusion Layers

    empirical result

    Evaluating different fusion operations and injection layers on UCF101 (split 1) using VGG-M-2048 network backbones demonstrates the performance and parameter efficiency of convolutional fusion at late convolutional layers:

    Fusion Method Fusion Layer Accuracy Parameters
    Sum (baseline late fusion) Softmax 85.94% 181.42M
    Max ReLU5 82.70% 97.31M
    Concatenation ReLU5 83.53% 172.81M
    Bilinear ReLU5 85.05% 6.61M+SVM
    Sum ReLU5 85.20% 97.31M
    Conv (1×11 \times 1) ReLU5 85.96% 97.58M
    Conv (1×11 \times 1) ReLU2 82.25% 91.90M
    Conv (1×11 \times 1) ReLU3 83.43% 93.08M
    Conv (1×11 \times 1) ReLU4 82.55% 95.48M
    Conv (1×11 \times 1) ReLU5 85.96% 97.57M
    Conv (1×11 \times 1) ReLU5 + FC8 86.04% 181.68M
    Conv (1×11 \times 1) ReLU3 + ReLU5 + FC6 81.55% 190.06M

    Key observations:

    1. Fusing with 1×11 \times 1 Conv at ReLU5 (85.96%) matches softmax score averaging (85.94%) while reducing parameters from 181.42M to 97.58M by removing one tower of fully connected layers.
    2. Fusing at early convolutional layers (ReLU2-ReLU4) degrades accuracy (82.25%–83.43%) compared to ReLU5.
    3. Combining ReLU5 fusion with prediction-layer fusion (ReLU5 + FC8) reaches 86.04% but requires keeping both fully connected towers (181.68M parameters), whereas multi-layer fusion incorporating early layers (ReLU3 + ReLU5 + FC6) drops accuracy to 81.55%.

  8. Knowl 8 — Empirical Comparison of Temporal Fusion Strategies and Network Depth

    empirical result

    Evaluating temporal aggregation methods on UCF101 (split 1) and HMDB51 (split 1) using a VGG-16 spatial net and a VGG-M temporal net shows steady gains from 3D operations:

    Fusion Method Temporal Pooling Fusion Layer UCF101 (split 1) HMDB51 (split 1)
    2D Conv 2D ReLU5 + 89.35% 56.93%
    2D Conv 3D ReLU5 + 89.64% 57.58%
    3D Conv 3D ReLU5 + 90.40% 58.63%

    Here + indicates keeping both network towers and averaging their predictions at test time. Replacing 2D pooling with 3D pooling increases accuracy (+0.29% on UCF101, +0.65% on HMDB51), and adding 3D convolutional filtering before 3D pooling yields further improvements (+0.76% on UCF101, +1.05% on HMDB51).

    Additionally, upgrading the network architecture from VGG-M-2048 to the 16-layer VGG-16 under softmax late fusion yields:

    • Spatial stream: UCF101 increases from 74.22% to 82.61% (+8.39%); HMDB51 increases from 36.77% to 47.06% (+10.29%).
    • Temporal stream: UCF101 increases from 82.34% to 86.25% (+3.91%); HMDB51 increases from 51.50% to 55.23% (+3.73%).
    • Late fusion of both streams: UCF101 improves from 85.94% to 90.62%; HMDB51 improves from 54.90% to 58.17%.

  9. Knowl 9 — Benchmark Action Recognition Performance on UCF101 and HMDB51

    empirical result

    Mean classification accuracy across all three splits on UCF101 and HMDB51 benchmarks highlights the state-of-the-art performance of spatiotemporal convolutional fusion and its complementarity with hand-crafted Improved Dense Trajectories (IDT):

    Method UCF101 HMDB51
    Spatiotemporal ConvNet 65.4% –
    LRCN 82.9% –
    Composite LSTM Model 84.3% 44.0%
    C3D 85.2% –
    Two-Stream ConvNet (VGG-M) 88.0% 59.4%
    Factorized ConvNet 88.1% 59.1%
    Two-Stream Conv Pooling 88.2% –
    Two-Stream ConvNet (VGG-16 baseline) 91.7% 58.7%
    Ours (Spatial: VGG-16, Temporal: VGG-M) 90.8% 62.1%
    Ours (Spatial: VGG-16, Temporal: VGG-16, single tower) 91.8% 64.6%
    Ours (Spatial: VGG-16, Temporal: VGG-16, two towers) 92.5% 65.4%
    IDT + higher dimensional Fisher Vector 87.9% 61.1%
    C3D + IDT 90.4% –
    TDD + IDT 91.5% 65.9%
    Ours + IDT (Spatial: VGG-16, Temporal: VGG-M) 92.5% 67.3%
    Ours + IDT (Spatial: VGG-16, Temporal: VGG-16) 93.5% 69.2%

    The two-tower 3D Conv + 3D Pooling model using VGG-16 achieves 92.5% on UCF101 and 65.4% on HMDB51 (and 91.8% / 64.6% using a single tower after fusion). Averaging the network predictions (pre-softmax) with SVM scores of Fisher Vector-encoded IDT descriptors achieves 93.5% on UCF101 and 69.2% on HMDB51.

  10. Knowl 10 — Sensitivity to Dataset Scale and Annotation Noise in Video ConvNets

    limitation

    Training 3D spatiotemporal ConvNets is constrained by existing action recognition datasets:

    1. Temporally consistent benchmarks such as UCF101 (~100 video clips per class) and HMDB51 contain substantially fewer examples per class than image recognition benchmarks such as ImageNet (~1000 examples per class), heightening the risk of overfitting when learning 3D spatio-temporal filter kernels.
    2. Larger datasets (e.g., Sports-1M and THUMOS) have significant label noise or untrimmed backgrounds where only a minor subset of frames contain the labeled action, impeding supervised end-to-end spatiotemporal filter learning.

    Consequently, empirical conclusions regarding spatiotemporal filter parameterizations must be interpreted with caution until larger, cleanly annotated video benchmarks are available.

Coverage note — No substantial contributed material was omitted.

References

  1. 1.T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV, 2004.
  2. 2.J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu. Semantic segmentation with second-order pooling. In Proc. ECCV, 2012.
  3. 3.K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In Proc. BMVC., 2014.
  4. 4.G. Chron, I. Laptev, and C. Schmid. P-CNN: Pose-based CNN features for action recognition. In Proc. ICCV, 2015.
  5. 5.M. Cimpoi, S. Maji, and A. Vedaldi. Deep filter banks for texture recognition and segmentation. In Proc. CVPR, 2015.
  6. 6.J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proc. CVPR, 2015.
  7. 7.G. Gkioxari and J. Malik. Finding action tubes. In Proc. CVPR, 2015.
  8. 8.A. Gorban, H. Indrees, Y. Jiang, A. R. Zamir, I. Laptev, M. Shah, and R. Sukthankar. Thumos challenge: Action recognition with a large number of classes. http://wwwthumos.info/, 2015.
  9. 9.S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML, 2015.
  10. 10.S. Ji, W. Xu, M. Yang, and K. Yu. 3D convolutional neural networks for human action recognition. IEEE PAMI, 35(1):221–231, 2013.
  11. 11.A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classication with convolutional neural networks. In Proc. CVPR, 2014.
  12. 12.A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
  13. 13.H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. In Proc. ICCV, 2011.
  14. 14.I. Laptev, M. Marszałek, C. Schmid, and B. Rozenfeld. Learning realistic human actions from movies. In Proc. CVPR, 2008.
  15. 15.T.-Y. Lin, A. RoyChowdhury, and S. Maji. Bilinear CNN models for fine-grained visual recognition. In Proc. ICCV, 2015.
  16. 16.A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proc. CVPR, 2015.
  17. 17.J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. Beyond short snippets: Deep networks for video classification. In Proc. CVPR, 2015.
  18. 18.J. Oh, X. Guo, H. Lee, S. Singh, and R. Lewis. Action-conditional video prediction using deep networks in atari game. In NIPS, 2015.
  19. 19.X. Peng, L. Wang, X. Wang, and Y. Qiao. Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice. CoRR, abs/1405.4506, 2014.
  20. 20.F. Perronnin, J. Sánchez, and T. Mensink. Improving the Fisher kernel for large-scale image classification. In Proc. ECCV, 2010.
  21. 21.F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proc. CVPR, 2015.
  22. 22.K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.
  23. 23.K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proc. ICLR, 2014.
  24. 24.K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions calsses from videos in the wild. Technical Report CRCV-TR-12-01, UCF Center for Research in Computer Vision, 2012.
  25. 25.N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using LSTMs. In Proc. ICML, 2015.
  26. 26.L. Sun, K. Jia, D.-Y. Yeung, and B. Shi. Human action recognition using factorized spatio-temporal convolutional networks. In Proc. ICCV, 2015.
  27. 27.C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proc. CVPR, 2015.
  28. 28.G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler. Convolutional learning of spatio-temporal features. In Proc. ECCV, 2010.
  29. 29.J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. Efficient object localization using convolutional networks. In Proc. CVPR, 2015.
  30. 30.D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. In Proc. ICCV, 2015.
  31. 31.A. Vedaldi and K. Lenc. MatConvNet – convolutional neural networks for MATLAB. In Proceeding of the ACM Int. Conf. on Multimedia, 2015.
  32. 32.S. Venugopalan, M. Rohrbach, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence video to text. In Proc. ICCV, 2015.
  33. 33.H. Wang and C. Schmid. Action recognition with improved trajectories. In Proc. ICCV, 2013.
  34. 34.L. Wang, Y. Qiao, and X. Tang. Action recognition with trajectory-pooled deep-convolutional descriptors. In Proc. CVPR, 2015.
  35. 35.L. Wang, Y. Xiong, Z. Wang, and Y. Qiao. Towards good practices for very deep two-stream convnets. arXiv preprint arXiv:1507.02159, 2015.
  36. 36.P. Weinzaepfel, Z. Harchaoui, and C. Schmid. Learning to track for spatio-temporal action localization. In Proc. ICCV, 2015.
  37. 37.C. Zach, T. Pock, and H. Bischof. A duality based approach for realtime TV-L1 optical flow. In Proc. DAGM, pages 214–223, 2007.
  38. 38.M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Proc. ECCV, 2014.

Citation

MLA
Feichtenhofer, C., et al. “Convolutional Two-Stream Network Fusion for Video Action Recognition”. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1933–41, https://doi.org/10.1109/CVPR.2016.213.
APA
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2016). Convolutional Two-Stream Network Fusion for Video Action Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1933–1941. https://doi.org/10.1109/CVPR.2016.213
Chicago
Feichtenhofer, C., A. Pinz, and A. Zisserman. 2016. “Convolutional Two-Stream Network Fusion for Video Action Recognition”. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1933–41. https://doi.org/10.1109/CVPR.2016.213.
Harvard
Feichtenhofer, C., Pinz, A. and Zisserman, A. (2016) “Convolutional Two-Stream Network Fusion for Video Action Recognition”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 1933–1941. Available at: https://doi.org/10.1109/CVPR.2016.213.
Vancouver
1. Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional Two-Stream Network Fusion for Video Action Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1933–1941

BibTeX

@inproceedings{Feichtenhofer_2016, title={Convolutional Two-Stream Network Fusion for Video Action Recognition}, url={http://dx.doi.org/10.1109/CVPR.2016.213}, DOI={10.1109/cvpr.2016.213}, booktitle={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, publisher={IEEE}, author={Feichtenhofer, Christoph and Pinz, Axel and Zisserman, Andrew}, year={2016}, month=June, pages={1933–1941} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE