keyword

efficient temporal modeling

Efficient temporal modeling is an approach in computer vision and machine learning that captures time-dependent dynamics, motion patterns, and sequential relationships across video frames while minimizing computational complexity, parameter overhead, memory usage, and latency. Unlike traditional methods such as dense three-dimensional convolutions that require significant hardware resources, efficient temporal modeling employs lightweight techniques—such as channel shifting, separable convolutions, and sparse or factorized attention mechanisms—to enable temporal information exchange within standard, computationally inexpensive architectures. This allows neural networks to achieve high accuracy in video understanding and recognition tasks while remaining fast and lightweight enough for real-time deployment on resource-constrained edge devices.

1 item

TSM: Temporal Shift Module for Efficient Video Understanding

Ji Lin, Chuang Gan, Song Han

OrganizationsMassachusetts Institute of TechnologyMIT-IBM Watson AI Lab

Why you should read this

Introduces the Temporal Shift Module, an approach that shifts feature channels across neighboring frames to equip standard 2D CNNs with 3D-level temporal modeling at zero extra parameters and computational cost, making real-time video recognition feasible on edge devices.

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: this https URL.

Source

https://arxiv.org/pdf/1811.08383

Added

2026-09-16