Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

low-latency online video recognition

Low-latency online video recognition is the computer vision process of identifying actions, behaviors, or visual events within continuous, live video streams with minimal processing delay. Unlike offline video recognition methods that process entire pre-recorded clips using access to future frames, online recognition operates under causal constraints, analyzing incoming frames sequentially as they are captured. Achieving low latency requires computationally efficient temporal modeling and inference architectures that can process frames in real time, enabling immediate decision-making in interactive and time-critical applications such as autonomous navigation, robotics, smart surveillance, and real-time human-computer interaction.

1 item

TSM: Temporal Shift Module for Efficient Video Understanding

TSM: Temporal Shift Module for Efficient Video Understanding

Ji Lin, Chuang Gan, Song Han

OrganizationsMassachusetts Institute of TechnologyMIT-IBM Watson AI Lab

Why you should read this

Introduces the Temporal Shift Module, an approach that shifts feature channels across neighboring frames to equip standard 2D CNNs with 3D-level temporal modeling at zero extra parameters and computational cost, making real-time video recognition feasible on edge devices.

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: this https URL.

Added

2026-09-16