Back

Abnormal Event Detection at 150 FPS in MATLAB

Cewu LuJianping ShiJiaya Jia

article2013ICCV1,351 citations

Proposes a sparse combination learning framework that replaces costly sparse coding with small-scale least-squares projections, enabling abnormal event detection in surveillance video at 150 frames per second in MATLAB without sacrificing accuracy.

Listen

Surveillance systems generate vast amounts of video data, yet manual monitoring is labor-intensive and inefficient because abnormal events occur rarely. While automated detection using sparse representation achieves high accuracy, existing methods are computationally intensive and require seconds to process a single frame. This computational bottleneck causes significant response delays and prevents real-time deployment on standard hardware.

The article evaluates a sparse combination learning framework designed to detect abnormal events in surveillance video at real-time speeds while preserving high detection accuracy.

The proposed method replaces complex per-frame optimization with a set of pre-learned sparse basis combinations. Video frames are decomposed into multi-scale spatial-temporal cubes, from which three-dimensional gradient features are extracted and compressed. During training, the system iteratively learns compact basis combinations bounded by a maximum reconstruction error. During testing, the system assesses incoming video features via simple matrix projections to identify anomalies. The authors validated the method on over 107 hours of surveillance video (spanning 31,200 feature groups) and benchmarked it on three public datasets: Avenue, Subway (Exit and Entrance gates), and UCSD Ped1.

The evaluation yielded several key findings. First, the framework achieved processing speeds of 140 to 150 frames per second on standard desktop hardware using MATLAB, representing a speed improvement of over 400 times compared to prior sparsity-based techniques (which took 2 to 4.6 seconds per frame). Second, surveillance video exhibited high structural redundancy; approximately 10 basis combinations per region were sufficient to represent normal activity, with 99% of regions requiring fewer than 45 combinations. Third, detection accuracy remained highly competitive. On the UCSD Ped1 dataset, the model achieved a 15% frame-level equal error rate (outperforming alternative sparse coding and subspace clustering baselines) and an area under the ROC curve of 91.8%. On the Subway and Avenue datasets, it maintained high detection rates with low false alarm counts (e.g., detecting 19 out of 19 ground-truth events at the Subway Exit Gate with only 2 false alarms).

These results demonstrate that automated surveillance can achieve true real-time processing without requiring specialized supercomputing hardware or sacrificing accuracy. By drastically cutting per-frame computation, security systems can issue immediate alerts and scale across multiple camera feeds on standard infrastructure, lowering hardware and operational costs.

Organizations seeking to implement or upgrade automated surveillance should consider adopting sparse combination structures over traditional dictionary-searching methods. Future engineering efforts should focus on extending this framework to other video analysis domains and implementing parallel processing pipelines to further minimize latency.

Confidence in these findings is supported by extensive validation across multiple diverse benchmark datasets and long-duration video feeds. However, stakeholders should note that the training phase relies on a sufficient volume of normal activity to establish reliable baseline representations. Unusual normal events not captured in the training footage may initially trigger false alarms until the baseline combination sets are updated.

The Chinese University of Hong Kongdetection algorithmdetection ratesframesabnormal event detectionsparse combination learningsparse representation
Lu et al (2013).pdf
Cover for Abnormal Event Detection at 150 FPS in MATLAB

Abstract

Speedy abnormal event detection meets the growing demand to process an enormous number of surveillance videos. Based on inherent redundancy of video structures, we propose an efficient sparse combination learning framework. It achieves decent performance in the detection phase without compromising result quality. The short running time is guaranteed because the new method effectively turns the original complicated problem to one in which only a few costless small-scale least square optimization steps are involved. Our method reaches high detection rates on benchmark datasets at a speed of 140~150 frames per second on average when computing on an ordinary desktop PC using MATLAB.

Topics

detection algorithmdetection ratesframesmatlaboptimizationvideo surveillanceoptimization problemssparse approximations

Keywords

abnormal event detectionsparse combination learningsparse representationsubspace clusteringsurveillance videosAvenue datasetreconstruction errorscrowded scenes

Table of Contents

  • 1.1. Sparsity Based Abnormality Detection
  • 1. Introduction
  • 1.2. Our Contribution
  • 2. Method
  • 2.1. Learning Combinations on Training Data
  • 2.2. Optimization for Training
  • 2.3. Testing
  • 2.4. Relation to Subspace Clustering
  • 3. Experiments
  • 3.1. System Setting
  • 3.2. Verification of Sparse Combinations
  • 3.3. Avenue Data Benchmark
  • 3.4. Subway Dataset
  • 3.5. UCSD Ped1 Dataset
  • 3.6. Separate Cost Analysis
  • 4. Conclusion
  • Acknowledgments
  • References

Knowls

  1. Knowl 1 — Sparse Combination Learning Formulation

    model/method

    Traditional sparsity-based abnormality detection models a test feature vector x∈Rpx \in \mathbb{R}^p by solving an ℓ0\ell_0-constrained reconstruction problem min⁡β∥x−Dβ∥22 s.t. ∥β∥0≤s\min_{\beta} \|x - D\beta\|_2^2 \text{ s.t. } \|\beta\|_0 \le s over a learned dictionary D∈Rp×qD \in \mathbb{R}^{p \times q}. Because searching among (qs)\binom{q}{s} basis combinations during inference is computationally expensive, sparse combination learning pre-learns a small collection of KK fixed basis combinations S={S1,…,SK}\mathcal{S} = \{S_1, \dots, S_K\}, where each Si∈Rp×sS_i \in \mathbb{R}^{p \times s} contains s≪qs \ll q unit-norm basis vectors (s=0.1×ps = 0.1 \times p).

    Given training features X={x1,…,xn}⊂RpX = \{x_1, \dots, x_n\} \subset \mathbb{R}^p at a specific spatial location, the learning objective ensures that every training vector xjx_j can be reconstructed with an error upper-bounded by a fixed tolerance λ>0\lambda > 0:

    ∀j∈{1,…,n},tj=∑i=1Kγji(∥xj−Siβji∥22−λ)≤0,s.t.∑i=1Kγji=1,  γji∈{0,1}\forall j \in \{1, \dots, n\}, \quad t_j = \sum_{i=1}^K \gamma_j^i \left( \|x_j - S_i \beta_j^i\|_2^2 - \lambda \right) \le 0, \quad \text{s.t.} \quad \sum_{i=1}^K \gamma_j^i = 1, \; \gamma_j^i \in \{0, 1\}

    where βji∈Rs\beta_j^i \in \mathbb{R}^s is the least-squares coefficient vector representing xjx_j via combination SiS_i, and binary indicator γji\gamma_j^i selects exactly one combination for sample xjx_j. The bound λ\lambda guarantees reconstruction fidelity without inflating the combination cardinality KK.

  2. Knowl 2 — Greedy Maximum Representation Algorithm for Training Sparse Combinations

    algorithm

    Sparse basis combinations S={S1,…,SK}\mathcal{S} = \{S_1, \dots, S_K\} are learned greedily in an iterative sequence of passes. In pass ii, given the subset of remaining training features Xc⊆X\mathcal{X}_c \subseteq \mathcal{X} not yet represented by {S1,…,Si−1}\{S_1, \dots, S_{i-1}\}, the algorithm optimizes Si∈Rp×sS_i \in \mathbb{R}^{p \times s} to maximize the number of samples in Xc\mathcal{X}_c whose reconstruction error is below tolerance λ\lambda by minimizing:

    min⁡Si,γ,β∑j∈Ωcγji(∥xj−Siβji∥22−λ)s.t.γji∈{0,1}\min_{S_i, \gamma, \beta} \sum_{j \in \Omega_c} \gamma_j^i \left( \|x_j - S_i \beta_j^i\|_2^2 - \lambda \right) \quad \text{s.t.} \quad \gamma_j^i \in \{0, 1\}

    where Ωc\Omega_c indexes Xc\mathcal{X}_c. The optimization alternates between solving for coefficients β\beta and basis matrix SiS_i using block-coordinate descent, and updating indicators γ\gamma:

    1. Fixed γ\gamma: For each sample xjx_j with γji=1\gamma_j^i = 1, the closed-form coefficient is βji=(SiTSi)−1SiTxj\beta_j^i = (S_i^T S_i)^{-1} S_i^T x_j. The basis matrix SiS_i is updated via projected gradient descent: Si←Π[Si−δt∇SiL(β,Si)]S_i \leftarrow \Pi [S_i - \delta_t \nabla_{S_i} L(\beta, S_i)], where L(β,Si)=∑j∈Ωcγji∥xj−Siβji∥22L(\beta, S_i) = \sum_{j \in \Omega_c} \gamma_j^i \|x_j - S_i \beta_j^i\|_2^2, δt=10−4\delta_t = 10^{-4}, and Π\Pi normalizes each column of SiS_i to unit norm.
    2. Fixed {Si,β}\{S_i, \beta\}: The indicator is updated in closed form as γji=1\gamma_j^i = 1 if ∥xj−Siβji∥22<λ\|x_j - S_i \beta_j^i\|_2^2 < \lambda, and 00 otherwise.
    Input: Training feature set X\mathcal{X}, error bound λ\lambda, basis dimension ss
    Xc=X\mathcal{X}_c = \mathcal{X}
    S=∅\mathcal{S} = \emptyset
    i=1i = 1
    repeat
      Initialize SiS_i by running K-means on Xc\mathcal{X}_c with ss cluster centers
      repeat
        Update coefficients βji=(SiTSi)−1SiTxj\beta_j^i = (S_i^T S_i)^{-1} S_i^T x_j for all jj where γji=1\gamma_j^i = 1
        Update Si=Π[Si−δt∇SiL(β,Si)]S_i = \Pi[S_i - \delta_t \nabla_{S_i} L(\beta, S_i)]
        Update γji=1\gamma_j^i = 1 if ∥xj−Siβji∥22<λ\|x_j - S_i \beta_j^i\|_2^2 < \lambda, else 00
      until objective function L(β,Si)L(\beta, S_i) converges
      Add SiS_i to S\mathcal{S}
      Remove all xjx_j with γji=1\gamma_j^i = 1 from Xc\mathcal{X}_c
      i=i+1i = i + 1
    until Xc=∅\mathcal{X}_c = \emptyset
    Output: Combination set S\mathcal{S}

  3. Knowl 3 — Fast Testing via Precomputed Auxiliary Projection Matrices

    algorithm

    Testing a query feature vector x∈Rpx \in \mathbb{R}^p against a learned combination set S={S1,…,SK}\mathcal{S} = \{S_1, \dots, S_K\} (Si∈Rp×sS_i \in \mathbb{R}^{p \times s}) requires finding if any combination reconstructs xx within error threshold TT. The least-squares solution for combination SiS_i is βi∗=(SiTSi)−1SiTx\beta^{*}_i = (S_i^T S_i)^{-1} S_i^T x. Substituting this yields the reconstruction error:

    ∥x−Siβi∗∥22=∥(Si(SiTSi)−1SiT−Ip)x∥22=∥Rix∥22\|x - S_i \beta^{*}_i\|_2^2 = \| (S_i(S_i^T S_i)^{-1} S_i^T - I_p) x \|_2^2 = \| R_i x \|_2^2

    where IpI_p is the p×pp \times p identity matrix, and Ri=Si(SiTSi)−1SiT−Ip∈Rp×pR_i = S_i(S_i^T S_i)^{-1}S_i^T - I_p \in \mathbb{R}^{p \times p} is an auxiliary projection matrix precomputed offline for each combination SiS_i.

    Input: Query feature vector x∈Rpx \in \mathbb{R}^p, auxiliary matrices {R1,…,RK}\{R_1, \dots, R_K\}, threshold TT
    for j=1j = 1 to KK do
      if ∥Rjx∥22<T\|R_j x\|_2^2 < T then
        return normal event
      end if
    end for
    return abnormal event

    Because combinations are ordered by descending frequency of represented training data, testing terminates on average after evaluating only a fraction (average ratio of 0.325) of the combinations, with each combination check requiring 10−610^{-6} to 10−710^{-7} seconds.

  4. Knowl 4 — Multi-Scale Spatio-Temporal Cube Pipeline and Frame Abnormality Scoring

    model/method

    To capture both local details and global scene context, each video frame is resized into three pyramid scales: 20×2020 \times 20, 30×4030 \times 40, and 120×160120 \times 160 pixels. Each scale is partitioned into non-overlapping spatial patches of 10×1010 \times 10 pixels, yielding 2×2=42 \times 2 = 4, 3×4=123 \times 4 = 12, and 12×16=19212 \times 16 = 192 patches respectively (208 spatial sub-regions per frame in total).

    Corresponding patches across 5 consecutive frames are stacked into spatio-temporal cubes of size 10×10×510 \times 10 \times 5. A 3D gradient descriptor of dimension 1500 is computed on each cube, projected to p=100p = 100 dimensions via PCA, and normalized to zero mean and unit variance. Feature vectors are trained and tested independently per spatial location.

    For each video frame, an abnormal response score VV aggregates anomaly counts across scales with exponential weighting:

    V=∑i=1n2n−iviV = \sum_{i=1}^n 2^{n-i} v_i

    where n=3n = 3 is the total number of pyramid scales (index i=1i = 1 corresponding to the coarsest scale and index nn to the finest scale), and viv_i is the number of abnormal cubes detected at scale ii.

  5. Knowl 5 — The CUHK Avenue Benchmark Dataset and Detection Performance

    experimental setup

    The CUHK Avenue dataset is an abnormal event detection benchmark consisting of 15 video sequences (each approximately 2 minutes long, totaling 35,240 frames). The dataset contains 14 ground-truth unusual events comprising running, throwing objects, and loitering. The training split uses 4 video sequences containing 8,478 frames in total, while the remaining sequences are reserved for testing.

    Run Loiter Throw False Alarm
    Ground Truth 4 5 5 N/A
    Sparse Combination Learning 4 4 4 1

    On this benchmark, the sparse combination learning method accurately identifies 12 out of the 14 anomalous events with only 1 false alarm, while achieving an average processing throughput of 141.34 frames per second.

  6. Knowl 6 — Empirical Distribution of Basis Combination Cardinality in Surveillance Video

    empirical result

    To evaluate the structural redundancy of surveillance video, 150 normal surveillance videos totaling 107.8 hours were gathered from subway exits, shopping malls, traffic cameras, elevators, and outdoor squares (combining clips from UCSD Ped1, Subway, YouTube, and newly captured footage). Across all videos and 208 patch regions per frame, a total of 31,200 spatial patch groups (each containing 6,000 to 120,000 cube features) were trained with error bound λ=0.04\lambda = 0.04.

    The number of learned basis combinations KK per spatial group exhibited a mean of μ=9.75\mu = 9.75 and a variance of σ2=10.62\sigma^2 = 10.62. Static background regions required only K=1K = 1 combination, while dynamic regions required dozens of combinations (maximum K=108K = 108). Approximately 99% of all spatial patch groups required K<45K < 45 combinations. The statistical regression reconstruction error across all normal features was 0.0132±1.38×10−40.0132 \pm 1.38 \times 10^{-4}, demonstrating that a very small dictionary of combinations suffices to span normal video patterns.

  7. Knowl 7 — UCSD Ped1 Benchmark Performance

    data/table

    The UCSD Ped1 dataset contains 34 training clips and 36 testing clips (200 frames each), with 10 clips annotated with pixel-level ground truth. Evaluation metrics include Equal Error Rate (EER), Equal Detected Rate (EDR), and Area Under the ROC Curve (AUC) evaluated at pixel and frame levels.

    Metric SF MPPCA SF-MPPCA MDT Sparse Adam Antic Subspace Ours
    Pixel-level EDR 21% 18% 18% 45% 46% 24% 68% 39.3% 59.1%
    Pixel-level AUC 19.7% 20.5% 21.3% 44.1% 13.3% 46.1% 76% 43.2% 63.8%
    Frame-level EER 31% - 40% 25% 19% - 18% 29.6% 15%
    Frame-level AUC 67.5% - 59% 81.8% 86% - 91% 68.4% 91.8%

    Sparse combination learning achieves a frame-level EER of 15% and frame-level AUC of 91.8%, outperforming traditional sparse reconstruction methods and subspace clustering while remaining competitive with complex spatial-temporal models.

  8. Knowl 8 — Anomaly Detection Performance on the Subway Dataset

    data/table

    The Subway dataset consists of 2 hours of surveillance video (209,150 frames, 512×384512 \times 384 resolution) across Exit-Gate and Entrance-Gate cameras, using the first 15 minutes of each sequence for training.

    Exit-Gate Wrong Dir. Loitering Misc. Total False Alarms
    Ground Truth 9 3 7 19 0
    Zhao et al. 9 3 7 19 2
    Kim Grauman 9 3 7 19 3
    Cong et al. 9 - - - 2
    Subspace Clustering 6 3 5 14 4
    Sparse Combination Learning 9 3 7 19 2
    Entrance-Gate Wrong Dir. No Pay Loiter Irreg. Int. Misc. Total False Alarms
    Ground Truth 26 13 14 4 9 66 0
    Zhao et al. 25 9 14 4 8 60 5
    Kim Grauman 24 8 13 4 8 57 6
    Cong et al. 21 6 - - - - 4
    Subspace Clustering 21 6 9 3 7 46 7
    Sparse Combination Learning 25 7 13 4 8 57 4

    On the Exit-Gate video, the model detects all 19 abnormal events with 2 false alarms. On the Entrance-Gate video, it detects 57 out of 66 events with 4 false alarms, matching or exceeding prior state-of-the-art sparse coding approaches.

  9. Knowl 9 — Computational Speed and Per-Frame Execution Breakdown

    data/table

    Inference speed was evaluated on an ordinary desktop PC (Intel 3.4 GHz CPU, 8 GB RAM, MATLAB 2012) and compared against existing abnormal event detection methods.

    Method Subway (s/frame) UCSD Ped1 (s/frame) Hardware / Platform
    Mahadevan et al. - 25.0 3.0 GHz CPU, 2.0 GB RAM
    Antic Ommer - 5.0–10.0 MATLAB
    Cong et al. 4.60 3.80 2.6 GHz CPU, 2.0 GB RAM
    Zhao et al. 2.00 - 2.6 GHz CPU, 2.0 GB RAM, MATLAB 7.0
    Sparse Combination Learning 0.00641 0.00697 3.4 GHz CPU, 8.0 GB RAM, MATLAB 2012
    Dataset Feature Extr. (ms) Comb. Testing (ms) Others (ms) Total (ms) FPS
    Avenue 4.513 1.792 0.770 7.075 141.34
    UCSD Ped1 4.496 1.724 0.743 6.965 143.57
    Subway 4.634 1.409 0.625 6.412 155.97

    By replacing online ℓ0/ℓ1\ell_0 / \ell_1 dictionary optimization with least-squares matrix projections against precomputed auxiliary matrices, the method achieves 140--156 FPS in MATLAB, running over 400 times faster than prior sparse coding pipelines.

Coverage note — None was omitted; all key contributions including the sparse combination learning formulation, training algorithm, projection matrix testing method, spatio-temporal feature pyramid, Avenue dataset creation, and empirical benchmark evaluations on Avenue, UCSD Ped1, and Subway are fully represented.

References

  1. 1.A. Adam, E. Rivlin, I. Shimshoni, and D. Reinitz. Robust real-time unusual event detection using multiple fixed-location monitors. IEEE TPAMI, 30(3):555–560, 2008.
  2. 2.B. Antic and B. Ommer. Video parsing for abnormality detection. In ICCV, pages 2415–2422, 2011.
  3. 3.Y. Benezeth, P.-M. Jodoin, V. Saligrama, and C. Rosemberger. Abnormal events detection based on spatio-temporal cooccurences. In CVPR, 2009.
  4. 4.D. Bertsekas. Nonlinear programming. Athena Scientific Belmont, MA, 1999.
  5. 5.Y. Cong, J. Yuan, and J. Liu. Sparse reconstruction costs for abnormal event detection. In CVPR, pages 3449–3456, 2011.
  6. 6.X. Cui, Q. Liu, M. Gao, and D. Metaxas. Abnormal detection using interaction energy potentials. In CVPR, pages 3161–3167, 2011.
  7. 7.E. Ehsan and R. Vidal. Sparse subspace clustering. In CVPR, 2009.
  8. 8.F. Jianga, J. Yuan, S. A. Tsaftarisa, and A. K. Katsaggelosa. Anomalous video event detection using spatiotemporal context. Computer Vision and Image Understanding, 115(3):323–333, 2011.
  9. 9.K. Jouseok and L. Kyoungmu. A unified framework for event summarization and rare event detection. In CVPR, 2012.
  10. 10.J. Kim and K. Grauman. Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates. In CVPR, pages 2921–2928, 2009.
  11. 11.L. Kratz and K. Nishino. Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In CVPR, pages 1446–1453, 2009.
  12. 12.C. Lu, J. Shi, and J. Jia. Online robust dictionary learning. In CVPR, 2013.
  13. 13.V. Mahadevan, W. Li, V. Bhalodia, and N. Vasconcelos. Anomaly detection in crowded scenes. In CVPR, 2010.
  14. 14.J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online learning for matrix factorization and sparse coding. The Journal of Machine Learning Research, 11:19–60, 2010.
  15. 15.R. Mehran, A. Oyama, and M. Shah. Abnormal crowd behavior detection using social force model. In CVPR, 2009.
  16. 16.V. Saligrama and Z. Chen. Video anomaly detection based on local statistical aggregates. In CVPR, pages 2112–2119, 2012.
  17. 17.J. Shi, X. Ren, G. Dai, J. Wang, and Z. Zhang. A non-convex relaxation approach to sparse dictionary learning. In CVPR, pages 1809–1816, 2011.
  18. 18.H. Trevor, T. Robert, and J. H. Friedman. The elements of statistical learning. Springer New York, 2001.
  19. 19.X. Wang, X. Ma, and E. Grimson. Unsupervised activity perception by hierarchical bayesian models. In CVPR, pages 1–8, 2007.
  20. 20.S. Wu, B. E. Moore, and M. Shah. Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes. In CVPR, 2010.
  21. 21.D. Zhang, D. Gatica-Perez, S. Bengio, and I. McCowan. Semi-supervised adapted hmms for unusual event detection. In CVPR, 2005.
  22. 22.B. Zhao, L. Fei-Fei, and E. Xing. Online detection of unusual events in videos via dynamic sparse coding. In CVPR, 2011.
  23. 23.H. Zhong, J. Shi, and M. Visontai. Detecting unusual activity in video. In CVPR, 2004.

Citation

MLA
Lu, C., et al. “Abnormal Event Detection at 150 FPS in MATLAB”. 2013 IEEE International Conference on Computer Vision, 2013, pp. 2720–27, https://doi.org/10.1109/ICCV.2013.338.
APA
Lu, C., Shi, J., & Jia, J. (2013). Abnormal Event Detection at 150 FPS in MATLAB. 2013 IEEE International Conference on Computer Vision, 2720–2727. https://doi.org/10.1109/ICCV.2013.338
Chicago
Lu, C., J. Shi, and J. Jia. 2013. “Abnormal Event Detection at 150 FPS in MATLAB”. 2013 IEEE International Conference on Computer Vision, 2720–27. https://doi.org/10.1109/ICCV.2013.338.
Harvard
Lu, C., Shi, J. and Jia, J. (2013) “Abnormal Event Detection at 150 FPS in MATLAB”, 2013 IEEE International Conference on Computer Vision. IEEE, pp. 2720–2727. Available at: https://doi.org/10.1109/ICCV.2013.338.
Vancouver
1. Lu C, Shi J, Jia J (2013) Abnormal Event Detection at 150 FPS in MATLAB. In: 2013 IEEE International Conference on Computer Vision. IEEE, pp 2720–2727

BibTeX

@inproceedings{Lu_2013, title={Abnormal Event Detection at 150 FPS in MATLAB}, url={http://dx.doi.org/10.1109/ICCV.2013.338}, DOI={10.1109/iccv.2013.338}, booktitle={2013 IEEE International Conference on Computer Vision}, publisher={IEEE}, author={Lu, Cewu and Shi, Jianping and Jia, Jiaya}, year={2013}, month=Dec, pages={2720–2727} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: IEEE