Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection

Bo ZongQi SongMartin Renqiang MinWei ChengCristian LumezanuDaeki ChoHaifeng Chen

article2018ICLR2,164 citations

Proposes an end-to-end unsupervised anomaly detection framework that jointly optimizes deep autoencoding reconstruction and Gaussian mixture density estimation, eliminating decoupled two-stage training to significantly improve detection accuracy on high-dimensional data.

Listen

Detecting anomalies in high-dimensional data is vital for high-stakes operational settings such as cybersecurity monitoring, medical diagnostics, and complex system management. In real-world environments, manually labeling abnormal events is expensive and often impossible, making unsupervised anomaly detection essential. However, existing techniques struggle with the curse of dimensionality. Standard approaches typically compress data first and estimate sample probability density second, but this disconnected process often discards crucial anomalous signals and leads to suboptimal performance.

The article develops and evaluates the Deep Autoencoding Gaussian Mixture Model (DAGMM), a deep learning framework designed to unify dimensionality reduction and density estimation. The primary goal is to demonstrate that simultaneously optimizing data compression and probability density modeling in an end-to-end framework substantially improves unsupervised anomaly detection accuracy.

To evaluate this framework, the authors conducted empirical experiments across four standard public benchmark datasets spanning cybersecurity and healthcare: KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev. DAGMM couples two sub-networks: a compression network that uses an autoencoder to produce both reduced latent dimensions and reconstruction error metrics, and an estimation network that models sample density using a Gaussian Mixture Model. The authors benchmarked DAGMM against traditional approaches like one-class support vector machines, state-of-the-art deep learning baselines, and several decoupled architectural variants.

The experimental findings show that DAGMM significantly outperforms existing methods across all evaluated benchmarks. On cybersecurity datasets, DAGMM achieved balanced accuracy improvements of up to 14% over competing baselines, reaching an F1 score of roughly 0.937 to 0.938 compared to baseline scores around 0.776 to 0.832. Second, combining the autoencoder's latent features with explicit reconstruction error metrics proved essential for isolating difficult anomalies that closely resemble normal data in reduced dimensional space alone. Third, the joint, end-to-end training process allowed the autoencoder to achieve reconstruction error as low as pre-trained models (approximately 0.26 versus 1.13 for end-to-end autoencoders without the estimation network), effectively escaping suboptimal local traps. Finally, DAGMM demonstrated strong resilience against noisy training environments, maintaining robust detection performance even when the baseline training data contained up to 5% unlabelled anomalies.

These findings indicate that anomaly detection pipelines do not need to choose between reconstruction-based filtering and statistical density modeling. Integrating both into a single end-to-end architecture reduces false alarms, captures sophisticated operational threats, and eliminates the engineering complexity and inflexibility of multi-stage pre-training pipelines. This offers a more reliable, cost-effective framework for critical automated monitoring systems.

Organizations implementing automated anomaly detection should transition away from decoupled, two-stage compression and clustering pipelines toward unified end-to-end frameworks that jointly monitor latent coordinates and reconstruction errors. When deploying these models, practitioners should ensure training data remains as clean as possible, as data contamination beyond 5% progressively degrades detection precision. Before broad operational deployment, technical teams should conduct domain-specific pilot studies to tune the baseline number of mixture components and validate performance on production-specific data streams.

Confidence in these findings is high for structured numerical tabular and network data within the tested anomaly ratios (ranging from 2.5% to 20%). However, practitioners should exercise caution, as the article primarily assesses static benchmark datasets with fixed pre-selected anomaly thresholds. Further operational validation is warranted for high-throughput, non-stationary streaming data or unstructured data types such as text and imagery.

NEC Laboratories America, Inc.Washington State University anomaly detection methods gaussian mixture model gaussian mixtures Deep Autoencoding Gaussian Mixture Model unsupervised anomaly detection unsupervised dimensionality reduction

Zong et al (2018).pdf

Paper: Deep One-Class Classification, Lukas Ruff et al. (2018). Understanding deep one-class classification provides essential foundational knowledge for single-class density-based anomaly detection.
Paper: Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion, Pascal Vincent et al. (2010). Mastering stacked denoising autoencoders is a prerequisite for learning robust latent representations used in joint reconstruction frameworks.

Paper: A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, Kimin Lee et al. (2018). This work extends unsupervised latent-space anomaly detection by introducing unified Mahalanobis distance scores for out-of-distribution inputs.
Paper: Deep Anomaly Detection with Outlier Exposure, Dan Hendrycks et al. (2019). This paper builds directly on anomaly detection foundations by introducing outlier exposure strategies to improve deep density models.

Cover for Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection

Abstract

Unsupervised anomaly detection on multi- or high-dimensional data is of great importance in both fundamental machine learning research and industrial applications, for which density estimation lies at the core. Although previous approaches based on dimensionality reduction followed by density estimation have made fruitful progress, they mainly suffer from decoupled model learning with inconsistent optimization goals and incapability of preserving essential information in the low-dimensional space. In this paper, we present a Deep Autoencoding Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection. Our model utilizes a deep autoencoder to generate a low-dimensional representation and reconstruction error for each input data point, which is further fed into a Gaussian Mixture Model (GMM). Instead of using decoupled two-stage training and the standard Expectation-Maximization (EM) algorithm, DAGMM jointly optimizes the parameters of the deep autoencoder and the mixture model simultaneously in an end-to-end fashion, leveraging a separate estimation network to facilitate the parameter learning of the mixture model. The joint optimization, which well balances autoencoding reconstruction, density estimation of latent representation, and regularization, helps the autoencoder escape from less attractive local optima and further reduce reconstruction errors, avoiding the need of pre-training. Experimental results on several public benchmark datasets show that, DAGMM significantly outperforms state-of-the-art anomaly detection techniques, and achieves up to 14% improvement based on the standard F1 score.

Topics

anomaly detection methods gaussian mixture model gaussian mixtures gaussians supervised machine learning unsupervised learning method network anomaly detection detection algorithm

Keywords

Deep Autoencoding Gaussian Mixture Model unsupervised anomaly detection unsupervised dimensionality reduction outlier detection neural variational inference Principal Component Analysis support vector machine latent representation density estimation tasks

1 INTRODUCTION
2 RELATED WORK
3 DEEP AUTOENCODING GAUSSIAN MIXTURE MODEL
3.1 OVERVIEW
3.2 COMPRESSION NETWORK
3.3 ESTIMATION NETWORK
3.4 OBJECTIVE FUNCTION
3.5 RELATION TO VARIATIONAL INFERENCE
3.6 TRAINING STRATEGY
4 EXPERIMENTAL RESULTS
4.1 DATASET
4.2 BASELINE METHODS
4.3 DAGMM CONFIGURATION
4.4 ACCURACY
4.5 VISUALIZATION ON THE LEARNED LOW-DIMENSIONAL REPRESENTATION
5 CONCLUSION
REFERENCES
A BASELINE CONFIGURATION
B CUMULATIVE DISTRIBUTION FUNCTION OF THE ENERGY FUNCTION LEARNED BY DAGMM
C LOW-DIMENSIONAL REPRESENTATION LEARNED BY DSEBM
D RECONSTRUCTION FEATURES IN DAGMM
E CASE STUDY: WHEN JOINT TRAINING OUTPERFORMS DECOUPLED TRAINING?
F HOW THE HYPERPARAMETERS IN THE OBJECTIVE FUNCTION IMPACT DAGMM

Knowls

Knowl 1 — Deep Autoencoding Gaussian Mixture Model Framework
model/method
The Deep Autoencoding Gaussian Mixture Model (DAGMM) is an end-to-end deep learning framework for unsupervised anomaly detection on multi- and high-dimensional data. DAGMM addresses the limitations of decoupled dimensionality reduction and density estimation by coupling two sub-networks:
1. Compression Network: A deep autoencoder that performs non-linear dimensionality reduction. For an input vector $x \in \mathbb{R}^D$ , the encoder maps $x$ to a low-dimensional latent representation $z_c = h(x; \theta_e) \in \mathbb{R}^{d_c}$ , and the decoder reconstructs the input as $x' = g(z_c; \theta_d) \in \mathbb{R}^D$ . It then computes a reconstruction error feature vector $z_r = f(x, x') \in \mathbb{R}^{d_r}$ and forms a combined low-dimensional representation $z = [z_c, z_r] \in \mathbb{R}^{d_c + d_r}$ .
2. Estimation Network: A multi-layer neural network parameterized by $\theta_m$ that maps $z$ to soft mixture-component membership predictions $\hat{\gamma} = \text{softmax}(\text{MLN}(z; \theta_m)) \in \mathbb{R}^K$ for a Gaussian Mixture Model (GMM) with $K$ components.
Rather than using two-stage decoupled training with Expectation-Maximization (EM), DAGMM jointly optimizes the autoencoder parameters $(\theta_e, \theta_d)$ and estimation network parameters $\theta_m$ simultaneously via backpropagation. Anomaly detection is conducted by evaluating the sample energy $E(z)$ under the learned GMM; samples with energy exceeding a threshold are classified as anomalies.
Knowl 2 — Compression Network and Reconstruction Feature Representation
model/method
In the Deep Autoencoding Gaussian Mixture Model (DAGMM), the compression network prepares a low-dimensional representation $z$ for an input sample $x \in \mathbb{R}^D$ by combining the latent representation from the autoencoder with explicit reconstruction error features:

$z_c = h(x; \theta_e)$

$x' = g(z_c; \theta_d)$

$z_r = f(x, x')$

$z = [z_c, z_r]$

where $h(\cdot; \theta_e)$ is the encoder parameterized by $\theta_e$ , $g(\cdot; \theta_d)$ is the decoder parameterized by $\theta_d$ , $x'$ is the reconstructed counterpart of $x$ , $z_c \in \mathbb{R}^{d_c}$ is the reduced representation, and $z_r \in \mathbb{R}^{d_r}$ is the reconstruction error feature vector.

To detect anomalies that lurk near normal samples in the latent space $z_c$ or exhibit reconstruction distortions, $f(x, x')$ computes continuous and differentiable error metrics with moderate numerical ranges:
1. Relative Euclidean Distance: $\frac{\|x - x'\|_2}{\|x\|_2}$
2. Cosine Similarity: $\frac{x \cdot x'}{\|x\|_2 \|x'\|_2}$
Concatenating $z_c$ with $z_r$ ensures that both geometric deviation in the reduced feature space and reconstruction fidelity inform subsequent density estimation.
Knowl 3 — Estimation Network and Closed-Form GMM Parameter Estimation
equation
The estimation network in the Deep Autoencoding Gaussian Mixture Model (DAGMM) estimates the parameters of a $K$ -component Gaussian Mixture Model (GMM) directly across a mini-batch of $N$ samples without requiring iterative Expectation-Maximization (EM) steps.

For each sample representation $z_i \in \mathbb{R}^d$ ( $i \in \{1, \dots, N\}$ ), the estimation network computes soft mixture membership probabilities $\hat{\gamma}_i \in \mathbb{R}^K$ :

$p_i = \text{MLN}(z_i; \theta_m), \quad \hat{\gamma}_i = \text{softmax}(p_i)$

where $\text{MLN}(\cdot; \theta_m)$ is a multi-layer neural network parameterized by $\theta_m$ , and $\hat{\gamma}_{ik}$ is the predicted probability that sample $z_i$ belongs to component $k \in \{1, \dots, K\}$ .

Given a batch of $N$ samples, the GMM parameters for each component $k$ are computed in closed form:

$\hat{\phi}_k = \sum_{i=1}^N \frac{\hat{\gamma}_{ik}}{N}$

$\hat{\mu}_k = \frac{\sum_{i=1}^N \hat{\gamma}_{ik} z_i}{\sum_{i=1}^N \hat{\gamma}_{ik}}$

$\hat{\Sigma}_k = \frac{\sum_{i=1}^N \hat{\gamma}_{ik} (z_i - \hat{\mu}_k)(z_i - \hat{\mu}_k)^T}{\sum_{i=1}^N \hat{\gamma}_{ik}}$

where $\hat{\phi}_k \in \mathbb{R}$ is the mixture prior probability, $\hat{\mu}_k \in \mathbb{R}^d$ is the component mean vector, and $\hat{\Sigma}_k \in \mathbb{R}^{d \times d}$ is the covariance matrix.

The sample energy (negative log-likelihood) $E(z)$ for any representation $z$ is inferred as:

$E(z) = -\log \left( \sum_{k=1}^K \hat{\phi}_k \frac{\exp\left(-\frac{1}{2}(z - \hat{\mu}_k)^T \hat{\Sigma}_k^{-1}(z - \hat{\mu}_k)\right)}{\sqrt{|2\pi \hat{\Sigma}_k|}} \right)$

where $|\cdot|$ denotes the matrix determinant and $\hat{\Sigma}_k^{-1}$ is the matrix inverse.
Knowl 4 — DAGMM Joint Objective Function and Covariance Regularization
equation
The parameters of the Deep Autoencoding Gaussian Mixture Model (DAGMM)—including the encoder parameters $\theta_e$ , decoder parameters $\theta_d$ , and estimation network parameters $\theta_m$ —are optimized jointly over a training batch of $N$ samples using the objective function:

$J(\theta_e, \theta_d, \theta_m) = \frac{1}{N} \sum_{i=1}^N L(x_i, x'_i) + \frac{\lambda_1}{N} \sum_{i=1}^N E(z_i) + \lambda_2 P(\hat{\Sigma})$

where:
- $L(x_i, x'_i) = \|x_i - x'_i\|_2^2$ is the reconstruction loss penalizing deviation between the input sample $x_i$ and the autoencoder reconstruction $x'_i = g(h(x_i; \theta_e); \theta_d)$ .
- $E(z_i)$ is the sample energy under the estimated Gaussian Mixture Model evaluated on the concatenated low-dimensional representation $z_i = [z_{c,i}, z_{r,i}]$ .
- $P(\hat{\Sigma})$ is a covariance singularity penalty function preventing diagonal entries of the component covariance matrices from collapsing to 0:
$P(\hat{\Sigma}) = \sum_{k=1}^K \sum_{j=1}^d \frac{1}{\hat{\Sigma}_{kjj}}$

where $\hat{\Sigma}_{kjj}$ is the $j$ -th diagonal element of covariance matrix $\hat{\Sigma}_k \in \mathbb{R}^{d \times d}$ , and $d$ is the dimensionality of $z_i$ .
- $\lambda_1 > 0$ and $\lambda_2 > 0$ are weighting hyperparameters, set by default to $\lambda_1 = 0.1$ and $\lambda_2 = 0.005$ .
Knowl 5 — DAGMM Training and Anomaly Detection Algorithm
algorithm
The following algorithm details the end-to-end training and energy-based inference procedure of the Deep Autoencoding Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection.
Input: Training dataset $X = \{x_1, \dots, x_N\} \subset \mathbb{R}^D$ , test dataset $X_{\text{test}}$ , number of mixture components $K$ , hyperparameters $\lambda_1, \lambda_2$ , learning rate $\eta$ , batch size $B$ , number of epochs $T$ , expected anomaly ratio $\rho$ .
Output: Binary anomaly label for each test sample in $X_{\text{test}}$ .
Initialize network parameters $\theta_e$ (encoder), $\theta_d$ (decoder), $\theta_m$ (estimation network).
for epoch = 1 to $T$ do
    for each mini-batch $\{x_1, \dots, x_B\} \subset X$ do
        for $i = 1$ to $B$ do
             $z_{c,i} = h(x_i; \theta_e)$
             $x'_i = g(z_{c,i}; \theta_d)$
             $z_{r,i} = [\frac{\|x_i - x'_i\|_2}{\|x_i\|_2}, \frac{x_i \cdot x'_i}{\|x_i\|_2 \|x'_i\|_2}]$
             $z_i = [z_{c,i}, z_{r,i}]$
             $p_i = \text{MLN}(z_i; \theta_m)$
             $\hat{\gamma}_i = \text{softmax}(p_i)$
        end for
        for $k = 1$ to $K$ do
             $\hat{\phi}_k = \sum_{i=1}^B \frac{\hat{\gamma}_{ik}}{B}$
             $\hat{\mu}_k = \frac{\sum_{i=1}^B \hat{\gamma}_{ik} z_i}{\sum_{i=1}^B \hat{\gamma}_{ik}}$
             $\hat{\Sigma}_k = \frac{\sum_{i=1}^B \hat{\gamma}_{ik} (z_i - \hat{\mu}_k)(z_i - \hat{\mu}_k)^T}{\sum_{i=1}^B \hat{\gamma}_{ik}}$
        end for
         $P(\hat{\Sigma}) = \sum_{k=1}^K \sum_{j=1}^d \frac{1}{\hat{\Sigma}_{kjj}}$
         $J = \frac{1}{B} \sum_{i=1}^B \|x_i - x'_i\|_2^2 + \frac{\lambda_1}{B} \sum_{i=1}^B E(z_i) + \lambda_2 P(\hat{\Sigma})$
        Update $\theta_e, \theta_d, \theta_m$ via Adam optimizer using $\nabla J$
    end for
end for
for each $x \in X_{\text{test}}$ do
    Compute $z = [h(x; \theta_e), f(x, g(h(x; \theta_e); \theta_d))]$
    Compute sample energy $E(z)$ using the estimated parameters $\{\hat{\phi}_k, \hat{\mu}_k, \hat{\Sigma}_k\}_{k=1}^K$
end for
Determine energy threshold $\tau$ as the $(1 - \rho)$ -th percentile of sample energies across $X_{\text{test}}$ .
for each $x \in X_{\text{test}}$ do
    if $E(z) \ge \tau$ then
        Predict $x$ as anomalous
    else
        Predict $x$ as normal
    end if
end for
Knowl 6 — Variational Lower Bound Interpretation of the Estimation Network
theoretical result
In the Deep Autoencoding Gaussian Mixture Model (DAGMM), the estimation network predicts the latent mixture membership $k \in \{1, \dots, K\}$ for a compressed representation $z_i$ . This process is connected to neural variational inference through the following upper bound on the sample energy:

$E(z_i) = -\log p(z_i) = -\log \sum_{k=1}^K p(z_i, k)$

$= -\log \sum_{k=1}^K Q_{\theta_m}(k \mid z_i) \frac{p(z_i, k)}{Q_{\theta_m}(k \mid z_i)}$

$\le -\sum_{k=1}^K Q_{\theta_m}(k \mid z_i) \log \frac{p(z_i, k)}{Q_{\theta_m}(k \mid z_i)}$

$= -\mathbb{E}_{Q_{\theta_m}} [\log p(z_i, k) - \log Q_{\theta_m}(k \mid z_i)]$

$= -\mathbb{E}_{Q_{\theta_m}} [\log p(z_i \mid k)] + \text{KL}(Q_{\theta_m}(k \mid z_i) \parallel p(k))$

$= -\log p(z_i) + \text{KL}(Q_{\theta_m}(k \mid z_i) \parallel p(k \mid z_i))$

$= E(z_i) + \text{KL}(Q_{\theta_m}(k \mid z_i) \parallel p(k \mid z_i))$

where $Q_{\theta_m}(k \mid z_i) = \hat{\gamma}_{ik}$ is the membership distribution parameterized by the estimation network, $\text{KL}(\cdot \parallel \cdot)$ is the Kullback-Leibler divergence, $p(k) = \phi_k$ is the prior mixture coefficient, and $p(k \mid z_i)$ is the true posterior mixture component probability.

Minimizing the negative evidence lower bound forces $Q_{\theta_m}(k \mid z_i)$ to approximate the true posterior $p(k \mid z_i)$ . DAGMM uses the exact energy $E(z_i)$ directly in its objective function rather than the variational upper bound because evaluating $E(z_i)$ for a Gaussian Mixture Model is tractable and computationally efficient.

Knowl 7 — Anomaly Detection Performance Benchmark Across Datasets

data/table

Unsupervised anomaly detection performance was evaluated across four public benchmark datasets (KDDCUP, Thyroid, Arrhythmia, KDDCUP-Rev) under a clean training data setting (50% normal samples for training, 50% mixed samples for testing) across 20 runs. Anomalies were identified by setting the decision threshold to mark the top $\rho$ fraction of samples with the highest energy as anomalous, where $\rho$ is the true anomaly ratio in the dataset.

Method	KDDCUP ( $\rho=0.20$ )			Thyroid ( $\rho=0.025$ )
Method	Precision	Recall	$F_1$	Precision	Recall	$F_1$
OC-SVM	0.7457	0.8523	0.7954	0.3639	0.4239	0.3887
DSEBM-r	0.1972	0.2001	0.1987	0.0404	0.0403	0.0403
DSEBM-e	0.7369	0.7477	0.7423	0.1319	0.1319	0.1319
DCN	0.7696	0.7829	0.7762	0.3319	0.3196	0.3251
GMM-EN	0.1932	0.1967	0.1949	0.0213	0.0227	0.0220
PAE	0.7276	0.7397	0.7336	0.1894	0.2062	0.1971
E2E-AE	0.0024	0.0025	0.0024	0.1064	0.1316	0.1176
PAE-GMM-EM	0.7183	0.7311	0.7246	0.4745	0.4538	0.4635
PAE-GMM	0.7251	0.7384	0.7317	0.4532	0.4881	0.4688
DAGMM-p	0.7579	0.7710	0.7644	0.4723	0.4725	0.4713
DAGMM-NVI	0.9290	0.9447	0.9368	0.4383	0.4587	0.4470
DAGMM	0.9297	0.9442	0.9369	0.4766	0.4834	0.4782
Method	Arrhythmia ( $\rho=0.15$ )			KDDCUP-Rev ( $\rho=0.20$ )
Method	Precision	Recall	$F_1$	Precision	Recall	$F_1$
OC-SVM	0.5397	0.4082	0.4581	0.7148	0.9940	0.8316
DSEBM-r	0.1515	0.1513	0.1510	0.2036	0.2036	0.2036
DSEBM-e	0.4667	0.4565	0.4601	0.2212	0.2213	0.2213
DCN	0.3758	0.3907	0.3815	0.2875	0.2895	0.2885
GMM-EN	0.3000	0.2792	0.2886	0.1846	0.1746	0.1795
PAE	0.4393	0.4437	0.4403	0.7835	0.7817	0.7826
E2E-AE	0.4667	0.4538	0.4591	0.7434	0.7463	0.7448
PAE-GMM-EM	0.3970	0.4168	0.4056	0.2822	0.2847	0.2835
PAE-GMM	0.4575	0.4823	0.4684	0.6307	0.6278	0.6292
DAGMM-p	0.4909	0.4679	0.4787	0.2750	0.2810	0.2780
DAGMM-NVI	0.5091	0.4892	0.4981	0.9211	0.9211	0.9211
DAGMM	0.4909	0.5078	0.4983	0.9370	0.9390	0.9380

DAGMM achieves the highest $F_1$ score across all four datasets, outperforming both traditional models (OC-SVM) and deep learning baselines (DSEBM, DCN, PAE, and two-stage GMM variants) by up to 14.15% on KDDCUP and 10.64% on KDDCUP-Rev.

Knowl 8 — Robustness of Anomaly Detection Under Contaminated Training Data

data/table

The robustness of DAGMM against training data contamination was evaluated on the KDDCUP dataset by introducing anomaly samples into the training partition at contamination ratios $c \in \{1\%, 2\%, 3\%, 4\%, 5\%\}$ . Performance metrics are averaged across 20 runs:

Ratio $c$	DAGMM			DCN
Ratio $c$	Precision	Recall	$F_1$	Precision	Recall	$F_1$
1%	0.9201	0.9337	0.9268	0.7585	0.7611	0.7598
2%	0.9186	0.9340	0.9262	0.7380	0.7424	0.7402
3%	0.9132	0.9272	0.9201	0.7163	0.7293	0.7228
4%	0.8837	0.8989	0.8912	0.6971	0.7106	0.7037
5%	0.8504	0.8643	0.8573	0.6763	0.6893	0.6827
Ratio $c$	DSEBM-e			OC-SVM
Ratio $c$	Precision	Recall	$F_1$	Precision	Recall	$F_1$
1%	0.6995	0.7135	0.7065	0.7129	0.6785	0.6953
2%	0.6780	0.6876	0.6827	0.6668	0.5207	0.5847
3%	0.6213	0.6367	0.6289	0.6393	0.4470	0.5261
4%	0.5704	0.5813	0.5758	0.5991	0.3719	0.4589
5%	0.5345	0.5375	0.5360	0.1155	0.3369	0.1720

While all models experience performance degradation as the contamination ratio $c$ increases, DAGMM maintains strong anomaly detection accuracy ( $F_1 = 0.8573$ at $c = 5\%$ ), significantly exceeding DCN ( $F_1 = 0.6827$ ), DSEBM-e ( $F_1 = 0.5360$ ), and OC-SVM ( $F_1 = 0.1720$ ).

Knowl 9 — Mutual Boosting Effect and Local Optima Escape in End-to-End Training
empirical result
Training a standalone deep autoencoder from scratch with end-to-end gradient descent (E2E-AE) frequently causes it to get trapped in suboptimal local minima. On the KDDCUP dataset, E2E-AE results in an average per-sample reconstruction error of approximately $1.13$ . Conversely, when the identical autoencoder architecture is trained end-to-end within DAGMM alongside the estimation network, it achieves an average per-sample reconstruction error of approximately $0.26$ , matching the error obtained by a pre-trained autoencoder (PAE).

This demonstrates a mutual boosting effect between the compression and estimation components:
1. The density estimation loss and covariance regularization provided by the estimation network guide the autoencoder away from poor local optima during backpropagation, yielding low reconstruction error without pre-training.
2. The low-dimensional representation dynamically adapts during end-to-end optimization to favor Gaussian Mixture Model density estimation, avoiding the rigidity of fixed pre-trained representations.

Knowl 10 — Objective Function Hyperparameter Sensitivity Under Fixed Ratio Scaling

data/table

The sensitivity of DAGMM's joint objective function to hyperparameter scaling was evaluated on the KDDCUP dataset by multiplying the weight ratio $1 : \lambda_1 : \lambda_2 = 1 : 0.1 : 0.005$ by a scaling base multiplier $b \in \{1, 3, 5, 7, 9\}$ (i.e., setting $\lambda_1 = 0.1 \times b$ and $\lambda_2 = 0.005 \times b$ ). Performance metrics over 20 runs are:

Base $b$	Precision	Recall	$F_1$
1	0.9298	0.9445	0.9371
3	0.9301	0.9442	0.9371
5	0.9296	0.9451	0.9373
7	0.9300	0.9453	0.9376
9	0.9300	0.9439	0.9369

DAGMM's anomaly detection performance is invariant to proportional scaling of the base multiplier, with $F_1$ scores consistently residing within the narrow range $[0.9369, 0.9376]$ , indicating robust optimization whenever the relative balance $1 : 0.1 : 0.005$ between reconstruction loss, sample energy, and covariance regularization is maintained.

Coverage note — No substantial contributed material was omitted. The knowls comprehensively cover the model architecture, equations, training algorithm, theoretical variational bound, experimental benchmark results, contamination robustness, mutual boosting analysis, and hyperparameter sensitivity.

References

1.Martın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pp. 265–283, 2016.
2.V. Barnett and T. Lewis. Outliers in statistical data. Wiley, 1984.
3.Emmanuel J. Candes, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? `J. ACM, 58(3):11:1–11:37, 2011. ISSN 0004-5411.
4.Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM Comput. Surv, 41:15:1–15:58, 2009.
5.Yunqiang Chen, Xiang Sean Zhou, and Thomas S Huang. One-class svm for learning in image retrieval. In International Conference on Image Processing, volume 1, pp. 34–37, 2001.
6.Simon Gunter, Nicol N. Schraudolph, and S. V. N. Vishwanathan. Fast Iterative Kernel Principal Component Analysis. jmlr, 8:1893–1918.
7.Peter J Huber. Robust statistics. In International Encyclopedia of Statistical Science, pp. 1248–1251. Springer, 2011.
8.Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.
9.I. T. Jolliffe. Principal component analysis. In Principal Component Analysis. Springer Verlag, New York, 1986.
10.Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999.
11.Fabian Keller, Emmanuel Muller, and Klemens Bohm. Hics: High contrast subspaces for density-based outlier ranking. In International Conference on Data Engineering, pp. 1037–1048. IEEE, 2012.
12.JooSeuk Kim and Clayton D. Scott. Robust kernel density estimation. CoRR, abs/1107.3133, 2011.
13.Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015.
14.M. Lichman. UCI machine learning repository, 2013. URL http://archive.ics.uci.edu/ml.
15.Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In International Conference on Data Mining, pp. 413–422. IEEE, 2008.
16.Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In ICML, pp. 1791–1799, 2014.
17.Matthias Paulik. Lattice-based training of bottleneck feature extraction neural networks. In Interspeech, pp. 89–93, 2013.
18.Q. Song, W. J. Hu, and W. F. Xie. Robust support vector machine with bullet hole image classification. IEEE Trans. Systems, Man and Cybernetics, 32:440–448, 2002.
19.Swee Chuan Tan, Kai Ming Ting, and Tony Fei Liu. Fast anomaly detection for streaming data. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, volume 22, pp. 1511, 2011.
20.Zoltan T uske, Pavel Golik, Ralf Schl uter, and Hermann Ney. Speaker adaptive joint training of gaussian mixture models and bottleneck features. In IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 596–603, 2015a.
21.Zoltan T uske, Muhammad Ali Tahir, Ralf Schl uter, and Hermann Ney. Integrating gaussian mixtures into deep neural networks: softmax layer with hidden variables. In ICASSP, pp. 4285–4289, 2015b.
22.Ehsan Variani, Erik McDermott, and Georg Heigold. A gaussian mixture model layer jointly optimized with discriminative features within a deep neural network architecture. In ICASSP, pp. 4270–4274, 2015.
23.Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(Dec):3371–3408, 2010.
24.Graham Williams, Rohan Baxter, Hongxing He, and Simon Hawkins. A comparative study of RNN for outlier detection in data mining. In Proceedings of ICDM02, pp. 709–712, 2002.
25.Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pp. 478–487, 2016.
26.Liang Xiong, Barnabas P oczos, and Jeff G. Schneider. Group anomaly detection using flexible genre models. In Advances in Neural Information Processing Systems, pp. 1071–1079, 2011.
27.Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In International Conference on Machine Learning, 2017a.
28.Xi Yang, Kaizhu Huang, and Rui Zhang. Unsupervised dimensionality reduction for gaussian mixture model. In International conference on neural information processing, pp. 84–92. Springer, 2014.
29.Xi Yang, Kaizhu Huang, John Yannis Goulermas, and Rui Zhang. Joint learning of unsupervised dimensionality reduction and gaussian mixture model. Neural Processing Letters, 45(3):791–806, 2017b.
30.Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International Conference on Machine Learning, pp. 1100–1109, 2016.
31.C. Zhang and P. C. Woodland. Joint optimisation of tandem systems using gaussian mixture density neural network discriminative sequence training. In ICASSP, pp. 5015–5019, 2017.
32.Chong Zhou and Randy C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 665–674, 2017.
33.Arthur Zimek, Erich Schubert, and Hans-Peter Kriegel. A survey on unsupervised outlier detection in high-dimensional numerical data. Statistical Analysis and Data Mining, 5:363–387, 2012.

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Deep Autoencoding Gaussian Mixture Model Framework

Knowl 2 — Compression Network and Reconstruction Feature Representation

Knowl 3 — Estimation Network and Closed-Form GMM Parameter Estimation

Knowl 4 — DAGMM Joint Objective Function and Covariance Regularization

Knowl 5 — DAGMM Training and Anomaly Detection Algorithm

Knowl 6 — Variational Lower Bound Interpretation of the Estimation Network

Knowl 7 — Anomaly Detection Performance Benchmark Across Datasets

Knowl 8 — Robustness of Anomaly Detection Under Contaminated Training Data

Knowl 9 — Mutual Boosting Effect and Local Optima Escape in End-to-End Training

Knowl 10 — Objective Function Hyperparameter Sensitivity Under Fixed Ratio Scaling

References

Access the Paper